Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions cmd/awesome-sync/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,14 @@
package main

import (
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"

"github.com/moul/mygnoscan/pkg/registry"
Expand Down Expand Up @@ -60,6 +62,10 @@ func run() error {
if err != nil {
return err
}
// The list links repositories; the page wants apps. This is the step that
// turns one into the other, and it is the slow half of the run because it
// talks to every project's own server.
resolveSites(context.Background(), sections)

snap := registry.Awesome{
Source: fmt.Sprintf("https://github.com/%s/%s", owner, repo),
Expand Down Expand Up @@ -92,8 +98,20 @@ func run() error {
if err := os.WriteFile(*out, body, 0o644); err != nil {
return err
}
fmt.Printf("awesome-sync: wrote %s: %d sections, %d entries, %d realms, from %s\n",
*out, len(snap.Sections), snap.Count(), len(snap.Paths()), sha[:12])
apps := snap.Apps()
hosts := snap.SiteHosts()
fmt.Printf("awesome-sync: wrote %s: %d sections, %d entries, %d apps (%d realms, %d sites), from %s\n",
*out, len(snap.Sections), snap.Count(), len(apps), len(snap.Paths()), len(hosts), sha[:12])
// Printed rather than left to be worked out, because it is configuration on
// another box: gnoshot refuses a host it was not told about, and the
// failure is a card with no picture and nothing in any log here.
//
// This is the community list's hosts only. apps.json can name a website
// too, so the authoritative set is `/api/apps` plus that file; run this,
// then check nothing in apps.json points at a host missing from the line
// below.
fmt.Printf("\ngnoshot needs these hosts on its -allow-site flag:\n\n -allow-site '%s'\n",
strings.Join(hosts, ","))
return nil
}

Expand Down
209 changes: 209 additions & 0 deletions cmd/awesome-sync/sites.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
package main

import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"os"
"strings"
"sync"
"time"

"github.com/moul/mygnoscan/pkg/registry"
)

// Finding the app behind an entry.
//
// An awesome list is a list of links, and most of its links are repositories.
// That is right for the list and wrong for a grid of pictures: a screenshot of
// a GitHub page tells a reader nothing they did not already know from the word
// "GitHub". So for every entry that could be an app, this resolves the page you
// would actually open, and then checks that the page is really there.
//
// Two sources, and they are recorded separately because they are not the same
// claim. Either the list points straight at the app, or it points at a
// repository and that repository declares a homepage. The second is the
// project's own assertion about where its app lives, which is why it is worth
// following and why it is worth labelling.

// notAnApp are hosts where a URL is a document, a conversation or source code
// rather than something to use.
//
// A denylist rather than an allowlist, because the point is to follow the
// community wherever they go: a new app on a host nobody has seen before should
// work on the day it is added, and a new *forum* should not.
//
// github.io is deliberately absent. It hosts source for some projects and the
// deployed app for others, and the deployed app is the common case for a link
// somebody put in an apps section.
var notAnApp = map[string]bool{
"github.com": true,
"gist.github.com": true,
"raw.githubusercontent.com": true,
"gitlab.com": true,
"www.youtube.com": true,
"youtube.com": true,
"youtu.be": true,
"x.com": true,
"twitter.com": true,
"t.me": true,
"discord.com": true,
"discord.gg": true,
"medium.com": true,
"docs.gno.land": true,
"gno.link": true,
"awesome.re": true,
"calendar.google.com": true,
}

func isAppHost(raw string) bool {
u, err := url.Parse(raw)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") {
return false
}
h := strings.ToLower(u.Hostname())
if h == "" || notAnApp[h] || strings.HasSuffix(h, ".hashnode.dev") {
return false
}
// gno.land itself is the chain's own front end, not an app somebody built
// on it. It arrives here as the declared homepage of the gnolang/gno
// monorepo, which is the homepage of everything in that repository and
// therefore of nothing in particular.
if h == "gno.land" {
return false
}
return true
}

// repoRoot reports whether a GitHub URL names a whole repository rather than a
// directory inside one.
//
// This is the guard that keeps a monorepo's homepage off its contents. gnodev,
// gnobro and the gnoclient package are all linked as
// github.com/gnolang/gno/tree/master/..., and gnolang/gno declares its homepage
// as gno.land, so following it gave three CLI tools a picture of the chain's
// website and each other. A link into a subdirectory is a link to source; only
// a link to the repository itself is a claim about a project.
func repoRoot(raw string) bool {
u, err := url.Parse(raw)
if err != nil {
return false
}
return len(strings.Split(strings.Trim(u.Path, "/"), "/")) == 2
}

// repoHomepage asks GitHub what homepage a repository declares.
func repoHomepage(ctx context.Context, repoURL string) string {
u, err := url.Parse(repoURL)
if err != nil || strings.ToLower(u.Hostname()) != "github.com" {
return ""
}
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
if len(parts) < 2 {
return ""
}
body, err := get(fmt.Sprintf("https://api.github.com/repos/%s/%s", parts[0], parts[1]))
if err != nil {
return ""
}
var doc struct {
Homepage string `json:"homepage"`
Archived bool `json:"archived"`
}
if err := json.Unmarshal(body, &doc); err != nil || doc.Archived {
return ""
}
return strings.TrimSpace(doc.Homepage)
}

// siteClient follows redirects, which is most of the point: half of these
// homepages are written without the www or with a trailing slash the server
// moves, and the URL that answers is the one worth storing.
var siteClient = &http.Client{Timeout: 20 * time.Second}

// reachable checks that a candidate is a page a browser can open, and returns
// the URL it settled on.
//
// A screenshot service will happily photograph a 404, and the result is a card
// that looks deliberate and says "Not Found". gnockpit is the worked example:
// its repository still declares a homepage on a testnet that was retired, so
// without this the grid would carry a confident picture of a dead host.
func reachable(raw string) (string, bool) {
req, err := http.NewRequest(http.MethodGet, raw, nil)
if err != nil {
return "", false
}
// Some of these serve a different page, or nothing at all, to a client that
// does not look like a browser.
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; mygnoscan-awesome-sync/1)")
req.Header.Set("Accept", "text/html,application/xhtml+xml")
resp, err := siteClient.Do(req)
if err != nil {
return "", false
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", false
}
if ct := resp.Header.Get("Content-Type"); ct != "" && !strings.Contains(ct, "html") {
// A JSON API or a tarball is a real 200 and not a page. The picture
// would be Chrome's own rendering of raw text.
return "", false
}
return resp.Request.URL.String(), true
}

// resolveSites fills Site and SiteFrom on every entry that turns out to have an
// app behind it. Concurrent because it is dominated by waiting on other
// people's servers, bounded because several of them are one box.
func resolveSites(ctx context.Context, sections []registry.AwesomeSection) {
type job struct{ s, e int }
jobs := []job{}
for si, s := range sections {
if !registry.IsAwesomeAppSection(s.Slug) || s.Archived {
continue
}
for ei := range s.Entries {
jobs = append(jobs, job{si, ei})
}
}
sem := make(chan struct{}, 6)
var wg sync.WaitGroup
var mu sync.Mutex
for _, j := range jobs {
wg.Add(1)
go func(j job) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()

e := sections[j.s].Entries[j.e]
if e.Path != "" {
return // a realm; the chain's own gnoweb photographs it
}
site, from := "", ""
if isAppHost(e.URL) {
site, from = e.URL, registry.SiteListed
} else if repoRoot(e.URL) {
if hp := repoHomepage(ctx, e.URL); hp != "" && isAppHost(hp) {
site, from = hp, registry.SiteRepoHomepage
}
}
if site == "" {
return
}
final, ok := reachable(site)
mu.Lock()
defer mu.Unlock()
if !ok {
fmt.Fprintf(os.Stderr, " skip %-28s %s does not answer\n", e.Name, site)
return
}
sections[j.s].Entries[j.e].Site = final
sections[j.s].Entries[j.e].SiteFrom = from
}(j)
}
wg.Wait()
}
93 changes: 92 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,10 @@ bounded at 3650 days instead.
| `GET /api/networks` | configured network IDs — the fastest way to confirm which chains an instance is actually serving |
| `GET /api/watch` | activity digest for a watchlist, plus a `transactions` timeline: the 50 most recent rows across every watched realm and address, merged and deduplicated. Repeated `realm=` and `address=` parameters, each optionally `id@height` — that height is the baseline `new_since` counts against (the timeline itself is not filtered by it). Answered from stored rows only, so a watchlist costs no indexer round-trips. Capped at 100 items |
| `GET /api/labels` | display names for addresses: `{address: {label, kind, why}}`, the curated registry merged with what the chain proves |
| `GET /api/apps` | the app hub, assembled: the chain's ranked realms, the community's off-chain apps, this repo's overrides, and the skip list, merged into one array of cards. Each card says where each of its fields came from |
| `GET /api/registry/apps` | the curated app directory: `categories`, `apps`, a count of known tokens and an `awesome` summary of the community list. With a single `network`, also `stats` (per path: `deployed`, `calls`, `callers`, `calls_window`), `candidates` (the busiest realms with no entry, over `window`, default 30d) and `realms` (how many exist). Those are per-chain and absent without one, because the same path is a different deployment on each chain |
| `GET /api/registry/awesome` | the vendored [awesome-gno](https://github.com/gnoverse/awesome-gno) snapshot: `sections`, `entries`, the `commit` it was read at and the day it was `synced`, plus the cross-check against the directory in both directions (`missing_from_awesome`, `missing_from_directory`, `in_directory`, `directory_by_name`). With a single `network`, `stats` for every realm either list names |
| `GET /api/registry/awesome` | the vendored [awesome-gno](https://github.com/gnoverse/awesome-gno) snapshot: `apps` (the entries with a page you can open), `others` (what is on the list and has nothing to photograph, counted by section), `entries`, the `commit` it was read at and the day it was `synced`, plus the cross-check against the directory in both directions (`missing_from_awesome`, `missing_from_directory`, `in_directory`, `directory_by_name`). With a single `network`, `stats` for every realm either list names |
| `GET /api/shot/site` | a screenshot of one listed off-chain app. `url` must appear verbatim in the vendored snapshot; anything else is a 400 |

**Address labels are global, not per network.** An address is the same key on
every chain, so a name earned on one applies everywhere.
Expand Down Expand Up @@ -167,6 +169,61 @@ Nothing is derived from a namespace with more than one deployer. Seven exist on
the live chains, and naming one of their deployers would present a guess as a
fact.

### The app hub: three layers, and which one wins

`/apps` used to be a curated list of ten entries against two hundred and ten
realms, whose only contribution workflow was to stare at a JSON file. It now
assembles itself from three layers.

**The chain proposes.** `DiscoverApps` ranks every realm people actually call.
The ranking is two constants in `pkg/store/discover.go` and nothing else:

```
score = callers_in_window * 10 + calls_in_window * 1
```

Distinct callers outrank raw calls by an order of magnitude because they answer
different questions. Calls measure activity, and activity is trivially
manufactured: one script in a loop is thousands of calls and one caller. Callers
measure reach, and buying that costs a funded address per unit. Both are
windowed, so a realm that was busy last year does not outrank one that is busy
today. Realms only: a `p/` package has no page to open and belongs in
`/packages`.

**Every default is somebody's own word.** The description is the realm's own
package doc comment, first sentence, stored in `symbol_index.package_doc` and
written by whoever wrote the realm. The picture is a capture of the realm or of
its website. Neither is this repo inventing a sentence about somebody else's
code, which is what made auto-listing acceptable at all.

**Curation corrects, with three small levers and no large one.**

| lever | file | does |
|---|---|---|
| list | `apps.json`, awesome-gno | include something the ranking cannot see, and override its name, sentence, website or category |
| relate | `apps.json` `supersedes` | fold an older generation into the one that replaces it |
| skip | `moderation.toml` | remove something that should not be here, with a reason |

Being listed means *included*, not *first*: a listed entry is floored at
`scoreVouched` rather than promoted, so a realm people genuinely use still
outranks it. Ranking listed entries to the top put Boards2, which nobody had
called, above the busiest realm on the chain.

**Every field says where it came from**, and the card shows it as one quiet
mark: `curated` (this repo's registry), `community` (awesome-gno), `chain` (the
realm's own doc), `path` (nobody has said anything yet). A reader who cannot
tell a vouched-for sentence from a generated one has to trust both equally or
neither.

**The skip list is served, not merely applied.** A page that quietly dropped an
entry would be indistinguishable from one that lost it, and an unexplained
removal is indistinguishable from censorship, so `moderation` comes back in the
response with each entry's reason and the page prints them.

**The website leads and the realm is second.** Somebody sent here wants the
thing itself; the realm page is what they want next, and only if they are the
kind of person who wants it.

### The community list, and the gap

`gnoverse/awesome-gno` is the other curated answer to "what is being built on
Expand All @@ -185,6 +242,40 @@ all three. `make awesome-check` says whether the committed copy is behind
without writing. Neither runs in CI: a red build because somebody else edited
their README is a build nobody here can fix.

**Most of the list is not a realm, and the page shows the part that is an app.**
Measured 2026-09-23: of 71 entries, **14 resolve to a page you can open** and the
other 57 are repositories, SDKs, editor plugins, docs and talks. `/apps?view=ecosystem`
draws a card with a screenshot per app and prints a count per section for the
rest, because a grid of pictures with grey holes where the SDKs are reads as a
broken page rather than as a list of libraries.

The address behind an app comes from one of two places and the response says
which, because they are not the same claim:

| `site_from` | means |
|---|---|
| `listed` | the community list points straight at the app |
| `repo-homepage` | the list points at a repository, and that repository declares this homepage |

Every candidate is fetched at generation time and dropped unless it answers 200
with HTML. gnockpit is the worked example: its repository still declares a
homepage on a testnet that was retired, and without the check the grid would
carry a confident picture of a dead host. Two guards sit beside it: a link into
a *subdirectory* of a repository never inherits that repository's homepage (the
gnolang/gno monorepo declares gno.land, which would have given gnodev, gnobro
and gnoclient a picture of the chain's website and of each other), and gno.land
itself is never a site, because a realm is photographed through gnoweb.

**Screenshots of off-chain apps go through `/api/shot/site`, and the gate is the
design.** An unbounded `?url=` in front of a capture service is an open proxy
and a way to spend a box's CPU on headless Chrome. A URL is accepted only if it
appears verbatim in the vendored snapshot: not a host match, the exact string a
human merged, so another page on a listed app's own domain is refused too.
gnoshot has its own host allowlist (`-allow-site`) behind that, which is the
second gate and the one that survives a bug in the first. `make awesome` prints
the host list gnoshot needs, because it is configuration on another box and the
failure mode is a card with no picture and nothing in any log here.

**The two lists barely overlap, and that gap is the useful part.** Measured
2026-09-23: 71 entries on the community list, 10 in the directory, and 2 in
both. So each is mostly the other's worklist, and the endpoint reports both
Expand Down
Loading