Skip to content

Latest commit

 

History

History

examples

README.md

Examples

The scripts declare their dependencies inline (PEP 723), so they run from anywhere with uv and nothing pre-installed:

uv run examples/quotes_analysis.py
uv run examples/compare.py --repeat 5
uv run examples/neurips_topics.py
uv run examples/classify_pages.py   # needs TYPESAFE_API_KEY
uv run examples/triage_crawl.py     # needs TYPESAFE_API_KEY

Set LIGHTPANDA_BIN=/path/to/lightpanda to run against a local browser build instead of the binary bundled in the wheel.

quotes_analysis.py — scrape a JavaScript-rendered site, analyse with pandas

quotes.toscrape.com/js renders every quote client-side with jQuery. requests gets the page shell — 0 quotes. Lightpanda runs the JavaScript and extract returns structured records (quote, author, nested tag list) straight from the rendered DOM — 100 quotes, no BeautifulSoup in between:

SCHEMA = {
    "quotes": [{"selector": ".quote",
                "fields": {"text": ".text", "author": ".author", "tags": [".tag"]}}],
    "next": {"selector": "li.next a", "attr": "href"},
}

with Browser() as browser, browser.new_session() as page:
    url = "https://quotes.toscrape.com/js/"
    while url:
        page.goto(url=url)
        data = page.extract(schema=SCHEMA)
        rows.extend(data["quotes"])
        url = data["next"]

df = pd.DataFrame(rows)
df.explode("tags")["tags"].value_counts()   # love, inspirational, life, ...

The script prints the top tags, most-quoted authors and quote-length stats, and writes quotes.png (matplotlib).

compare.py — the same job with requests, Selenium and Lightpanda

Loads the ten pages of the site with each tool and measures how many quotes it saw, wall time including browser startup, and peak resident memory summed over the child processes it spawned. --repeat N interleaves the legs N times and reports medians with the min–max range. Selenium needs Chrome installed (Selenium Manager fetches a matching chromedriver on the first run); that leg is skipped if Chrome is missing.

Median of 5 runs on a Linux laptop (Core Ultra 7 258V, performance CPU profile):

tool quotes seconds (min–max) peak memory
requests 0 1.6 (1.5–1.7) 0 MB
selenium 100 6.1 (5.8–6.7) 1245 MB
lightpanda 100 3.4 (3.0–3.6) 35 MB

requests is the fastest way to get nothing. Selenium gets the data but drags a full Chrome (plus a chromedriver download and version matching) along for the ride; Lightpanda gets the same data from a single pip install, in about half the time and ~35× less memory. The script writes compare.png.

neurips_topics.py — find what a field started working on, with nothing told to it

neurips.cc/virtual/2025/papers.html lists every accepted paper and requests gets none of them: 800 KB of scripts around an empty card list and a <noscript> reading "Enable Javascript in your browser to see the papers page".

soup = BeautifulSoup(requests.get("https://neurips.cc/virtual/2025/papers.html").text)
len(soup.select(".myCard"))   # 0

Rendering it is not enough either. The page fetches a 27 MB index, keeps all 5858 papers in a JavaScript array, and only ever builds 400 cards from it, so a scraper that reads the DOM still sees a seventh of the conference. Lightpanda reads the array:

async with AsyncBrowser(args=["--http-timeout", "180000"]) as browser:   # the index is 27 MB
    async with browser.session() as page:
        await page.goto(url=f"https://neurips.cc/virtual/{year}/papers.html")
        await page.wait_for_script(script="typeof allPapers !== 'undefined' && allPapers.length > 0")
        rows = await page.evaluate(script="return allPapers.map(p => [p.title, p.abstract, p.url])")

One browser process per year, five years at once, about a minute for 19,219 papers with abstracts. (Several sessions inside one process contend badly on this workload and most of them fail; separate processes are both reliable and faster.)

Those get embedded once, and then each year is clustered on its own by Chinese Whispers: link every pair of papers alike enough to be worth linking, and let each paper repeatedly adopt the weighted-majority topic of whatever it is linked to. No number of clusters is chosen anywhere.

Clustering the years together would average a topic against the years it did not exist in, and lose the ones that only appeared recently. So every year nominates its own topics, near-duplicates are merged, and the centre of each surviving topic then labels every paper in every year. That last step is what makes a topic's share comparable across time. Each topic is named by the phrases its papers use far more than the rest of the conference does.

Nothing supplies a topic list. This is the whole output:

198 topics over all years, 69 after merging duplicates

year                                           2021  2022  2023  2024  2025
reasoning, models llms, large language models  0.43  0.90  1.84  4.30  8.19
harmful, safety, attack                        0.04  0.45  0.75  2.31  2.66
video generation, denoising, diffusion models  0.00  0.24  0.25  0.90  1.09
visual tokens, mllms, vision language          0.00  0.10  0.25  0.57  0.85
vision transformers, vits, self attention      2.87  2.00  0.86  0.95  0.43
domain, generalization, target                 2.57  2.51  1.48  0.93  0.43
low rank, matrix, matrices                     2.23  1.27  1.12  0.62  0.34
meta learning, shot, task                      1.67  0.93  0.59  0.15  0.12

neurips_topics.png

Passing a query searches the same embeddings by meaning instead, which costs nothing extra once they exist:

$ uv run examples/neurips_topics.py "making language models reason step by step"

  0.87  2022  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  0.86  2023  Why think step by step? Reasoning emerges from the locality of experience
  0.85  2022  Large Language Models are Zero-Shot Reasoners

Nothing told it the phrase "chain of thought". It also marks any hit that shares no word at all with the query, which is where the difference from grep shows: asking for "teaching machines to see the world in three dimensions" returns "Multistable Shape from Shading Emerges from Patch Diffusion", with not one word in common.

Why clustering and not counting

Four other ways of turning embeddings into a number were tried on these same 19,219 papers, and all of them fail:

approach result
mean similarity to a topic, per year GANs move 0.002 over five years while their share collapses 7×
forced nearest-topic assignment claims two thirds of NeurIPS 2021 was adversarial robustness
cutoff fitted to phrase labels admits 6.6× too many papers; every trend flattens
year-to-year corpus similarity 2025 looks more like the past than 2022 did, a corpus-size artifact

They share a cause. Cosine similarities here sit in a narrow band, roughly 0.65 to 0.90, so anything that compares a paper against a global bar drowns in the 19,000 papers that are not about the topic. Chinese Whispers never asks that question. It only asks which papers are near each other, and relative neighbourhood structure survives the compression intact.

One parameter remains, --threshold, deciding how alike two papers must be to be linked at all. Swept across the five per-year clusterings:

threshold topics found after merging largest with 60+ papers
0.855 225 59 10.2% 59
0.865 225 62 6.9% 61
0.875 198 69 5.9% 67
0.885 124 47 5.8% 47
0.895 59 30 7.2% 30

Too low and separate topics run together, so the largest swells to a tenth of the conference. Too high and papers stop being linked at all, so fewer topics are found in the first place: 124 at 0.885, then 59. 0.875 yields the most substantial topics while keeping the largest small.

Labelling every paper by its nearest topic is safe here only because the topics came from the corpus. An earlier version of this example tried the same assignment with six hand-written topics and reported that two thirds of NeurIPS 2021 was about adversarial robustness, which was simply the bucket catching everything else. With 69 topics the papers themselves produced, the largest holds 4% of the corpus and that distortion is gone.

The topics are reproducible in character rather than identical. Rerun against different embeddings and reasoning, safety and vision transformers reliably appear, while the precise split between adjacent topics moves.

On dlib, which has a well-known Chinese Whispers implementation: it is not needed here. Building the graph is numpy work either way and takes 2.5 s; the propagation itself is 0.88 s in plain Python against 0.38 s in dlib's C++. Half a second does not pay for a source build needing CMake and a C++ toolchain, and writing the loop out means the example can show the algorithm rather than hide it behind a call. (dlib's own all-in-one chinese_whispers_clustering, which builds edges itself, is far slower still — about 80× — because it does the pairwise comparison in Python objects rather than in numpy.)

Which model

Ranking the papers that genuinely match a phrase to the top, over a 1,250-paper sample:

model AUC to embed 1,250 papers
gemini-embedding-2 0.958 one API call per 100, needs a key
bge-base-en-v1.5 (ONNX) 0.923 481 s on CPU
bge-small-en-v1.5 (ONNX) — the keyless default 0.909 130 s on CPU
all-MiniLM-L6-v2 (ONNX) 0.854 22 s on CPU
potion-base-32M (model2vec, static) 0.844 0.4 s
potion-base-8M (model2vec, static) 0.790 0.5 s

Only the first needs a network and none need a GPU. gemini-embedding-001 scores the same as -2 within noise, but it is the legacy model. With a key the whole corpus embeds in about twenty seconds; without one it takes considerably longer, once, and the 27 MB cache under ~/.cache/neurips_topics makes every later run instant.

classify_pages.py — triage a crawl before scraping it

A 200 response can still be a login wall, a CAPTCHA or an empty search. page.judge() asks TypeSafe's page model about the rendered page and says whether to skip it, and why. Your own questions ride along in the same call, yes/no, a choice, or a score, so each page costs one round trip:

verdict = page.judge(questions={
    "has_price": "Does the page show a price?",                     # -> 0.99
    "kind": {"question": "What kind of page is this?",
             "options": ["product", "listing", "article", "other"]},  # -> {"choice": "product", ...}
    "content": {"question": "How much useful content would a scraper get from this page?",
                "levels": ["none", "little", "plenty"]},             # -> {"level": "plenty", "score": 1.90, ...}
})
if not verdict.ok:
    print("skip", verdict.reason)           # is_captcha
else:
    print(verdict.answers["kind"]["choice"])  # product
keep  books.toscrape.com/catalogue/a-light-in-  product  price 0.99  content plenty (1.94)  [load 1.11s, jev 0.37s]
keep  quotes.toscrape.com/js/                   listing  price 0.01  content plenty (1.91)  [load 0.95s, jev 0.23s]
keep  en.wikipedia.org/wiki/Headless_browser    article  price 0.02  content plenty (1.74)  [load 0.59s, jev 0.25s]
skip  news.ycombinator.com/login                is_login_wall                           [load 0.57s, jev 0.23s]
skip  en.wikipedia.org/w/index.php?search=xqzz  is_empty_catalog                        [load 1.18s, jev 0.26s]
skip  www.g2.com/                               is_captcha                              [load 0.16s, jev 0.20s]

A score falls between its levels (0 = none, 2 = plenty). Scores vary a little between runs. The jev time is the whole call: the browser reading the page, then the round trip to TypeSafe, which reuses one connection after the first.

Needs TYPESAFE_API_KEY and a lightpanda release with judge():

uv run examples/classify_pages.py

To run it from this checkout against a local browser build instead, use the project's environment:

LIGHTPANDA_BIN=../browser/zig-out/bin/lightpanda uv run python examples/classify_pages.py

triage_crawl.py — crawl real sites in parallel and act on what Jev sees

The same judge() call over 30 real pages, six at a time in one browser. The verdict decides what the crawler does next: back off when the site says we are going too fast, accept a cookie wall, wait for a page that is still loading, and then look again; or keep the page, or skip it and say why. "Too fast" is one of the checks, set by a 429 or by a page that says so in words; the kind of page is our own question, asked in the same call:

QUESTIONS = {
    "kind": {"question": "What kind of page is this?",
             "options": ["product", "listing", "article", "docs", "search", "other"]},
}

verdict = await page.judge(questions=QUESTIONS)
if verdict.is_rate_limited:
    await asyncio.sleep(3)                         # and load it again
elif verdict.is_consent_wall:
    found = await page.find_element(role="button", name="/^(accept|aceptar|akzeptieren)/i")
    await found[0].click()

Every verdict prints a line. One the crawler acts on says retry and what it does about it, and the page's last verdict says keep or skip, so the two cookie walls show up twice and a rate-limited page once per try. decision is the judge() call alone, total the page so far, load and any retries included. A page that answered 4xx or 5xx is judged by its status without asking Jev, so it shows the status instead of a decision time:

keep   en.wikipedia.org/wiki/Headless_browser        article                 decision 0.25s  total  0.53s
keep   en.wikipedia.org/wiki/Web_scraping            article                 decision 0.29s  total  0.58s
keep   developer.mozilla.org/en-US/docs/Web/API/Fet  docs                    decision 0.24s  total  0.59s
keep   docs.python.org/3/library/asyncio.html        docs                    decision 0.27s  total  0.66s
keep   quotes.toscrape.com/js/                       listing                 decision 0.15s  total  0.91s
keep   books.toscrape.com/catalogue/a-light-in-the-  product                 decision 0.16s  total  1.18s
keep   lobste.rs/                                    listing                 decision 0.14s  total  0.91s
keep   www.spiegel.de/                               listing                 decision 0.30s  total  1.01s
keep   news.ycombinator.com/                         listing                 decision 0.16s  total  1.09s
keep   news.ycombinator.com/newest                   listing                 decision 0.14s  total  1.07s
keep   github.com/lightpanda-io/browser              docs                    decision 0.22s  total  1.64s
skip   news.ycombinator.com/login                    login wall              decision 0.14s  total  0.68s
skip   www.g2.com/                                   blocked                 http 403        total  0.35s
skip   www.wsj.com/                                  login wall              http 401        total  0.46s
skip   www.economist.com/leaders/2024/01/04/what-to  blocked                 http 403        total  0.25s
retry  www.20minutes.fr/                             consent wall            decision 0.19s  total  1.39s  -> accept cookies
skip   stackoverflow.com/questions/231767/what-does  blocked                 http 403        total  0.23s
skip   www.zeit.de/index                             blocked                 http 403        total  0.07s
skip   github.com/settings/profile                   login wall              decision 0.18s  total  1.39s
skip   www.linkedin.com/feed/                        login wall              decision 0.17s  total  1.26s
skip   www.npmjs.com/search?q=xqzzzvqkplorbt         blocked                 http 403        total  0.39s
keep   pypi.org/project/lightpanda/                  product                 decision 0.22s  total  1.06s
retry  www.elmundo.es/                               consent wall            decision 0.18s  total  2.76s  -> accept cookies
skip   web.whatsapp.com/                             unsupported browser     decision 0.15s  total  1.05s
skip   www.python.org/no-such-page                   error page              http 404        total  0.46s
skip   arxiv.org/abs/1706.03762                      blocked                 http 403        total  0.07s
skip   en.wikipedia.org/w/index.php?search=xqzzzvqk  empty catalog           decision 0.17s  total  1.22s
skip   github.com/search?q=xqzzzvqkplorbt&type=repo  empty catalog           decision 0.35s  total  1.73s
skip   www.ebay.com/sch/i.html?_nkw=xqzzzvqkplorbt   blocked                 http 403        total  0.77s
skip   github.com/lightpanda-io/does-not-exist       error page              http 404        total  1.23s
keep   www.elmundo.es/                               other                   decision 0.16s  total  5.04s
keep   www.20minutes.fr/                             listing                 decision 0.15s  total  6.62s

30 pages in 8.2s with 6 crawlers (34.7s of page time): 14 kept, 16 skipped
Jev: 22 calls, median 0.18s, p90 0.30s
skipped: blocked 7, login wall 4, empty catalog 2, error page 2, unsupported browser 1
acted:   accept cookies 2

What the walls actually showed: El Mundo's and 20 Minutes' cookie walls are popups over the front page that one click on the accept button clears, Zeit's is a German notice that it reserves its content against data mining, arXiv's is a bare Varnish 403, eBay's is "Something went wrong on our end", and Stack Overflow's is empty. Hacker News rate-limits three tabs opened at once, and says so in a sentence the crawler had no rule for.

Sites change and the model is not deterministic, so a run differs a little: how often Hacker News pushes back, or whether a challenge page scores as blocked or captcha. Each crawler drives its own session, and the browser loads their pages in parallel. Known misses: Zillow's home page scores as an empty search, and Le Monde's front page leans paywall (0.71).

Runs like classify_pages.py, from a release or from this checkout:

uv run examples/triage_crawl.py
LIGHTPANDA_BIN=../browser/zig-out/bin/lightpanda uv run python examples/triage_crawl.py