Data collection, analysis, and charts for the Ravelry cohort study: what it took for a knitting or crochet designer to find an audience on Ravelry, by the year they started publishing, 2007–2024.
Python scripts pull designer, pattern, and project records from the
Ravelry API (plus the Wayback Machine and public Instagram pages), land
them as parquet under data/, a Jupyter notebook explores the result,
and export_*.py scripts write the aggregates that the Astro site in
astro/ charts.
Only aggregates are published. Per-designer datasets, raw API and HTML
dumps, and the executed notebook are git-ignored by design; the tracked
data files are pattern- and platform-level (data/manifests/,
data/project_dates.parquet) plus the anchor and roster lists used to
pick case-study designers. The JSON under astro/src/data/ holds the
chart series: cohort shares, accumulation curves, conversion rates, and
the per-pattern release timeline for a rule-selected cast of well-known
designers (public pattern metadata).
Research notes, drafts, and pitch material are kept locally and are not part of the repository.
Dependencies are managed with uv against the
pyproject.toml here (Python 3.12–3.13).
uv sync # runtime deps
uv sync --group dev # + jupyter, jupytext, seaborn (needed for the notebook)Ravelry API credentials are required for every fetch_*/probe_* script
that hits the API. Create a basic auth, read-only app at
https://www.ravelry.com/pro/developer, then copy .env.example to .env
and fill in the pair:
cp .env.example .env
# edit .env:
# RAVELRY_API_USERNAME=...
# RAVELRY_API_PASSWORD=...fetch_mkal.py and the --max-detail path of enrich_projects.py
call project-level endpoints that are not read-only API methods; they need
a second app of type basic auth: personal account (RAVELRY_PERSONAL_USERNAME
/ RAVELRY_PERSONAL_API_KEY in .env).
The client rate-limits itself to ~1 request/second (config.REQUEST_INTERVAL_S)
and retries on transient failures. The Instagram and Wayback scripts touch
public pages only (no login) and back off politely.
src/closingwindow/— shared library imported by every script (configpaths +.env,ravelryAPI client,idmappattern-ID↔year math,schemarow types,waybackcapture parsing).scripts/— runnable entry points (below).data/— inputs (anchors.yaml, rosters) and outputs (parquet, JSON manifests). See "Data posture" for what is tracked.notebooks/— the exploration notebook, kept as a.pyfile.astro/— the chart site (Astro + React + visx), deployed via Netlify (netlify.toml). Its inputs are the JSON files inastro/src/data/.reports/— rendered chart PNGs (git-ignored; regenerate with the screenshot pipeline below).docs/— method documentation: API field inventory, pilot report, hypotheses, the champion codebook, and the data-plan review.phase1-implementation-plan.md— the original technical plan for the API layer.
All scripts import the closingwindow package by relative name, so run
them through uv from the repo root (which puts scripts/ on the path):
uv run scripts/<name>.py [args]| Script | What it does | Output |
|---|---|---|
fetch_designers.py pilot|full |
Samples designers by random pattern ID across entering years (50 per year in full); keeps individuals with 5+ patterns; assigns each to a cohort by the year of their own first pattern. |
data/{pilot,full}/designers.parquet |
fetch_anchors.py |
Pulls records for the named case-anchor designers from data/anchors.yaml. |
data/pilot/anchors.parquet (+ .csv) |
audit_cohorts.py |
Re-dates designers whose platform entry predates their sampled first pattern; writes before/after corrections applied by the notebook and exports. | data/full/cohort_corrections.parquet |
fetch_pattern_level.py |
Full pattern catalogs (dates, favorites, projects, price) for cohort champions, anchors, and matched pairs. | data/full/pattern_level.parquet |
fetch_project_dates.py |
Project start dates for a basket of patterns — yearly engagement volume and per-pattern usage curves. | data/project_dates.parquet |
id_census.py |
Binary-searches pattern-ID year boundaries for patterns-added-per-year counts (handles the 2023 ID re-basing). | data/manifests/id_census.json |
project_census.py |
Scatter-samples project IDs and dates to estimate projects created per year — the "is Ravelry dying?" check. | data/manifests/project_census.json |
parse_wayback.py |
Parses archived designer-page captures in data/raw/html/ into per-pattern favorite counts over time. |
data/full/wayback_favorites.parquet |
fetch_ig_followers.py [max] |
Instagram follower counts from public profile meta tags; re-runs skip handles already fetched. | data/ig_followers.parquet |
fetch_ig_history.py [handle …] |
Follower/post history over time via Wayback captures. | data/ig_history.parquet |
fetch_mkal.py [--year …] [--refresh] |
Westknits MKAL study: pattern detail, the full project roster (dates, status, happiness) and pattern comments for every MKAL in data/mkal.yaml. Needs the personal-scope key. --refresh bypasses the cache and saves a dated snapshot (for the live year). |
data/mkal/patterns.parquet, data/mkal/projects/, data/mkal/comments/ |
enrich_projects.py --roster {mkal,yarn-brands} [--year …] [--only …] [--max-detail N] |
Adds project notes and yarn (packs) to every project on a roster pattern via projects/search.json (filtered locally by pattern id). For the MKAL roster, optional per-project fetches fill the residual; for the yarn-brand panel the search is the project list. |
data/mkal/notes/, data/yarn_brands/{notes,packs}/ |
Re-run these after any collection wave; the site reads their output
directly and git diff astro/src/data shows what moved.
| Script | Series | Output |
|---|---|---|
export_window.py |
Share of each entering class (2007–2024) reaching 100+ fans and 500–5,000 fans. | astro/src/data/window.json |
export_cadence.py |
Every pattern released by the rule-selected cast (2+ of: cohort champion, 20k+ fans, KnitStars roster, 3+ prestige-venue patterns), with favorites. | astro/src/data/cadence.json |
export_evergreen.py |
Wayback accumulation curve (share of lifetime favorites by pattern age) and favorites gained per year by age and period. | astro/src/data/evergreen.json |
export_conversion.py |
Projects per 100 favorites by publication year. | astro/src/data/conversion.json |
export_mkal_site.py |
Per-MKAL participation (KAL-window knitters, first-timers vs returning), project outcomes (on time / within 90 days / later / open / abandoned) and finishing curves for the /mkal/ page. |
astro/src/data/mkal.json |
export_mkal.py |
One aggregate row per Westknits MKAL: clue-1 date, projects, knitters, finish shares (ever / by clue 4 + 1 week / by month end), happiness, pattern-level demand. The only tracked file under data/mkal/. |
data/mkal/mkal_summary.csv |
One-off scripts used to map the API and validate assumptions. Safe to ignore for normal collection runs.
explore_api.py— dumps sample responses and a field inventory for the four endpoint families.probe_search.py— checkspatterns/search.jsonsorting/filtering behavior.probe_id_space.py— probes the sparse 1.3M–7.5M pattern-ID region.probe_html.py <username> …— checks whether join dates are visible on public HTML pages.probe_wayback.py [username …]— looks for join-date anchors in archived profile pages.probe_wayback_designer.py [permalink]— looks for fan counts / join info in archived designer pages.frame_screenshot.py <in> <out> "<label>"— wraps a screenshot in a browser-card frame for the essay.
notebooks/explore.py is the source of truth, paired to a notebook with
jupytext. It loads the full designer
collection if present (otherwise the pilot), applies cohort corrections if
they exist, and plots the shape of the data. Run it either way:
In VS Code — open notebooks/explore.py and run the # %% cells
directly in the interactive window.
In Jupyter Lab — convert and launch:
uv run jupytext --to ipynb notebooks/explore.py
uv run jupyter lab notebooks/explore.ipynbGenerated .ipynb files are git-ignored (their saved outputs contain
per-designer rows); edit the .py file and regenerate rather than
editing a notebook by hand.
notebooks/mkal.py is the companion notebook for the Westknits MKAL study
(clue windows, cast-on timing, finish rates, happiness, repeat
participation, yarn). Same jupytext workflow; it reads data/mkal/.
cd astro
npm install
npm run dev # http://localhost:4321
npm run screenshot # build, then render every chart to ../reports/*.pngThe screenshot pipeline (astro/scripts/screenshot.mjs, Playwright)
captures each chart twice: the full research view, and an _article
variant with badges, extra milestones, and method notes stripped for
use in the essay. The variant is selected with ?variant=article on the
page URL (astro/src/lib/variant.ts); the site's default view is the
full one.