- Single-project TypeScript CLI (pmx) plus Python utilities.
- Stack: Node 20+, TypeScript (strict), commander, better-sqlite3, Bottleneck, Vitest; Python uses pandas/numpy/sklearn.
- Sub-guides live in
src/AGENTS.md(TS data plane) andpython/AGENTS.md(model plane).
- Install:
npm ci - Build:
npm run build - Typecheck:
npm run typecheck - Test:
npm test - Dev CLI (hot):
npm run dev -- <cmd>e.g.npm run dev -- resolve "@handle" - Dashboard (web):
npm run dashboard:build, thennode scripts/pipeline-dashboard.js --out-root <dir> --port 9090(enable charts with--metrics-out auto; rebuild run registry withnode scripts/rebuild-run-registry.js --out-root out --write). - Command choice:
scripts/pipeline-coordinator.jsfor live/dashboard runs;scripts/pipeline-validate.jsfor CI-style pass/fail gating. - Docs routing for agents: see
docs/AGENTS.md(kept minimal and current). - Docs folder map:
docs/README.md. - Parquet-first outputs:
pmx extractandpmx watchwrite Parquet by default; use--emit-jsonto keep JSON tables. Parquet output is chunked to <=1000 rows per file to avoid high-cardinality string corruption.
- TS strict; reuse utilities (
src/utils/http.ts,src/utils/crypto.ts,src/utils/time.ts). - Config/env defaults only via
src/config/index.ts; avoid ad-hocprocess.envin modules. - Append-only outputs in
out/; runtime state in SQLite atstate/db.sqlite. - Keep commands copy-pasteable; no committed outputs/state/secrets.
- Coordinator defaults live in
scripts/pipeline-coordinator.jsand must match docs indocs/pipeline/pipeline-runbook.mdanddocs/pipeline/pipeline-status.md.
- Use the plan tool for multi-step or multi-sprint work; skip for trivial edits.
- Treat each sprint as a checklist item; sub-plan at sprint start.
- Update the plan after completing meaningful subtasks, not just at the end.
- Log evidence (runs, validation outputs, summaries) in docs as you go.
- If blocked or assumptions change, pause and ask before continuing.
- Use
python3in commands (notpython) to avoid missing interpreter errors. - Prefer windowed runs (
--start/--end) for large wallets; all-time runs are intentional and opt-in only. - Prefer small pure modules; structured logs where possible.
- Trade fetch modes:
strict(single-market, most reliable; default),packed(volume-target chunking using cached per-market trade counts), orcustom(manual chunk sizes/targets). - Data API baseline defaults:
minTime=52ms,concurrency=7,tradeChunkConcurrency=14(override viaDATA_API_RATE_LIMIT_MIN_TIME_MS,DATA_API_RATE_LIMIT_CONCURRENT,DATA_API_TRADE_CHUNK_CONCURRENCY). - Packed default behavior: activity-sliced trade count preseed is enabled for cold windows; disable with
--no-trade-activity-preseedif you need a pure cached baseline. - Trade pagination mitigations: trunc-split for multi-market chunks, time-sliced single-market fallback, optional verification that can auto-promote to strict.
- Trade repair pass (selective): enable with
--trade-repair-passorDATA_API_TRADE_REPAIR_PASS=true; use--trade-repair-repeat-pagesto target repeat-page markets only,--trade-repair-missing-markets/DATA_API_TRADE_REPAIR_MISSING_MARKETS=truefor missing markets, and--trade-repair-missing-start/DATA_API_TRADE_REPAIR_MISSING_START=truefor missing-start coverage; tune with--trade-repair-slice-minutesor--trade-repair-slice-secondsand--trade-repair-batch-size. - Repeat-page slicing override (opt-in):
DATA_API_TRADE_REPEAT_PAGE_SLICE_MINUTES=2(andDATA_API_TRADE_REPEAT_PAGE_SLICE_MAX_DEPTH=16) to tighten slicing only when repeat-pages are detected. - Activity slicing default is 5 minutes with max depth 6; 2-minute slices did not improve activity counts and triggered trade truncation in experiments.
- For windowed runs, prefer
--condition-enum-strategy goldsky-orderbookto avoid activity enumeration truncation. - When running Python snippets, use
python3(notpython), sincepythonmay be unavailable in some environments. - Merge windowed extracts with
pmx merge; usescripts/verify-merge-output.jsto validate merged outputs. - Merge windowed CLOB snapshots with
pmx merge-clobto produce a single CLOB directory forml:generate-dataset(avoids a second Dome backfill). - Merge runbook:
docs/merge/merge-runbook.mdfor fixtures, verification, and benchmarks. - Historical CLOB is decoupled: use
pmx dome:backfill(resumable, rate-limited) pluspmx ml:generate-dataset --clob;--with-clobonly hits live CLOB for active markets. Dome backfill defaults to L1-only; use--full-bookfor full depth. Snapshots are.ndjson.zstby default with L1 parquet intables/clob_snapshot_l1. - Dome backfill env baseline:
DOME_RATE_LIMIT_QPS=100,DOME_TOKEN_CONCURRENCY=2,DOME_BACKFILL_MAX_TOKEN_CONCURRENCY=48(override via env or--dome-token-concurrency). Coordinator default overrides this to 20; seedocs/pipeline/pipeline-runbook.md. - ML dataset generation zero-imputes nullable numeric features and adds
missing_<feature>indicators; include those in training. - Output timing debug:
OUTPUT_DEBUG_TIMINGS=truelogs NDJSON/JSON fallback/Parquet write timings. - Output timing summary:
node scripts/aggregate-output-timings.js --log <file> [--log <file> ...] [--out <json>]. - Trade fetch report:
OUTPUT_TRADE_FETCH_REPORT_PATH=truewritesreports/trade-fetch-report.json; aggregate withnode scripts/aggregate-trade-fetch-metrics.js --report <file> [--report <file> ...] [--out <json>]. UseDATA_API_TRADE_SLICE_FALLBACK_MINUTES=2to retry slicing with smaller windows when truncation occurs; useDATA_API_TRADE_REPEAT_PAGE_BATCH_SIZES=100,50,25to downshift batch size on repeat-page stalls. - Extract profiling:
pmx extract --profile-httpwritesreports/extract-http-metrics.json(per-host/route counts, bytes, throttle/queue waits) andreports/extract-stage-timings.json(stage timings). - Parquet verification:
node scripts/verify-parquet.js --out <dir> [--check-keys]samples partitions and checks parquet row counts (and optional key integrity) vs JSON fallback. - Raw outputs: compressed (
.ndjson.zst) by default; use--keep-raw-uncompressedorOUTPUT_KEEP_RAW_UNCOMPRESSED=trueto retain uncompressed NDJSON. - Parquet compaction:
pmx compact --out <dir> [--dry-run]compacts partitions; row-group sizing viaOUTPUT_PARQUET_ROW_GROUP_SIZEor--parquet-row-group-size. - Auto-compact is enabled by default for >=2h windows; use
--no-compact-afterto disable and--compact-after-hoursorOUTPUT_COMPACT_AFTER_HOURSto adjust the threshold. - Parquet wasm debug:
OUTPUT_DEBUG_PARQUET_WASM=truelogs parquet-wasm import + write details;node scripts/test-parquet-wasm.js --out <dir>runs a small smoke test. - Full extraction verification:
node scripts/verify-extract-output.js --out <dir> --mode fullvalidates structure, row counts, and parquet/JSON parity.
- Never commit tokens/keys; endpoints configurable via env (
GAMMA_API_URL,DATA_API_URL,CLOB_API_URL,STATE_DB_PATH,POLYGON_RPC_URLfor optional address resolution). - SQLite state is local runtime only; treat as disposable.
- No PII expected; avoid expanding identity data beyond API responses.
- TS data plane:
src/→ seesrc/AGENTS.md - Python model plane:
python/→ seepython/AGENTS.md - State/output:
state/(SQLite),out/(artifacts),watchlist.example.json(config sample) - Overview:
docs/overview.md(full project brief) - Quick find commands:
- CLI commands:
rg -n "command\(" src/cli.ts - Trade fetch modes/guards:
rg -n "tradeMode|verifyTrades|getTradesByMarkets" src/ingest/tradeFetch.ts - Trade chunking/pagination:
rg -n "getTradesByMarkets|trunc" src/clients/dataApiClient.ts - Trade count cache:
rg -n "market_trade_counts" src/state - Watcher loop:
rg -n "class Watcher" src/ingest/watcher.ts - Normalizers/schemas:
rg -n "export function normalize" src/normalize - Tests:
rg -n "describe" src/test - HTTP/rate limit:
rg -n "fetchWithRetry" src/utils/http.ts - WebSockets (optional enrichment):
rg -n "wsManager" src/clients/ws
- CLI commands:
npm run typecheck && npm test && npm run buildpass.- No stray TODOs in touched files; docs/commands accurate.
- Outputs/state not committed.
- Add unit tests for new helpers or logic (
src/test/*.test.ts). - If you touch orchestration/merge/dataset wiring, add or update pipeline coordinator tests.
- For storage/dataset changes, update or add verification scripts (
scripts/verify-*.js).