Skip to content

Latest commit

 

History

History
94 lines (86 loc) · 8.68 KB

AGENTS.md

File metadata and controls

94 lines (86 loc) · 8.68 KB

Polywinner AGENT Guide (Root)

Project Snapshot

  • Single-project TypeScript CLI (pmx) plus Python utilities.
  • Stack: Node 20+, TypeScript (strict), commander, better-sqlite3, Bottleneck, Vitest; Python uses pandas/numpy/sklearn.
  • Sub-guides live in src/AGENTS.md (TS data plane) and python/AGENTS.md (model plane).

Root Setup Commands

  • Install: npm ci
  • Build: npm run build
  • Typecheck: npm run typecheck
  • Test: npm test
  • Dev CLI (hot): npm run dev -- <cmd> e.g. npm run dev -- resolve "@handle"
  • Dashboard (web): npm run dashboard:build, then node scripts/pipeline-dashboard.js --out-root <dir> --port 9090 (enable charts with --metrics-out auto; rebuild run registry with node scripts/rebuild-run-registry.js --out-root out --write).
  • Command choice: scripts/pipeline-coordinator.js for live/dashboard runs; scripts/pipeline-validate.js for CI-style pass/fail gating.
  • Docs routing for agents: see docs/AGENTS.md (kept minimal and current).
  • Docs folder map: docs/README.md.
  • Parquet-first outputs: pmx extract and pmx watch write Parquet by default; use --emit-json to keep JSON tables. Parquet output is chunked to <=1000 rows per file to avoid high-cardinality string corruption.

Universal Conventions

  • TS strict; reuse utilities (src/utils/http.ts, src/utils/crypto.ts, src/utils/time.ts).
  • Config/env defaults only via src/config/index.ts; avoid ad-hoc process.env in modules.
  • Append-only outputs in out/; runtime state in SQLite at state/db.sqlite.
  • Keep commands copy-pasteable; no committed outputs/state/secrets.
  • Coordinator defaults live in scripts/pipeline-coordinator.js and must match docs in docs/pipeline/pipeline-runbook.md and docs/pipeline/pipeline-status.md.

Pre-implementation planning (agent workflow)

  • Use the plan tool for multi-step or multi-sprint work; skip for trivial edits.
  • Treat each sprint as a checklist item; sub-plan at sprint start.
  • Update the plan after completing meaningful subtasks, not just at the end.
  • Log evidence (runs, validation outputs, summaries) in docs as you go.
  • If blocked or assumptions change, pause and ask before continuing.
  • Use python3 in commands (not python) to avoid missing interpreter errors.
  • Prefer windowed runs (--start/--end) for large wallets; all-time runs are intentional and opt-in only.
  • Prefer small pure modules; structured logs where possible.
  • Trade fetch modes: strict (single-market, most reliable; default), packed (volume-target chunking using cached per-market trade counts), or custom (manual chunk sizes/targets).
  • Data API baseline defaults: minTime=52ms, concurrency=7, tradeChunkConcurrency=14 (override via DATA_API_RATE_LIMIT_MIN_TIME_MS, DATA_API_RATE_LIMIT_CONCURRENT, DATA_API_TRADE_CHUNK_CONCURRENCY).
  • Packed default behavior: activity-sliced trade count preseed is enabled for cold windows; disable with --no-trade-activity-preseed if you need a pure cached baseline.
  • Trade pagination mitigations: trunc-split for multi-market chunks, time-sliced single-market fallback, optional verification that can auto-promote to strict.
  • Trade repair pass (selective): enable with --trade-repair-pass or DATA_API_TRADE_REPAIR_PASS=true; use --trade-repair-repeat-pages to target repeat-page markets only, --trade-repair-missing-markets/DATA_API_TRADE_REPAIR_MISSING_MARKETS=true for missing markets, and --trade-repair-missing-start/DATA_API_TRADE_REPAIR_MISSING_START=true for missing-start coverage; tune with --trade-repair-slice-minutes or --trade-repair-slice-seconds and --trade-repair-batch-size.
  • Repeat-page slicing override (opt-in): DATA_API_TRADE_REPEAT_PAGE_SLICE_MINUTES=2 (and DATA_API_TRADE_REPEAT_PAGE_SLICE_MAX_DEPTH=16) to tighten slicing only when repeat-pages are detected.
  • Activity slicing default is 5 minutes with max depth 6; 2-minute slices did not improve activity counts and triggered trade truncation in experiments.
  • For windowed runs, prefer --condition-enum-strategy goldsky-orderbook to avoid activity enumeration truncation.
  • When running Python snippets, use python3 (not python), since python may be unavailable in some environments.
  • Merge windowed extracts with pmx merge; use scripts/verify-merge-output.js to validate merged outputs.
  • Merge windowed CLOB snapshots with pmx merge-clob to produce a single CLOB directory for ml:generate-dataset (avoids a second Dome backfill).
  • Merge runbook: docs/merge/merge-runbook.md for fixtures, verification, and benchmarks.
  • Historical CLOB is decoupled: use pmx dome:backfill (resumable, rate-limited) plus pmx ml:generate-dataset --clob; --with-clob only hits live CLOB for active markets. Dome backfill defaults to L1-only; use --full-book for full depth. Snapshots are .ndjson.zst by default with L1 parquet in tables/clob_snapshot_l1.
  • Dome backfill env baseline: DOME_RATE_LIMIT_QPS=100, DOME_TOKEN_CONCURRENCY=2, DOME_BACKFILL_MAX_TOKEN_CONCURRENCY=48 (override via env or --dome-token-concurrency). Coordinator default overrides this to 20; see docs/pipeline/pipeline-runbook.md.
  • ML dataset generation zero-imputes nullable numeric features and adds missing_<feature> indicators; include those in training.
  • Output timing debug: OUTPUT_DEBUG_TIMINGS=true logs NDJSON/JSON fallback/Parquet write timings.
  • Output timing summary: node scripts/aggregate-output-timings.js --log <file> [--log <file> ...] [--out <json>].
  • Trade fetch report: OUTPUT_TRADE_FETCH_REPORT_PATH=true writes reports/trade-fetch-report.json; aggregate with node scripts/aggregate-trade-fetch-metrics.js --report <file> [--report <file> ...] [--out <json>]. Use DATA_API_TRADE_SLICE_FALLBACK_MINUTES=2 to retry slicing with smaller windows when truncation occurs; use DATA_API_TRADE_REPEAT_PAGE_BATCH_SIZES=100,50,25 to downshift batch size on repeat-page stalls.
  • Extract profiling: pmx extract --profile-http writes reports/extract-http-metrics.json (per-host/route counts, bytes, throttle/queue waits) and reports/extract-stage-timings.json (stage timings).
  • Parquet verification: node scripts/verify-parquet.js --out <dir> [--check-keys] samples partitions and checks parquet row counts (and optional key integrity) vs JSON fallback.
  • Raw outputs: compressed (.ndjson.zst) by default; use --keep-raw-uncompressed or OUTPUT_KEEP_RAW_UNCOMPRESSED=true to retain uncompressed NDJSON.
  • Parquet compaction: pmx compact --out <dir> [--dry-run] compacts partitions; row-group sizing via OUTPUT_PARQUET_ROW_GROUP_SIZE or --parquet-row-group-size.
  • Auto-compact is enabled by default for >=2h windows; use --no-compact-after to disable and --compact-after-hours or OUTPUT_COMPACT_AFTER_HOURS to adjust the threshold.
  • Parquet wasm debug: OUTPUT_DEBUG_PARQUET_WASM=true logs parquet-wasm import + write details; node scripts/test-parquet-wasm.js --out <dir> runs a small smoke test.
  • Full extraction verification: node scripts/verify-extract-output.js --out <dir> --mode full validates structure, row counts, and parquet/JSON parity.

Security & Secrets

  • Never commit tokens/keys; endpoints configurable via env (GAMMA_API_URL, DATA_API_URL, CLOB_API_URL, STATE_DB_PATH, POLYGON_RPC_URL for optional address resolution).
  • SQLite state is local runtime only; treat as disposable.
  • No PII expected; avoid expanding identity data beyond API responses.

JIT Index (what to open, not what to paste)

  • TS data plane: src/ → see src/AGENTS.md
  • Python model plane: python/ → see python/AGENTS.md
  • State/output: state/ (SQLite), out/ (artifacts), watchlist.example.json (config sample)
  • Overview: docs/overview.md (full project brief)
  • Quick find commands:
    • CLI commands: rg -n "command\(" src/cli.ts
    • Trade fetch modes/guards: rg -n "tradeMode|verifyTrades|getTradesByMarkets" src/ingest/tradeFetch.ts
    • Trade chunking/pagination: rg -n "getTradesByMarkets|trunc" src/clients/dataApiClient.ts
    • Trade count cache: rg -n "market_trade_counts" src/state
    • Watcher loop: rg -n "class Watcher" src/ingest/watcher.ts
    • Normalizers/schemas: rg -n "export function normalize" src/normalize
    • Tests: rg -n "describe" src/test
    • HTTP/rate limit: rg -n "fetchWithRetry" src/utils/http.ts
    • WebSockets (optional enrichment): rg -n "wsManager" src/clients/ws

Definition of Done

  • npm run typecheck && npm test && npm run build pass.
  • No stray TODOs in touched files; docs/commands accurate.
  • Outputs/state not committed.

Testing expectations

  • Add unit tests for new helpers or logic (src/test/*.test.ts).
  • If you touch orchestration/merge/dataset wiring, add or update pipeline coordinator tests.
  • For storage/dataset changes, update or add verification scripts (scripts/verify-*.js).