Skip to content

feat(evaluation): fault capture + golden set runner (Phase I/II slice) - #5

Closed
Ian-Tharp wants to merge 5 commits into
developfrom
claude/core-evaluation-phase-1-2-i7ol19
Closed

Ian-Tharp wants to merge 5 commits into
developfrom
claude/core-evaluation-phase-1-2-i7ol19

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

What Changed

New package backend/app/evaluation/ (no DB, no LLM, no OTEL, no new services):

  • models.py: FaultReport / StepRecord / ToolCallRecord (pydantic v2, JSON round-trip, derived latency/total tokens).
  • capture.py: fail-safe capture_fault() hook, default-off (CORE_EVAL_CAPTURE_FAULTS), appends to faults.jsonl.
  • golden_set.py: JSONL GoldenSet with load/validate, stable-signature duplicate rejection, corrupt-line tolerance, add_fault().
  • evaluators.py: exact match, contains, regex, max-latency, forbidden-substring guardrail.
  • runner.py: run_cases(cases, callable) returning typed per-case results plus an aggregate summary.
  • docs/evaluation/golden/seed.jsonl: 10 seed cases; docs/evaluation_spec.md roadmap updated (sub-items only; no phase fully ticked).

Why

Implements the smallest working slice of Phase I (fault capture) and Phase II (golden set) from docs/evaluation_spec.md.

How to Test

cd backend
uv run black --check app/ tests/
uv run pytest --tb=short -q      # 2768 passed (2741 baseline + 27 new)

Deviations / notes

  • No call site in core_graph.py: it is a dead, non-functional stub (nothing imports it; the live graph is core_graph_v2.py). capture_fault() is ready for the engine failure path; wiring is a follow-up.
  • Roadmap Phase I (OTEL) remains open; faults are plain JSON.
  • Separate from the existing app/eval drift benchmark.
  • Branch is based on the chore/cloud-wave1-seed docs commit (4cc9cd9), which is not yet in develop, so that commit appears in this PR.
  • Seed cases are fixtures validated against a fake callable, not recorded from real runs.

Checklist

  • Tests pass
  • Code follows project style guidelines
  • Documentation updated (if applicable)
  • Conventional commit messages used

UI / UX changes

N/A: no frontend changes.


Generated by Claude Code

Ian-Tharp and others added 5 commits November 2, 2026 09:52
Adds evaluation_spec.md and AGENTS.md so cloud sessions can see them.
.codex/ and the WIP core_graph_workspace test are intentionally left out.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4Wm82kwv8haybKWL4gf
@Ian-Tharp

Copy link
Copy Markdown
Owner Author

Duplicate of #6 (same task, parallel cloud run). #6 merged.

@Ian-Tharp Ian-Tharp closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants