Skip to content

feat(evaluation): fault capture, golden set and replay runner (Phase I/II slice) - #6

Merged
Ian-Tharp merged 3 commits into
developfrom
claude/core-evaluation-phase-1-2-yzv5oj
Nov 2, 2026
Merged

Ian-Tharp merged 3 commits into
developfrom
claude/core-evaluation-phase-1-2-yzv5oj

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

What Changed

New package backend/app/evaluation/:

  • FaultReport (pydantic v2, JSON round-trip) plus fault_report_from_state() / capture_fault() (JSONL on disk).
  • GoldenSet: JSONL store with stable-signature dedupe, append_fault(), and corrupt-line tolerance.
  • Deterministic evaluators: exact match, contains, regex, max latency, forbidden-substring guardrail.
  • run_cases() / arun_cases(): replay cases through an injected callable, return typed per-case results plus a summary.
  • docs/evaluation/golden/seed.jsonl: 10 seed cases. docs/evaluation_spec.md gets status and deviation notes; no roadmap box is ticked.

Why

Phase I/II groundwork from docs/evaluation_spec.md: failures become data, and a Golden Set can be replayed offline. No LLM, DB, OTEL or new services.

How to Test

cd backend
uv run black --check app/ tests/
uv run pytest tests/evaluation -q   # 33 tests
uv run pytest -q                    # full suite: 2774 passed, 1 skipped

Notes / deviations

  • Not wired into the pipeline: core/langgraph/core_graph.py is an unused stub; the live graph is core_graph_v2.py. A default-off hook in conversation_node is a follow-up.
  • No OpenTelemetry, so Phase I stays open. 10 of the 100 Phase II cases exist.
  • app/eval (verdict-drift harness) is untouched.

Checklist

  • Tests pass
  • Code follows project style guidelines (black)
  • Documentation updated (if applicable)
  • Conventional commit messages used

Generated by Claude Code

Ian-Tharp and others added 3 commits November 2, 2026 09:52
Adds evaluation_spec.md and AGENTS.md so cloud sessions can see them.
.codex/ and the WIP core_graph_workspace test are intentionally left out.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds app/evaluation: FaultReport model + COREState adapter, JSONL golden
set with signature dedupe and corrupt-line tolerance, deterministic
evaluators (exact, contains, regex, max latency, forbidden substring) and
a runner that replays cases through an injected callable.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4Wi8WgwdUJKScfnBeGG
@Ian-Tharp
Ian-Tharp marked this pull request as ready for review November 2, 2026 22:07
@Ian-Tharp
Ian-Tharp merged commit d40bf9f into develop Nov 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants