Skip to content

feat(evaluation): fail-safe fault capture wired into core_graph_v2 - #8

Merged
Ian-Tharp merged 3 commits into
developfrom
claude/fault-capture-fail-safe-v2bxyj
Nov 3, 2026
Merged

Ian-Tharp merged 3 commits into
developfrom
claude/fault-capture-fail-safe-v2bxyj

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

What Changed

  • Settings — new backend/app/config/evaluation.py. It follows the app/config/discord.py pattern: a pydantic model with env-backed defaults plus get_fault_capture_config(), read on every run.
    • CORE_EVAL_CAPTURE_FAULTS: off by default. true/1/yes/on turns it on.
    • CORE_EVAL_FAULTS_DIR: defaults to backend/data/evaluation/faults. Relative paths resolve against backend/, not the cwd, so output never lands in the repo root. In Docker this is /app/data/... on the existing backend-data volume. backend/data/ is now gitignored (it was already in .dockerignore).
  • app/evaluation/capture.py
    • fault_reasons(state) returns errors, eval_failure, step_failure and unresolved_evaluation.
    • state_failed() now calls fault_reasons(). Existing behaviour stays the same, and the new unresolved_evaluation case is added.
    • maybe_capture_fault(state, config=None) never raises. Any failure is logged at WARNING with the run_id and swallowed. Each report's metadata records fault_reasons, gate_outcome, plan_revisions, eval_status and eval_next_action.
    • capture_fault() still raises, for callers that want to know.
  • core_graph_v2.py — conversation_node calls maybe_capture_fault(state) once, right after record_pipeline_run, inside a second try/except.
  • Docs / config — docs/evaluation_spec.md status section and a new "Live fault capture" section. .env.example gets commented knobs. docker-compose.yml passes the two variables through, defaulting to off.

Why: where a fault is detected

Following docs/evaluation_spec.md §2 ("every failed prompt … is captured") and §3.C:

  • One place: the end of conversation_node. Every path through the graph ends there: task runs, plain chat, gate clarify/refuse, and each node's exception fallback. So the check runs exactly once per run, on the finished state with the response already set. That gives "exactly one report per faulting run" without dedupe logic. Hooking each node's exception path would miss evaluator verdicts and could write several reports per run.
  • What counts as a fault:
    • Exception paths (state.errors). This matters because an evaluation crash is otherwise hidden behind a default overall_status="success"; capture catches it through errors.
    • Failed evaluation (overall_status == "failure").
    • Failed plan step.
    • Loop-guard forced finalization (unresolved_evaluation): the run ended while the evaluator still asked to revise or retry. This covers the spec's §3.B.2 "infinite loops" failure pattern.
  • Not a fault: a front-gate refusal or clarification on its own. That is the gate working as designed. Wrong refusals should become Golden Set cases, not automatic captures.

How to Test

cd backend
uv run pytest tests/evaluation -q           # 56 passed
uv run pytest -q                            # 2850 passed, 1 skipped (baseline 2827)
uv run black --check app/ tests/            # clean

tests/evaluation/test_fault_capture_wiring.py runs the real compiled LangGraph with stubbed agents. There are no LLM calls, and everything is written under tmp_path.

  • Flag off: a faulting run writes no file, and capture_fault and fault_report_from_state are never called.
  • Flag on, faulting run: exactly one JSONL line, with matching run_id, input, reasons and step records.
  • Flag on, clean run or gate refusal: no file.
  • Unwritable dir (a regular file where the directory should be, so it also fails as root): the request still succeeds, there is no new state error, and exactly one WARNING containing the run_id is logged.
  • A bug while building the report: swallowed and logged.
  • Config: default off, default dir under backend/data, truthy/falsey values, relative paths anchored to backend/.

Mutation-checked:

  • Removing the call site fails 3 tests.
  • Breaking the inner guard is still caught by the outer one, so the request survives; only the run_id-in-log assertion fails.

Gaps / follow-ups

  • Capture is synchronous. It is one small append, and only when the flag is on and the run faulted. Spec §2 prefers decoupled evaluation, so a background writer is a follow-up if latency ever shows it.
  • No rotation or size cap on faults.jsonl.
  • No automatic promotion of faults into the Golden Set. GoldenSet.append_fault() exists but needs an expected outcome, which still needs a human or the async analyzer (Phase III).
  • Token counts are not recorded, because COREState doesn't carry them yet (Phase I / OTEL).
  • The async engine.py paths that bypass the graph (if any) are not covered. Only runs that reach conversation_node are captured.

Checklist

  • Tests pass
  • Code follows project style guidelines (black)
  • Documentation updated
  • Conventional commit messages used

Generated by Claude Code

Add app/config/evaluation.py (CORE_EVAL_CAPTURE_FAULTS, default off;
CORE_EVAL_FAULTS_DIR, default backend/data/evaluation/faults, relative
paths anchored to backend/). Add fault_reasons() (errors, eval_failure,
step_failure, unresolved_evaluation) and maybe_capture_fault(), which
never raises and logs capture failures at WARNING. Gitignore backend/data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjE8Jvaj5tWHWkMdEb
Call maybe_capture_fault() once per run in the terminal conversation node,
next to record_pipeline_run, behind a second try/except. Tests drive the
real compiled graph with stubbed agents: flag off writes nothing and never
calls capture; flag on writes exactly one report for a faulting run and
none for clean or refused runs; an unwritable dir still answers the
request and logs one warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjE8Jvaj5tWHWkMdEb
Update the spec status, explain where faults are detected, and expose the
two settings in .env.example and docker-compose (default off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjE8Jvaj5tWHWkMdEb
@Ian-Tharp
Ian-Tharp marked this pull request as ready for review November 3, 2026 14:34
@Ian-Tharp
Ian-Tharp merged commit 4c4f6e2 into develop Nov 3, 2026
2 checks passed
Ian-Tharp added a commit that referenced this pull request Nov 3, 2026
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration),
#10 (coverage pass) and #11 (comprehension gate Phase 3).

Conflict resolutions:
- backend/app/evaluation/__init__.py: union of exports (develop's
  fault_reasons / maybe_capture_fault plus the claims API).
- backend/app/evaluation/capture.py: kept develop's fault_reasons /
  state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate,
  never raises); the branch's unsupported-claim check becomes a new
  "unsupported_claims" fault reason instead of an early return in
  state_failed. Branch's dispatched_tool filter and refusal /
  blocked_actions / unsupported_claims metadata kept in
  fault_report_from_state.
- backend/app/models/core_state.py: kept both field sets (refusal,
  blocked_actions, claim_report from the branch; step_checks from
  develop).
- docs/evaluation_spec.md: auto-merged; added the unsupported_claims
  fault reason and the extra metadata keys to "Live fault capture".
- tests/evaluation/test_claims.py: asserts fault_reasons ==
  ["unsupported_claims"] for the unsupported-claim run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants