Repository navigation
feat(evaluation): fail-safe fault capture wired into core_graph_v2 - #8
Merged
Merged
Conversation
Add app/config/evaluation.py (CORE_EVAL_CAPTURE_FAULTS, default off; CORE_EVAL_FAULTS_DIR, default backend/data/evaluation/faults, relative paths anchored to backend/). Add fault_reasons() (errors, eval_failure, step_failure, unresolved_evaluation) and maybe_capture_fault(), which never raises and logs capture failures at WARNING. Gitignore backend/data/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjE8Jvaj5tWHWkMdEb
Call maybe_capture_fault() once per run in the terminal conversation node, next to record_pipeline_run, behind a second try/except. Tests drive the real compiled graph with stubbed agents: flag off writes nothing and never calls capture; flag on writes exactly one report for a faulting run and none for clean or refused runs; an unwritable dir still answers the request and logs one warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjE8Jvaj5tWHWkMdEb
Update the spec status, explain where faults are detected, and expose the two settings in .env.example and docker-compose (default off). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjE8Jvaj5tWHWkMdEb
4 tasks done
Ian-Tharp
marked this pull request as ready for review
November 3, 2026 14:34
Ian-Tharp
added a commit
that referenced
this pull request
Nov 3, 2026
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration), #10 (coverage pass) and #11 (comprehension gate Phase 3). Conflict resolutions: - backend/app/evaluation/__init__.py: union of exports (develop's fault_reasons / maybe_capture_fault plus the claims API). - backend/app/evaluation/capture.py: kept develop's fault_reasons / state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate, never raises); the branch's unsupported-claim check becomes a new "unsupported_claims" fault reason instead of an early return in state_failed. Branch's dispatched_tool filter and refusal / blocked_actions / unsupported_claims metadata kept in fault_report_from_state. - backend/app/models/core_state.py: kept both field sets (refusal, blocked_actions, claim_report from the branch; step_checks from develop). - docs/evaluation_spec.md: auto-merged; added the unsupported_claims fault reason and the extra metadata keys to "Live fault capture". - tests/evaluation/test_claims.py: asserts fault_reasons == ["unsupported_claims"] for the unsupported-claim run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
backend/app/config/evaluation.py. It follows theapp/config/discord.pypattern: a pydantic model with env-backed defaults plusget_fault_capture_config(), read on every run.CORE_EVAL_CAPTURE_FAULTS: off by default.true/1/yes/onturns it on.CORE_EVAL_FAULTS_DIR: defaults tobackend/data/evaluation/faults. Relative paths resolve againstbackend/, not the cwd, so output never lands in the repo root. In Docker this is/app/data/...on the existingbackend-datavolume.backend/data/is now gitignored (it was already in.dockerignore).app/evaluation/capture.pyfault_reasons(state)returnserrors,eval_failure,step_failureandunresolved_evaluation.state_failed()now callsfault_reasons(). Existing behaviour stays the same, and the newunresolved_evaluationcase is added.maybe_capture_fault(state, config=None)never raises. Any failure is logged at WARNING with therun_idand swallowed. Each report'smetadatarecordsfault_reasons,gate_outcome,plan_revisions,eval_statusandeval_next_action.capture_fault()still raises, for callers that want to know.core_graph_v2.py—conversation_nodecallsmaybe_capture_fault(state)once, right afterrecord_pipeline_run, inside a second try/except.docs/evaluation_spec.mdstatus section and a new "Live fault capture" section..env.examplegets commented knobs.docker-compose.ymlpasses the two variables through, defaulting to off.Why: where a fault is detected
Following
docs/evaluation_spec.md§2 ("every failed prompt … is captured") and §3.C:conversation_node. Every path through the graph ends there: task runs, plain chat, gate clarify/refuse, and each node's exception fallback. So the check runs exactly once per run, on the finished state with the response already set. That gives "exactly one report per faulting run" without dedupe logic. Hooking each node's exception path would miss evaluator verdicts and could write several reports per run.state.errors). This matters because an evaluation crash is otherwise hidden behind a defaultoverall_status="success"; capture catches it througherrors.overall_status == "failure").unresolved_evaluation): the run ended while the evaluator still asked to revise or retry. This covers the spec's §3.B.2 "infinite loops" failure pattern.How to Test
tests/evaluation/test_fault_capture_wiring.pyruns the real compiled LangGraph with stubbed agents. There are no LLM calls, and everything is written undertmp_path.capture_faultandfault_report_from_stateare never called.run_id, input, reasons and step records.run_idis logged.backend/data, truthy/falsey values, relative paths anchored tobackend/.Mutation-checked:
Gaps / follow-ups
faults.jsonl.GoldenSet.append_fault()exists but needs anexpectedoutcome, which still needs a human or the async analyzer (Phase III).COREStatedoesn't carry them yet (Phase I / OTEL).engine.pypaths that bypass the graph (if any) are not covered. Only runs that reachconversation_nodeare captured.Checklist
Generated by Claude Code