Repository navigation
feat(evaluation): block actions after unresolved refusal; verify claims against tool trace - #13
Merged
Ian-Tharp merged 4 commits intoNov 4, 2026
Conversation
…ms against tool trace
An unresolved refusal now stops the run. The comprehension gate's
clarify/refuse, or an LLM step whose output declines, trips a per-run
latch: every later step is blocked (never dispatched, recorded in
COREState.blocked_actions), and route_from_evaluation no longer sends
the run to retry_step/revise_plan around it. The response opens with
what stopped and what did not run. CORE_BLOCK_AFTER_REFUSAL=0 turns it
off for ablation.
Evaluation now checks first-person action claims in LLM step outputs
("I wrote config.yaml", "I committed the fix") against the dispatched
tool-call trace. StepResult.dispatched_tool records which tool actually
ran, so a planned tool is no longer evidence that it ran. Unsupported
claims cap eval confidence at 0.5, fail meets_requirements, and are
flagged in the response; next_action is unchanged, so no new loops.
Evaluation layer: claims_supported Golden Set evaluator, state_failed
counts unsupported claims, and fault reports stop listing
never-dispatched steps as tool calls and record refusal/blocked/claims
metadata.
Docs: docs/architecture/abstention-guard.md, plus status notes in the
eval spec, gate spec, docs index and changelog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4Wjwj7RkzmHFfcjuYcm
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration), #10 (coverage pass) and #11 (comprehension gate Phase 3). Conflict resolutions: - backend/app/evaluation/__init__.py: union of exports (develop's fault_reasons / maybe_capture_fault plus the claims API). - backend/app/evaluation/capture.py: kept develop's fault_reasons / state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate, never raises); the branch's unsupported-claim check becomes a new "unsupported_claims" fault reason instead of an early return in state_failed. Branch's dispatched_tool filter and refusal / blocked_actions / unsupported_claims metadata kept in fault_report_from_state. - backend/app/models/core_state.py: kept both field sets (refusal, blocked_actions, claim_report from the branch; step_checks from develop). - docs/evaluation_spec.md: auto-merged; added the unsupported_claims fault reason and the extra metadata keys to "Live fault capture". - tests/evaluation/test_claims.py: asserts fault_reasons == ["unsupported_claims"] for the unsupported-claim run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only conflict: docs/evaluation_spec.md, kept both the abstention-guard and gate-suite bullets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ian-Tharp
marked this pull request as ready for review
November 4, 2026 19:13
Ian-Tharp
added a commit
that referenced
this pull request
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opened from a finished Claude Code cloud session branch (the session pushed but didn't open a PR).
Commits
feat(evaluation): block actions after unresolved refusal; verify claims against tool trace
An unresolved refusal now stops the run. The comprehension gate's
clarify/refuse, or an LLM step whose output declines, trips a per-run
latch: every later step is blocked (never dispatched, recorded in
COREState.blocked_actions), and route_from_evaluation no longer sends
the run to retry_step/revise_plan around it. The response opens with
what stopped and what did not run. CORE_BLOCK_AFTER_REFUSAL=0 turns it
off for ablation.
Evaluation now checks first-person action claims in LLM step outputs
("I wrote config.yaml", "I committed the fix") against the dispatched
tool-call trace. StepResult.dispatched_tool records which tool actually
ran, so a planned tool is no longer evidence that it ran. Unsupported
claims cap eval confidence at 0.5, fail meets_requirements, and are
flagged in the response; next_action is unchanged, so no new loops.
Evaluation layer: claims_supported Golden Set evaluator, state_failed
counts unsupported claims, and fault reports stop listing
never-dispatched steps as tool calls and record refusal/blocked/claims
metadata.
Docs: docs/architecture/abstention-guard.md, plus status notes in the
eval spec, gate spec, docs index and changelog.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019KQ4Wjwj7RkzmHFfcjuYcm
🤖 Generated with Claude Code