Skip to content

feat(evaluation): block actions after unresolved refusal; verify claims against tool trace - #13

Merged
Ian-Tharp merged 4 commits into
developfrom
claude/block-actions-unresolved-refusal-vec5hb
Nov 4, 2026
Merged

Ian-Tharp merged 4 commits into
developfrom
claude/block-actions-unresolved-refusal-vec5hb

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

Opened from a finished Claude Code cloud session branch (the session pushed but didn't open a PR).

Commits

feat(evaluation): block actions after unresolved refusal; verify claims against tool trace

An unresolved refusal now stops the run. The comprehension gate's
clarify/refuse, or an LLM step whose output declines, trips a per-run
latch: every later step is blocked (never dispatched, recorded in
COREState.blocked_actions), and route_from_evaluation no longer sends
the run to retry_step/revise_plan around it. The response opens with
what stopped and what did not run. CORE_BLOCK_AFTER_REFUSAL=0 turns it
off for ablation.

Evaluation now checks first-person action claims in LLM step outputs
("I wrote config.yaml", "I committed the fix") against the dispatched
tool-call trace. StepResult.dispatched_tool records which tool actually
ran, so a planned tool is no longer evidence that it ran. Unsupported
claims cap eval confidence at 0.5, fail meets_requirements, and are
flagged in the response; next_action is unchanged, so no new loops.

Evaluation layer: claims_supported Golden Set evaluator, state_failed
counts unsupported claims, and fault reports stop listing
never-dispatched steps as tool calls and record refusal/blocked/claims
metadata.

Docs: docs/architecture/abstention-guard.md, plus status notes in the
eval spec, gate spec, docs index and changelog.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019KQ4Wjwj7RkzmHFfcjuYcm

🤖 Generated with Claude Code

claude and others added 3 commits November 3, 2026 03:37
…ms against tool trace

An unresolved refusal now stops the run. The comprehension gate's
clarify/refuse, or an LLM step whose output declines, trips a per-run
latch: every later step is blocked (never dispatched, recorded in
COREState.blocked_actions), and route_from_evaluation no longer sends
the run to retry_step/revise_plan around it. The response opens with
what stopped and what did not run. CORE_BLOCK_AFTER_REFUSAL=0 turns it
off for ablation.

Evaluation now checks first-person action claims in LLM step outputs
("I wrote config.yaml", "I committed the fix") against the dispatched
tool-call trace. StepResult.dispatched_tool records which tool actually
ran, so a planned tool is no longer evidence that it ran. Unsupported
claims cap eval confidence at 0.5, fail meets_requirements, and are
flagged in the response; next_action is unchanged, so no new loops.

Evaluation layer: claims_supported Golden Set evaluator, state_failed
counts unsupported claims, and fault reports stop listing
never-dispatched steps as tool calls and record refusal/blocked/claims
metadata.

Docs: docs/architecture/abstention-guard.md, plus status notes in the
eval spec, gate spec, docs index and changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4Wjwj7RkzmHFfcjuYcm
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration),
#10 (coverage pass) and #11 (comprehension gate Phase 3).

Conflict resolutions:
- backend/app/evaluation/__init__.py: union of exports (develop's
  fault_reasons / maybe_capture_fault plus the claims API).
- backend/app/evaluation/capture.py: kept develop's fault_reasons /
  state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate,
  never raises); the branch's unsupported-claim check becomes a new
  "unsupported_claims" fault reason instead of an early return in
  state_failed. Branch's dispatched_tool filter and refusal /
  blocked_actions / unsupported_claims metadata kept in
  fault_report_from_state.
- backend/app/models/core_state.py: kept both field sets (refusal,
  blocked_actions, claim_report from the branch; step_checks from
  develop).
- docs/evaluation_spec.md: auto-merged; added the unsupported_claims
  fault reason and the extra metadata keys to "Live fault capture".
- tests/evaluation/test_claims.py: asserts fault_reasons ==
  ["unsupported_claims"] for the unsupported-claim run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only conflict: docs/evaluation_spec.md, kept both the abstention-guard and gate-suite bullets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Ian-Tharp
Ian-Tharp marked this pull request as ready for review November 4, 2026 19:13
@Ian-Tharp
Ian-Tharp merged commit ffc6e28 into develop Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants