Skip to content

feat(evaluation): paired-twin suite for the comprehension gate - #12

Merged
Ian-Tharp merged 1 commit into
developfrom
claude/comprehension-gate-eval-suite-5konn4
Nov 4, 2026
Merged

Ian-Tharp merged 1 commit into
developfrom
claude/comprehension-gate-eval-suite-5konn4

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

Opened from a finished Claude Code cloud session branch (the session pushed but didn't open a PR).

Commits

feat(evaluation): paired-twin suite for the comprehension gate

Scores the proceed/clarify/refuse front gate on 35 paired act/abstain
twins. Reports paired accuracy (both twins right), act vs abstain accuracy,
clarify/refuse accuracy and over-refusal, per category, with the regex
ambiguity merge as an ablation and an always-act baseline.

Baseline (gate only): paired 68.6%, act 85.7%, abstain 82.9%,
over-refusal 14.3%. Failures: missing actions on present tool roots
(git push, file delete) proceed instead of refusing; the regex ambiguity
detector blocks fully specified requests. Tests pin these as known gaps
and enforce floors.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019KQ4WioYDCWsAkJGW511z4

🤖 Generated with Claude Code

Scores the proceed/clarify/refuse front gate on 35 paired act/abstain
twins. Reports paired accuracy (both twins right), act vs abstain accuracy,
clarify/refuse accuracy and over-refusal, per category, with the regex
ambiguity merge as an ablation and an always-act baseline.

Baseline (gate only): paired 68.6%, act 85.7%, abstain 82.9%,
over-refusal 14.3%. Failures: missing actions on present tool roots
(git push, file delete) proceed instead of refusing; the regex ambiguity
detector blocks fully specified requests. Tests pin these as known gaps
and enforce floors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WioYDCWsAkJGW511z4
Ian-Tharp added a commit that referenced this pull request Nov 3, 2026
Only conflict: docs/evaluation_spec.md, kept both the abstention-guard and gate-suite bullets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Ian-Tharp
Ian-Tharp marked this pull request as ready for review November 4, 2026 19:13
@Ian-Tharp
Ian-Tharp merged commit 540f79a into develop Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants