Skip to content

feat(gate): Phase 3 — memory grounding + per-step evaluation - #11

Merged
Ian-Tharp merged 6 commits into
developfrom
claude/core-comprehension-gate-phase-3-zt5s4c
Nov 3, 2026
Merged

Ian-Tharp merged 6 commits into
developfrom
claude/core-comprehension-gate-phase-3-zt5s4c

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

What Changed

Implements Phase 3 of docs/architecture/comprehension-grounded-gate.md: both verifiers now get more self-knowledge.

1. Memory grounding (front verifier: "do I know what this relies on?") — backend/app/core/memory/

  • A small interface: MemoryGrounder.ground(query, user_id=, conversation_id=, k=) -> list[MemoryHit] (content, kind fact|episode, score).
  • StoreMemoryGrounder over LangGraph's BaseStore: InMemoryStore by default and in tests, PostgresStore when CORE_MEMORY_STORE=postgres and it's available (otherwise it logs a warning and falls back to the in-memory store). Namespaces are ("core_memory", <user|global>, <kind>). It uses the store's semantic score when one exists, otherwise a deterministic lexical score.
  • detect_knowledge_dependencies() finds strong cues that the request relies on prior knowledge ("like last time", "the usual", "as we discussed", "my preferred X", "my manager"). A bare "my"/"our" doesn't count, so "fix my code in main.py" doesn't clarify.
  • A cue that no memory hit covers is a knowledge gap. Gaps are appended to intent.ambiguities, so the existing tri-state gate clarifies; the gate logic is untouched. Hits are appended to intent.context_retrieved, and the assessment is recorded in COREState.memory_grounding.
  • Behind CORE_COMPREHENSION_MEMORY_GROUNDING, default off. When it's off, the grounder is never built and routing is identical to today.

2. Per-step evaluation (back verifier) — backend/app/evaluation/step_checks.py

  • A deterministic check per executed step: has a result, no error or partial status, non-empty output, and outputs match the step's declared PlanStep.output_schema (new optional field, validated with the existing _schema_validate).
  • Recorded in COREState.step_checks. Failing checks are passed to EvaluationAgent.evaluate_execution(step_checks=...): they're listed in the LLM prompt, and the rule-based fallback counts them as failures. A clean run calls the evaluator with the same kwargs and prompt as before. No extra LLM calls.
  • The evaluation system prompt now names retry_step (execution flaked) vs revise_plan (intent misalignment), per spec §4.4.

3. core_graph_v2: additive changes only (+58/−1), and no routing function changed:

Why the LangGraph store and not memory_service

  • comprehension_node is synchronous and the front gate must stay cheap, deterministic and offline-testable.
  • memory_service is async, needs the asyncpg pool and an Ollama embedding call per query, and is scoped per agent rather than per user.
  • BaseStore has a sync API, swaps cleanly between InMemoryStore and PostgresStore, and is what spec §4.5 already names ("LangGraph PostgresStore over CORE pgvector").
  • A memory_service adapter can implement the same MemoryGrounder interface later.
  • No new dependencies: PostgresStore lives in langgraph-checkpoint-postgres, which isn't installed, so it's imported optionally.

Why

CORE is a deliberation loop between two verifiers. Phase 1/2 grounded comprehension in what it can do (Capability Registry). This adds what it knows: requests that depend on facts CORE doesn't have now clarify instead of running O→R→E on a guess. Evaluation also gets a cheap deterministic signal per step, so a step that "succeeded" with empty or ill-shaped output can't be silently finalized.

How to Test

cd backend
uv run black --check app/ tests/
uv run pytest -q   # 2922 passed, 1 skipped (baseline on develop: 2827)
uv run pytest tests/test_memory_grounding.py tests/test_step_checks.py tests/test_phase3_graph_integration.py -q
  • test_memory_grounding.py: cue detection, scoring, user/global scoping, error tolerance, gaps → gate clarify, store factory fallbacks.
  • test_step_checks.py: each failure kind, retry semantics, prompt and fallback integration in EvaluationAgent.
  • test_phase3_graph_integration.py: flag off = identical gate outcome, message and route to a graph with grounding stubbed, across proceed/clarify/refuse/conversation/cue inputs. With the flag on, gaps clarify and short-circuit to ["comprehension", "conversation"]. Also covers covered cues proceeding, grounder errors degrading safely, and full compiled-graph runs. All LLMs are mocked; no network.

Follow-ups (also in spec §9)

  • When checks run: after the reasoning node, not between steps. Stopping early on a bad step needs a hook in ReasoningAgent.execute_plan.
  • No write path into memory yet: nothing consolidates conversations into the store. With the flag on and an empty store, every cue clarifies. That's why the flag defaults off.
  • Fault capture: state_failed doesn't count failing step checks yet. I'm leaving that until feat(evaluation): fail-safe fault capture wired into core_graph_v2 #8 lands.
  • Reasoning crashes: if reasoning raises before producing results, no step checks are recorded. The error is still in state.errors.

Checklist

  • Tests pass
  • Code follows project style guidelines (black)
  • Documentation updated (spec §9 Implementation status)
  • Conventional commit messages used

Generated by Claude Code

Adds app/core/memory: a small MemoryGrounder interface and a default
StoreMemoryGrounder over LangGraph's BaseStore (InMemoryStore in tests,
PostgresStore when CORE_MEMORY_STORE=postgres and available). Returns
scored facts/episodes for a request, detects strong knowledge-dependency
cues ("like last time", "my preferred stack", "as we discussed") and
flags the ones memory does not cover as knowledge gaps. Pure, sync and
model-free; flag CORE_COMPREHENSION_MEMORY_GROUNDING defaults off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
…uation

app/evaluation/step_checks.py checks each executed plan step: produced a
result, no error/partial status, non-empty output, and outputs match the
step's declared output_schema (new optional PlanStep field). COREState
gains step_checks and memory_grounding (both optional/defaulted).

EvaluationAgent.evaluate_execution takes optional failing step_checks:
they are listed in the LLM prompt and counted as failures by the
rule-based fallback. Without them the call and prompt are unchanged. The
system prompt now names retry_step (execution flaked) vs revise_plan
(intent misalignment), per spec 4.4. No extra LLM calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
…h_v2

Additive hooks only, no routing changes:
- comprehension_node: _ground_in_memory() runs between ambiguity
  detection and the gate when the flag is on; gaps become ambiguities so
  the existing gate clarifies. No-op (grounder never built) when off.
- reasoning_node: _record_step_checks() fills state.step_checks.
- evaluation_node: passes failing checks to the evaluation agent only
  when some exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Stemmed forms ("discuss", "preferr") no longer matched the cue list, so
"discussed"/"preferred" inflated relevance scores. Regression test added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Pins flag off = identical gate outcome, message and route to the
pre-Phase-3 node across proceed/clarify/refuse/conversation/cue inputs;
flag on knowledge gaps clarify and short-circuit the loop; covered cues
proceed; grounder errors degrade safely; step checks are recorded and
reach evaluation only when failing; full compiled-graph runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
@Ian-Tharp
Ian-Tharp marked this pull request as ready for review November 3, 2026 20:50
@Ian-Tharp
Ian-Tharp merged commit bae71c7 into develop Nov 3, 2026
2 checks passed
Ian-Tharp added a commit that referenced this pull request Nov 3, 2026
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration),
#10 (coverage pass) and #11 (comprehension gate Phase 3).

Conflict resolutions:
- backend/app/evaluation/__init__.py: union of exports (develop's
  fault_reasons / maybe_capture_fault plus the claims API).
- backend/app/evaluation/capture.py: kept develop's fault_reasons /
  state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate,
  never raises); the branch's unsupported-claim check becomes a new
  "unsupported_claims" fault reason instead of an early return in
  state_failed. Branch's dispatched_tool filter and refusal /
  blocked_actions / unsupported_claims metadata kept in
  fault_report_from_state.
- backend/app/models/core_state.py: kept both field sets (refusal,
  blocked_actions, claim_report from the branch; step_checks from
  develop).
- docs/evaluation_spec.md: auto-merged; added the unsupported_claims
  fault reason and the extra metadata keys to "Live fault capture".
- tests/evaluation/test_claims.py: asserts fault_reasons ==
  ["unsupported_claims"] for the unsupported-claim run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants