Repository navigation
feat(gate): Phase 3 — memory grounding + per-step evaluation - #11
Merged
Merged
Conversation
Adds app/core/memory: a small MemoryGrounder interface and a default
StoreMemoryGrounder over LangGraph's BaseStore (InMemoryStore in tests,
PostgresStore when CORE_MEMORY_STORE=postgres and available). Returns
scored facts/episodes for a request, detects strong knowledge-dependency
cues ("like last time", "my preferred stack", "as we discussed") and
flags the ones memory does not cover as knowledge gaps. Pure, sync and
model-free; flag CORE_COMPREHENSION_MEMORY_GROUNDING defaults off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
…uation app/evaluation/step_checks.py checks each executed plan step: produced a result, no error/partial status, non-empty output, and outputs match the step's declared output_schema (new optional PlanStep field). COREState gains step_checks and memory_grounding (both optional/defaulted). EvaluationAgent.evaluate_execution takes optional failing step_checks: they are listed in the LLM prompt and counted as failures by the rule-based fallback. Without them the call and prompt are unchanged. The system prompt now names retry_step (execution flaked) vs revise_plan (intent misalignment), per spec 4.4. No extra LLM calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
…h_v2 Additive hooks only, no routing changes: - comprehension_node: _ground_in_memory() runs between ambiguity detection and the gate when the flag is on; gaps become ambiguities so the existing gate clarifies. No-op (grounder never built) when off. - reasoning_node: _record_step_checks() fills state.step_checks. - evaluation_node: passes failing checks to the evaluation agent only when some exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Stemmed forms ("discuss", "preferr") no longer matched the cue list, so
"discussed"/"preferred" inflated relevance scores. Regression test added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Pins flag off = identical gate outcome, message and route to the pre-Phase-3 node across proceed/clarify/refuse/conversation/cue inputs; flag on knowledge gaps clarify and short-circuit the loop; covered cues proceed; grounder errors degrade safely; step checks are recorded and reach evaluation only when failing; full compiled-graph runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WjmjqnwL1cSQzGC8LA
Ian-Tharp
marked this pull request as ready for review
November 3, 2026 20:50
Ian-Tharp
added a commit
that referenced
this pull request
Nov 3, 2026
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration), #10 (coverage pass) and #11 (comprehension gate Phase 3). Conflict resolutions: - backend/app/evaluation/__init__.py: union of exports (develop's fault_reasons / maybe_capture_fault plus the claims API). - backend/app/evaluation/capture.py: kept develop's fault_reasons / state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate, never raises); the branch's unsupported-claim check becomes a new "unsupported_claims" fault reason instead of an early return in state_failed. Branch's dispatched_tool filter and refusal / blocked_actions / unsupported_claims metadata kept in fault_report_from_state. - backend/app/models/core_state.py: kept both field sets (refusal, blocked_actions, claim_report from the branch; step_checks from develop). - docs/evaluation_spec.md: auto-merged; added the unsupported_claims fault reason and the extra metadata keys to "Live fault capture". - tests/evaluation/test_claims.py: asserts fault_reasons == ["unsupported_claims"] for the unsupported-claim run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
Implements Phase 3 of
docs/architecture/comprehension-grounded-gate.md: both verifiers now get more self-knowledge.1. Memory grounding (front verifier: "do I know what this relies on?") —
backend/app/core/memory/MemoryGrounder.ground(query, user_id=, conversation_id=, k=) -> list[MemoryHit](content, kindfact|episode, score).StoreMemoryGrounderover LangGraph'sBaseStore:InMemoryStoreby default and in tests,PostgresStorewhenCORE_MEMORY_STORE=postgresand it's available (otherwise it logs a warning and falls back to the in-memory store). Namespaces are("core_memory", <user|global>, <kind>). It uses the store's semantic score when one exists, otherwise a deterministic lexical score.detect_knowledge_dependencies()finds strong cues that the request relies on prior knowledge ("like last time", "the usual", "as we discussed", "my preferred X", "my manager"). A bare "my"/"our" doesn't count, so "fix my code in main.py" doesn't clarify.intent.ambiguities, so the existing tri-state gate clarifies; the gate logic is untouched. Hits are appended tointent.context_retrieved, and the assessment is recorded inCOREState.memory_grounding.CORE_COMPREHENSION_MEMORY_GROUNDING, default off. When it's off, the grounder is never built and routing is identical to today.2. Per-step evaluation (back verifier) —
backend/app/evaluation/step_checks.pyPlanStep.output_schema(new optional field, validated with the existing_schema_validate).COREState.step_checks. Failing checks are passed toEvaluationAgent.evaluate_execution(step_checks=...): they're listed in the LLM prompt, and the rule-based fallback counts them as failures. A clean run calls the evaluator with the same kwargs and prompt as before. No extra LLM calls.retry_step(execution flaked) vsrevise_plan(intent misalignment), per spec §4.4.3.
core_graph_v2: additive changes only (+58/−1), and no routing function changed:comprehension_nodecalls_ground_in_memory()between ambiguity detection and the gate.reasoning_nodecalls_record_step_checks()after its try/except.evaluation_nodepasses failing checks only when some exist.route_from_evaluation(test: coverage pass on the least-tested cognitive-engine modules (+5 bug fixes) #10) or the end ofconversation_node(feat(evaluation): fail-safe fault capture wired into core_graph_v2 #8), so it should merge cleanly with both.Why the LangGraph store and not
memory_servicecomprehension_nodeis synchronous and the front gate must stay cheap, deterministic and offline-testable.memory_serviceis async, needs the asyncpg pool and an Ollama embedding call per query, and is scoped per agent rather than per user.BaseStorehas a sync API, swaps cleanly betweenInMemoryStoreandPostgresStore, and is what spec §4.5 already names ("LangGraphPostgresStoreover CORE pgvector").memory_serviceadapter can implement the sameMemoryGrounderinterface later.PostgresStorelives inlanggraph-checkpoint-postgres, which isn't installed, so it's imported optionally.Why
CORE is a deliberation loop between two verifiers. Phase 1/2 grounded comprehension in what it can do (Capability Registry). This adds what it knows: requests that depend on facts CORE doesn't have now clarify instead of running O→R→E on a guess. Evaluation also gets a cheap deterministic signal per step, so a step that "succeeded" with empty or ill-shaped output can't be silently finalized.
How to Test
test_memory_grounding.py: cue detection, scoring, user/global scoping, error tolerance, gaps → gate clarify, store factory fallbacks.test_step_checks.py: each failure kind, retry semantics, prompt and fallback integration inEvaluationAgent.test_phase3_graph_integration.py: flag off = identical gate outcome, message and route to a graph with grounding stubbed, across proceed/clarify/refuse/conversation/cue inputs. With the flag on, gaps clarify and short-circuit to["comprehension", "conversation"]. Also covers covered cues proceeding, grounder errors degrading safely, and full compiled-graph runs. All LLMs are mocked; no network.Follow-ups (also in spec §9)
ReasoningAgent.execute_plan.state_faileddoesn't count failing step checks yet. I'm leaving that until feat(evaluation): fail-safe fault capture wired into core_graph_v2 #8 lands.state.errors.Checklist
Generated by Claude Code