Repository navigation
test: coverage pass on the least-tested cognitive-engine modules (+5 bug fixes) - #10
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
…ACTORY The verdict check was `"satisf" in text.lower()`, which also matches "unsatisfactory", so every negative verdict was reported as a pass. Check for the negative form before accepting the positive one. Reproduced by tests/test_core_entry_controller.py::TestSteps::test_evaluation_unsatisfactory_verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
42 tests for controllers/core_entry.py (auth, 422s, step outputs, SSE streams, stub fallback, LLM error + temperature retry) and 28 for services/llm_provider.py (provider detection, local endpoints, client kwargs). Both modules now at 100% line coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
…workspace prefix _safe_path compared paths with str.startswith, so a path like ../ws_evil/secret.txt passed the check for workspace "ws". Writes landed outside the workspace and reads crashed with an uncaught ValueError. Use Path.is_relative_to for a component-wise check. Reproduced by tests/test_dispatcher_behavior.py::TestFileOperationsBehavior::test_sibling_dir_sharing_name_prefix_is_blocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
Error/edge paths for every dispatcher tool (unknown tool, traversal, git timeouts and blocked writes, HTTP errors, DB tool misconfig) and for orchestration plan parsing (fenced/repaired JSON, malformed plans, LLM failure fallback, cost/risk levels). Both modules at 100%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
route_from_evaluation capped plan revisions but not step retries. When a step kept failing, the evaluator returned retry_step every pass and the graph looped reasoning -> evaluation until LangGraph raised GraphRecursionError, crashing the run. The built-in fallback evaluator produces this on any partial failure. Add CORE_MAX_STEP_RETRIES (default 3), counted per plan revision from execution_history, routing to conversation once reached, mirroring the existing plan-revision guard. Reproduced by tests/test_core_graph_v2_behavior.py::TestFullPipeline::test_persistent_retry_step_is_capped_instead_of_recursing_forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
Full-graph runs with mocked agents (task, chat, clarify gate, step retry, capped plan revision), per-node failure paths, ambiguity merging, invented-tool filtering, conversation node branches; telemetry exporter setup, fallbacks and reset. Both modules at 100%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
35 tests with a mocked asyncpg pool: CRUD, delivery logging, pruning and stats, SQL params/filters/limit cap, JSON columns as str or dict, not-found and empty results, and each function's DB-error contract. 100% coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
controllers/engine.py awaits run_repository.save_run(state.model_dump()) in six places, but the function never existed. The AttributeError was caught and logged as a warning, so no /engine run was ever persisted. Add save_run as an upsert into core_runs (schema from dependencies.py), deriving status from the snapshot (END/completed_at -> completed, errors -> failed, else running). Reproduced by tests/test_run_repository.py::TestSaveRun and tests/test_engine_controller.py::test_engine_only_calls_repository_functions_that_exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
- GET /engine/runs called .isoformat() on timestamps the repository already
returns as ISO strings; the AttributeError was swallowed, so database
runs were never listed. It also read user_input instead of the
input_text column.
- GET /engine/runs/{id} fallback passed current_node=None, which COREState
rejects, turning a stored run without a state snapshot into a 404. It
also read user_input instead of input_text.
Reproduced by tests/test_engine_controller.py::TestListRuns::test_merges_repository_rows_without_duplicates
and ::TestGetRun::test_db_run_without_state_built_from_repository_columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
43 tests for every /engine endpoint (run, streams, stats, get/delete/list runs; graph errors -> 500 + RUN_FAILED webhook, DB/webhook failures tolerated, 401/403/404/422, SSE error events) and 35 for run_repository (SQL params, row mapping, filters/pagination, not-found, DB errors). Both modules at 100%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
4 tasks done
Ian-Tharp
marked this pull request as ready for review
November 3, 2026 14:34
Ian-Tharp
added a commit
that referenced
this pull request
Nov 3, 2026
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration), #10 (coverage pass) and #11 (comprehension gate Phase 3). Conflict resolutions: - backend/app/evaluation/__init__.py: union of exports (develop's fault_reasons / maybe_capture_fault plus the claims API). - backend/app/evaluation/capture.py: kept develop's fault_reasons / state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate, never raises); the branch's unsupported-claim check becomes a new "unsupported_claims" fault reason instead of an early return in state_failed. Branch's dispatched_tool filter and refusal / blocked_actions / unsupported_claims metadata kept in fault_report_from_state. - backend/app/models/core_state.py: kept both field sets (refusal, blocked_actions, claim_report from the branch; step_checks from develop). - docs/evaluation_spec.md: auto-merged; added the unsupported_claims fault reason and the extra metadata keys to "Live fault capture". - tests/evaluation/test_claims.py: asserts fault_reasons == ["unsupported_claims"] for the unsupported-claim run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
Behavior-level tests for the 9 lowest-covered KEEP modules in the cognitive engine, as classified in
docs/ARCHIVE_PLAN.md(PR #7). All 9 now have 100% line coverage. Writing the tests turned up 5 bugs. Each fix is in its ownfix:commit, and the test that reproduces it is kept.pytest-covadded as a dev dependency. No runtime dependency changes.Module selection
The scope is the cognitive engine: C/O/R/E pipeline,
/coreand/enginecontrollers, and the repositories and services they use. KEEP modules outside that scope were left out on purpose, even though some are covered less:controllers/knowledgebase.py(0%)controllers/local_llm.py(0%)migrations/runner.py(0%)repository/knowledgebase_repository.py(17.5%)dependencies.py(33%)These are good candidates for a follow-up pass. UNSURE/ARCHIVE modules, such as the standalone comprehension and evaluation services (C1), were not touched.
Coverage: targeted modules
controllers/core_entry.pytest_core_entry_controller.py(42)controllers/engine.pytest_engine_controller.py(43)repository/run_repository.pytest_run_repository.py(35)services/llm_provider.pytest_llm_provider.py(28)core/langgraph/core_graph_v2.pytest_core_graph_v2_behavior.py(28)repository/webhook_repository.pytest_webhook_repository.py(35)core/telemetry.pytest_telemetry_behavior.py(7)core/tools/dispatcher.pytest_dispatcher_behavior.py(40 cases)core/agents/orchestration_agent.pytest_orchestration_agent_behavior.py(21 cases)Whole
app/package: 53.9% → 57.4%. Suite: 2,827 → 3,106 passed, 1 skipped.Baseline → after for all 50 KEEP modules
auth.pyconfig/models.pyconfig/startup_validator.pycontrollers/chat.pycontrollers/conversations.pycontrollers/core_entry.pycontrollers/engine.pycontrollers/health.pycontrollers/knowledgebase.pycontrollers/local_llm.pycontrollers/metrics.pycore/agents/comprehension_agent.pycore/agents/evaluation_agent.pycore/agents/orchestration_agent.pycore/agents/reasoning_agent.pycore/circuit_breaker.pycore/intelligence_log.pycore/langgraph/core_graph_v2.pycore/logging_config.pycore/middleware.pycore/security.pycore/telemetry.pycore/tools/dispatcher.pydependencies.pymiddleware/correlation.pymiddleware/logging.pymiddleware/rate_limit.pymigrations/runner.pymodels/*(8 modules)repository/api_key_repository.pyrepository/conversation_repository.pyrepository/health_repository.pyrepository/knowledgebase_repository.pyrepository/run_repository.pyrepository/webhook_repository.pyservices/chat_service.pyservices/health_aggregator.pyservices/knowledgebase_service.pyservices/llm_provider.pyservices/metrics_service.pyservices/ollama_embeddings.pyservices/title_generation_service.pyservices/webhook_service.pyutils/json_repair.pyBugs found and fixed (each in its own commit)
/core/evaluationreported UNSATISFACTORY verdicts as SATISFACTORY. It checked"satisf" in text, which also matches "unsatisfactory". Commit09d69e1. Reproducer:test_core_entry_controller.py::TestSteps::test_evaluation_unsatisfactory_verdict.The file tool's workspace check could be bypassed by a sibling directory.
_safe_pathcompared paths withstr.startswith, so../ws_evil/xpassed for workspacews. Writes landed outside the workspace. Now usesPath.is_relative_to. Commitb35a1cd. Reproducer:test_dispatcher_behavior.py::TestFileOperationsBehavior::test_sibling_dir_sharing_name_prefix_is_blocked.Unbounded
retry_steploop crashed runs. A step that kept failing looped reasoning → evaluation until LangGraph raisedGraphRecursionError. The built-in fallback evaluator triggers this on any partial failure. Fixed by addingCORE_MAX_STEP_RETRIES(default 3, counted per plan revision), which mirrors the existing plan-revision guard. Commit1cd9281. Reproducer:test_core_graph_v2_behavior.py::TestFullPipeline::test_persistent_retry_step_is_capped_instead_of_recursing_forever.Engine runs were never saved to the database.
engine.pycallsrun_repository.save_run(...)in 6 places, but that function did not exist. TheAttributeErrorwas swallowed as a warning. Addedsave_runas an upsert intocore_runs. Commit57a18d7. Reproducers:test_run_repository.py::TestSaveRun,test_engine_controller.py::test_engine_only_calls_repository_functions_that_exist.GET /engine/runsandGET /engine/runs/{id}mishandled stored runs..isoformat()on timestamps that are already ISO strings, so stored runs were silently dropped from the list.current_node=None, which turned a valid run into a 404.user_inputinstead of theinput_textcolumn.Commit
10fdefe. Reproducers:TestListRuns::test_merges_repository_rows_without_duplicates,TestGetRun::test_db_run_without_state_built_from_repository_columns.Worth a look (not changed)
core_runsschema mismatch.backend/migrations/004_run_persistence.sql(the unused legacy dir, D5 in the archive plan) definescore_runswithuser_inputandcurrent_node.app/dependencies.pyusesinput_text. Fix feat: add agent spawn templates for reusable specialist configurations #4 followsdependencies.py, whichcreate_runalready writes against./core/orchestrationkeeps "1." numbering on plan steps.OrchestrationAgent.create_planfall back to a single-step plan.get_openai_client_sync()sits outside thetryinOrchestrationAgent.create_plan. A client construction failure propagates instead of falling back to a single-step plan.Why
Makes the KEEP cognitive-engine modules safe to refactor ahead of the archive plan (PR #7).
How to Test
cd backend uv sync uv run black --check app/ tests/ uv run pytest -q --cov=app --cov-report=termUse
--cov=appor a path such as--cov=app/controllers. Dotted module targets like--cov=app.controllers.engineimportapptoo early and break collection on a pydantic/langchain discriminator error. That predates this PR.Checklist
black --checkclean)CORE_MAX_STEP_RETRIESis documented inline next toCORE_MAX_PLAN_REVISIONS.Generated by Claude Code