Skip to content

test: coverage pass on the least-tested cognitive-engine modules (+5 bug fixes) - #10

Merged
Ian-Tharp merged 11 commits into
developfrom
claude/core-backend-coverage-pass-7fwnje
Nov 3, 2026
Merged

Ian-Tharp merged 11 commits into
developfrom
claude/core-backend-coverage-pass-7fwnje

Conversation

@Ian-Tharp

Copy link
Copy Markdown
Owner

What Changed

Behavior-level tests for the 9 lowest-covered KEEP modules in the cognitive engine, as classified in docs/ARCHIVE_PLAN.md (PR #7). All 9 now have 100% line coverage. Writing the tests turned up 5 bugs. Each fix is in its own fix: commit, and the test that reproduces it is kept.

  • 9 new test files, 341 tests in total. LLM and DB boundaries are mocked: no network, no live Postgres or Redis.
  • pytest-cov added as a dev dependency. No runtime dependency changes.
  • No modules deleted or moved.

Module selection

The scope is the cognitive engine: C/O/R/E pipeline, /core and /engine controllers, and the repositories and services they use. KEEP modules outside that scope were left out on purpose, even though some are covered less:

  • controllers/knowledgebase.py (0%)
  • controllers/local_llm.py (0%)
  • migrations/runner.py (0%)
  • repository/knowledgebase_repository.py (17.5%)
  • dependencies.py (33%)
  • the chat stack (46–58%)

These are good candidates for a follow-up pass. UNSURE/ARCHIVE modules, such as the standalone comprehension and evaluation services (C1), were not touched.

Coverage: targeted modules

Module Before After New test file(s)
controllers/core_entry.py 0.0% 100% test_core_entry_controller.py (42)
controllers/engine.py 17.1% 100% test_engine_controller.py (43)
repository/run_repository.py 21.6% 100% test_run_repository.py (35)
services/llm_provider.py 36.0% 100% test_llm_provider.py (28)
core/langgraph/core_graph_v2.py 80.1% 100% test_core_graph_v2_behavior.py (28)
repository/webhook_repository.py 80.1% 100% test_webhook_repository.py (35)
core/telemetry.py 80.5% 100% test_telemetry_behavior.py (7)
core/tools/dispatcher.py 81.7% 100% test_dispatcher_behavior.py (40 cases)
core/agents/orchestration_agent.py 87.9% 100% test_orchestration_agent_behavior.py (21 cases)

Whole app/ package: 53.9% → 57.4%. Suite: 2,827 → 3,106 passed, 1 skipped.

Baseline → after for all 50 KEEP modules
Module Before After
auth.py 63.4% 63.4%
config/models.py 100.0% 100.0%
config/startup_validator.py 93.3% 93.3%
controllers/chat.py 58.1% 58.1%
controllers/conversations.py 55.6% 55.6%
controllers/core_entry.py 0.0% 100.0%
controllers/engine.py 17.1% 100.0%
controllers/health.py 74.1% 74.1%
controllers/knowledgebase.py 0.0% 0.0%
controllers/local_llm.py 0.0% 0.0%
controllers/metrics.py 100.0% 100.0%
core/agents/comprehension_agent.py 90.3% 90.3%
core/agents/evaluation_agent.py 98.8% 98.8%
core/agents/orchestration_agent.py 87.9% 100.0%
core/agents/reasoning_agent.py 94.8% 94.8%
core/circuit_breaker.py 98.8% 98.8%
core/intelligence_log.py 100.0% 100.0%
core/langgraph/core_graph_v2.py 80.1% 100.0%
core/logging_config.py 100.0% 100.0%
core/middleware.py 90.7% 90.7%
core/security.py 96.3% 96.3%
core/telemetry.py 80.5% 100.0%
core/tools/dispatcher.py 81.7% 100.0%
dependencies.py 33.3% 33.3%
middleware/correlation.py 100.0% 100.0%
middleware/logging.py 100.0% 100.0%
middleware/rate_limit.py 88.3% 88.3%
migrations/runner.py 0.0% 0.0%
models/* (8 modules) 100.0% 100.0%
repository/api_key_repository.py 73.2% 73.2%
repository/conversation_repository.py 54.8% 54.8%
repository/health_repository.py 96.5% 96.5%
repository/knowledgebase_repository.py 17.5% 17.5%
repository/run_repository.py 21.6% 100.0%
repository/webhook_repository.py 80.1% 100.0%
services/chat_service.py 46.1% 46.1%
services/health_aggregator.py 65.1% 65.1%
services/knowledgebase_service.py 57.6% 57.6%
services/llm_provider.py 36.0% 100.0%
services/metrics_service.py 76.6% 76.6%
services/ollama_embeddings.py 82.0% 82.0%
services/title_generation_service.py 100.0% 100.0%
services/webhook_service.py 85.4% 85.4%
utils/json_repair.py 96.7% 96.7%

Bugs found and fixed (each in its own commit)

  1. /core/evaluation reported UNSATISFACTORY verdicts as SATISFACTORY. It checked "satisf" in text, which also matches "unsatisfactory". Commit 09d69e1. Reproducer: test_core_entry_controller.py::TestSteps::test_evaluation_unsatisfactory_verdict.

  2. The file tool's workspace check could be bypassed by a sibling directory. _safe_path compared paths with str.startswith, so ../ws_evil/x passed for workspace ws. Writes landed outside the workspace. Now uses Path.is_relative_to. Commit b35a1cd. Reproducer: test_dispatcher_behavior.py::TestFileOperationsBehavior::test_sibling_dir_sharing_name_prefix_is_blocked.

  3. Unbounded retry_step loop crashed runs. A step that kept failing looped reasoning → evaluation until LangGraph raised GraphRecursionError. The built-in fallback evaluator triggers this on any partial failure. Fixed by adding CORE_MAX_STEP_RETRIES (default 3, counted per plan revision), which mirrors the existing plan-revision guard. Commit 1cd9281. Reproducer: test_core_graph_v2_behavior.py::TestFullPipeline::test_persistent_retry_step_is_capped_instead_of_recursing_forever.

  4. Engine runs were never saved to the database. engine.py calls run_repository.save_run(...) in 6 places, but that function did not exist. The AttributeError was swallowed as a warning. Added save_run as an upsert into core_runs. Commit 57a18d7. Reproducers: test_run_repository.py::TestSaveRun, test_engine_controller.py::test_engine_only_calls_repository_functions_that_exist.

  5. GET /engine/runs and GET /engine/runs/{id} mishandled stored runs.

    • The list endpoint called .isoformat() on timestamps that are already ISO strings, so stored runs were silently dropped from the list.
    • The single-run fallback passed current_node=None, which turned a valid run into a 404.
    • Both read user_input instead of the input_text column.

    Commit 10fdefe. Reproducers: TestListRuns::test_merges_repository_rows_without_duplicates, TestGetRun::test_db_run_without_state_built_from_repository_columns.

Worth a look (not changed)

  • core_runs schema mismatch. backend/migrations/004_run_persistence.sql (the unused legacy dir, D5 in the archive plan) defines core_runs with user_input and current_node. app/dependencies.py uses input_text. Fix feat: add agent spawn templates for reusable specialist configurations #4 follows dependencies.py, which create_run already writes against.
  • Odd but possibly intended behaviour, now pinned by tests:
    • /core/orchestration keeps "1." numbering on plan steps.
    • In stub mode, comprehension always routes to orchestration.
    • One malformed step makes OrchestrationAgent.create_plan fall back to a single-step plan.
  • get_openai_client_sync() sits outside the try in OrchestrationAgent.create_plan. A client construction failure propagates instead of falling back to a single-step plan.

Why

Makes the KEEP cognitive-engine modules safe to refactor ahead of the archive plan (PR #7).

How to Test

cd backend
uv sync
uv run black --check app/ tests/
uv run pytest -q --cov=app --cov-report=term

Use --cov=app or a path such as --cov=app/controllers. Dotted module targets like --cov=app.controllers.engine import app too early and break collection on a pydantic/langchain discriminator error. That predates this PR.

Checklist

  • Tests pass locally (3,106 passed, 1 skipped). CI is pending on this PR.
  • Code follows project style guidelines (black --check clean)
  • Documentation updated (if applicable): not needed. CORE_MAX_STEP_RETRIES is documented inline next to CORE_MAX_PLAN_REVISIONS.
  • Conventional commit messages used

Generated by Claude Code

claude added 11 commits November 2, 2026 22:45
…ACTORY

The verdict check was `"satisf" in text.lower()`, which also matches
"unsatisfactory", so every negative verdict was reported as a pass.
Check for the negative form before accepting the positive one.

Reproduced by tests/test_core_entry_controller.py::TestSteps::test_evaluation_unsatisfactory_verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
42 tests for controllers/core_entry.py (auth, 422s, step outputs, SSE
streams, stub fallback, LLM error + temperature retry) and 28 for
services/llm_provider.py (provider detection, local endpoints, client
kwargs). Both modules now at 100% line coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
…workspace prefix

_safe_path compared paths with str.startswith, so a path like
../ws_evil/secret.txt passed the check for workspace "ws". Writes landed
outside the workspace and reads crashed with an uncaught ValueError.
Use Path.is_relative_to for a component-wise check.

Reproduced by tests/test_dispatcher_behavior.py::TestFileOperationsBehavior::test_sibling_dir_sharing_name_prefix_is_blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
Error/edge paths for every dispatcher tool (unknown tool, traversal,
git timeouts and blocked writes, HTTP errors, DB tool misconfig) and for
orchestration plan parsing (fenced/repaired JSON, malformed plans,
LLM failure fallback, cost/risk levels). Both modules at 100%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
route_from_evaluation capped plan revisions but not step retries. When
a step kept failing, the evaluator returned retry_step every pass and
the graph looped reasoning -> evaluation until LangGraph raised
GraphRecursionError, crashing the run. The built-in fallback evaluator
produces this on any partial failure.

Add CORE_MAX_STEP_RETRIES (default 3), counted per plan revision from
execution_history, routing to conversation once reached, mirroring the
existing plan-revision guard.

Reproduced by tests/test_core_graph_v2_behavior.py::TestFullPipeline::test_persistent_retry_step_is_capped_instead_of_recursing_forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
Full-graph runs with mocked agents (task, chat, clarify gate, step retry,
capped plan revision), per-node failure paths, ambiguity merging,
invented-tool filtering, conversation node branches; telemetry exporter
setup, fallbacks and reset. Both modules at 100%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
35 tests with a mocked asyncpg pool: CRUD, delivery logging, pruning and
stats, SQL params/filters/limit cap, JSON columns as str or dict, not-found
and empty results, and each function's DB-error contract. 100% coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
controllers/engine.py awaits run_repository.save_run(state.model_dump())
in six places, but the function never existed. The AttributeError was
caught and logged as a warning, so no /engine run was ever persisted.

Add save_run as an upsert into core_runs (schema from dependencies.py),
deriving status from the snapshot (END/completed_at -> completed,
errors -> failed, else running).

Reproduced by tests/test_run_repository.py::TestSaveRun and
tests/test_engine_controller.py::test_engine_only_calls_repository_functions_that_exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
- GET /engine/runs called .isoformat() on timestamps the repository already
  returns as ISO strings; the AttributeError was swallowed, so database
  runs were never listed. It also read user_input instead of the
  input_text column.
- GET /engine/runs/{id} fallback passed current_node=None, which COREState
  rejects, turning a stored run without a state snapshot into a 404. It
  also read user_input instead of input_text.

Reproduced by tests/test_engine_controller.py::TestListRuns::test_merges_repository_rows_without_duplicates
and ::TestGetRun::test_db_run_without_state_built_from_repository_columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
43 tests for every /engine endpoint (run, streams, stats, get/delete/list
runs; graph errors -> 500 + RUN_FAILED webhook, DB/webhook failures
tolerated, 401/403/404/422, SSE error events) and 35 for run_repository
(SQL params, row mapping, filters/pagination, not-found, DB errors).
Both modules at 100%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KQ4WiAWznYknyr7UteiL9
@Ian-Tharp
Ian-Tharp marked this pull request as ready for review November 3, 2026 14:34
@Ian-Tharp
Ian-Tharp merged commit 018772d into develop Nov 3, 2026
2 checks passed
Ian-Tharp added a commit that referenced this pull request Nov 3, 2026
Brings in #8 (fail-safe fault capture wiring), #9 (kb world_id migration),
#10 (coverage pass) and #11 (comprehension gate Phase 3).

Conflict resolutions:
- backend/app/evaluation/__init__.py: union of exports (develop's
  fault_reasons / maybe_capture_fault plus the claims API).
- backend/app/evaluation/capture.py: kept develop's fault_reasons /
  state_failed / maybe_capture_fault (CORE_EVAL_CAPTURE_FAULTS gate,
  never raises); the branch's unsupported-claim check becomes a new
  "unsupported_claims" fault reason instead of an early return in
  state_failed. Branch's dispatched_tool filter and refusal /
  blocked_actions / unsupported_claims metadata kept in
  fault_report_from_state.
- backend/app/models/core_state.py: kept both field sets (refusal,
  blocked_actions, claim_report from the branch; step_checks from
  develop).
- docs/evaluation_spec.md: auto-merged; added the unsupported_claims
  fault reason and the extra metadata keys to "Live fault capture".
- tests/evaluation/test_claims.py: asserts fault_reasons ==
  ["unsupported_claims"] for the unsupported-claim run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants