Skip to content

fix: prevent backend lock contention from freezing the server - #874

Merged
RetroGameDeveloper merged 1 commit into
mainfrom
fix/server-lock-contention
Nov 8, 2026
Merged

RetroGameDeveloper merged 1 commit into
mainfrom
fix/server-lock-contention

Conversation

@RetroGameDeveloper

@RetroGameDeveloper RetroGameDeveloper commented Nov 8, 2026 •

Copy link
Copy Markdown

Problem and scope

The live ASGI main thread was blocked on a Python lock while background work continued; two independent health probes timed out. The same problem reproduces deterministically with a held job-admission path. The new contention regression fails on unchanged base b10d1bc and passes with this patch. Native sampling did not reveal the exact live Python lock owner; elevated Python stack attachment was unavailable.

Audited asynchronous app routes, shared job/control entry points, project/progress/database and cache boundaries, room/MCP dispatch, login/OAuth, diagnostics and shutdown. Details: docs/SERVER_LOCK_AUDIT.md.

Changes

  • Offload synchronous job admission/reservations, cancellation, progress status, browser capability operations, settings/session/OAuth writes, agent refresh and logging. Atomic duplicate-job reservations stay on one worker and retain their RLock scope; no lock is acquired on one thread and released by another.
  • Give short control operations independent worker capacity, so ordinary builds/UI snapshot saturation cannot consume cancellation/admission capacity.
  • Read a project snapshot when validating repository-sync admission instead of running a full progress import before its job ID exists.
  • Make cookie authorization a read-only current-session lookup with expiry validation, without waiting for a login writer. State issuance/revocation remains locked and is offloaded.
  • Bound snapshot consumers' waits and reject recursive waits for their own builder; do not cancel or replace successful background builders.
  • Move diagnostics logging off the event loop. Add an independent heartbeat watchdog that records bounded local Python thread code locations if the loop stalls for five seconds. No frame locals, source lines, SQL or request URLs are recorded.
  • Add a structural test against direct blocking locks and critical job/database calls in async routes. Preserve transaction, verification, worker claim and shutdown semantics.

Verification

311 targeted Python tests passed across route security, server/worker contention, settings/login/OAuth, room/browser behavior, project caches, source updates, repair routes, startup, MCP responsiveness and shutdown. The contention cases use a single ASGI event loop and require concurrent health responses while job admission, reservation, progress and a real database writer are held. The delayed database write commits and passes SQLite integrity checking. Watchdog and recursive/bounded cache waits are covered.

Four Node GUI suites pass (repository setup/sync/push and room reconnect); Ruff (src tests projects) and git diff --check pass. Demo-ROM validation uses local LLVM tools. No paid model requests or live-project mutations/restarts were used.

One unrelated room-browser mocked imported-build test (test_compile_dispatches_to_normal_engine_without_leaking_diff[True]) fails identically on unmodified b10d1bc. It was excluded; its ROM counterpart and the real browser byte-match case passed. This is a targeted regression run, not a claim that the full repository suite passes.


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@RetroGameDeveloper
RetroGameDeveloper merged commit 765e06c into main Nov 8, 2026
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant