[chrismccord]

Summary

A restarted server claims its existing key in try_lock_object_via_update/2 with an If-Match PUT. That PUT had no retries. The first-ever claim (ObjectStore.try_claim/3) and regular syncs (@max_sync_retries) both retry. So a single transient object-storage failure failed the start.

Object storage throttles rapid writes to one key, and a restart claims the key right after the previous owner's final writes. Sprites staging hit exactly that: a sprite start issued right after a stop returned internal_error, and the retained logs show:

01:12:31.117 DurableServer orgs/23430/clusters/default/sprite/dkr-src shutting down gracefully via user stop (:permanent)
01:12:31.439 [error] Failed to put object with etag: [key: "durable/orgs/23430/.../dkr-src", response: %Req.Response{status: 503, ...
             <Code>SlowDown</Code><Message>Please reduce your request rate.</Message> ...]
01:12:31.442 [error] Failed to initialize DurableServer: %Req.Response{status: 503, ...}
01:12:31.892 won the lock for "orgs/23430/clusters/default/sprite/dkr-src"   <- the next caller, 450 ms later

The lock claim now passes max_retries: 2, the same budget as try_claim. The object store retries 408/429/500/502/503/504 and transport timeouts and closes.

Safety. A retry whose earlier attempt actually landed gets 412. Backends.ObjectStore.put_object/4 already resolves a 412 on an :etag put through resolve_ambiguous_conditional_put/5. That reads the object back and adopts it only if it holds exactly this attempt's encoded state from the same boot owner. The claim's state carries this attempt's pid and heartbeat, so a different winner is still {:error, :conflict} → "raced the lock".

Testing

New DurableServerTest cases run against LocalStack with a Req adapter in front of it:

• "a restart retries a throttled lock claim": the restart's lock claim gets one 503 SlowDown. Without this change the test fails with the same two log lines as staging. With it, the restart wins the lock.
• "a retried lock claim whose first attempt landed is this boot's claim": the claim reaches storage, but its response is replaced with 503. The retry gets 412 and resolves as this boot's claim.

mix test: 454 passed, 37 excluded. mix compile --warnings-as-errors is clean.