feat(testing): checkCrashRecovery + three crash-recovery fixes

.changeset/crash-recovery.md

@@ -0,0 +1,11 @@
+---
+"@statelyai/agent": patch
+---
+
+Add `checkCrashRecovery` to `@statelyai/agent/testing`: it kills a durable run after every log append and while every effect is in flight, resumes each from the store's log, and checks the resume against an uninterrupted run (no durable effect re-runs, none skipped, same `callKey` and request, same outcome, the log replays). Pass `store` to check your own store adapter.
+
+Fix: a child machine that finished just before a crash (its last entry journaled, its parent's `xstate.done.actor` not yet) no longer leaves the resumed parent waiting forever. The restored child's completion is now delivered to its parent, at any depth.
+
+Fix: a call re-executed after a crash no longer reports its usage twice. `@agent.usage` now carries the call's `callKey`; when the log already holds usage for that key, the repeat is dropped (`usage.dropped` with `reason: "repeat"`), so a token budget counts each call once.
+
+Fix: a parent's `sendTo` into a journaled child machine, and the child's reply, are no longer lost when the run dies between appends. The store now receives every entry since the loop last waited in one atomic append (`expectedIndex` is the batch's first index), so an event and the child inputs it caused land together.

docs/event-log.md

@@ -74,7 +74,8 @@ const result = await runToQuiescence(
 );
 ```
 
-- Each entry is written at its own `index` as `expectedIndex`, so a concurrent writer conflicts instead of interleaving.
+- Entries are written in batches, each at its first entry's `index` as `expectedIndex`, so a concurrent writer conflicts instead of interleaving.
+- A batch holds every entry since the loop last waited: an event, the child inputs its transition sent, and the events those children sent back land in one atomic append. A crash never keeps a cause without its effect.
 - No model call starts until every entry before it is durable. Pure transitions never wait.
 - A rejected write (an `AgentEventLogConflictError`, or any storage failure) aborts in-flight work and settles the run `{ status: "error", cause: "journal" }`. No further calls run.
 - A `@agent.usage` entry that arrives after the run settled is still written, but the result does not wait for it. Await `result.drain()` before terminating the process if you care about those straggler entries; it resolves once every write issued so far has landed, and never rejects (a failed write already settled the run with `cause: "journal"`).

docs/persistence.md

@@ -70,6 +70,72 @@ if (result.ignored) {
 - A log that already reached a final state settles immediately with the recorded output.
 - The resumed result's `events` extends the same log, so the whole history stays replayable.
 
+## Crash testing
+
+`checkCrashRecovery` from `@statelyai/agent/testing` kills a run at every point it could die and checks that the resume matches a run that never died.
+
+```ts
+import { createAsyncLogic } from "xstate";
+import { z } from "zod";
+import { setupAgent, createTextLogic } from "@statelyai/agent";
+import { checkCrashRecovery } from "@statelyai/agent/testing";
+
+const machine = setupAgent({
+  context: z.object({ order: z.string() }),
+  events: {},
+  actors: {
+    lookup: createAsyncLogic({ run: async () => "order-7" }),
+    reply: createTextLogic({ model: "m", prompt: "Reply." }),
+  },
+}).createMachine({
+  context: { order: "" },
+  initial: "lookup",
+  states: {
+    lookup: {
+      invoke: {
+        src: "lookup",
+        onDone: ({ event }) => ({ target: "reply", context: { order: event.output } }),
+      },
+    },
+    reply: { invoke: { src: "reply", onDone: { target: "done" } } },
+    done: { type: "final" },
+  },
+});
+
+const report = await checkCrashRecovery(machine, {
+  executors: { generateText: async () => ({ result: "Your order shipped." }) },
+});
+report.failures; // [] when every crash point recovers
+```
+
+It runs the machine once uninterrupted, then once per crash point, each resumed from the store's log by a fresh runtime:
+
+- **Log points:** before the first append, and after each append. The next write never lands.
+- **Effect points:** while each model call or plain actor is in flight. It ran, but its result was never recorded.
+
+Each resumed run is checked against the uninterrupted one. Each failure names its crash point and one of these problems:
+
+| `problem`         | Meaning                                                                                   |
+| ----------------- | ----------------------------------------------------------------------------------------- |
+| `reran`           | An effect ran again although its completion was durable before the crash                  |
+| `skipped`         | An effect of the uninterrupted run never ran                                              |
+| `unexpected`      | An effect ran that the uninterrupted run never made, such as a call under a new `callKey` |
+| `request-changed` | A model call re-executed under its `callKey` with a different request                     |
+| `outcome`         | The resume ended with a different status, output, state, or context                       |
+| `replay`          | The final log does not replay to the resumed state                                        |
+| `stuck`           | The crashed run or its resume did not settle within `timeout` (default 5000 ms)           |
+| `threw`           | The resume threw                                                                          |
+
+- An effect in flight at the crash may run twice. That is the at-least-once contract (see [Idempotency keys](hosts.md#idempotency-keys)), and it is not reported.
+- Model calls are keyed by `callKey` without the execution id, and plain actors by invoke site. Only actors bound through string-keyed sources are seen.
+- Model calls run live only in the uninterrupted run. After a crash, a call it made gets its recorded answer, after the same delay, so a fake may answer by call order.
+- Plain actors run live every time, so they must give the same result however often they run. A fake that keeps state passes `reset`, which runs before every run.
+- The outcome check compares final states, so a machine whose result depends on which concurrent call finishes first can fail it after a crash. Make the result order-independent, or test that part without concurrency.
+- `events` are host events, delivered one per idle settle. A resume skips the ones its log already holds.
+- `store` takes a factory for your own store adapter, one fresh store per run. A store that acknowledges a write it then loses shows up as `reran`.
+
+Run it in CI next to the machine's other tests. It needs no API key.
+
 ## Snapshot as cache
 
 `persist()` output carries `agentMeta: { machineId, version, logId, logIndex }` — the lineage and position of the log it caches.

docs/usage-and-budgets.md

@@ -190,6 +190,7 @@ A wrapper is still host-side. The machine cannot read its counts or react to the
 - Delivery is opt-in. The event is delivered only when an active state declares a transition that matches it, either `'@agent.usage'` or the wildcard `'*'`. Otherwise no usage event is sent to the machine. Wildcard matching follows XState semantics, so a catch-all `on: { '*': … }` does receive `@agent.usage`. Narrow the handler on `event.type` when the catch-all is meant for other events.
 - Declare the handler at machine level. A root `on: { '@agent.usage': … }` catches every call regardless of which state made it. A state-scoped handler drops calls made in other states.
 - Delivery goes to the run's root machine, including usage from requests inside an invoked child machine. Attribute those calls with `id`, `src`, and `model`.
+- Each call is counted once across a crash. A call in flight at a crash re-executes on resume, and when the log already holds its usage, matched by `callKey`, the repeat is not delivered or journaled. `onTrace` reports it as `usage.dropped` with `reason: "repeat"`.
 
 The payload has this shape.
 
@@ -202,6 +203,7 @@ The payload has this shape.
   src?: string,                         // the request's invoke src
   model?: string,                       // the model ref it targeted
   name?: string,                        // a text request's registered name
+  callKey?: string,                     // the call's idempotency key, with an event log
   budget?: { modelCalls, maxModelCalls }, // counted against maxModelCalls so far, this call included
 }
 ```

docs/verify.md

@@ -480,6 +480,7 @@ For machines authored as data, validate the config with `validateAgentConfig(con
 
 ## Related
 
+- [Crash testing](persistence.md#crash-testing): `checkCrashRecovery` kills a run at every log append and in-flight effect, and checks each resume.
 - [Debugging](debugging.md): scripted reproduction and the diagnostic codes in context.
 - [Evals](evals.md): scoring runs on output, trajectory, and budget.
 - [Quickstart](quickstart.md#define-and-run-one-artifact): a plain-function executor for `runToQuiescence`.

skills/build-machine/SKILL.md

@@ -62,7 +62,10 @@ expect(result).toMatchObject({ reachable: false, complete: true });
 
 ## 5. Crash and resume
 
-Run with a store and a thread id, kill the run mid-flight (throw from an executor, or exit the process), then call `runAgent` again with the same store and `threadId`.
+Call `checkCrashRecovery(machine, { input, events, executors })` from `@statelyai/agent/testing` for each scenario, and assert `report.failures` is `[]`. It kills the run after every log append and while every effect is in flight, resumes each from the log, and checks the resume against an uninterrupted run. See `docs/persistence.md#crash-testing`.
+
+- Executors and fake tools must answer from the request, not call order. A fake that keeps state passes `reset`.
+- A failure names the crash point and the problem (`reran`, `skipped`, `outcome`, `stuck`, …). Fix the machine or the effect, never the check.
 
 - A completed model call never runs again. An in-flight one re-runs with the same `info.callKey`.
 - A plain actor (`createAsyncLogic`) in flight at the crash re-runs. It gets no call key, so derive an idempotency key from the domain (the ticket id, the booking key) and dedupe in the effect.

src/agent-usage-event.test.ts

@@ -105,7 +105,7 @@ describe("@agent.usage (reserved per-call usage event)", () => {
     expect(result.usage).toEqual({ totalTokens: 1200, inputTokens: 900, modelCalls: 3 });
   });
 
-  test("carries attribution: kind, invoke id, src, model, and the request name", async () => {
+  test("carries attribution: kind, invoke id, src, model, the request name, and the call key", async () => {
     const delivered: AgentUsageEvent[] = [];
     await runToQuiescence(
       createAgentRuntime(budgetMachine, {
@@ -129,6 +129,7 @@ describe("@agent.usage (reserved per-call usage event)", () => {
       src: "researchStep",
       model: "test-model",
       name: "researchStep",
+      callKey: expect.stringMatching(/:research#1$/),
       usage: { totalTokens: 400, inputTokens: 300 },
       budget: { modelCalls: 1, maxModelCalls: 100 },
     });