[zhaoyuuu0624]

fix(replay): isolate blocking resource sampling from trajectory scheduling

Problem

Concurrent replay performs synchronous host resource reads (including cgroup files and /proc/<pid>/smaps_rollup), JSON serialization, and sample writes on the asyncio thread that also schedules trajectories. Slow sampling delays timer wakeups and command-control callbacks, adding measurement overhead to the workload being measured. Sampling also needs to finish before the associated VM is torn down.

In the previous 80-trajectory UKSM run, mean extra waiting per trajectory was 249.11 seconds after subtracting the original planned sleeps. This is replay-controller overhead, not evidence that the task needed additional computation or that the VM runtime itself spent that time sleeping.

Upstream scope

This PR extracts the validated sampling and wait-timing helpers for explicit adoption by concurrent deployment controllers. It does not change the current synchronous single-task runner or VM runtime. The historical controller integration (including the UKSM-only scan change) is retained as a diagnostic diff, not an automatically applied upstream backend change. This is independent of the checked-exec startup changes in PR #135; their older optional sampler can subsequently be consolidated.

Changes

• Move blocking collection, serialization, and sample writes into a dedicated bounded four-worker executor, separate from tool execution. Keep one outstanding read per VM and the existing 0.5-second delay after each completed sample; do not generate catch-up bursts.
• Drain in-flight sampling on cancellation before stopping a VM. Propagate unexpected collection failures instead of reporting missing data as zero usage.
• Add sample queue/read latency, event-loop delay, and pause/sleep/resume timestamps to distinguish planned waits from measurement and scheduling overhead. Preserve the full original sleep duration.
• Skip the offload-only guest-process safety scan when entering a UKSM wait. Keep sampler STOP/ack/CONT, offload safety checks, rejection of offload outside offload mode, and final shutdown audits.

The shared sampling fix is implemented for both replay adapters. The full-run validation below covers the updated pVisor UKSM adapter; the AgentENV result is the existing reference run, not a new run of its updated adapter.

Validation

The frozen deployment package passes 13 local tests; the extracted upstream helpers pass 7 standalone tests. mise is unavailable on this host, so validation used standard-library unittest discovery directly. The deployment tests cover covering event-loop responsiveness under blocking reads, bounded workers, executor separation, cancellation draining, error propagation, real observer integration, unchanged sleep budgets, and offload/shutdown safeguards.

The four-trajectory A/B completed 290 commands in each arm with clean shutdowns. Mean extra waiting fell from 6.65 to 1.85 seconds; batch time changed only from 29m51s to 29m45s. The larger effect appears in the full concurrent workload below.

Full-run conditions: same 80 trajectories, five waves of 16 arriving every five minutes, 24 physical CPU cores, a 48 GiB parent cgroup limit, 2 vCPU/5 GiB guests, 4 GiB admission headroom, at most one launch per second, and a 30-second command timeout. Original commands and waits are unchanged; a cleaned command timeout does not abort its trajectory. No filesystem access-path or runtime-binary optimization is included.

Metric	Previous UKSM	Fixed UKSM	Existing AgentENV reference
Replay makespan	59m50s	50m55s	49m25s
Mean extra wait per trajectory	249.11 s	1.02 s	1.60 s
Mean initialization	58.22 s	30.58 s	0.77 s
Mean admission queue	14.54 s	8.01 s	11.00 s
Peak concurrent VMs	77	74	70
Mean parent memory	34.34 GiB	29.57 GiB	41.10 GiB
Peak parent memory	45.86 GiB	45.83 GiB	48.00 GiB
CPU consumption	53,620 core-seconds	49,655 core-seconds	4,742 core-seconds
Command timeouts	31	30	6
Completed trajectories / commands	80 / 3,007	80 / 3,007	80 / 3,007
Strict exit audits passed	80 / 80	80 / 80	80 / 80

The fixed run finished 8m55s earlier (14.9%) than the previous UKSM run, and 1m30s later than the AgentENV reference. Mean extra waiting is nearly eliminated; tool execution cost remains largely unchanged (97.96 vs 98.83 guest-command seconds per trajectory). Final audits report no live VMs, mounts, or workspaces, with no OOM or swap use.