cargo xtask m02-qualificationprovisions dependencies and runs independent producer, relay and consumer processes against isolated PostgreSQL schemas and owned authenticated NATS brokers, retaining scenario evidence.- For the focused
event-spine-processescheck only, setEDGEAGENT_POSTGRES_URLto an isolated administrative test database. Docker provisions the broker automatically; alternativelyEDGEAGENT_NATS_SERVERnames the absolute pinned 2.14.7 native executable. - Acceptance/outbox and inbox/probe/fact intent each share one commit. External publication occurs only through a separate production relay invocation.
- Semantic receipts and aggregate-local revision guards tolerate distinct message IDs, out-of-order work and competing replicas without global FIFO.
- The qualification command additionally proves specific worker kills, same-store broker restart and directional wire ACK loss. This does not establish model durability or production service readiness.
The conformance-worker binary belongs to the tooling-only
edgeagent-event-spine-conformance package. It is not an image/deployable and
does not alter product shells. ADR-0026
records ownership, lifecycle and compatibility decisions.
- A fixture administrator creates fresh producer/consumer schemas, applies outbox migrations to both, inbox migrations to the consumer, and only the owning synthetic acceptance or probe/semantic-receipt tables to each schema.
- It creates seven restricted logins: producer, producer relay, consumer, consumer relay, each owning schema's operator and the consumer dispatcher. None owns a schema or receives migration authority. A relay can select/update only its outbox. Peer-schema writes and runtime DDL fail.
- The authenticated provisioner creates registry-derived topology before any runtime child starts. Runtime broker permissions come from the same plan.
- A producer child connects, reports
ready, waits for one parent action byte, writes acceptance plus command outbox, commits and exitsaccepted. - A fresh producer-relay child claims/publishes/records one intent and exits
published; it holds no database transaction while publishing. - A consumer child receives one command, validates it, atomically writes inbox,
semantic receipt, revision/value and fact intent, commits, confirms ACK and
exits
applied. A committed transport duplicate exitsduplicatewithout invoking the callback. A semantic duplicate commits its distinct inbox but writes no second receipt, transition or fact. - A consumer-relay child publishes the fact. The audit subscriber decodes the typed payload and verifies correlation, causation, revision and value.
Each child receives its runtime database URL through its private environment,
never process arguments. Only schema identifiers beginning m02_ and containing
bounded lowercase ASCII/digits/underscores enter connection search-path options.
Broker namespace and runtime component are validated against the topology plan.
Producer requires no broker connection. There is no automatic topology mutation
or schema creation inside a worker.
independent_processes_commit_acceptance_transition_and_fact_once starts fresh
processes for every operation, including identical producer re-entry and idle
relay re-entry. It verifies one acceptance, published command, inbox, probe and
fact intent. It republishes identical command bytes without Nats-Msg-Id so the
second transport delivery cannot be hidden by broker publication deduplication.
The new consumer returns duplicate, one transition/fact remains, and command
ACK state is empty after commit. Actual SQLSTATE 42501 proves peer-write and DDL
denials, not an inference from a timeout.
process_rollbacks_leave_no_partial_acceptance_or_inbound_fact asks the tooling
producer to roll back after enqueue, proving neither row survives. It then
accepts/publishes normally. A tooling callback returns a transient failure after
inbox, state, semantic receipt and fact enqueue, proving all four roll back before retry is
confirmed. A fresh consumer receives the retained command and commits normally.
These synthetic controls remain outside production adapter APIs.
semantics::competing_semantic_commands_commit_one_transition_and_retain_conflicts
holds two independent deliveries before releasing competing replicas. Distinct
transport IDs for the same probe/revision/value produce two inbox receipts but
one state transition, semantic receipt and causal fact. A changed value enters
semantic_conflict quarantine with the exact original bytes; accepted state,
receipt and fact remain unchanged.
semantics::missing_predecessors_recover_in_order_or_exhaust_into_quarantine
delivers revision one before revision zero. It proves the gap rolls back all
writes, then holds its redelivery while another process commits the predecessor.
Releasing the successor advances state to revision two. An old semantic duplicate
still compares accepted history, not current value. A separate unresolved gap
requests four retries, then retains missing_predecessor quarantine on delivery
five without a partial inbox, provisional state, receipt or fact.
replicas::competing_replicas_respect_shared_inflight_high_water provisions a
durable bound of two outstanding ACKs with four waiting pulls allowed. Two child
processes hold deliveries and a third has a confirmed waiting pull, with one
message still pending. No transaction/state exists yet. Only releasing a held
delivery through committed processing/ACK admits the third; all three finish
with one receipt/state/fact each. Broker state, not a timing threshold, proves
the admission bound. Each worker owns one connection/delivery and does not prefetch.
replicas::locked_partition_and_connection_loss_preserve_other_progress_and_recovery
observes a worker blocked by an administrator-held aggregate row lock. Another
partition commits before the lock is released. The fixture then terminates only
that observed worker connection mid-transaction; confirmed retry and durable
inspection show no partial inbox/receipt/state/fact. A fresh process consumes the
retained command and advances the aggregate once.
ADR-0027
defines the conformance-specific policy. Semantic identity is
(probe_id, expected_revision); content compares validated value, not message
ID, envelope metadata or unknown optional fields. Accepted receipts are immutable
and remain available after later revisions. Under the probe row lock, an identical
receipt is a no-op; changed content conflicts. With no receipt, only the exact
current predecessor permits a guarded update and receipt/fact insertion. A future
revision defers; an unexplained stale revision fails permanently.
A provisional revision-zero row serializes contenders for a previously absent
aggregate and rolls back with an unfulfilled predecessor. Locks are per aggregate,
not global, and have the existing two-second timeout. This proves safe concurrent
admission, not FIFO, starvation freedom, throughput, or unbounded storage. The
portable inbox does not acquire business-key policy. applied means that a first
transport identity committed; inspect receipt/state/fact counts to distinguish a
new transition from an acknowledged semantic duplicate.
Facts derive stable identity from probe/revision and causation/correlation from the winning command. Facts also preserve that command's validated W3C carrier; the proof exporter's stage spans link it without rewriting accepted bytes. Producer acceptance is still transport-identity based; consumer semantics decide whether its independently published request changes state.
[Prerequisite from M10] Each child handles one operation only: two runtime
threads, one connection, one delivery or claim. Readiness follows actual
connection, not elapsed time. The parent starts work with one byte (c commit,
r fixture rollback), closes stdin, then checks a bounded result and exit.
For an explicit producer command, bounded structured envelope bytes follow the
action and end at EOF; absent bytes select the original default fixture. The
consumer-only h action emits delivered after receive and waits for a c
release byte before opening any transaction. This fixture barrier exists only
inside tooling, not production adapters. A unique worker application name lets
the administrator identify owned lock waits without exposing credentials.
Children have a 30-second total deadline; the proof has a tighter 15-second child
budget and 90-second database-scenario budget. SQL statements are limited to 5 seconds,
locks to 2 seconds, broker operations/receive to 5 seconds. Runtime shutdown is
bounded even if a parent abandons stdin. No arbitrary sleep coordinates work.
The parent kills/reaps owned children on ordinary errors/deadlines; kill-on-drop is a cancellation backstop. Owned database and broker fixtures attempt bounded cleanup on errors, panics and deadlines. Disposable CI teardown handles process abort. Unknown or cancelled outcomes are not confirmed rollback/success; retained identity and outbox/inbox evidence determine safe re-entry. Never rerun with new message IDs to conceal an uncertain result.
Worker errors intentionally omit source chains and payloads. A failed child
returns unresolved failure, not a fabricated durable success. A classified
settlement failure reports confirmation_unknown; it never reports applied.
Oversized raw intake reports protocol_halt without ACK/quarantine. Tests distinguish
explicit rollback, committed success, duplicate and idle outcomes. Neither child
exit nor broker health alone constitutes transaction evidence.
Default cargo xtask verify compiles the worker and discovers the ignored T3
suite; it does not execute service-backed proofs. cargo xtask m02-qualification
provisions isolated dependencies, runs protected suites, retains reports and
cleans up. Missing prerequisites fail, never skip. The focused check below
still accepts an explicitly managed isolated PostgreSQL environment.
cargo xtask verify --check event-spine-processes
All service-owned statements use checked sqlx::query! and committed
offline metadata. The metadata action also applies
crates/event-spine-conformance/fixtures/process-schema.sql in its authorized
metadata database before preparing all targets. Test-only schema/grant/inspection
queries remain dynamic; no untrusted message fields enter SQL identifiers.
This is bounded synthetic probe behavior, not product research/order policy. Use fresh disposable schemas: old retained fixtures constrain revision to one and lack semantic receipt history. Neither mixed-version workers nor an upgrade of retained synthetic state is supported. Never infer old receipts from current state or apply the fixture to a product deployment.
The canonical gate admits at most two scenarios concurrently, independently of host CPU count. Each scenario retains its internal competing workers and its original deadlines; this bounds fixture pressure, not the behavior under test.
The nineteen-case protected suite includes four
authorized replay proofs: authenticated/atomic authorization,
inbound result-loss recovery and bounded fencing, original-route fan-out, and
outbound audited release. Replay authorization needs no broker connection;
dispatch uses a separate login and exact-subject broker scope. Its fixture-only
p barrier reports persisted after a real broker receipt, then waits for c
outside any transaction so a parent can kill or fence that holder.
Two exported telemetry proofs add an owned OTLP/HTTP receiver, non-default W3C links, exact metric dimensions/counts, raw-wire privacy sentinels and real child PID evidence. Rejected or stalled exports must preserve commits, duplicate suppression and confirmed settlement. Export is opt-in in child-only environment; normal process cases remain unconfigured. Bounded SDK threads and HTTP transport are installed before Tokio and shut down after it; their diagnostic result never replaces the worker's operation result.
Seven failure qualification cases add producer, relay
and consumer process kills, same-store broker restart, directional wire ACK
loss, duplicates after broker deduplication expiry and invalid-input bounds.
Fixture-only b/e producer actions hold before commit/after enqueue, u
holds a real relay receipt before completion, and a holds postcommit
settlement. These closed actions accept no arbitrary payload. Consumer v
reports actual delivery attempt before a c release. All stay out of adapter APIs.
The proof does not establish production telemetry operation, arbitrary crash or network faults, PostgreSQL restart, power-loss, HA or load capacity. Connection interruption is not a database restart or an unknown-commit proof. Those remain in the M02 completion plan or their owning milestones. No model-provider calls, local model runtime or live trading are involved.