Check the evidence behind performance claims.
Latest maintenance: verified experimental 0.8.1 preserves failed outcomes and original exceptions through competing observer/ cleanup faults. Eleven new regressions; 328 tests per hosted matrix passed.
Status: Experimental / Research Prototype. V10 collection is complete; latest verified release is 0.8.1. All agreed stages are complete; prior releases and negative results remain preserved.
Latest increment: V10 usable observation collection, verified experimental 0.8.1. A bounded stdlib recorder and explicit optional OTel exporter bridge supply caller-owned lifecycle/edge evidence. Usage and limits. The pinned Pydantic example retains partial coverage and UNKNOWN model demand. Release proof and measured collection cost.
Latest application study: V5 HotpotQA qualification completed eight real requests but failed its predeclared quality floor (4/8 exact answers, mean answer F1 0.542). Evidence and limits. Its conditional performance study was not unlocked; useful-agent capacity is still unestablished.
TraceWarrant checks whether observations support a performance model and whether frozen predictions survive real interventions. Use its trace conformance checker and simulator-neutral validator without adopting our simulator. UNKNOWN, unsupported claims and a simple baseline beating a complex model are useful results. Returning to the project? Start with the short refresher, then PROJECT_COMPLETION.md, PROJECT_STATUS.md and the continuation checkpoint.
Former internal name: Workload Lab. Source is available at G26karthik/tracewarrant, licensed Apache-2.0. The workload-lab command, workload_lab Python imports and versioned artifact/telemetry bindings remain compatible. GitHub releases, release qualification.
Publication status, checked 2026-10-15: experimental prerelease v0.8.1 is verified. All five exact-source CI jobs passed, with 328 tests per matrix job. All thirteen public downloads, fresh installed core/SDK and failure checks, new stdlib collection and exact trace/bundle replay passed. Proof, cost and limits. Private security reporting is enabled.
V3 accepts the pilot's PIVOT RECOMMENDED conclusion. V2, its final report, research-pilot-1 and all frozen evidence remain unchanged. This is a local research prototype, not a capacity planner or production release. Refreshed research establishes substantial overlap; no uniqueness is claimed.
Export application quality inputs and reports with bundle-quality, then
recompute their JSON/HTML using verify-quality-bundle on another machine.
Portable quality guide. Integrity checks do not
authenticate authors, scores or preregistration chronology.
The 0.6.0 release can bind a quality policy and declared task roster
with freeze-quality, then require --plan/--freeze during qualification.
Quality-plan guide. Byte/membership verification does
not authenticate preregistration chronology or collector honesty.
Check application quality before creating performance predictions:
uv run tracewarrant qualify-application examples/qualification/hotpot/sample.json --policy examples/qualification/hotpot/policy.json --output artifacts/quality-reportThis retained sample correctly returns 1 for its failed exact/F1 quality gate. It needs no model or prediction freeze. Quality workflow and schemas.
The 0.4.0 workflow adds one manifest, CI pass/fail, standalone HTML/JSON and portable evidence verification. Complete guide, job schema, application CI example.
uv sync --locked
python -c "from pathlib import Path; Path('artifacts').mkdir(exist_ok=True)"
uv run tracewarrant validate-job examples/validation/frames/job.json --output artifacts/strict-reportThis retained study deliberately returns 1: exact-answer quality failed, interventions extrapolate beyond calibration, and added decision value is unestablished. Open artifacts/strict-report/report.html. The separate performance-job.json permits a limited numerical replay and says quality was not assessed. Use bundle-job and verify-bundle to share and replay checksummed evidence offline. Existing outputs are never overwritten.
Python 3.11+; core runtime has no third-party dependencies, service or GPU requirement.
git clone https://github.com/G26karthik/tracewarrant.git
cd tracewarrant
uv sync --locked
uv run tracewarrant inspect examples/traces/pydantic-ai-2.51.0.otlp.json
uv run tracewarrant inspect examples/v3/frames-heldout-base.otlp.json
uv run tracewarrant schema
uv run tracewarrant check docs/v3/frames-frozen/simpy.jsonThe framework capture cannot identify queue/service boundaries. The external FRAMES capture exposes a misspelled completeness binding; inspection refuses it. An explicit adapter and the retained failure show how to fix an existing assertion without guessing topology.
Re-evaluate the frozen external experiment entirely offline:
uv run workload-lab evaluate --protocol docs/v3/frames-frozen/protocol.json --freeze docs/v3/frames-frozen/freeze.json --prediction docs/v3/frames-frozen/simpy.json --prediction docs/v3/frames-frozen/fixed.json --prediction docs/v3/frames-frozen/utilization.json --measurement docs/v3/frames-evaluation/measured-r0.json --measurement docs/v3/frames-evaluation/measured-r1.jsonIn that study, SimPy and the utilization heuristic selected the same intervention. Exact-answer quality failed, so measured speedups do not establish a useful agent. Results and limits, reproduction, observation contract, artifact schema/evaluation.
from workload_lab.conformance import inspect_observations
from workload_lab.ingest import ingest
report = inspect_observations(ingest("traces.json"))
print(report["claims"]["inference_service_demand"]) # UNKNOWN, with required instrumentsPython 3.11+; no database, GPU, model, agent framework or server needed. From this directory:
uv sync --locked
uv run workload-lab analyze examples/research-workflow.otlp.json --origin synthetic
uv run workload-lab analyze examples/research-workflow.otlp.json --format json
uv run workload-lab ingest examples/research-workflow.otlp.json --output workload-ir.json
uv run workload-lab simulate examples/scenario.json --output simulation.json
uv run pytest--output creates a new file and refuses overwrite. analyze consumes original OTLP JSON/JSONL, not compiled IR. simulate accepts a separate explicit scenario and never executes traced tools. Runtime-only installation works with python -m pip install ., or install a wheel from GitHub Releases. There is no PyPI release; the package retains its registry upload-prevention classifier.
from workload_lab import analyze, compile_workload, ingest
dataset = ingest("examples/research-workflow.otlp.json", origin="synthetic")
for workflow in compile_workload(dataset):
report = analyze(workflow)
print(report.critical_path_elapsed.value) # nanoseconds, with evidence beside itThe synthetic example has a hand-derived 11 s declared path, 14 s total work and 1 s unattributed wall time. These are fixture arithmetic, not benchmark claims. Example explanation.
OTel nesting expresses containment, not dependency. Declare finish-to-start prerequisites with workload_lab.depends_on; otherwise the graph remains incomplete. Work spans are atomic leaves; container spans are excluded from additive work. Longest path uses fixed observed elapsed weights and zero gaps. It cannot predict another load, infer service time or prove a capacity bottleneck. Contributions along one tied path may not be unique.
Missing queue/service values are UNKNOWN. Uncovered time is unattributed. GPU utilization and capacity recommendations remain unavailable. Every timing quantity carries origin, provenance, source and derivation method. Synthetic resources downgrade evidence automatically; no confidence interval is invented from one trace.
Accepts OTLP JSON resourceSpans/scopeSpans/spans envelopes, or one envelope per JSONL line. Not arbitrary framework exports or SDK console output. Default limits: 16 MiB and 50,000 spans. Exact mapping, IR semantics.
No upload or telemetry. The allowlist removes payload-bearing names, prompts, arguments, messages and events. Selected service/model/tool labels and IDs remain potentially sensitive. Security.
The controlled experiment achieved 1.70% median p95 error across eight frozen held-out cells. A utilization heuristic selected the same useful intervention; the workload uses stub inference. This does not establish real-agent differentiation. See instrumentation and simulation semantics before interpreting results.
For fresh artifacts, create artifacts/ and use new output names:
uv run python -m examples.capture_controlled --output artifacts/actual.json
uv run --group integration python -m examples.capture_pydantic_ai --output artifacts/framework.json
uv run --group validation python -m benchmarks.validate_simulation --output artifacts/analytic.json
uv run python -m benchmarks.simulation_benchmark --output artifacts/scale.jsonCore tests use the committed content-free framework trace and need no framework. Optional groups isolate PydanticAI/OTel and the independent SimPy baseline. The real-model probe uses already-cached local llama3.1:8b after preserving a failed Qwen warmup. It never downloads weights or contacts a paid provider.
Architecture, roadmap, ADRs, contributing, benchmark methodology, validation plan, license inventory.
The two ContextForge handoffs remain unchanged historical context. TraceWarrant replaces the internal Workload Lab name; historical names remain in frozen evidence. New project code is Apache-2.0; reviewed competitor code has not been imported.