Skip to content

About

Content-free telemetry conformance and simulator-neutral validation of performance predictions for agentic systems.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

67 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

TraceWarrant

Check the evidence behind performance claims.

Latest maintenance: verified experimental 0.8.1 preserves failed outcomes and original exceptions through competing observer/ cleanup faults. Eleven new regressions; 328 tests per hosted matrix passed.

Status: Experimental / Research Prototype. V10 collection is complete; latest verified release is 0.8.1. All agreed stages are complete; prior releases and negative results remain preserved.

Latest increment: V10 usable observation collection, verified experimental 0.8.1. A bounded stdlib recorder and explicit optional OTel exporter bridge supply caller-owned lifecycle/edge evidence. Usage and limits. The pinned Pydantic example retains partial coverage and UNKNOWN model demand. Release proof and measured collection cost.

Latest application study: V5 HotpotQA qualification completed eight real requests but failed its predeclared quality floor (4/8 exact answers, mean answer F1 0.542). Evidence and limits. Its conditional performance study was not unlocked; useful-agent capacity is still unestablished.

Correctness

TraceWarrant checks whether observations support a performance model and whether frozen predictions survive real interventions. Use its trace conformance checker and simulator-neutral validator without adopting our simulator. UNKNOWN, unsupported claims and a simple baseline beating a complex model are useful results. Returning to the project? Start with the short refresher, then PROJECT_COMPLETION.md, PROJECT_STATUS.md and the continuation checkpoint.

Former internal name: Workload Lab. Source is available at G26karthik/tracewarrant, licensed Apache-2.0. The workload-lab command, workload_lab Python imports and versioned artifact/telemetry bindings remain compatible. GitHub releases, release qualification.

Publication status, checked 2026-10-15: experimental prerelease v0.8.1 is verified. All five exact-source CI jobs passed, with 328 tests per matrix job. All thirteen public downloads, fresh installed core/SDK and failure checks, new stdlib collection and exact trace/bundle replay passed. Proof, cost and limits. Private security reporting is enabled.

V3 accepts the pilot's PIVOT RECOMMENDED conclusion. V2, its final report, research-pilot-1 and all frozen evidence remain unchanged. This is a local research prototype, not a capacity planner or production release. Refreshed research establishes substantial overlap; no uniqueness is claimed.

Run an explicit validation policy

Export application quality inputs and reports with bundle-quality, then recompute their JSON/HTML using verify-quality-bundle on another machine. Portable quality guide. Integrity checks do not authenticate authors, scores or preregistration chronology.

The 0.6.0 release can bind a quality policy and declared task roster with freeze-quality, then require --plan/--freeze during qualification. Quality-plan guide. Byte/membership verification does not authenticate preregistration chronology or collector honesty.

Check application quality before creating performance predictions:

uv run tracewarrant qualify-application examples/qualification/hotpot/sample.json --policy examples/qualification/hotpot/policy.json --output artifacts/quality-report

This retained sample correctly returns 1 for its failed exact/F1 quality gate. It needs no model or prediction freeze. Quality workflow and schemas.

The 0.4.0 workflow adds one manifest, CI pass/fail, standalone HTML/JSON and portable evidence verification. Complete guide, job schema, application CI example.

uv sync --locked
python -c "from pathlib import Path; Path('artifacts').mkdir(exist_ok=True)"
uv run tracewarrant validate-job examples/validation/frames/job.json --output artifacts/strict-report

This retained study deliberately returns 1: exact-answer quality failed, interventions extrapolate beyond calibration, and added decision value is unestablished. Open artifacts/strict-report/report.html. The separate performance-job.json permits a limited numerical replay and says quality was not assessed. Use bundle-job and verify-bundle to share and replay checksummed evidence offline. Existing outputs are never overwritten.

Inspect and validate without a simulator

Python 3.11+; core runtime has no third-party dependencies, service or GPU requirement.

git clone https://github.com/G26karthik/tracewarrant.git
cd tracewarrant
uv sync --locked
uv run tracewarrant inspect examples/traces/pydantic-ai-2.51.0.otlp.json
uv run tracewarrant inspect examples/v3/frames-heldout-base.otlp.json
uv run tracewarrant schema
uv run tracewarrant check docs/v3/frames-frozen/simpy.json

The framework capture cannot identify queue/service boundaries. The external FRAMES capture exposes a misspelled completeness binding; inspection refuses it. An explicit adapter and the retained failure show how to fix an existing assertion without guessing topology.

Re-evaluate the frozen external experiment entirely offline:

uv run workload-lab evaluate --protocol docs/v3/frames-frozen/protocol.json --freeze docs/v3/frames-frozen/freeze.json --prediction docs/v3/frames-frozen/simpy.json --prediction docs/v3/frames-frozen/fixed.json --prediction docs/v3/frames-frozen/utilization.json --measurement docs/v3/frames-evaluation/measured-r0.json --measurement docs/v3/frames-evaluation/measured-r1.json

In that study, SimPy and the utilization heuristic selected the same intervention. Exact-answer quality failed, so measured speedups do not establish a useful agent. Results and limits, reproduction, observation contract, artifact schema/evaluation.

from workload_lab.conformance import inspect_observations
from workload_lab.ingest import ingest

report = inspect_observations(ingest("traces.json"))
print(report["claims"]["inference_service_demand"])  # UNKNOWN, with required instruments

Preserved reference backend

Python 3.11+; no database, GPU, model, agent framework or server needed. From this directory:

uv sync --locked
uv run workload-lab analyze examples/research-workflow.otlp.json --origin synthetic
uv run workload-lab analyze examples/research-workflow.otlp.json --format json
uv run workload-lab ingest examples/research-workflow.otlp.json --output workload-ir.json
uv run workload-lab simulate examples/scenario.json --output simulation.json
uv run pytest

--output creates a new file and refuses overwrite. analyze consumes original OTLP JSON/JSONL, not compiled IR. simulate accepts a separate explicit scenario and never executes traced tools. Runtime-only installation works with python -m pip install ., or install a wheel from GitHub Releases. There is no PyPI release; the package retains its registry upload-prevention classifier.

from workload_lab import analyze, compile_workload, ingest

dataset = ingest("examples/research-workflow.otlp.json", origin="synthetic")
for workflow in compile_workload(dataset):
    report = analyze(workflow)
    print(report.critical_path_elapsed.value)  # nanoseconds, with evidence beside it

The synthetic example has a hand-derived 11 s declared path, 14 s total work and 1 s unattributed wall time. These are fixture arithmetic, not benchmark claims. Example explanation.

What the report means

OTel nesting expresses containment, not dependency. Declare finish-to-start prerequisites with workload_lab.depends_on; otherwise the graph remains incomplete. Work spans are atomic leaves; container spans are excluded from additive work. Longest path uses fixed observed elapsed weights and zero gaps. It cannot predict another load, infer service time or prove a capacity bottleneck. Contributions along one tied path may not be unique.

Missing queue/service values are UNKNOWN. Uncovered time is unattributed. GPU utilization and capacity recommendations remain unavailable. Every timing quantity carries origin, provenance, source and derivation method. Synthetic resources downgrade evidence automatically; no confidence interval is invented from one trace.

Input and privacy

Accepts OTLP JSON resourceSpans/scopeSpans/spans envelopes, or one envelope per JSONL line. Not arbitrary framework exports or SDK console output. Default limits: 16 MiB and 50,000 spans. Exact mapping, IR semantics.

No upload or telemetry. The allowlist removes payload-bearing names, prompts, arguments, messages and events. Selected service/model/tool labels and IDs remain potentially sensitive. Security.

Development and direction

The controlled experiment achieved 1.70% median p95 error across eight frozen held-out cells. A utilization heuristic selected the same useful intervention; the workload uses stub inference. This does not establish real-agent differentiation. See instrumentation and simulation semantics before interpreting results.

For fresh artifacts, create artifacts/ and use new output names:

uv run python -m examples.capture_controlled --output artifacts/actual.json
uv run --group integration python -m examples.capture_pydantic_ai --output artifacts/framework.json
uv run --group validation python -m benchmarks.validate_simulation --output artifacts/analytic.json
uv run python -m benchmarks.simulation_benchmark --output artifacts/scale.json

Core tests use the committed content-free framework trace and need no framework. Optional groups isolate PydanticAI/OTel and the independent SimPy baseline. The real-model probe uses already-cached local llama3.1:8b after preserving a failed Qwen warmup. It never downloads weights or contacts a paid provider.

Architecture, roadmap, ADRs, contributing, benchmark methodology, validation plan, license inventory.

The two ContextForge handoffs remain unchanged historical context. TraceWarrant replaces the internal Workload Lab name; historical names remain in frozen evidence. New project code is Apache-2.0; reviewed competitor code has not been imported.

About

Content-free telemetry conformance and simulator-neutral validation of performance predictions for agentic systems.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages