SURE Harness

SURE Harness is an agent-based, system-level harness for reproducible model onboarding and evaluation. Built on the pi agent framework, it provides a TUI workflow that coding agents such as Codex, DeepSeek, and Kimi Code can operate through structured slash commands.

SURE turns model deployment, environment adaptation, inference-code generation, and dataset evaluation into one auditable workflow. The resulting runtime identity, dependency locks, parameters, model binding, predictions, metric route, and reports remain available as structured evidence, so an experiment is easier to reproduce, compare, and trust than an undocumented sequence of shell commands.

One discover, onboard, approve, infer, evaluate session, replayed as a self-contained SVG animation. Regenerate with node scripts/generate-readme-terminal.mjs.

Use SURE from another agent

skills/sure/SKILL.md is the single entry point for the complete SURE workflow: init, feed, onboard, trans, approve, infer, eval, agent_eval and resume. This checkout exposes it through .agents/skills/sure. Start a new agent session here and invoke $sure. To use it from another project, link the absolute skills/sure directory into your agent's skill directory (Codex: ~/.agents/skills/sure). Keep the installed harness checkout available.

The skill includes a Python 3.10+ helper that can be called from any directory (use uv run --no-project --python 3.11 in place of python if needed):

python /path/to/sure/skills/sure/scripts/run.py list
python /path/to/sure/skills/sure/scripts/run.py feed --arguments 'url=<model-url>' --dry-run

After setup, replace --dry-run with --trust to invoke the native harness in JSON print mode through the structured native bridge. --trust explicitly trusts this checkout for that invocation; without it the harness keeps its project-trust check. The helper returns JSON, checks correlated persisted terminal evidence and exits nonzero for non-success outcomes. It keeps the native state machines, runtime locks and human approval gates. Read the skill's workflow examples for initialization, handoffs, inference, scoring, approval and recovery. Provider configuration belongs to SURE, even when the calling agent already has a login. The existing-prediction scoring command in this version is sure_eval; sure_infer runs the model.

Distribution

This repository supports two distributions from one codebase:

• A distribution with config/site.bundled.yaml carries a trusted site policy. Users do not create a local policy before running SURE workflows.
• A public or self-hosted distribution carries no operator policy, no approved models, and no datasets. It falls back to the zero-configuration default at config/site.default.yaml, which keeps every root under ~/.sure. Users write their own storage and execution boundaries into config/site.local.yaml when those defaults do not fit.

The public core, command parameters, state machines, gates, runtime locks, artifact schemas, and evaluation configuration precedence are the same in both distributions. A site policy supplies deployment-specific roots and execution resources; it does not redefine workflow behavior.

Provider credentials and site policy are separate:

• /sure_init configures model providers and authentication.
• The site policy configures approved storage roots, forbidden output roots, dataset roots, a declared site runtime cache root, execution surfaces, and optional container delivery naming.

The locked Harness and Evaluation runtimes remain repository-local. A sealed local Model Python runtime is content-addressed below storage.runtime_root; its portable identity is stored with the model bundle and resolved against the active site at inference time.

Quick Start

Node.js 22.19 or newer (see .nvmrc), Git, uv, and the repository submodules are required. On Windows also install Git for Windows: the agent's bash tool and the skills' .sh templates run through it. You do not need a system Python: uv fetches and pins the interpreters the runtimes need.

Install uv on Linux or macOS:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install uv on Windows:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

The Harness and Evaluation runtimes are uv virtual environments that uv builds on first use. The first /sure_* command therefore needs network access and takes a few minutes while uv downloads its CPython 3.11 interpreter and installs the locked dependencies.

Bundled site policy

Use this path when the distribution contains config/site.bundled.yaml:

# Linux, macOS, or Git Bash on Windows
git clone --recurse-submodules <repository-url> sure-harness
cd sure-harness
npm install --ignore-scripts
npm run sure:site-info
npm run sure:site-check
npm run sure:doctor
PI_OFFLINE=1 ./pi-test.sh

# Windows PowerShell
git clone --recurse-submodules <repository-url> sure-harness
cd sure-harness
npm.cmd install --ignore-scripts
npm.cmd run sure:site-info
npm.cmd run sure:site-check
npm.cmd run sure:doctor
$env:PI_OFFLINE = "1"; powershell -ExecutionPolicy Bypass -File .\pi-test.ps1

:: Windows cmd.exe
git clone --recurse-submodules <repository-url> sure-harness
cd sure-harness
npm install --ignore-scripts
npm run sure:site-info
npm run sure:site-check
npm run sure:doctor
set PI_OFFLINE=1
.\pi-test.bat

The PowerShell blocks call npm.cmd and forward the launcher through powershell -ExecutionPolicy Bypass -File because Windows refuses to run .ps1 files, npm's own npm.ps1 included, under its default execution policy.

sure:site-info reports which policy is active; in a bundled distribution it must name the bundled source rather than the shipped default. Do not copy the public example over the bundled policy. A bundled distribution may include site-specific operating documentation below private/.

Public/self-hosted site policy

The public distribution contains no private storage paths, internal gateways, cluster defaults, or credentials. /sure_infer and /sure_eval need an approved model rather than a configured deployment: a fresh machine starts with an empty approved root, and both commands refuse until /sure_onboard and /sure_approve have published a model into it. Configure the deployment when the default roots do not fit, or to point at approved storage that already exists:

# Linux, macOS, or Git Bash on Windows
git clone --recurse-submodules --branch harness-tui-agent https://github.com/PigeonDan1/sure.git sure-harness
cd sure-harness
npm install --ignore-scripts
cp config/site.example.yaml config/site.local.yaml
# Edit every path and execution surface in config/site.local.yaml.
npm run sure:site-check
npm run sure:doctor
PI_OFFLINE=1 ./pi-test.sh

# Windows PowerShell
git clone --recurse-submodules --branch harness-tui-agent https://github.com/PigeonDan1/sure.git sure-harness
cd sure-harness
npm.cmd install --ignore-scripts
Copy-Item config/site.example.yaml config/site.local.yaml
# Edit every path and execution surface in config/site.local.yaml.
npm.cmd run sure:site-check
npm.cmd run sure:doctor
$env:PI_OFFLINE = "1"; powershell -ExecutionPolicy Bypass -File .\pi-test.ps1

config/site.local.yaml is ignored by Git and must remain local. Advanced deployments can instead set SURE_SITE_POLICY to an absolute path outside the repository.

npm run sure:doctor reports every prerequisite it can see — Node, uv, Git, rg, fd, a bash for the agent on Windows, and where the site policy came from. Warnings are safe to start with; rg and fd only make search faster, and the tools fall back to a built-in scan without them.

Without a local or bundled policy, SURE selects the shipped default at config/site.default.yaml, which keeps approved models, promoted results, the runtime cache, datasets, and dataset projections under ~/.sure. The default relaxes no gate: its approved root starts out empty, only /sure_approve publishes into it, and until it holds the model you name, a command that requires site resources still fails closed and points back to this Quick Start and the site configuration guide.

Local execution profiles

Docker registry delivery remains the default for local models and is required for VC execution. A self-hosted site that does not need Docker can explicitly permit sealed local Python execution:

execution:
  surfaces:
    - local
  local_runtimes:
    - python

Then select the Python profile during onboarding and local execution during inference:

/sure_onboard model=<model> package=none
/sure_approve model_dir=/path/to/sure/models/<model>
/sure_approve mode=approve review_manifest=/path/to/review_packet.json decision=approve
/sure_infer model=<approved-model> datasets=<source-path> execution=local
/sure_eval model=<approved-model> datasets=<dataset> metrics=<metric>

For keyword spotting, the dataset source declares task: KWS plus per-sample keywords and positive/negative references. Run inference normally, then select the canonical wake-word route with metrics=accuracy or metrics=macro_recall; the resulting metric report includes FAR, FRR, false alarms per hour, and the DET curve. The current SURE JSON route counts a sample positive only when detected is true and its score reaches the scanned threshold; complete WekWS-style DET/FAR requires threshold-independent candidate scores from the model wrapper.

For spoken language identification, declare task: LID and expose the model's language result as language or label. /sure_infer normalizes that result to the label projection consumed by sure-evaluation route lid.any.accuracy.lid_label_canonical_v1.classify_v1; see the LID onboarding playbook.

The two approval calls are intentionally separate: the first creates an immutable candidate and review packet; the second requires an explicit human decision and publishes the verified package. This path currently requires the uv backend and a hash-locked requirements file. SURE materializes a content-addressed Model Runtime below the configured storage.runtime_root, seals its portable manifest into the approved model bundle, and verifies the runtime plus model-core hashes before inference. A model-local .venv or arbitrary host Python is never accepted as an Eval runtime. Omitting package keeps the Docker registry default.

Site Configuration

Configuration sources use this fixed order:

1. Absolute path in SURE_SITE_POLICY.
2. Trusted distribution policy at config/site.bundled.yaml.
3. Local untracked policy at config/site.local.yaml.
4. Shipped default policy at config/site.default.yaml, which keeps every root under ~/.sure.

An explicit invalid SURE_SITE_POLICY never falls back to another source. Roots inside a policy are absolute, either POSIX (/srv/sure) or Windows drive-letter (D:\sure), and may refer to ${HOME} and ${REPO}. The complete schema, field semantics, symlink rules, examples, and diagnostics are documented in docs/site-configuration.md.

Configure storage.approved_models_roots[0] once. /sure_approve publishes approved packages only to that root, and /sure_infer model=<name> resolves the exact child directory from the same root. There is no command-level approval-root override.

For docker-registry delivery, the active site policy supplies the registry and repository template. SURE resolves the image destination and version before planning, rather than allowing an agent to invent a namespace; registry credentials remain in the deployment's Docker credential store.

Site policy is independent from the evaluation engine configuration. /sure_infer resolves its config= parameter first, then SURE_EVAL_CONFIG, then the evaluation submodule's config/default.yaml. /sure_eval rejects an explicit config=; it always scores the source bundle as resolved.

Verify Installation

npm run sure:site-info
npm run sure:site-check
npm run sure:doctor
npm run check

• sure:site-info reports the selected source, site_id, policy version, path, and SHA256 without printing credentials.
• sure:site-check validates the policy contract and storage-boundary relationships. It does not create directories or contact production services.
• sure:doctor checks the local harness installation and runtime prerequisites.
• npm run check runs repository-wide static checks.

In PowerShell, call npm.cmd instead of npm here too, for the execution-policy reason given in the Quick Start.

Seven SURE Commands

Command	Purpose	Main product
/sure_feed	Discover a model and create the onboarding handoff	sure/handoffs/<model>/model_input.yaml and evidence
/sure_onboard	Build and validate a runnable model package	wrapper, model package, and deployment_ready.json
/sure_trans	Transform an existing Dockerfile, model, and inference entrypoint into the standard Eval runtime contract	adapter wrapper, digest-pinned image, and deployment_ready.json
/sure_approve	Audit a completed model package, bind an explicit human decision, and publish it atomically	review packet, approval decision, and approval_ready.json
/sure_infer	Run an approved model over the selected datasets	an inference bundle: predictions, protocol.yaml, generation status, and reference projections
/sure_eval	Score an inference bundle with the evaluation engine, without running the model	an appended evaluation_runs/<batch>/ with metric artifacts and eval_run_report.json
/sure_agent_eval	Run an Agent (an ordered chain of approved models, e.g. ASR -> LLM translator) over speech datasets and score its answers	an agent product bundle (predictions, protocol, references) plus evaluation_runs/<batch>/ and eval_run_report.json

If an agent ends a turn without sure_finish, the harness continues it up to three times per unchanged checkpoint. Advancing to another unit resets this budget. If the provider fails, continuation cannot start, or the agent still does not finish, the harness closes the run as failed and writes .sure/runs/<run-id>/system_failure_manifest.json. That manifest records validation.gate=system_failure: it does not claim the skill's final gate passed. Explicit user interruption remains cancelled. /sure_resume <run-id> remains available for older interrupted runs with resumable checkpoints, but not for automatically closed runs.

Model publication is a two-run, human-gated /sure_approve workflow; ordinary workflow outputs cannot write into protected roots. Promotion of evaluation results remains a separate human-reviewed operation. Runtime products are otherwise staged in repository-local ignored directories or an explicit output_dir.

Repository Content

Source code, skills, schemas, documentation, and small fixtures belong in Git. Credentials, local policy, model weights, datasets, run state, and generated results do not. See .gitignore and AGENTS.md.

sure/external/sure-evaluation is pinned as a Git submodule. See docs/evaluation_engine.md for the evaluator boundary and route selection model.

Documentation

• docs/site-configuration.md: full site policy schema, source precedence, and diagnostics
• docs/evaluation_engine.md: evaluation engine boundary, routes, and runtime locking
• docs/pi-0.85.1-upgrade.md: what the pi 0.85.1 upgrade changed, and what the vendored pi copy is frozen at
• AGENTS.md: maintainer-facing development and hygiene guide
• Distributions with a bundled site policy may also carry additional site-specific documentation under a private directory that is not part of the public export.

License

MIT