Image Evals

A small, auditable Python harness for comparing image generation and editing models. The first suite uses gpt-image-2 for images and the pinned gpt-5.4-2026-03-05 snapshot as a multimodal rubric judge.

The included smoke run covers:

• UI mockup generation
• Logo generation and editing
• Marketing-flyer generation and editing
• Multi-reference virtual try-on editing

Setup

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev,viewer]'

Keep OPENAI_API_KEY in the ignored .env or .env.local file. The CLI loads it without displaying or copying it.

Inspect the existing results

.venv/bin/streamlit run app.py

The viewer only reads local artifacts. It cannot generate or grade images; its only write action is an explicit human-feedback submission.

The self-contained report is available at docs/implementation_summary.html. Rebuild it after future runs with:

.venv/bin/python scripts/build_summary.py

Run an evaluation

Review the dry validation and its estimated guardrail first:

.venv/bin/image-evals validate --suite smoke

Then explicitly authorize the bounded live run:

.venv/bin/image-evals run --suite smoke --confirm-spend

Generation and grading can also be separated:

.venv/bin/image-evals generate --suite smoke --confirm-spend
.venv/bin/image-evals grade <run-id> --confirm-spend

The default smoke suite permits at most six image calls, six judge calls, no automatic retries, and an estimated $0.50 maximum. These are local safety guards, not provider billing quotes.

Edit the experiment

• configs/models.yaml — model names, quality/detail, cost estimates, limits
• configs/suites.yaml — suite membership and order
• configs/graders.yaml — metric definitions, deterministic verdict rules, human-feedback fields, and failure tags
• cases/smoke/*.yaml — individual prompts, references, exact text, lineage, and per-case generation settings
• prompts/generators/*.md — generation/edit instructions
• prompts/graders/*.md — grader instructions

Each run snapshots the case, rendered prompts, model/rubric configuration, references, output, request metadata, usage, hashes, and deterministic verdict under runs/<run-id>/. Existing run artifacts are immutable.

API traceability

API tracking is enabled in configs/models.yaml. Every OpenAI call receives a unique X-Client-Request-Id, and both that client ID and OpenAI's x-request-id are saved with the local artifact.

Judge calls use the Responses API with store: true. Each stored Response is tagged with non-secret metadata for project, run_id, case_id, use_case, task, and grader_profile; its resp_... ID is saved locally and shown in the viewer's API trace tab. The Images API does not create stored Response objects, so generation and editing calls are correlated through their request IDs instead.

Stored judge Responses are retrievable through the API and may be inspected in the OpenAI dashboard. OpenAI documents Responses storage and metadata in the Responses API reference. For sensitive cases such as real-person VTO inputs, disable remote persistence before creating the run:

api_tracking:
  enabled: true
  project: image-evals
  store_judge_responses: false
  send_client_request_ids: true

The immutable local run remains the canonical evaluation record either way.

Verify

.venv/bin/python -m pytest
.venv/bin/python -m compileall -q src viewer app.py scripts

The frozen VTO reference assets can be recreated, if deliberately desired, with two additional low-quality image calls:

.venv/bin/python scripts/bootstrap_vto_assets.py --confirm-spend --force