[zh820]

Summary

• Update the pinned sure-evaluation engine to the merged LLM Judge version.
• Support passing llm_config through the existing SLU evaluation workflow.
• Preserve native 0–100 scores, sample judgments, and rubric/configuration provenance.
• Keep existing accuracy behavior and avoid reusing accuracy baselines for Judge RPS.
• Enable all 14 Judge regression tests in the existing CI workflow.

[PigeonDan1]

CI is now green on the current commit. The Judge implementation and engine update look reasonable; please refine the user-facing documentation before merging.

Please update sure/skills/sure_eval/SKILL.md and add or link a focused docs/llm_judge.md covering:

• A complete judge.json example with base_url, model, rubric, api_key_env, temperature, and max_tokens. State that non-local endpoints require HTTPS and that the OpenAI-compatible endpoint is /chat/completions.
• The exact SLU JSONL contract: prompt/question, context, choices/options, candidate answer, and optional reference answer. Include one minimal input and prediction example.
• A complete command from an existing /sure_infer bundle through /sure_eval metrics=llm_judge llm_config=..., including where the config file should live.
• Expected artifacts and fields: 0–100 score, per-sample reason, insufficient_evidence, batch score=null when any sample fails, rps=null, and where to inspect the pipeline trace.
• Reproducibility limits: pin the engine commit, Judge model/provider, rubric, temperature, and input bundle; explain that an external LLM with temperature 0 still does not guarantee bit-for-bit identical scores. The current Judge prompt hash covers the instruction/rubric, so document that scope precisely.
• Operational and security behavior: one request per sample, timeout/error behavior, required network/API access, API-key handling, and the fact that credentials are excluded from reports.
• Scope boundaries: Judge is for textual SLU answer quality, does not process audio, and does not replace existing SLU accuracy or RPS baselines.

This will make the feature usable by someone who has not read the implementation, while keeping the current code and CI behavior unchanged.

This comment is AI-generated by /wr.

[zh820]

Added docs/llm_judge.md and linked it from sure/skills/sure_eval/SKILL.md.

The guide covers:

• Complete Judge configuration, HTTPS and /chat/completions requirements.
• SLU input contracts and minimal input/prediction examples.
• An existing /sure_infer bundle → /sure_eval command.
• Scores, sample reasons, failure codes, null aggregation, RPS and trace locations.
• Reproducibility limits and the exact scope of prompt_sha256.
• Request/timeout behavior, API-key handling and textual-only scope.