Skip to content

Fix: Bug 288 + Tests - #545

Merged
codelion merged 3 commits into
algorithmicsuperintelligence:mainfrom
SUPR1M3:fix/288-fail-on-zero-successful-generations
Oct 25, 2026
Merged

codelion merged 3 commits into
algorithmicsuperintelligence:mainfrom
SUPR1M3:fix/288-fail-on-zero-successful-generations

Conversation

@SUPR1M3

@SUPR1M3 SUPR1M3 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #288

Problem:

When every iteration fails before producing a candidate program (LLM endpoint down, deployment missing, model returning empty replies), ProcessParallelController.run_evolution logs each error as a warning and moves on. Nothing tracks whether any iteration succeeded, so the run ends exactly like a healthy one:

  • the CLI prints Evolution complete! with the seed's metrics and exits with code 0
  • best/best_program.py is written as a byte-identical copy of initial_program.py

Scripted pipelines check the exit code and the final banner, so a run that generated nothing looks like a successful run that simply found nothing better than the seed.

What:

  • ProcessParallelController.run_evolution counts iterations whose worker produced a candidate program and records the last iteration error (worker error, timeout, or result-processing exception). At the end of the run it sets produced_no_candidates when iterations completed but none produced a candidate, and logs an error. The flag is computed inside run_evolution rather than as a property, because stop() sets the shutdown event during normal cleanup.
  • OpenEvolve.run raises a new EvolutionFailedError (exported from openevolve) in that case, before the best program is saved, so the seed is never presented as an evolved result. The message includes the count and the last error, e.g. Evolution failed: 0/10 iterations produced a candidate program. Last error: LLM generation failed: Error code: 404 - ....
  • cli.py catches EvolutionFailedError, prints the message without a traceback and returns exit code 1.
  • Behaviour is unchanged for intermittent failures (a single successful iteration is enough for the run to finish normally) and for runs interrupted by a shutdown request, which are not treated as failures. Checkpoints are still written as before.
  • Add tests/test_all_iterations_failed.py (6 tests): controller counting (all errors, one success among errors, shutdown), OpenEvolve.run raising and not writing best/, and an end-to-end CLI run against a local HTTP server that answers every LLM request with 404 (the reproduction from the issue).
  • Existing tests that replace ProcessParallelController with a bare Mock/MagicMock now set produced_no_candidates = False, the same way they already set shutdown_event (test_checkpoint_resume.py, test_initial_program_artifacts.py, test_island_system_messages.py).

"Candidate produced" is used as the success signal rather than "LLM call succeeded", so a run where the model answers but never returns usable output (e.g. No valid diffs found in response on every iteration) also fails. Nothing was evolved in that case either.

Red -> green evidence:

On main, the new tests fail:

ERROR: test_all_errors_produce_no_candidates      AttributeError: 'ProcessParallelController' object has no attribute 'produced_no_candidates'
ERROR: test_shutdown_is_not_a_failure             AttributeError: (same)
ERROR: test_single_success_counts_as_progress     AttributeError: (same)
FAIL: test_exits_non_zero_without_success_banner  AssertionError: 0 != 1
FAIL: test_raises_and_does_not_save_seed_as_best  AssertionError: EvolutionFailedError not raised
FAILED (failures=2, errors=3)

With this PR: Ran 6 tests ... OK.

Testing:

  • Unit tests: 887 pass on Python 3.14 (881 on main plus the 6 new tests); no other per-test differences from main

  • End to end with the issue's reproduction: examples/function_minimization with only api_base pointed at a local stub, 10 iterations, default retries, 3 workers:

    Endpoint returns Branch Exit code Evolution complete! best/
    HTTP 404 main 0 printed copy of the seed
    HTTP 404 this PR 1 not printed not written
    HTTP 500 main 0 printed copy of the seed
    HTTP 500 this PR 1 not printed not written
    HTTP 200, empty completion main 0 printed copy of the seed
    HTTP 200, empty completion this PR 1 not printed not written
  • black 25.12.0 (pre-commit version) clean on all changed files. isort 7.0.0 is clean on all changed files except tests/test_checkpoint_resume.py, whose import json / import time ordering is already flagged on main; this PR does not touch its imports.

SUPR1M3 and others added 3 commits October 24, 2026 04:36
…idates

dhara-250m never emits a well-formed SEARCH/REPLACE block, so in diff
mode every integration iteration ended with "No valid diffs found". The
tests passed only because such runs finished like successful ones; with
EvolutionFailedError (algorithmicsuperintelligence#288) all 17 real-LLM integration tests failed.

Switch the shared integration config and the library API test config to
full-rewrite mode, whose parser falls back to the raw response, so every
iteration produces a candidate and the tests exercise the evaluate, add,
migrate and checkpoint paths that diff mode never reached.

That exposed a stale assertion in test_checkpoint_resume_functionality:
checkpoints contain metadata.json and programs/, not database.json. The
test now checks that layout and actually resumes from the checkpoint,
verifying the saved programs load and the run continues past it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_evolve_code_real_integration ran a single iteration, and dhara-250m
occasionally returns an empty reply ("No valid code found in response"),
which leaves the run with no candidate and now raises EvolutionFailedError.
Run each library API evolution for three iterations so one empty reply
does not fail the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codelion
codelion merged commit 6f8c953 into algorithmicsuperintelligence:main Oct 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openevolve does not correctly clean up when there are failures that lead to retries

3 participants