Repository navigation
Conversation
…idates dhara-250m never emits a well-formed SEARCH/REPLACE block, so in diff mode every integration iteration ended with "No valid diffs found". The tests passed only because such runs finished like successful ones; with EvolutionFailedError (algorithmicsuperintelligence#288) all 17 real-LLM integration tests failed. Switch the shared integration config and the library API test config to full-rewrite mode, whose parser falls back to the raw response, so every iteration produces a candidate and the tests exercise the evaluate, add, migrate and checkpoint paths that diff mode never reached. That exposed a stale assertion in test_checkpoint_resume_functionality: checkpoints contain metadata.json and programs/, not database.json. The test now checks that layout and actually resumes from the checkpoint, verifying the saved programs load and the run continues past it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_evolve_code_real_integration ran a single iteration, and dhara-250m
occasionally returns an empty reply ("No valid code found in response"),
which leaves the run with no candidate and now raises EvolutionFailedError.
Run each library API evolution for three iterations so one empty reply
does not fail the test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #288
Problem:
When every iteration fails before producing a candidate program (LLM endpoint down, deployment missing, model returning empty replies),
ProcessParallelController.run_evolutionlogs each error as a warning and moves on. Nothing tracks whether any iteration succeeded, so the run ends exactly like a healthy one:Evolution complete!with the seed's metrics and exits with code 0best/best_program.pyis written as a byte-identical copy ofinitial_program.pyScripted pipelines check the exit code and the final banner, so a run that generated nothing looks like a successful run that simply found nothing better than the seed.
What:
ProcessParallelController.run_evolutioncounts iterations whose worker produced a candidate program and records the last iteration error (worker error, timeout, or result-processing exception). At the end of the run it setsproduced_no_candidateswhen iterations completed but none produced a candidate, and logs an error. The flag is computed insiderun_evolutionrather than as a property, becausestop()sets the shutdown event during normal cleanup.OpenEvolve.runraises a newEvolutionFailedError(exported fromopenevolve) in that case, before the best program is saved, so the seed is never presented as an evolved result. The message includes the count and the last error, e.g.Evolution failed: 0/10 iterations produced a candidate program. Last error: LLM generation failed: Error code: 404 - ....cli.pycatchesEvolutionFailedError, prints the message without a traceback and returns exit code 1.tests/test_all_iterations_failed.py(6 tests): controller counting (all errors, one success among errors, shutdown),OpenEvolve.runraising and not writingbest/, and an end-to-end CLI run against a local HTTP server that answers every LLM request with 404 (the reproduction from the issue).ProcessParallelControllerwith a bareMock/MagicMocknow setproduced_no_candidates = False, the same way they already setshutdown_event(test_checkpoint_resume.py,test_initial_program_artifacts.py,test_island_system_messages.py)."Candidate produced" is used as the success signal rather than "LLM call succeeded", so a run where the model answers but never returns usable output (e.g.
No valid diffs found in responseon every iteration) also fails. Nothing was evolved in that case either.Red -> green evidence:
On
main, the new tests fail:With this PR:
Ran 6 tests ... OK.Testing:
Unit tests: 887 pass on Python 3.14 (881 on
mainplus the 6 new tests); no other per-test differences frommainEnd to end with the issue's reproduction:
examples/function_minimizationwith onlyapi_basepointed at a local stub, 10 iterations, default retries, 3 workers:Evolution complete!best/mainmainmainblack 25.12.0 (pre-commit version) clean on all changed files. isort 7.0.0 is clean on all changed files except
tests/test_checkpoint_resume.py, whoseimport json/import timeordering is already flagged onmain; this PR does not touch its imports.