diff --git a/examples/alphaevolve_interop/README.md b/examples/alphaevolve_interop/README.md index 414779106..46fb3475e 100644 --- a/examples/alphaevolve_interop/README.md +++ b/examples/alphaevolve_interop/README.md @@ -1,12 +1,14 @@ -# AlphaEvolve evaluator interoperability +# AlphaEvolve interoperability -This is the evaluator-contract slice of [#468](https://github.com/algorithmicsuperintelligence/openevolve/issues/468). +This walkthrough covers opt-in evaluator and configuration slices of +[#468](https://github.com/algorithmicsuperintelligence/openevolve/issues/468). It runs the **unchanged** `program.py` and `evaluate.py` from Google's [AlphaEvolve examples](https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud) through an explicit adapter. A reverse adapter also lets a synchronous OpenEvolve evaluator score AlphaEvolve candidates. Separate opt-in helpers import a limited -subset of experiment configuration for local Python runs. These helpers do not -replace the cloud SDK, export bundles, or implement all of #468. +subset of experiment configuration for local Python runs and export a limited +configuration subset in the other direction. These helpers do not replace the +cloud SDK, export bundles, or implement all of #468. ## Evaluate an upstream seed locally @@ -140,6 +142,72 @@ text, images, and `alphaevolve_insights` artifacts are not round-tripped. This i only the reverse evaluator-contract slice of #468, not a cloud export bundle, SDK shim, or an assertion that a managed-service experiment has been run. +## Export an OpenEvolve experiment configuration + +`config_to_alphaevolve` returns a plain experiment dictionary for the upstream +SDK, without creating files, loading templates, importing Google dependencies, +looking up credentials, or making requests: + +```python +from openevolve.compat.alphaevolve_export import config_to_alphaevolve +from openevolve.config import Config + +config = Config.from_dict({ + "language": "python", + "max_iterations": 20, + "llm": {"models": [{"name": "my-local-model", "weight": 1.0}]}, + "prompt": {"system_message": "Pack circles into a unit square."}, +}) +experiment = config_to_alphaevolve( + config, + title="Circle Packing", + # Pass literal problem text, not the name of a prompt template. + problem_description=config.prompt.system_message, + model_mapping={"my-local-model": "gemini-2.5-flash"}, + generation_concurrency=2, +) +# UserWarning: OpenEvolve settings not exported (including defaults): ... +# UserWarning: Model names explicitly remapped: llm.models[0].name +``` + +The result can be checked offline with the upstream SDK's +`AlphaEvolveExperimentConfig.model_validate(experiment)`. Supplying it to a real +SDK experiment's `create_experiment` is a separate, authenticated cloud operation +that may incur charges; this helper does not perform that operation. + +| OpenEvolve / explicit input | Exported setting | +| --- | --- | +| `config.max_iterations` | `run_settings.max_programs` | +| `config.language="python"` | `program_language="python"` | +| Resolved `llm.models` names and relative weights | `generation_settings.models` | +| Required literal `title` and `problem_description` arguments | Top-level text fields | +| Optional `generation_concurrency` argument | `run_settings.concurrency` | + +Only single-file Python interoperability is covered. Non-Gemini models require +an explicit name mapping; no fallback model is selected. Duplicate target names +are merged by summing weights, with at most two distinct targets. Weights must be +finite and positive. Names are checked for Gemini form, not against a live model +catalog. Service availability, project permissions and model-tier mixture limits +(such as caps on some Pro models) remain the caller/service's responsibility. + +The generation budget counts attempts, not successful evaluations, API requests, +tokens, elapsed time, or money; seed evaluation is separate. Evaluation worker +counts are **not** copied into generation concurrency. Configure the upstream +experiment's local evaluation workers and `max_programs_evaluated` separately. +Without `generation_concurrency`, the service's concurrency setting is omitted. + +Export always warns by field path about omitted settings, including defaults: +MAP-Elites/islands/migration, local evaluator settings and timeouts, LLM feedback, +prompt construction, per-island prompts, local endpoints/keys, sampling/retry +controls, hooks, checkpointing, tracing and early stopping. Warning text contains +field names, not values or arbitrary dictionary keys. Prompt templates are not +resolved: explicitly supply the literal problem text you intend to use. + +The input config and mapping are not modified. Export followed by +`config_from_alphaevolve` preserves the supported budget/language/model subset; +the importer prefixes literal problem text and requires a new local endpoint. +This does not preserve search behavior or provide a runnable cloud export bundle. + ## Import a local Python experiment configuration `config_from_alphaevolve` imports an explicit experiment dictionary without @@ -321,6 +389,7 @@ python -m unittest discover -s tests -p test_alphaevolve_config.py python -m unittest discover -s tests -p test_alphaevolve_env.py python -m unittest discover -s tests -p test_alphaevolve_runner.py python -m unittest discover -s tests -p test_alphaevolve_reverse.py +python -m unittest discover -s tests -p test_alphaevolve_export.py ``` The walkthrough additionally permits direct validation against an upstream diff --git a/openevolve/compat/alphaevolve_export.py b/openevolve/compat/alphaevolve_export.py new file mode 100644 index 000000000..40144143a --- /dev/null +++ b/openevolve/compat/alphaevolve_export.py @@ -0,0 +1,170 @@ +"""Explicit, lossy configuration export for the AlphaEvolve experiment contract.""" + +import math +import re +import warnings +from collections.abc import Mapping +from dataclasses import fields +from numbers import Real +from typing import Optional + +from openevolve.config import Config, LLMConfig, LLMModelConfig + + +def _positive_integer(value, path: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + raise ValueError(f"{path} must be a positive integer") + return value + + +def _gemini_name(value, path: str) -> str: + if not isinstance(value, str) or not re.fullmatch(r"gemini-[A-Za-z0-9][A-Za-z0-9._-]*", value): + raise ValueError(f"{path} must explicitly name a Gemini model") + return value + + +def _warn_omitted_settings(config: Config) -> None: + # Inspect fixed dataclass field names, never values or keys from arbitrary + # dictionaries (e.g. evaluator_args). Do not serialize or deepcopy clients. + omitted = [] + for field in fields(Config): + if field.name in {"max_iterations", "language", "llm"}: + continue + value = getattr(config, field.name) + if field.name in {"prompt", "database", "evaluator", "evolution_trace"}: + omitted.extend( + f"{field.name}.{child.name}" + for child in fields(value) + if getattr(value, child.name) is not None + ) + elif value is not None: + omitted.append(field.name) + for field in fields(LLMConfig): + # Legacy model aliases have already been expanded into llm.models. + if field.name in { + "models", + "primary_model", + "primary_model_weight", + "secondary_model", + "secondary_model_weight", + }: + continue + if getattr(config.llm, field.name) is not None: + omitted.append(f"llm.{field.name}") + for index, model in enumerate(config.llm.models): + omitted.extend( + f"llm.models[{index}].{field.name}" + for field in fields(LLMModelConfig) + if field.name not in {"name", "weight"} and getattr(model, field.name) is not None + ) + warnings.warn( + "OpenEvolve settings not exported (including defaults): " + ", ".join(omitted), + UserWarning, + stacklevel=3, + ) + + +def config_to_alphaevolve( + config: Config, + *, + title: str, + problem_description: str, + model_mapping: Optional[Mapping[str, str]] = None, + generation_concurrency: Optional[int] = None, +) -> dict: + """Export a Python experiment dictionary without files, credentials or requests. + + Copy max_iterations as the generation-attempt budget, language='python', and + model names/relative weights. Require literal title/problem_description text; + OpenEvolve prompt template names and per-island prompts are not resolved. + + Non-Gemini models require explicit model_mapping entries; no model is chosen + automatically. Merge duplicate target names by summing weights and allow at + most two distinct targets. Service availability, permissions and model-tier + mixture restrictions still require validation by the caller/service. + + generation_concurrency, if supplied, sets the service's generation parallelism. + OpenEvolve evaluation worker counts have a different meaning and are omitted. + The generation budget is not an evaluation, token, time or monetary limit. + + Warn by field path for omitted settings, including defaults, without printing + values. Local endpoints/keys, evaluator configuration, MAP-Elites, stopping, + LLM sampling/runtime controls and prompt construction are not transferred. + The input is not mutated. This is not a runnable bundle or a cloud client. + """ + if not isinstance(config, Config) or not isinstance(config.llm, LLMConfig): + raise TypeError("config must be an OpenEvolve Config with an LLMConfig") + for path, value in (("title", title), ("problem_description", problem_description)): + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{path} must be non-empty literal text") + if config.language != "python": + raise ValueError("Set config.language='python' explicitly for this export") + run_settings = {"max_programs": _positive_integer(config.max_iterations, "max_iterations")} + if generation_concurrency is not None: + run_settings["concurrency"] = _positive_integer( + generation_concurrency, "generation_concurrency" + ) + + if model_mapping is None: + model_mapping = {} + if not isinstance(model_mapping, Mapping): + raise TypeError("model_mapping must be a mapping") + for source, target in model_mapping.items(): + if not isinstance(source, str) or not source.strip(): + raise ValueError("model_mapping keys must be non-empty model names") + _gemini_name(target, "model_mapping target") + if not isinstance(config.llm.models, (list, tuple)) or not config.llm.models: + raise ValueError("llm.models must be a non-empty list") + + weights_by_name = {} + source_names = set() + remapped = [] + for index, model in enumerate(config.llm.models): + path = f"llm.models[{index}]" + if not isinstance(model, LLMModelConfig): + raise TypeError(f"{path} must be an LLMModelConfig") + if not isinstance(model.name, str) or not model.name.strip(): + raise ValueError(f"{path}.name must be a non-empty model name") + source_names.add(model.name) + name = _gemini_name(model_mapping.get(model.name, model.name), f"{path}.name") + weight = model.weight + if isinstance(weight, bool) or not isinstance(weight, Real): + raise ValueError(f"{path}.weight must be finite and positive") + try: + weight = float(weight) + except OverflowError: + raise ValueError(f"{path}.weight must be finite and positive") from None + if not math.isfinite(weight) or weight <= 0: + raise ValueError(f"{path}.weight must be finite and positive") + weights_by_name.setdefault(name, []).append(weight) + if name != model.name: + remapped.append(f"{path}.name") + if set(model_mapping) - source_names: + raise ValueError("model_mapping contains names absent from llm.models") + if len(weights_by_name) > 2: + raise ValueError("AlphaEvolve supports at most two distinct generation models") + try: + models = [ + {"name": name, "weight": math.fsum(weights)} + for name, weights in weights_by_name.items() + ] + total_weight = math.fsum(model["weight"] for model in models) + except OverflowError: + raise ValueError("model weights must have a finite sum") from None + if not math.isfinite(total_weight): + raise ValueError("model weights must have a finite sum") + + _warn_omitted_settings(config) + if remapped: + warnings.warn( + "Model names explicitly remapped: " + ", ".join(remapped), + UserWarning, + stacklevel=2, + ) + return { + "title": title, + "problem_description": problem_description, + "program_language": "python", + "run_settings": run_settings, + "generation_settings": {"models": models}, + } diff --git a/tests/test_alphaevolve_export.py b/tests/test_alphaevolve_export.py new file mode 100644 index 000000000..571e761c1 --- /dev/null +++ b/tests/test_alphaevolve_export.py @@ -0,0 +1,295 @@ +"""Offline tests for the explicit OpenEvolve-to-AlphaEvolve config subset.""" + +import copy +import json +import secrets +import unittest +import warnings +from unittest.mock import patch + +from openevolve.compat.alphaevolve import config_from_alphaevolve +from openevolve.compat.alphaevolve_export import config_to_alphaevolve +from openevolve.config import Config, LLMModelConfig + + +def config(): + return Config.from_dict( + { + "language": "python", + "max_iterations": 12, + "llm": { + "models": [ + {"name": "gemini-2.5-flash", "weight": 7}, + {"name": "gemini-3.1-pro-preview", "weight": 3}, + ] + }, + "prompt": {"system_message": "Pack circles into a unit square. 中文。"}, + } + ) + + +class TestAlphaEvolveConfigExport(unittest.TestCase): + def convert(self, supplied=None, **options): + if supplied is None: + supplied = config() + settings = { + "title": "Circle Packing", + "problem_description": supplied.prompt.system_message, + **options, + } + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + return config_to_alphaevolve(supplied, **settings) + + def test_supported_subset_is_json_serializable_and_does_not_mutate_config(self): + supplied = config() + original = copy.deepcopy(supplied) + result = self.convert(supplied) + self.assertEqual( + result, + { + "title": "Circle Packing", + "problem_description": supplied.prompt.system_message, + "program_language": "python", + "run_settings": {"max_programs": 12}, + "generation_settings": { + "models": [ + {"name": "gemini-2.5-flash", "weight": 7.0}, + {"name": "gemini-3.1-pro-preview", "weight": 3.0}, + ] + }, + }, + ) + self.assertEqual(supplied, original) + json.dumps(result, ensure_ascii=False, allow_nan=False) + + def test_non_gemini_model_requires_an_explicit_mapping(self): + supplied = config() + supplied.llm.models[0].name = "local-private-model" + with self.assertRaisesRegex(ValueError, r"llm.models\[0\].name") as caught: + self.convert(supplied) + self.assertNotIn("local-private-model", str(caught.exception)) + result = self.convert(supplied, model_mapping={"local-private-model": "gemini-2.5-flash"}) + self.assertEqual(result["generation_settings"]["models"][0]["name"], "gemini-2.5-flash") + self.assertEqual(supplied.llm.models[0].name, "local-private-model") + + def test_remapping_warns_by_path_without_disclosing_source_names(self): + supplied = config() + supplied.llm.models[0].name = "private-model" + mapping = {"private-model": "gemini-2.5-flash"} + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + config_to_alphaevolve( + supplied, title="Example", problem_description="Problem", model_mapping=mapping + ) + messages = "\n".join(str(w.message) for w in caught) + self.assertIn("Model names explicitly remapped: llm.models[0].name", messages) + self.assertNotIn("private-model", messages) + self.assertEqual(mapping, {"private-model": "gemini-2.5-flash"}) + + def test_duplicate_targets_are_merged_by_summing_relative_weights(self): + supplied = config() + supplied.llm.models.append(LLMModelConfig(name="local", weight=2)) + result = self.convert(supplied, model_mapping={"local": "gemini-2.5-flash"}) + self.assertEqual( + result["generation_settings"]["models"], + [ + {"name": "gemini-2.5-flash", "weight": 9.0}, + {"name": "gemini-3.1-pro-preview", "weight": 3.0}, + ], + ) + self.assertEqual(len(supplied.llm.models), 3) + + def test_more_than_two_distinct_targets_are_rejected(self): + supplied = config() + supplied.llm.models.append(LLMModelConfig(name="gemini-third", weight=1)) + with self.assertRaisesRegex(ValueError, "at most two"): + self.convert(supplied) + + def test_invalid_model_names_are_rejected(self): + for name in (None, "", " ", "Gemini-test", "gemini-", "gemini-name with space"): + supplied = config() + supplied.llm.models[0].name = name + with self.subTest(name=name), self.assertRaises(ValueError): + self.convert(supplied) + + def test_invalid_weights_are_rejected(self): + for weight in (None, True, "1", 0, -1, float("nan"), float("inf"), 10**1000): + supplied = config() + supplied.llm.models[0].weight = weight + with self.subTest(weight=weight), self.assertRaisesRegex(ValueError, "weight"): + self.convert(supplied) + + def test_overflowing_mixture_or_merged_weight_is_rejected(self): + for same_target in (False, True): + supplied = config() + for model in supplied.llm.models: + model.weight = 1e308 + if same_target: + supplied.llm.models[1].name = supplied.llm.models[0].name + with ( + self.subTest(same_target=same_target), + self.assertRaisesRegex(ValueError, "finite sum"), + ): + self.convert(supplied) + + def test_generation_budget_must_be_an_explicit_positive_integer(self): + for budget in (None, True, 0, -1, 1.5, "12"): + supplied = config() + supplied.max_iterations = budget + with self.subTest(budget=budget), self.assertRaisesRegex(ValueError, "max_iterations"): + self.convert(supplied) + + def test_generation_concurrency_is_explicit_and_not_copied_from_workers(self): + supplied = config() + supplied.evaluator.parallel_evaluations = 8 + self.assertNotIn("concurrency", self.convert(supplied)["run_settings"]) + self.assertEqual( + self.convert(supplied, generation_concurrency=3)["run_settings"], + {"max_programs": 12, "concurrency": 3}, + ) + self.assertEqual(supplied.evaluator.parallel_evaluations, 8) + for concurrency in (True, 0, -1, "3", 1.5): + with ( + self.subTest(concurrency=concurrency), + self.assertRaisesRegex(ValueError, "generation_concurrency"), + ): + self.convert(supplied, generation_concurrency=concurrency) + + def test_title_and_problem_are_required_nonempty_literal_text(self): + for field in ("title", "problem_description"): + for value in (None, "", " ", 12): + with ( + self.subTest(field=field, value=value), + self.assertRaisesRegex(ValueError, field), + ): + self.convert(**{field: value}) + result = self.convert(title="标题", problem_description="system_message") + self.assertEqual(result["title"], "标题") + self.assertEqual(result["problem_description"], "system_message") + + def test_language_must_explicitly_be_python(self): + for language in (None, "cpp", "Python", ""): + supplied = config() + supplied.language = language + with self.subTest(language=language), self.assertRaisesRegex(ValueError, "language"): + self.convert(supplied) + + def test_invalid_mapping_contracts_are_rejected(self): + for mapping in ([], "gemini-name", {None: "gemini-name"}, {"": "gemini-name"}): + with self.subTest(mapping=mapping), self.assertRaises((TypeError, ValueError)): + self.convert(model_mapping=mapping) + for target in (None, "", "local-name", "gemini-name with space"): + with self.subTest(target=target), self.assertRaisesRegex(ValueError, "target"): + self.convert(model_mapping={"gemini-2.5-flash": target}) + with self.assertRaisesRegex(ValueError, "absent"): + self.convert(model_mapping={"typo": "gemini-2.5-flash"}) + + def test_invalid_config_or_model_collection_is_rejected(self): + for supplied in (None, {}, "config"): + with self.subTest(config=supplied), self.assertRaises(TypeError): + config_to_alphaevolve(supplied, title="Example", problem_description="Problem") + for models in (None, [], {}, "models", [None], [{"name": "gemini-test"}]): + supplied = config() + supplied.llm.models = models + with self.subTest(models=models), self.assertRaises((TypeError, ValueError)): + self.convert(supplied) + + def test_legacy_model_aliases_use_the_resolved_ensemble(self): + supplied = Config.from_dict( + { + "language": "python", + "llm": { + "primary_model": "gemini-2.5-flash", + "primary_model_weight": 4, + "secondary_model": "gemini-3.1-pro-preview", + "secondary_model_weight": 1, + }, + } + ) + self.assertEqual( + [entry["weight"] for entry in self.convert(supplied)["generation_settings"]["models"]], + [4.0, 1.0], + ) + + def test_omission_warning_covers_defaults_and_custom_local_controls_without_values(self): + supplied = config() + # A generated sentinel is not an account credential or a hardcoded key. + sentinel = "private-" + secrets.token_urlsafe(24) + supplied.llm.api_key = sentinel + supplied.llm.models[0].api_key = sentinel + supplied.llm.models[0].api_base = "https://private-endpoint.example" + supplied.llm.models[0].max_budget_usd = 17 + supplied.prompt.template_dir = "private-template-directory" + supplied.evaluator.evaluator_args = {"private-argument-name": "private-argument-value"} + supplied.database.db_path = "private-database-directory" + supplied.early_stopping_patience = 5 + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + result = config_to_alphaevolve(supplied, title="Example", problem_description="Problem") + messages = "\n".join(str(w.message) for w in caught) + for path in ( + "database.num_islands", + "database.feature_dimensions", + "database.migration_interval", + "evaluator.parallel_evaluations", + "evaluator.timeout", + "evaluator.evaluator_args", + "prompt.template_dir", + "llm.evaluator_models", + "llm.models[0].api_key", + "llm.models[0].api_base", + "llm.models[0].max_budget_usd", + "early_stopping_patience", + ): + self.assertIn(path, messages) + self.assertIn("including defaults", messages) + self.assertNotIn(sentinel, messages + json.dumps(result)) + self.assertNotIn("private-", messages + json.dumps(result)) + + def test_export_does_not_load_templates_copy_clients_write_files_or_access_network(self): + class LocalClient: + def __call__(self, *args, **kwargs): + raise AssertionError("client must not be called") + + def __deepcopy__(self, memo): + raise AssertionError("client must not be copied") + + supplied = config() + supplied.prompt.template_dir = "/nonexistent-template-directory" + supplied.llm.models[0].init_client = LocalClient() + supplied.database.novelty_llm = LocalClient() + with ( + patch("builtins.open", side_effect=AssertionError("file access forbidden")), + patch("socket.socket.connect", side_effect=AssertionError("network forbidden")), + ): + result = self.convert(supplied, problem_description="Literal problem text") + self.assertEqual(result["problem_description"], "Literal problem text") + + def test_output_is_independent_of_the_input_and_other_exports(self): + supplied = config() + first, second = self.convert(supplied), self.convert(supplied) + first["generation_settings"]["models"][0]["weight"] = 100 + first["run_settings"]["max_programs"] = 1 + self.assertEqual(second["generation_settings"]["models"][0]["weight"], 7) + self.assertEqual(supplied.llm.models[0].weight, 7) + self.assertEqual(supplied.max_iterations, 12) + + def test_supported_fields_round_trip_through_the_existing_importer(self): + supplied = config() + data = self.convert(supplied) + imported = config_from_alphaevolve(data, api_base="http://localhost:8080/v1") + self.assertEqual(imported.max_iterations, supplied.max_iterations) + self.assertEqual(imported.language, supplied.language) + self.assertEqual( + [(model.name, model.weight) for model in imported.llm.models], + [(model.name, model.weight) for model in supplied.llm.models], + ) + self.assertEqual( + imported.prompt.system_message, "Problem description:\n" + data["problem_description"] + ) + self.assertEqual(imported.llm.models[0].api_base, "http://localhost:8080/v1") + + +if __name__ == "__main__": + unittest.main()