Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 73 additions & 4 deletions examples/alphaevolve_interop/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# AlphaEvolve evaluator interoperability
# AlphaEvolve interoperability

This is the evaluator-contract slice of [#468](https://github.com/algorithmicsuperintelligence/openevolve/issues/468).
This walkthrough covers opt-in evaluator and configuration slices of
[#468](https://github.com/algorithmicsuperintelligence/openevolve/issues/468).
It runs the **unchanged** `program.py` and `evaluate.py` from Google's
[AlphaEvolve examples](https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud)
through an explicit adapter. A reverse adapter also lets a synchronous OpenEvolve
evaluator score AlphaEvolve candidates. Separate opt-in helpers import a limited
subset of experiment configuration for local Python runs. These helpers do not
replace the cloud SDK, export bundles, or implement all of #468.
subset of experiment configuration for local Python runs and export a limited
configuration subset in the other direction. These helpers do not replace the
cloud SDK, export bundles, or implement all of #468.

## Evaluate an upstream seed locally

Expand Down Expand Up @@ -140,6 +142,72 @@ text, images, and `alphaevolve_insights` artifacts are not round-tripped. This i
only the reverse evaluator-contract slice of #468, not a cloud export bundle,
SDK shim, or an assertion that a managed-service experiment has been run.

## Export an OpenEvolve experiment configuration

`config_to_alphaevolve` returns a plain experiment dictionary for the upstream
SDK, without creating files, loading templates, importing Google dependencies,
looking up credentials, or making requests:

```python
from openevolve.compat.alphaevolve_export import config_to_alphaevolve
from openevolve.config import Config

config = Config.from_dict({
"language": "python",
"max_iterations": 20,
"llm": {"models": [{"name": "my-local-model", "weight": 1.0}]},
"prompt": {"system_message": "Pack circles into a unit square."},
})
experiment = config_to_alphaevolve(
config,
title="Circle Packing",
# Pass literal problem text, not the name of a prompt template.
problem_description=config.prompt.system_message,
model_mapping={"my-local-model": "gemini-2.5-flash"},
generation_concurrency=2,
)
# UserWarning: OpenEvolve settings not exported (including defaults): ...
# UserWarning: Model names explicitly remapped: llm.models[0].name
```

The result can be checked offline with the upstream SDK's
`AlphaEvolveExperimentConfig.model_validate(experiment)`. Supplying it to a real
SDK experiment's `create_experiment` is a separate, authenticated cloud operation
that may incur charges; this helper does not perform that operation.

| OpenEvolve / explicit input | Exported setting |
| --- | --- |
| `config.max_iterations` | `run_settings.max_programs` |
| `config.language="python"` | `program_language="python"` |
| Resolved `llm.models` names and relative weights | `generation_settings.models` |
| Required literal `title` and `problem_description` arguments | Top-level text fields |
| Optional `generation_concurrency` argument | `run_settings.concurrency` |

Only single-file Python interoperability is covered. Non-Gemini models require
an explicit name mapping; no fallback model is selected. Duplicate target names
are merged by summing weights, with at most two distinct targets. Weights must be
finite and positive. Names are checked for Gemini form, not against a live model
catalog. Service availability, project permissions and model-tier mixture limits
(such as caps on some Pro models) remain the caller/service's responsibility.

The generation budget counts attempts, not successful evaluations, API requests,
tokens, elapsed time, or money; seed evaluation is separate. Evaluation worker
counts are **not** copied into generation concurrency. Configure the upstream
experiment's local evaluation workers and `max_programs_evaluated` separately.
Without `generation_concurrency`, the service's concurrency setting is omitted.

Export always warns by field path about omitted settings, including defaults:
MAP-Elites/islands/migration, local evaluator settings and timeouts, LLM feedback,
prompt construction, per-island prompts, local endpoints/keys, sampling/retry
controls, hooks, checkpointing, tracing and early stopping. Warning text contains
field names, not values or arbitrary dictionary keys. Prompt templates are not
resolved: explicitly supply the literal problem text you intend to use.

The input config and mapping are not modified. Export followed by
`config_from_alphaevolve` preserves the supported budget/language/model subset;
the importer prefixes literal problem text and requires a new local endpoint.
This does not preserve search behavior or provide a runnable cloud export bundle.

## Import a local Python experiment configuration

`config_from_alphaevolve` imports an explicit experiment dictionary without
Expand Down Expand Up @@ -321,6 +389,7 @@ python -m unittest discover -s tests -p test_alphaevolve_config.py
python -m unittest discover -s tests -p test_alphaevolve_env.py
python -m unittest discover -s tests -p test_alphaevolve_runner.py
python -m unittest discover -s tests -p test_alphaevolve_reverse.py
python -m unittest discover -s tests -p test_alphaevolve_export.py
```

The walkthrough additionally permits direct validation against an upstream
Expand Down
170 changes: 170 additions & 0 deletions openevolve/compat/alphaevolve_export.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
"""Explicit, lossy configuration export for the AlphaEvolve experiment contract."""

import math
import re
import warnings
from collections.abc import Mapping
from dataclasses import fields
from numbers import Real
from typing import Optional

from openevolve.config import Config, LLMConfig, LLMModelConfig


def _positive_integer(value, path: str) -> int:
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
raise ValueError(f"{path} must be a positive integer")
return value


def _gemini_name(value, path: str) -> str:
if not isinstance(value, str) or not re.fullmatch(r"gemini-[A-Za-z0-9][A-Za-z0-9._-]*", value):
raise ValueError(f"{path} must explicitly name a Gemini model")
return value


def _warn_omitted_settings(config: Config) -> None:
# Inspect fixed dataclass field names, never values or keys from arbitrary
# dictionaries (e.g. evaluator_args). Do not serialize or deepcopy clients.
omitted = []
for field in fields(Config):
if field.name in {"max_iterations", "language", "llm"}:
continue
value = getattr(config, field.name)
if field.name in {"prompt", "database", "evaluator", "evolution_trace"}:
omitted.extend(
f"{field.name}.{child.name}"
for child in fields(value)
if getattr(value, child.name) is not None
)
elif value is not None:
omitted.append(field.name)
for field in fields(LLMConfig):
# Legacy model aliases have already been expanded into llm.models.
if field.name in {
"models",
"primary_model",
"primary_model_weight",
"secondary_model",
"secondary_model_weight",
}:
continue
if getattr(config.llm, field.name) is not None:
omitted.append(f"llm.{field.name}")
for index, model in enumerate(config.llm.models):
omitted.extend(
f"llm.models[{index}].{field.name}"
for field in fields(LLMModelConfig)
if field.name not in {"name", "weight"} and getattr(model, field.name) is not None
)
warnings.warn(
"OpenEvolve settings not exported (including defaults): " + ", ".join(omitted),
UserWarning,
stacklevel=3,
)


def config_to_alphaevolve(
config: Config,
*,
title: str,
problem_description: str,
model_mapping: Optional[Mapping[str, str]] = None,
generation_concurrency: Optional[int] = None,
) -> dict:
"""Export a Python experiment dictionary without files, credentials or requests.

Copy max_iterations as the generation-attempt budget, language='python', and
model names/relative weights. Require literal title/problem_description text;
OpenEvolve prompt template names and per-island prompts are not resolved.

Non-Gemini models require explicit model_mapping entries; no model is chosen
automatically. Merge duplicate target names by summing weights and allow at
most two distinct targets. Service availability, permissions and model-tier
mixture restrictions still require validation by the caller/service.

generation_concurrency, if supplied, sets the service's generation parallelism.
OpenEvolve evaluation worker counts have a different meaning and are omitted.
The generation budget is not an evaluation, token, time or monetary limit.

Warn by field path for omitted settings, including defaults, without printing
values. Local endpoints/keys, evaluator configuration, MAP-Elites, stopping,
LLM sampling/runtime controls and prompt construction are not transferred.
The input is not mutated. This is not a runnable bundle or a cloud client.
"""
if not isinstance(config, Config) or not isinstance(config.llm, LLMConfig):
raise TypeError("config must be an OpenEvolve Config with an LLMConfig")
for path, value in (("title", title), ("problem_description", problem_description)):
if not isinstance(value, str) or not value.strip():
raise ValueError(f"{path} must be non-empty literal text")
if config.language != "python":
raise ValueError("Set config.language='python' explicitly for this export")
run_settings = {"max_programs": _positive_integer(config.max_iterations, "max_iterations")}
if generation_concurrency is not None:
run_settings["concurrency"] = _positive_integer(
generation_concurrency, "generation_concurrency"
)

if model_mapping is None:
model_mapping = {}
if not isinstance(model_mapping, Mapping):
raise TypeError("model_mapping must be a mapping")
for source, target in model_mapping.items():
if not isinstance(source, str) or not source.strip():
raise ValueError("model_mapping keys must be non-empty model names")
_gemini_name(target, "model_mapping target")
if not isinstance(config.llm.models, (list, tuple)) or not config.llm.models:
raise ValueError("llm.models must be a non-empty list")

weights_by_name = {}
source_names = set()
remapped = []
for index, model in enumerate(config.llm.models):
path = f"llm.models[{index}]"
if not isinstance(model, LLMModelConfig):
raise TypeError(f"{path} must be an LLMModelConfig")
if not isinstance(model.name, str) or not model.name.strip():
raise ValueError(f"{path}.name must be a non-empty model name")
source_names.add(model.name)
name = _gemini_name(model_mapping.get(model.name, model.name), f"{path}.name")
weight = model.weight
if isinstance(weight, bool) or not isinstance(weight, Real):
raise ValueError(f"{path}.weight must be finite and positive")
try:
weight = float(weight)
except OverflowError:
raise ValueError(f"{path}.weight must be finite and positive") from None
if not math.isfinite(weight) or weight <= 0:
raise ValueError(f"{path}.weight must be finite and positive")
weights_by_name.setdefault(name, []).append(weight)
if name != model.name:
remapped.append(f"{path}.name")
if set(model_mapping) - source_names:
raise ValueError("model_mapping contains names absent from llm.models")
if len(weights_by_name) > 2:
raise ValueError("AlphaEvolve supports at most two distinct generation models")
try:
models = [
{"name": name, "weight": math.fsum(weights)}
for name, weights in weights_by_name.items()
]
total_weight = math.fsum(model["weight"] for model in models)
except OverflowError:
raise ValueError("model weights must have a finite sum") from None
if not math.isfinite(total_weight):
raise ValueError("model weights must have a finite sum")

_warn_omitted_settings(config)
if remapped:
warnings.warn(
"Model names explicitly remapped: " + ", ".join(remapped),
UserWarning,
stacklevel=2,
)
return {
"title": title,
"problem_description": problem_description,
"program_language": "python",
"run_settings": run_settings,
"generation_settings": {"models": models},
}
Loading
Loading