Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 56 additions & 7 deletions examples/alphaevolve_interop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,9 @@ python -m openevolve.compat.alphaevolve_bundle \

# Review/install the evaluator's dependencies in your environment first.
python ./alphaevolve-bundle/check.py

# Offline preflight; does not import the SDK/evaluator or execute either source.
python ./alphaevolve-bundle/run_alphaevolve.py
```

The destination must not exist, and its parent must already exist. Exporting
Expand All @@ -251,6 +254,7 @@ alphaevolve-bundle/
openevolve_evaluator.py # Original evaluator bytes
evaluate.py # Reverse adapter entry point
check.py
run_alphaevolve.py # Offline preflight by default; explicit --run for cloud use
README.md
requirements.txt # Only when explicitly supplied
```
Expand All @@ -275,19 +279,64 @@ evaluation to stdout and redirects Python-level evaluator/import stdout to
stderr. Evaluators must capture their own subprocess output to keep stdout
clean. The check also works by absolute path from another directory.
Import/contract errors and adapter-reported failures produce a nonzero exit
status. The check executes
trusted code with the current process's privileges; it adds no sandbox or
timeout and cannot recognize failures silently converted into numeric scores.
status. The check executes trusted code with the current process's privileges;
it adds no sandbox or timeout and cannot recognize failures silently converted
into numeric scores.

### Preflight and opt-in cloud launch

`run_alphaevolve.py` defaults to a basic offline JSON summary of the bundle,
models and generation budget. It checks the exported configuration subset,
evaluation options and Python sources without importing the SDK or evaluator,
executing a seed, looking up credentials or making requests. `--dry-run` is an
explicit alias for this default. Dependency installation, full SDK schemas,
permissions and live model availability are not checked in this mode.

For cloud use, install the official SDK from a reviewed checkout of
[Google's repository](https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud)
and follow its README for ADC, project, engine and service access setup. Review
the generation budget and select currently available models explicitly during
export; the Gemini-name check is not a live model catalog. Then opt in:

```bash
python ./alphaevolve-bundle/run_alphaevolve.py --run \
--project-id YOUR_PROJECT_ID --engine YOUR_ENGINE_ID \
--max-programs-evaluated 1
```

This is an offline-checkable export slice. It does not create a cloud client,
start an experiment, enforce a cost limit or reproduce the OpenEvolve evaluation
pipeline and search behavior. Managed-service authentication and execution
remain separate operations that may incur charges.
The project, engine and positive evaluation target are required for `--run`.
`--location` accepts `global` (default), `us` or `eu`. No project/engine values
are inferred from `.env` or environment variables; Google credentials follow the
SDK's ADC behavior only after cloud use is requested. There is no endpoint override.

The launcher validates the configuration with the real SDK and evaluates the
seed locally before constructing a client. It creates a new experiment, uploads
the seed with its actual scores, checks seed creation and start responses, then
invokes the SDK controller with one sampler and one synchronous evaluator. This
avoids requiring thread-safe or signal-free evaluators. The generated entry point
resolves the bundle relative to itself and works from another directory. Existing
bundles can also use `python -m openevolve.compat.alphaevolve_launch /path/to/bundle`.

The final run JSON includes the experiment name, SDK statistics and whether the
evaluation target was reached; failures or unmet targets exit nonzero. The target
cannot exceed the exported generation-attempt budget, but it is not a strict
evaluation cap: SDK polling/queued work can overshoot it, and seed evaluation is
additional. The positive `--idle-timeout-seconds` (default 120) is an idle threshold,
not a wall-clock timeout or a cost limit. Running trusted sources adds no sandbox.

Cloud operations may incur charges. Every `--run` creates a new experiment;
reruns do not resume existing work. Failure, interruption or local controller exit
does not automatically stop or delete remote resources. Inspect the reported
experiment in your project. Export and preflight never start cloud work, and
offline validation is not evidence of a managed-service run. This remains a
limited #468 export slice and does not reproduce OpenEvolve's evaluator pipeline,
search behavior or the broader SDK shim.

Run the offline bundle tests with:

```bash
python -m unittest discover -s tests -p 'test_alphaevolve_bundle.py' -v
python -m unittest discover -s tests -p 'test_alphaevolve_launch.py' -v
```

## Import a local Python experiment configuration
Expand Down
70 changes: 62 additions & 8 deletions openevolve/compat/alphaevolve_bundle.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,16 @@ def main():
raise SystemExit(main())
'''

_LAUNCH = '''"""Offline preflight by default; --run explicitly opts in to cloud use."""

from pathlib import Path

from openevolve.compat.alphaevolve_launch import main

if __name__ == "__main__":
raise SystemExit(main(bundle_dir=Path(__file__).resolve().parent))
'''

_README = """# Minimal AlphaEvolve evaluator bundle

This directory packages a single Python seed and a synchronous
Expand All @@ -74,12 +84,13 @@ def main():
- `evaluation.json`: the objective metric and finite failure score.
- `experiment.json`: the limited AlphaEvolve experiment configuration.
- `check.py`: an explicit local seed-evaluation check.
- `run_alphaevolve.py`: offline preflight or an explicitly requested cloud launch.
- `requirements.txt`, only if explicitly supplied during export.

## Install dependencies

Use Python 3.10 or newer and an OpenEvolve checkout containing
`openevolve.compat.alphaevolve_reverse`. Install that checkout in your chosen
`openevolve.compat.alphaevolve_launch`. Install that checkout in your chosen
environment, for example `python -m pip install /path/to/openevolve`.
The exporter does not infer, install or vendor dependencies. Review and install
your evaluator's packages; if `requirements.txt` is included, review it before
Expand Down Expand Up @@ -120,11 +131,52 @@ def main():
# Pass `evaluate` as evaluator_function to your configured AlphaEvolveExperiment.
```

Cloud authentication, client construction, experiment creation and controller
execution are separate, explicit operations and may incur charges. This bundle
does not perform them or claim a managed-service run. Generation limits are not
cost or evaluation limits. Configure evaluation workers separately from service
generation concurrency and check model availability/mixture rules yourself.
## Preflight or explicitly launch an experiment

Run `python run_alphaevolve.py` (or add `--dry-run`) for an offline JSON summary.
The script resolves the bundle relative to itself, including from another working
directory. It checks the exported configuration subset, evaluation options and
source files without importing the SDK or evaluator, executing the seed,
discovering credentials or making requests. This basic check does not validate
installed dependencies, SDK schemas, permissions or live model availability.
Use `check.py` separately to execute and test your trusted evaluator locally.

For cloud use, install the official SDK from a reviewed checkout of
https://github.com/Google-Cloud-AI/alphaevolve-on-googlecloud, for example
`python -m pip install /path/to/alphaevolve-on-googlecloud`. Follow that project's
README for Google application-default credentials, the GCP project, Gemini
Enterprise engine and access requirements. Choose models currently supported
by your service; the exporter's Gemini-name syntax check is not a model catalog.

After reviewing the exported generation budget, explicitly opt in:

```bash
python run_alphaevolve.py --run \\
--project-id YOUR_PROJECT_ID --engine YOUR_ENGINE_ID \\
--max-programs-evaluated 1
```

Only `--run` imports the optional SDK and trusted evaluator. It validates with
the real SDK and evaluates the seed before constructing a client, then creates
a new experiment, uploads the seed with its actual scores, starts generation
and invokes the SDK controller with one sampler and one synchronous evaluator.
`--location` accepts `global` (default), `us` or `eu`. Credentials use the SDK's
normal Google ADC flow; no key, endpoint or .env discovery is added by the runner.

The evaluation target is required for `--run` and cannot exceed the exported
generation-attempt budget. It is a controller stopping target, not a strict
evaluation cap: SDK polling/queued work can overshoot it, and seed evaluation
is additional. Generation limits are not cost limits. The positive
`--idle-timeout-seconds` (default 120) is an idle threshold, not a wall-clock
timeout, sandbox, cloud cancellation or money limit.

The final run JSON includes the experiment name, SDK statistics and whether the
evaluation target was reached. An unmet target or launch failure exits nonzero.
SDK diagnostics go to stderr. Cloud operations may incur charges. Each `--run`
creates a new experiment; reruns do not resume an earlier experiment. Failures
or interruptions do not delete or stop cloud resources automatically; inspect
the reported experiment in your project. No managed-service run is claimed by
the export or offline checks.

## Scope

Expand All @@ -138,7 +190,7 @@ def main():
non-finite results, and removes temporary candidate files. It omits artifacts and
does not reproduce OpenEvolve's evaluator pipeline, retries, LLM feedback,
cascade stages, resource limits or search behavior. This is a limited export
slice of OpenEvolve #468, not a complete cloud exporter or SDK shim.
slice of OpenEvolve #468, not a complete SDK shim or search-behavior translation.
"""


Expand Down Expand Up @@ -207,7 +259,8 @@ def export_to_alphaevolve(
the two named UTF-8 .py files and an optional explicit dependency file. Reject
existing destinations (including symlinks); their parent must already exist.
Validate inputs before creating output and clean up new output on write errors.
No dependencies, data, local modules, cloud runners or credentials are inferred.
No dependencies, data, local modules or credentials are inferred. The generated
launcher defaults to offline preflight; cloud use requires explicit --run.
Read the generated README and explicitly run check.py after installing deps.
"""
destination = Path(out_dir).absolute()
Expand Down Expand Up @@ -238,6 +291,7 @@ def export_to_alphaevolve(
"src/__init__.py": b"",
"src/evaluate.py": _EVALUATOR.encode("utf-8"),
"check.py": _CHECK.encode("utf-8"),
"run_alphaevolve.py": _LAUNCH.encode("utf-8"),
"README.md": _README.encode("utf-8"),
}
)
Expand Down
Loading
Loading