Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 45 additions & 14 deletions benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,30 +396,61 @@ python -m benchmark.edgebench.checkpoint_judge --checkpoint-capture \
--tasks-dir /path/to/tasks --log-dir /path/to/private-run-logs
```

The supported public checkpoint provider is the compatibility fork at
[`071a884012754127f28f5b84be10f533a42db527`](https://github.com/loopx-agent/EdgeBench/commit/071a884012754127f28f5b84be10f533a42db527).
It contains `sforge.harness.checkpoint` and the native
`create_app(..., checkpoints=True)` seam used by the controller below. The earlier
`0e20fb499ee248f012d6467c00b3dec00a29fe85` compatibility pin and upstream
`sforge==1.1.0` do not contain that API and cannot qualify joint continuation.
Enabling capture against an unsupported provider fails before adding routes.
This is pinned proposed source, not an upstream release. From this LoopX checkout,
prepare an isolated Python 3.12 environment and verify the imported API:
Select the checkpoint provider through the same
[shared EdgeBench source protocol](edgebench/README.md#select-and-freeze-the-native-provider)
as the workers and judge. A new qualification cohort resolves the canonical
integration branch and applies the reviewed native compatibility patch under
that protocol; its `.edgebench-source/receipt.json` records the effective provider
commit, upstream revision and patch digest. Existing cohorts remain frozen.
Patch representation changes receive a new digest and effective commit even
when the provider tree is unchanged; the original frozen bytes and provenance
remain independently verifiable for an older cohort.
An explicit source selection uses unmodified provider code and may not include
the fix; follow the linked protocol when qualifying or replaying a known source.

The provider must expose `sforge.harness.checkpoint` and native
`create_app(..., checkpoints=True)`. API presence alone does not qualify
continuation: a released result with a non-finite score or nested metric must
remain capturable without changing native ranking, history or consumed rounds.
The controller transports native v1 and v2 manifests and verifies their bound
bytes; the provider owns non-finite encoding and restoration. New default cohorts
receive this fix through the bundled compatibility patch. Updating the LoopX
runner alone does not change an older cohort's frozen provider.
Run the HTTP continuity regression below against the selected source before
using it for a new cohort; an older provider can expose the API and still fail
this regression. From this LoopX checkout:

```sh
CKPT_PROVIDER_ROOT=$(mktemp -d)
git clone https://github.com/loopx-agent/EdgeBench.git "$CKPT_PROVIDER_ROOT/provider"
git -C "$CKPT_PROVIDER_ROOT/provider" checkout --detach 071a884012754127f28f5b84be10f533a42db527
python -m benchmark.edgebench.source --log-dir "$CKPT_PROVIDER_ROOT/cohort"
uv venv --python 3.12 "$CKPT_PROVIDER_ROOT/venv"
uv pip install --python "$CKPT_PROVIDER_ROOT/venv/bin/python" \
-e '.[test]' -e "$CKPT_PROVIDER_ROOT/provider" 'httpx>=0.28,<1'
"$CKPT_PROVIDER_ROOT/venv/bin/python" -c 'import inspect,sforge,loopx,sforge.harness.checkpoint; from sforge.harness.judge_server import create_app; assert "checkpoints" in inspect.signature(create_app).parameters; print(sforge.__file__); print(loopx.__file__)'
"$CKPT_PROVIDER_ROOT/venv/bin/python" -m pytest -q benchmark/tests/test_checkpoint_judge.py
-e '.[test]' -e "$CKPT_PROVIDER_ROOT/cohort/.edgebench-source/checkout" 'httpx>=0.28,<1'
LOOPX_EDGEBENCH_SOURCE_RECEIPT="$CKPT_PROVIDER_ROOT/cohort/.edgebench-source/receipt.json" \
"$CKPT_PROVIDER_ROOT/venv/bin/python" - <<'PY'
import inspect
import json
import sforge.harness.checkpoint
from benchmark.edgebench.source import active_source_receipt
from sforge.harness.judge_server import create_app
assert "checkpoints" in inspect.signature(create_app).parameters
print(json.dumps(active_source_receipt(), indent=2))
PY
LOOPX_EDGEBENCH_SOURCE_RECEIPT="$CKPT_PROVIDER_ROOT/cohort/.edgebench-source/receipt.json" \
"$CKPT_PROVIDER_ROOT/venv/bin/python" -m pytest -q \
benchmark/tests/test_checkpoint_judge.py \
benchmark/tests/test_checkpoint_controller.py::test_native_controller_transport_preserves_original_point_and_fresh_authority \
benchmark/tests/test_checkpoint_controller.py::test_native_controller_transport_retains_nonfinite_released_results
# Linux with an owned disposable Docker daemon; grades only synthetic fixtures.
LOOPX_EDGEBENCH_DOCKER_SMOKE=1 "$CKPT_PROVIDER_ROOT/venv/bin/python" -m pytest -q \
benchmark/tests/test_edgebench_feedback_docker.py::test_checkpoint_opportunity_uses_native_admission_and_real_grade
```

The HTTP regression uses synthetic grading and a synthetic backend, with the real
native HTTP service and LoopX capture/export/restore transport. It retains a
finite incumbent beside a partial non-finite result, preserves nested native
values and consumed opportunities, and accepts the next eligible submission
without mutating the source. It does not run a model, Docker or benchmark task.
The Docker journey checks a real grade, remaining rounds/cooldown, incumbent bytes,
and the cold-registration counterexample; it does not restore evaluator authority.
Native session qualification is separate: the tested Host is `codex-cli 0.160.0`.
Expand Down
59 changes: 51 additions & 8 deletions benchmark/edgebench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,50 @@ iptables permissions; a macOS Docker socket alone does not provide that boundary
## Select and freeze the native provider

The first CLI entry for a log directory fetches the selected branch into
`.edgebench-source/checkout` and atomically records its exact commit in
`.edgebench-source/checkout` and atomically freezes the effective source in
`.edgebench-source/receipt.json`. Concurrent worker/judge startup shares that lock.
Later entries reuse the frozen checkout without fetching, even if the branch or
repository default changes. Dirty/moved source, conflicting explicit selection,
and existing runs without a source receipt fail before model/container entry.
Use a new log directory to adopt a newer integration revision; never repin an
active or historical cohort.
Later entries reuse the frozen checkout without fetching, even if the branch,
repository default, or bundled compatibility patch changes. Dirty/moved source,
conflicting explicit selection, and existing runs without a source receipt fail
before model/container entry. Use a new log directory to adopt a newer source;
never repin an active or historical cohort.

For a **new cohort with no explicit source options**, `source-default.json`
selects the shared branch and the versioned
[`nonfinite-checkpoint-v1` compatibility patch](compatibility/nonfinite-checkpoint-v1.patch).
This LoopX-owned patch fixes the native checkpoint representation of nonfinite
released scores and metrics. Native scoring, report values and selection remain
unchanged. Preparation verifies the asset digest and each touched file's exact
upstream blob and mode, applies the patch before provider import/installation,
and creates a clean, reproducible Git commit with the fetched commit as its
parent. The snapshot uses an explicit LoopX compatibility identity and the
upstream commit time; it does not attribute the patch to an upstream author.
Unrelated upstream changes can advance normally. A changed patch preimage,
including a later upstream implementation of the fix, stops preparation without
publishing a source directory. Review and update the bundled patch, or explicitly
select the original source, before starting that new cohort.

Patched sources use `edgebench_source_v2` receipts. `upstream_commit` is the
fetched remote revision; `compatibility_patch` records the patch ID and SHA-256;
`commit` is the **effective local derived revision** actually imported and bound
by native checkpoint provenance. This derived SHA is not an upstream revision
that a fresh remote fetch can retrieve. The frozen `compatibility.patch` copy,
Git parent/tree, complete upstream-to-effective diff, and deterministic commit
contents are cross-checked on reuse. Verification uses the frozen copy rather
than the current LoopX asset, so later asset changes do not invalidate an older
cohort. Preserve the complete `.edgebench-source` directory for offline reuse or
exact patched-source replay. Older `edgebench_source_v1` cohorts retain their
original unmodified source and receipt; preparation never upgrades them.

The bundled patch uses one-line unified context and Git's
`diff.suppressBlankEmpty=true` representation. This keeps its full-index file
identities and changed content readable while avoiding unrelated example text
and trailing-space context markers in the containing repository. Validation
accepts that exact canonical form or the original three-line, space-prefixed
context form retained by earlier v2 cohorts; it does not ignore arbitrary
whitespace. Changing the representation changes its digest and derived commit,
even when the provider tree is identical. Existing cohorts keep their original
patch bytes, receipt and commit and remain verifiable offline.

Prepare dependencies in an isolated environment before starting a study:

Expand Down Expand Up @@ -51,8 +88,14 @@ or a full `--sforge-revision SHA` on the first entry (including preparation).
For example, `--sforge-repository https://github.com/ByteDance-Seed/EdgeBench.git
--sforge-ref refs/heads/main` selects upstream instead. These source options also
apply to the online judge CLI; `SFORGE_LOG_DIR` is equivalent to `--log-dir` there.
Use an explicit SHA to replay a known revision. A server in another log directory
is a separate cohort: the launcher cannot attest a remote server's loaded code.
**Any explicit repository, ref, or revision selects unmodified source for a new
cohort**, even when it names the default repository/branch. These selections use
the original v1 receipt contract and do not apply the bundled patch. Use an
explicit upstream SHA to replay an unmodified known revision; retain its native
checkpoint limitations. For an already frozen cohort, options only assert that
its existing selection matches; they cannot apply/remove a patch or change its
commit. A server in another log directory is a separate cohort: the launcher
cannot attest a remote server's loaded code.

These defaults apply to the LoopX EdgeBench CLI. Direct `sforge serve` and imported
Python APIs remain caller-managed compatibility entrypoints; use them only with
Expand Down
2 changes: 1 addition & 1 deletion benchmark/edgebench/checkpoint_controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ def _bound_bank(folder: Path, reference: dict) -> dict:
witnessed = _physical(folder / "manifest.sha256").read_text()
packet = json.loads(raw)
if (digest != reference["manifest_sha256"] or witnessed != digest or
packet.get("schema_version") != "sforge_judge_checkpoint_v1" or
packet.get("schema_version") not in {"sforge_judge_checkpoint_v1", "sforge_judge_checkpoint_v2"} or
any(packet.get(key) != reference[key] for key in (
"task_id", "source_run_id", "opportunity", "incumbent"))):
raise ValueError("Native bank bytes differ from the captured reference")
Expand Down
Loading
Loading