feat: integrate with ethrex over the Engine API#367
Open
pablodeymo wants to merge 20 commits into
Open
Conversation
Add ethlambda-ethrex-client crate speaking JWT HS256-authenticated JSON-RPC to the EL auth endpoint, with typed V3 wrappers for the four engine_* methods we use (exchangeCapabilities, forkchoiceUpdatedV3, newPayloadV3, getPayloadV3) and field-for-field schema match against the canonical execution-apis spec. The blockchain actor takes an optional EngineClient and fires engine_forkchoiceUpdatedV3 at interval 0 of every slot, fire-and-forget; errors are logged but never block consensus. Integration is opt-in via --execution-endpoint + --execution-jwt-secret flags (clap enforces both-or-neither). Verified end-to-end against real ethrex: capability handshake returns the 18 advertised methods, FCUs round-trip in sub-ms with SYNCING (expected -- Lean blocks do not carry an executionPayload yet; that schema change is deferred to an upstream leanSpec proposal, see docs/plans/engine-api-integration.md). Tests: 12 unit + 2 wire smoke tests covering JWT signing, V3 type serde, RPC error surfacing, and full request/response against a hand-rolled mock HTTP server.
🤖 Kimi Code ReviewOverall Assessment: Well-structured PR implementing Engine API scaffolding. Correctly keeps EL integration off the consensus critical path. Minor serialization bugs in wire types need fixing. Critical Issues
Security & Safety
**Code Quality Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code ReviewFindings
Open Questions / Assumptions
Outside the new EL client and tick hook, I didn’t see direct changes to LMD GHOST / 3SF-mini, attestation validation, justification/finalization, XMSS, or SSZ paths. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
Contributor
Greptile SummaryThis PR adds a new
Confidence Score: 3/5Safe for current FCU-only usage; two type definition bugs in types.rs will break wire format when payload building or INVALID_BLOCK_HASH responses are encountered The JWT auth, transport, and blockchain-actor wiring are solid. However types.rs has two defects that matter as soon as deferred milestones are picked up: PayloadStatusKind would produce a deserialization error on any INVALID_BLOCK_HASH EL response, and the [u8; 20] address fields emit integer arrays instead of hex strings, guaranteed to be rejected by the EL when payload building is attempted. crates/net/ethrex-client/src/types.rs needs the PayloadStatusKind rename attribute corrected and hex serializers added to the three [u8; 20] address fields before payload flow is wired
|
| Filename | Overview |
|---|---|
| crates/net/ethrex-client/src/types.rs | New Engine API V3 wire types; two bugs: PayloadStatusKind uses UPPERCASE rename (should be SCREAMING_SNAKE_CASE) and [u8; 20] address fields lack hex serde |
| crates/net/ethrex-client/src/auth.rs | JWT HS256 auth: correct iat-only claims, 32-byte length enforcement, hex parsing with/without 0x prefix |
| crates/net/ethrex-client/src/client.rs | EngineClient wraps reqwest with JWT bearer auth, JSON-RPC envelope, and typed method wrappers |
| crates/blockchain/src/lib.rs | Threads optional EngineClient into BlockChainServer; fires FCU via tokio::spawn fire-and-forget at interval 0 |
| bin/ethlambda/src/main.rs | Adds CLI flags with clap mutual-require; build_execution_client degrades gracefully but has a misleading comment |
| docs/plans/engine-api-integration.md | Detailed integration plan; References section contains absolute local filesystem paths |
| crates/net/ethrex-client/tests/wire_smoke.rs | End-to-end integration test using a hand-rolled TCP mock |
Sequence Diagram
sequenceDiagram
participant CLI as bin/ethlambda
participant EC as EngineClient
participant EL as ethrex Engine API
Note over CLI: startup
CLI->>EC: new(endpoint, JwtSecret)
CLI->>EC: exchange_capabilities
EC->>EL: POST engine_exchangeCapabilities
EL-->>EC: capability list
EC-->>CLI: Ok or warn and continue
Note over CLI: BlockChain spawned with optional client
loop every slot at interval 0
CLI->>EC: notify_execution_layer via tokio::spawn
EC->>EC: sign fresh JWT
EC->>EL: POST engine_forkchoiceUpdatedV3
EL-->>EC: SYNCING scaffold response
EC-->>CLI: logged, never blocks consensus
end
Comments Outside Diff (1)
-
bin/ethlambda/src/main.rs, line 321-325 (link)The warning message "will keep retrying on each tick" implies
engine_exchangeCapabilitiesis called on every slot. In practice onlyengine_forkchoiceUpdatedV3runs on each tick; the capability handshake is one-shot at startup. The comment should describe what actually happens, e.g. "EL will still receive FCU calls; capability list is unknown".Prompt To Fix With AI
This is a comment left during a code review. Path: bin/ethlambda/src/main.rs Line: 321-325 Comment: The warning message "will keep retrying on each tick" implies `engine_exchangeCapabilities` is called on every slot. In practice only `engine_forkchoiceUpdatedV3` runs on each tick; the capability handshake is one-shot at startup. The comment should describe what actually happens, e.g. "EL will still receive FCU calls; capability list is unknown". How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 4
crates/net/ethrex-client/src/types.rs:68-76
`UPPERCASE` rename collapses multi-word variant names without inserting underscores, so `InvalidBlockHash` serializes/deserializes as `"INVALIDBLOCKHASH"` rather than the Engine API spec value `"INVALID_BLOCK_HASH"`. Any EL response carrying that status will fail to deserialize, surfacing as an `EngineClientError::DeserializeResponse` instead of the correct typed variant. Using `SCREAMING_SNAKE_CASE` fixes all five variants simultaneously.
```suggestion
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub enum PayloadStatusKind {
Valid,
Invalid,
Syncing,
Accepted,
InvalidBlockHash,
}
```
### Issue 2 of 4
crates/net/ethrex-client/src/types.rs:35
`[u8; 20]` with no custom serializer is emitted by `serde_json` as a JSON integer array (`[0,1,2,...]`), not the `"0x..."` DATA string the Engine API spec requires. The same issue affects `fee_recipient` in `ExecutionPayloadV3` (line 102) and `address` in `Withdrawal` (line 48). Both `engine_newPayloadV3` and any FCU call that passes `PayloadAttributesV3` will produce a malformed request body that the EL will reject.
### Issue 3 of 4
docs/plans/engine-api-integration.md:168-170
The References section embeds absolute paths from the author's local machine (`/Users/pablodeymonnaz/Lambda/ethrex/...`). These should be replaced with GitHub URLs so other contributors can follow them. The same issue appears in the "Starting state" section (line 33).
### Issue 4 of 4
bin/ethlambda/src/main.rs:321-325
The warning message "will keep retrying on each tick" implies `engine_exchangeCapabilities` is called on every slot. In practice only `engine_forkchoiceUpdatedV3` runs on each tick; the capability handshake is one-shot at startup. The comment should describe what actually happens, e.g. "EL will still receive FCU calls; capability list is unknown".
Reviews (1): Last reviewed commit: "feat: integrate with ethrex over Engine ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+68
to
+76
| #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] | ||
| #[serde(rename_all = "UPPERCASE")] | ||
| pub enum PayloadStatusKind { | ||
| Valid, | ||
| Invalid, | ||
| Syncing, | ||
| Accepted, | ||
| InvalidBlockHash, | ||
| } |
Contributor
There was a problem hiding this comment.
UPPERCASE rename collapses multi-word variant names without inserting underscores, so InvalidBlockHash serializes/deserializes as "INVALIDBLOCKHASH" rather than the Engine API spec value "INVALID_BLOCK_HASH". Any EL response carrying that status will fail to deserialize, surfacing as an EngineClientError::DeserializeResponse instead of the correct typed variant. Using SCREAMING_SNAKE_CASE fixes all five variants simultaneously.
Suggested change
| #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] | |
| #[serde(rename_all = "UPPERCASE")] | |
| pub enum PayloadStatusKind { | |
| Valid, | |
| Invalid, | |
| Syncing, | |
| Accepted, | |
| InvalidBlockHash, | |
| } | |
| #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] | |
| #[serde(rename_all = "SCREAMING_SNAKE_CASE")] | |
| pub enum PayloadStatusKind { | |
| Valid, | |
| Invalid, | |
| Syncing, | |
| Accepted, | |
| InvalidBlockHash, | |
| } |
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/net/ethrex-client/src/types.rs
Line: 68-76
Comment:
`UPPERCASE` rename collapses multi-word variant names without inserting underscores, so `InvalidBlockHash` serializes/deserializes as `"INVALIDBLOCKHASH"` rather than the Engine API spec value `"INVALID_BLOCK_HASH"`. Any EL response carrying that status will fail to deserialize, surfacing as an `EngineClientError::DeserializeResponse` instead of the correct typed variant. Using `SCREAMING_SNAKE_CASE` fixes all five variants simultaneously.
```suggestion
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub enum PayloadStatusKind {
Valid,
Invalid,
Syncing,
Accepted,
InvalidBlockHash,
}
```
How can I resolve this? If you propose a fix, please make it concise.| #[serde(with = "hex_u64")] | ||
| pub timestamp: u64, | ||
| pub prev_randao: H256, | ||
| pub suggested_fee_recipient: [u8; 20], |
Contributor
There was a problem hiding this comment.
[u8; 20] with no custom serializer is emitted by serde_json as a JSON integer array ([0,1,2,...]), not the "0x..." DATA string the Engine API spec requires. The same issue affects fee_recipient in ExecutionPayloadV3 (line 102) and address in Withdrawal (line 48). Both engine_newPayloadV3 and any FCU call that passes PayloadAttributesV3 will produce a malformed request body that the EL will reject.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/net/ethrex-client/src/types.rs
Line: 35
Comment:
`[u8; 20]` with no custom serializer is emitted by `serde_json` as a JSON integer array (`[0,1,2,...]`), not the `"0x..."` DATA string the Engine API spec requires. The same issue affects `fee_recipient` in `ExecutionPayloadV3` (line 102) and `address` in `Withdrawal` (line 48). Both `engine_newPayloadV3` and any FCU call that passes `PayloadAttributesV3` will produce a malformed request body that the EL will reject.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+168
to
+170
| - ethrex Engine API: `/Users/pablodeymonnaz/Lambda/ethrex/crates/networking/rpc/engine/` | ||
| - ethrex auth client (template): `/Users/pablodeymonnaz/Lambda/ethrex/crates/networking/rpc/clients/auth/mod.rs` | ||
| - ethrex JWT auth: `/Users/pablodeymonnaz/Lambda/ethrex/crates/networking/rpc/authentication.rs` |
Contributor
There was a problem hiding this comment.
/Users/pablodeymonnaz/Lambda/ethrex/...). These should be replaced with GitHub URLs so other contributors can follow them. The same issue appears in the "Starting state" section (line 33).
Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/plans/engine-api-integration.md
Line: 168-170
Comment:
The References section embeds absolute paths from the author's local machine (`/Users/pablodeymonnaz/Lambda/ethrex/...`). These should be replaced with GitHub URLs so other contributors can follow them. The same issue appears in the "Starting state" section (line 33).
How can I resolve this? If you propose a fix, please make it concise.
🤖 Claude Code ReviewNow I have a thorough view of all the critical files. Let me compile the review. Review: feat: integrate with ethrex over the Engine APIOverviewThis PR adds the Engine API plumbing between ethlambda and an execution layer (ethrex). It is deliberately scoped to Option B ("scaffold") — the JWT/transport/tick-wiring layer — while deferring actual payload flow until leanSpec defines Correctness Issues1.
#[serde(rename_all = "UPPERCASE")]
pub enum PayloadStatusKind {
...
InvalidBlockHash,
}
#[serde(rename = "INVALID_BLOCK_HASH")]
InvalidBlockHash,2.
pub suggested_fee_recipient: [u8; 20], // PayloadAttributesV3
pub fee_recipient: [u8; 20], // ExecutionPayloadV3
pub address: [u8; 20], // Withdrawal
The fix is a newtype or a custom Minor Issues3. Misleading log message in
Err(err) => warn!(
%err,
"EL capability handshake failed (will keep retrying on each tick)"
),
4. No HTTP status check before body parsing in
let raw = self
.http
.post(&self.url)
...
.send()
.await?
.text()
.await?;If the EL returns a 401 (wrong JWT), 403, or 500, the body may not be valid JSON-RPC. The error will surface as 5. Fixed
id: 1,Fine for fire-and-forget sequential calls, but if concurrent requests are ever issued (possible if two rapid tick intervals fire before the first completes given the 8s timeout), request/response correlation by 6.
pub struct JwtSecret {
bytes: Vec<u8>,
}Since Positive Observations
SummaryThe transport and auth foundations are solid. Two correctness issues need fixing before M6 exercises the types:
Items 3–6 are non-blocking improvements. Automated review by Claude (Anthropic) · sonnet · custom prompt |
MegaRedHand
reviewed
May 13, 2026
Comment on lines
+234
to
+237
| let state = ForkChoiceState { | ||
| head_block_hash: head, | ||
| safe_block_hash: safe, | ||
| finalized_block_hash: finalized, |
…d FCU - types.rs: PayloadStatusKind now uses SCREAMING_SNAKE_CASE so `InvalidBlockHash` round-trips as `INVALID_BLOCK_HASH` (was `INVALIDBLOCKHASH`, which would silently fail to deserialize from any spec-compliant EL). - types.rs: PayloadId serializes/deserializes as a hex DATA string (`"0x..."`) instead of `[serde(transparent)]` over `[u8; 8]` (which emitted a JSON integer array). - types.rs: Added `hex_address` serde helper and applied it to `PayloadAttributesV3.suggested_fee_recipient`, `Withdrawal.address`, and `ExecutionPayloadV3.fee_recipient` — previously these `[u8; 20]` fields were emitted as integer arrays rather than the spec-required hex DATA strings. - types.rs: `hex_u256::deserialize` now returns a serde error on >32-byte input rather than panicking via `copy_from_slice`. - client.rs: HTTP responses now run through `.error_for_status()` before body parsing so 401/403/5xx surface as `EngineClientError::Transport` with a readable message instead of `DeserializeResponse`. - blockchain/lib.rs: `notify_execution_layer` now sends `H256::ZERO` for head/safe/finalized instead of beacon roots. Beacon roots are not EL block hashes; passing them confuses the EL into syncing to garbage. Zero is the spec-friendly "unknown head" sentinel until Lean blocks carry an executionPayload. - bin/ethlambda/main.rs: Fixed misleading warn log — the capability handshake is one-shot at startup, not retried on each tick. - docs/plans/engine-api-integration.md: Replaced absolute local filesystem paths with GitHub URLs. Added unit tests for each bug fix (6 new tests, 16 total in ethrex-client lib). All targeted tests pass, `cargo fmt --all -- --check` clean, `cargo clippy --workspace --all-targets -- -D warnings` clean.
Phase 1a of the M6 plan (docs/plans/engine-api-integration.md, also updated in this commit). The canonical block-component types — ExecutionPayloadV3, Withdrawal, HexBytes, and the hex_* serde helpers — move from the engine-client crate into the foundational types crate, where the Lean BlockBody can later embed them directly. ethlambda-ethrex-client's public API stays stable through re-exports. No SSZ derives yet; those land in Phase 2 alongside the BlockBody embed.
Phase 2a of the M6 plan. Adds SszEncode/SszDecode/HashTreeRoot derives
so the canonical execution-payload types can later be embedded in
BlockBody and State (Phase 2c).
Variable-length list fields move to bounded SSZ types — the spec
requires limits at compile time for merkle tree layout:
extra_data: Vec<u8> → ByteList<MAX_EXTRA_DATA_BYTES>
transactions: Vec<HexBytes> → SszList<ByteList<MAX_BYTES_PER_TX>, MAX_TXS>
withdrawals: Vec<Withdrawal> → SszList<Withdrawal, 16>
Fixed-size byte fields move to plain arrays:
logs_bloom: Vec<u8> → [u8; 256]
JSON wire format is preserved byte-for-byte through new helper modules
(byte_list_hex, hex_bytes_fixed, transactions_serde, withdrawals_serde).
HexBytes is removed; its role is subsumed by ByteList<N> plus the new
transactions serde wrapper.
Manual Default impl on ExecutionPayloadV3: stdlib only auto-derives
Default for arrays up to length 32, and logs_bloom is 256 bytes.
Verified: 32 ethlambda-types tests pass (new SSZ + JSON roundtrips
check hash_tree_root consistency across both encodings); 12
ethrex-client lib tests pass; fmt clean; clippy clean.
Phase 2b of the M6 plan. Adds the cached projection that the consensus
state will carry between blocks (Capella+Deneb spec): every fixed-size
field copies from the payload verbatim, and the two variable-length
lists (transactions, withdrawals) collapse to their SSZ hash-tree roots
so the header itself stays bounded.
ExecutionPayloadV3::to_header() — explicit method
From<&ExecutionPayloadV3> for ExecutionPayloadHeader — sugar
Default — manual (same [u8; 256] reason as ExecutionPayloadV3)
Genesis convention: ExecutionPayloadHeader::default() is all-zeros.
The first Lean block carrying a real payload will assert its
parent_hash matches state.latest_execution_payload_header.block_hash —
which is H256::ZERO at genesis. Subsequent blocks chain forward
normally.
35 ethlambda-types tests pass (3 new: header default, header
SSZ+JSON roundtrip, to_header projects transactions/withdrawals to
their hash tree roots and copies every other field verbatim).
…reak)
Phase 2c of the M6 plan. Adds the canonical execution-payload fields
to the two SSZ containers that block import and STF revolve around:
BlockBody { attestations,
execution_payload: ExecutionPayloadV3 }
State { ...,
latest_execution_payload_header: ExecutionPayloadHeader }
State::from_genesis seeds the header all-zero so the first non-genesis
blocks payload must have parent_hash = H256::ZERO to be accepted —
clean genesis convention without pinning a real EL block hash.
This is the schema-breaking commit in the M6 sequence. Hash tree roots
for BlockBody, Block, and State all change. Consequences:
* Pinned genesis state_root + block_root unit test in
crates/common/types/src/genesis.rs updated to the new values.
* Every fixture-driven spec test that exercises these containers is
gated behind a FIXTURES_AWAIT_M6_REGEN: bool = true flag at the
top of its fn run(). To re-enable a group, flip the flag and
make leanSpec/fixtures after upstream lands the schema. Groups
gated wholesale: forkchoice_spectests (84 cases), stf_spectests
(49 cases), signature_spectests (11 cases). The ssz_spectests
dispatch skips only the BlockBody / Block / State / SignedBlock
arms (127 unrelated cases keep running).
* All other workspace tests pass; ethlambda-types lib tests still
cover ExecutionPayloadV3 / Header SSZ + JSON roundtrips.
Trade-off taken (vs. cargo feature flag, see plan doc Phase 7): the
fixture skips are explicit and tracked in code, but a real cargo
feature would have inflated every BlockBody / State construction with
cfg pollution. The feature-flag alternative was rejected in favor of
the localized skip.
Phase 2d will wire process_execution_payload into the STF —
parent_hash and timestamp assertions per the Capella spec.
Phase 2d of the M6 plan, closing out Phase 2. The STF now enforces the Capella-style payload assertions Pablo flagged on PR #367: 1. payload.parent_hash == state.latest_execution_payload_header.block_hash 2. payload.timestamp == compute_time_at_slot(slot) Both checks run inside process_block between header processing and attestation processing — same ordering as the spec. On success the new payloads header is cached onto state so the next block can chain forward. New error variants InvalidPayloadParentHash and InvalidPayloadTimestamp. Omitted vs. the spec, by design: * verify_and_notify_new_payload (engine_newPayloadV3 roundtrip): lives in the blockchain actor (Phase 3). The STF runs in-process, fork-choice testing, and spec-test harness contexts — none want a network call. * prev_randao: Lean state has no randao mix and leanSpec hasnt defined one. Re-add when upstream lands the field. * SECONDS_PER_SLOT is a duplicate const that must track ethlambda_blockchain::MILLISECONDS_PER_SLOT (currently 4000). state_transition cant depend on blockchain (wrong direction) and the millisecond resolution is wasted in STF. Documented in the consts doc comment. To keep non-EL proposers minting valid blocks until Phase 4 wires engine_getPayloadV3, build_block now calls a synthetic_payload helper that fills in (parent_hash, timestamp) deterministically from state. Phase 4 will swap this for the real EL response when an endpoint is configured. The 20 existing blockchain lib tests (including the two build_block tests) continue to pass. 4 new state_transition unit tests cover the happy path, parent-hash mismatch, timestamp mismatch, and a two-block chain-forward case.
…oadV3
Phase 3 of the M6 plan. Every block arriving over the network — gossip
or BlocksByRoot req-resp — now passes through the EL before fork-choice
insertion. The Handler\<NewBlock\> in BlockChainServer awaits
engine_newPayloadV3(payload, \[\], H256::ZERO) and drops the block on
explicit INVALID / INVALID_BLOCK_HASH verdicts; VALID, SYNCING, and
ACCEPTED all proceed to the existing on_block sync path. Transport
failures are permissive — same policy as notify_execution_layer:
warn-and-accept so EL flakes cant gridlock consensus.
Design notes:
* The EL call lives in the actors async handler, not in
store::on_block. Keeping the store layer sync preserves the
on_block_without_verification seam that fork-choice spec tests
rely on, and avoids fanning async fn across the whole import
pipeline. The validate_payload_with_el helper is private to the
actor.
* Own-built blocks (proposer path at line 453) bypass the pre-check
intentionally — they were either built from a real
engine_getPayloadV3 response (Phase 4, future) or via the
synthetic_payload helper, neither of which the EL needs to
re-validate.
* Pending children inherit validation: every block enters via
Handler<NewBlock>, so anything in pending_blocks already
passed the EL once. The cascade processing at line 610 stays
sync.
* The V3 calls last two params — expected_blob_versioned_hashes
and parent_beacon_block_root — are stubbed to vec![] and
H256::ZERO. Lean blocks dont carry blob transactions or a
beacon-root analogue yet; refine when those land.
Phase 4 next will replace synthetic_payload in build_block with the
real engine_getPayloadV3 response when an EL endpoint is configured.
Phase 4 of the M6 plan. Closes the proposer loop end-to-end when an EL
endpoint is configured:
interval 4 of slot N-1:
request_payload_id_for_next_slot(N-1) — if any of our validators
will propose at slot N, fire engine_forkchoiceUpdatedV3 with
PayloadAttributesV3 (correct slot-timestamp). Stash the returned
payload_id.
interval 0 of slot N:
take_prepared_payload(N) — pop the stashed id, call
engine_getPayloadV3, parse executionPayload, hand to
produce_block_with_signatures. build_block embeds it directly into
BlockBody.execution_payload; STFs process_execution_payload from
Phase 2d then enforces parent_hash + timestamp.
Fallback paths (any of which trigger the Phase 2d synthetic_payload):
* no EL configured
* we didnt queue a build (interval-4 path skipped)
* EL was syncing at interval 4 (payload_id = None on the FCU response)
* stashed slot doesnt match the proposal slot (we skipped a tick)
* engine_getPayloadV3 transport / parse failure
API touches:
* EngineClient::get_payload_v3 now returns ExecutionPayloadV3 directly
(extracts executionPayload from the envelope; drops blobsBundle
and blockValue for now).
* produce_block_with_signatures and the private build_block take an
Option<ExecutionPayloadV3>. None → synthesize.
* propose_block is now async (was sync) and accepts the optional
payload. The two on_tick call sites adjust accordingly.
* New BlockChainServer field pending_payload_id: Option<(u64, PayloadId)>.
What still wont work end-to-end until M6 is fully complete:
* notify_execution_layer still sends H256::ZERO for head/safe/finalized
(Phase 5). Until that goes, the EL has no idea what we consider the
canonical head and may stay in SYNCING.
* suggested_fee_recipient and prev_randao are hardcoded zero. A real
devnet needs CLI flags / RANDAO accumulation.
* Lean blocks still dont propagate blob transactions or a meaningful
parent_beacon_block_root.
These are the next phase-5/6/7 items in docs/plans/engine-api-integration.md.
All workspace tests pass; wire_smoke is sandbox-only.
… draft
Phase 7 of the M6 plan, closing out the in-repo work:
* build_block_embeds_provided_execution_payload — unit test in
crates/blockchain/src/store.rs. Confirms that when produce_block_with_signatures
is handed an engine_getPayloadV3-style ExecutionPayloadV3, build_block
embeds it verbatim (block_hash + full hash_tree_root match) instead of
falling back to synthetic_payload. The other M6-related units are
already covered: process_execution_payloads parent/timestamp guards in
state_transition (Phase 2d) and the ExecutionPayloadV3/Header SSZ + JSON
roundtrips in ethlambda-types (Phases 2a/2b).
* docs/plans/lean-execution-payload-schema.md — draft of the leanSpec
issue proposing the schema upstream. Frames the missing-EL-payload
problem, the canonical-V3 mirror choice, the genesis convention, and
enumerates ethlambdas reference commits as proof of feasibility.
File this verbatim on leanSpec when ready.
* docs/plans/engine-api-integration.md — Phase 7 section now reflects
what landed vs whats deferred. The two EL-mocked tests
(on_block_rejects_when_el_says_invalid /
notify_execution_layer_sends_real_hashes_after_first_block) sit behind
an EngineClient-trait-abstraction refactor that isnt worth blocking
on for this PR; tracked as follow-up in the same section.
21 blockchain lib tests pass (was 20). fmt + clippy clean.
…datedV3 Phase 5 of the M6 plan, retiring the H256::ZERO placeholder that MegaRedHand flagged on PR #367 (\"This is wrong\") and that started this whole expansion. notify_execution_layer now resolves each of the head, safe, and finalized Lean roots to its blocks body.execution_payload.block_hash via a small el_hash_at helper. At genesis the helper naturally rolls back to H256::ZERO (because BlockBody::default() carries ExecutionPayloadV3::default()), so the existing "first FCU is all zeros" startup behavior is preserved — no extra slot-0 special case needed. From the first non-genesis block onward the EL receives the hash it actually minted via engine_getPayloadV3 (Phase 4), so it can chain its own fork choice forward off blocks its already seen via engine_newPayloadV3 (Phase 3) instead of chasing zeros indefinitely. el_hash_at defensively falls back to H256::ZERO when a root is missing from storage. That shouldnt fire for head/safe/finalized (which are always present), but a torn write or pruning bug shouldnt crash the EL notifier; warn-on-failure semantics are preserved further down by the spawned forkchoice_updated_v3 call. Doc comment on notify_execution_layer updated; the in-body comment before the call site no longer claims "we send all-zero hashes". All workspace tests pass; wire_smoke is sandbox-only.
`crates/common/types/src/genesis.rs::GenesisValidatorEntry` was already on the dual-pubkey schema (`attestation_pubkey` / `proposal_pubkey`), but the validators-file parser in `bin/ethlambda/src/main.rs` was still on the older single-pubkey, role-by-filename layout. A node booting from a current `lean-quickstart` bundle would crash on `missing field \`pubkey_hex\`` before reaching the consensus stack — this commit aligns the two. New `AnnotatedValidator` carries both pubkeys and both privkey filenames on the same entry, matching the on-disk format the genesis generator emits today. `read_validator_keys` loses the role-by-filename indirection (classify_role + RoleSlots) since each entry is now self-describing — one entry per validator, both files explicit. No behavior change for the success path on the new format. The single- pubkey/by-filename path is dropped entirely; no client is producing that shape anymore. Net: +22 / -71.
…ml schema" This reverts commit dc25b97.
…lock-hash New CLI flag `--execution-genesis-block-hash` takes the ELs genesis block hash (32-byte hex, `0x`-prefixed or bare) and stores it in `state.latest_execution_payload_header.block_hash` at genesis-state construction. Without this seed the first `engine_forkchoiceUpdatedV3` carries an all-zero head, ethrex replies `SYNCING`, the build-mode FCU at interval 4 returns `payload_id = None`, and the chain never bootstraps a real EL payload. With the seed, the very first FCU references the ELs actual genesis block, ethrex accepts, and the get-payload / new-payload loop starts producing real execution payloads. The flag requires `--execution-endpoint` (clap enforces) and is parsed through the new `parse_h256_hex` helper which rejects wrong-length input. `State::from_genesis` is untouched — the seed happens in `fetch_initial_state` right after construction, so the dozen other test/internal call sites of `from_genesis` dont change. Operators find the value in the EL boot log (`Genesis Block Hash: 0xb923...` for ethrex).
After `propose_block` successfully runs STF on a freshly-built block, fire `engine_newPayloadV3` to the EL so it imports the payload as a real block in its chain. Without this call the EL knows the payload only as a candidate from `engine_getPayloadV3`, so subsequent FCU `head_block_hash` lookups against that hash bounce as SYNCING and the chain doesnt advance on the EL side. For received blocks the same call already happens in `Handler<NewBlock>`s EL pre-check (Phase 3 of M6); this commit closes the parallel gap for own-built blocks, which never re-enter the gossip-handler path. Fire-and-forget via `tokio::spawn` (~ms roundtrip, next FCU is 4s away). INVALID/error verdicts are logged but dont reverse the local `process_block` — by design, mirroring `notify_execution_layer`s "consensus must keep running regardless of EL state" stance.
Three intertwined changes that together unblock real `engine_newPayload`
acceptance on ethrex and end the "FCU always carries ZERO head" loop:
1. **V4 newPayload.** Add `engine_newPayloadV4` to ethrex-client and
switch both call sites (Phase 3 receive-side import check; Phase 5
follow-up own-built notify) from V3 to V4. V4 takes the same
`ExecutionPayloadV3` shape plus an `executionRequests` parameter
(EIP-7685 system contracts — empty for Lean blocks). ethrex rejects
V3 with `-38005 Unsupported fork: Prague` once the payload timestamp
crosses `pragueTime`, which our devnet genesis sets at 0. The
capability advertisement is updated to include V4.
2. **Genesis BLOCK body seed.** The previous commit seeded
`state.latest_execution_payload_header.block_hash` (which drives
STFs parent_hash check) but `el_hash_at` — Phase 5s FCU
`head_block_hash` source — reads
`block.body.execution_payload.block_hash`, and the genesis blocks
body was synthesized as `BlockBody::default()` (all zero) regardless
of any state seeding. `fetch_initial_state` now also builds an
explicit genesis BlockBody whose `execution_payload.block_hash`
equals the seed, updates the headers `body_root`, and uses
`Store::get_forkchoice_store` (which persists the body) instead of
`from_anchor_state` (which assumed the empty body). With both seeds
in place, the very first FCU at interval 0 of slot 0 carries the
real EL genesis hash.
3. **Build-mode FCU uses real hashes.**
`request_payload_id_for_next_slot` (Phase 4 — interval-4 FCU+attrs
that asks the EL to start building) was hardcoding the
`ForkChoiceState` triplet to ZERO, so even with everything else
correct the EL would never recognize our head and return
`payload_id = None`. Factored both that path and
`notify_execution_layer` onto a shared
`current_el_forkchoice_state()` helper.
After these three: at slot 0 interval 4 ethlambda fires
FCU+attrs(head=EL genesis hash); ethrex accepts, returns a payload_id;
at slot 1 interval 0 ethlambda fetches via `engine_getPayloadV3`,
Three intertwined fixes that finally close the loop: ethrex now
receives real, non-zero execution payloads from ethlambda every slot
and reports the new head back via FCU, slot-by-slot.
1. **Anchor consistency at genesis seed.** `Store::get_forkchoice_store`s
`anchor_pair_is_consistent` requires `block.state_root` to equal
`state.hash_tree_root()` (with header.state_root zeroed) exactly —
`ZERO` is not accepted. `fetch_initial_state` now zero-passes
`latest_block_header.state_root`, computes the canonical state root,
stamps it on both the state header and the genesis block before
calling get_forkchoice_store.
2. **Build-mode FCU carries real EL hashes.** `request_payload_id_for_next_slot`
(interval-4 FCU+attrs that asks the EL to start building) was
hardcoding head/safe/finalized to ZERO. Refactored both this path
and `notify_execution_layer` onto a shared
`current_el_forkchoice_state()` helper that reads via `el_hash_at`,
so the EL sees the same head whether the call is heartbeat or
build-mode.
3. **V4 + V5 newPayload / getPayload.** Added typed wrappers for both
on `EngineClient`; advertised in `ETHLAMBDA_ENGINE_CAPABILITIES`.
The actor calls V5 because ethrex on main implements V5 for
Amsterdam-era (EIP-7928 BAL) and rejects V4 with
`-38005 Unsupported fork` once `timestamp >= amsterdamTime`.
The genesis JSON must activate Amsterdam at 0 for V5 to apply.
Confirmed end-to-end against ethrex main: per-slot FCU carries
0xb923…c7af (ethrexs genesis), interval-4 FCU+attrs returns a real
`payload_id`, getPayloadV5 returns a payload ethrex minted, newPayloadV5
on the proposed block lands cleanly, next slots FCU advances to the
new ethrex-minted block_hash. Real EL chain advancing in lockstep.
ilitteri
added a commit
to lambdaclass/ethereum-package
that referenced
this pull request
May 20, 2026
the standard `participants:` block, and remove the parallel
`lean_participants:` schema entirely. The new shape collapses Lean and
EL+CL into one input surface:
participants:
- el_type: ethrex
cl_type: ethlambda
is_aggregator: true
- el_type: none
cl_type: ream
ethlambda is the only Lean client that implements Engine API today
(lambdaclass/ethlambda#367); when paired with an EL (`el_type` != none)
the Lean launcher reads the EL's genesis block hash via
`eth_getBlockByNumber 0x0` after the EL is up, stages the network JWT
into the ethlambda container, and adds the three Engine API flags
(`--execution-endpoint`, `--execution-jwt-secret`,
`--execution-genesis-block-hash`) to its CLI. The other seven Lean
clients run client-only — `el_type: none` skips EL launch entirely
(the package already supports this for `consensoor` etc.).
Cl_TYPE additions: ethlambda, ream, zeam, qlean, lantern, gean,
lean_grandine, lean_lighthouse. The last two are prefixed because
`grandine` and `lighthouse` already exist in CL_TYPE for the Eth1 CLs
of the same name (different binaries from different repos).
`LEAN_CL_TYPES` is the set the cl_launcher dispatcher checks to
decide whether to skip a participant (Lean cl_types are launched by
src/lean/lean_launcher.star, not the standard CL launchers); main.star
then builds a Lean record per such participant and hands the list to
the Lean launcher with the network jwt_file attached.
Other plumbing changes that fall out of this:
- `is_aggregator` is now a first-class per-participant field. Ignored
on non-Lean cl_types.
- The "first participant cannot have el_type=none without bootnodoor"
guard is relaxed when every participant has a Lean cl_type — Lean
uses its own libp2p QUIC mesh and doesn't need an Eth1 bootnode.
- The Fulu/PeerDAS validation skips Lean cl_types (they don't speak
PeerDAS).
- The VC / remote-signer / snooper / metrics-exporter pipeline is
skipped for Lean cl_types in participant_network.star — Lean
validators live inside the consensus binary, not a separate VC.
- shared_utils.get_client_names is None-safe: when cl_context is None
(Lean participants), it falls back to the cl_type string from the
participant config so downstream consumers (validator-ranges, dora,
etc.) still get a usable row name.
`lean_network_params:` stays as a separate config block for Lean-only
knobs (`active_epoch`, `attestation_committee_count`,
`num_validator_keys_per_node`, `metrics_enabled`, ...).
`parse_lean_participants` and `DEFAULT_LEAN_IMAGES` are deleted; the
DEFAULT_CL_IMAGES table now carries the Lean defaults too.
Args files migrated:
- `.github/tests/lean-devnet4.yaml` — every entry moved to
`participants:` with `el_type: none`.
- `.github/tests/lean-smoke.yaml` — same shape, two ethlambda nodes.
- `.github/tests/ethlambda-el-pair.yaml` — new, single ethrex+ethlambda
pair.
- `.github/tests/ethlambda-el-pair-2node.yaml` — new, two pairs with
one aggregator + one non-aggregator on the Lean side.
Validated locally: the 2-node ethrex+ethlambda pair finalizes
slot-by-slot, both ethrex ELs converge on identical block hashes via
the Lean libp2p mesh between the two ethlambdas.
…oadV5 d0c5b72 switched the build path (engine_getPayload + the FCU-then-build chain) to V5 but missed the import-validation path (`validate_payload_with_el`) and the post-propose self-notification path. Both kept calling new_payload_v4 while every surrounding log line, comment, and the commit message itself claimed V5. This worked against the demos ethrex genesis only because that genesis activates forks through Osaka @0 with no `amsterdamTime` set — without an Amsterdam timestamp the EL doesnt gate V4 yet. The moment a paired EL activates Amsterdam, ethrex would have returned `-38005 Unsupported fork: Osaka/Amsterdam` on those two paths and our import-validation would silently flip to the permissive "accepting block" branch. - validate_payload_with_el now calls new_payload_v5; surrounding doc comment rewritten to describe Amsterdam-era V5 (BAL on the payload, same JSON-RPC param shape as V4). - The own-built fire-and-forget notification after propose_block now calls new_payload_v5; surrounding comment updated to reference engine_getPayloadV5 (the version that minted the candidate) and engine_newPayloadV5 (the version that promotes it). - Module docstring and ETHLAMBDA_ENGINE_CAPABILITIES doc-comment in ethrex-client/src/lib.rs both now reflect V3/V4/V5 across newPayload and getPayload, with the version-selection rule (timestamp against the EL fork schedule) made explicit. - cargo fmt picks up three .await line breaks in get_payload_v3/v4/v5 that d0c5b72 introduced but didnt reformat. Caught by an automated three-agent code review; two of the agents flagged the V4 calls independently as a functional drift between the code and the commit message.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Description / Motivation
Lands the full Engine API integration between ethlambda (consensus) and any Engine-API-speaking execution layer, validated live against ethrex. This is Option C from
docs/plans/engine-api-integration.md:ExecutionPayloadV3is embedded in LeanBlockBody, processed by the STF, fetched from the EL on proposal, and validated against the EL on import. The transport / auth / per-slot FCU foundation (Option B) is the substrate it rides on.executionPayloadto LeanBlockBody+ STF + newPayload/getPayload on the slot loopEnd-to-end result: ethrex receives real execution payloads from ethlambda every slot and reports the resulting head back via FCU, slot-by-slot, chain advancing on both sides. Logs in "Live end-to-end" below.
What Changed
New crate:
crates/net/ethrex-client/src/lib.rsETHLAMBDA_ENGINE_CAPABILITIESadvertised listsrc/auth.rsJwtSecret— HS256 token minting, hex/file loaders, deterministic-per-iatsrc/client.rsEngineClient— typed wrappers forengine_*methods over reqwest+JWTsrc/types.rsForkChoiceState,PayloadAttributesV3,ExecutionPayloadV3,PayloadStatus,Withdrawal, hex serde helperssrc/error.rsEngineClientError— distinct variants for transport / RPC / serialize / empty-responseexamples/smoke.rs--loop <N>mode at 4s slot cadencetests/wire_smoke.rstokio::net::TcpListenermockEngine methods wrapped:
engine_exchangeCapabilities— startup handshakeengine_forkchoiceUpdatedV3— per-tick FCU + build-mode (withPayloadAttributesV3) at interval 4engine_newPayloadV3/engine_newPayloadV4/engine_newPayloadV5— import path; V4 addsexecutionRequests, V5 covers Amsterdam-era (BAL / EIP-7928)engine_getPayloadV3/engine_getPayloadV4/engine_getPayloadV5— fetch built payloads at interval 0engine_getClientVersionV1— diagnosticsELs validate method version against payload
timestamp; the actor calls V5 against ethrex main (which advertises V5). Field names match canonical execution-apis JSON character-for-character.Schema change: payload in
BlockBody+ header inStatecrates/common/types/src/block.rs—BlockBody { attestations, execution_payload: ExecutionPayloadV3 }crates/common/types/src/state.rs—State { ..., latest_execution_payload_header: ExecutionPayloadHeader }crates/common/types/src/execution_payload.rs— full SSZ + JSON dual encoding forExecutionPayloadV3,ExecutionPayloadHeader,Withdrawal; hex serde helpers for primitives;BYTES_PER_LOGS_BLOOM=256,MAX_EXTRA_DATA_BYTES=32,MAX_BYTES_PER_TRANSACTION=1073741824,MAX_TRANSACTIONS_PER_PAYLOAD=1048576,MAX_WITHDRAWALS_PER_PAYLOAD=16per execution-apis.STF:
process_execution_payloadcrates/blockchain/state_transition/src/lib.rsadds:process_execution_payload(state, payload)— checkspayload.parent_hash == state.latest_execution_payload_header.block_hash, checkspayload.timestamp == compute_time_at_slot(state.slot), caches the projected header on the post-stateInvalidPayloadParentHash,InvalidPayloadTimestampSECONDS_PER_SLOT = 4andcompute_time_at_slot(slot) = genesis_time + slot * 4crates/blockchain/actor: real EL pairingpending_payload_id: Option<(u64, PayloadId)>onBlockChainServer— caches the EL's payload-id between interval 4 (request) and interval 0 (consume)current_el_forkchoice_state()shared helper reads real EL block hashes fromblock.body.execution_payload.block_hash; used by both heartbeat FCU and build-mode FCU (no more zeroed placeholders)request_payload_id_for_next_slotfiresforkchoiceUpdatedV3withPayloadAttributesV3to ask the EL to start buildingtake_prepared_payloadconsumes the cached id viagetPayloadV5, embeds the resultingExecutionPayloadV3in the proposed block, and firesnewPayloadV5so the EL imports its own payloadvalidate_payload_with_elcallsnewPayloadV5before STF; failures abort importbin/ethlambda/New CLI flags:
--execution-genesis-block-hashis required when pairing with an EL because the lean anchor state must agree on the EL's genesis. The flag:genesis_state.latest_execution_payload_header.block_hash = el_hashBlockBodywithexecution_payload.block_hash = el_hashstate.hash_tree_root()withlatest_block_header.state_rootzeroed, then stamps it on both the state header and the genesis block — required byanchor_pair_is_consistent'sblock.state_root == state_rootcheckStore::get_forkchoice_storeWithout the flag, integration is disabled (unchanged behavior).
Docs
docs/plans/engine-api-integration.md— full plan including M1–M7docs/plans/lean-execution-payload-schema.md— leanSpec proposal draft for the BlockBody/State schema (Phase 7 artifact, to upstream)Correctness / Behavior Guarantees
--execution-endpoint+--execution-jwt-secret,BlockChainreceivesNone, the EL hooks short-circuit, and the genesis seed code path is bypassed. Existing nodes are bitwise-unaffected.tokio::spawn-ed off-tick; the actor never awaits them. RPC failures becomewarn!log entries — never panics, never tick stalls.iat(per execution-apis auth spec); EL accepts ±60s skew; HS256 with a 32-byte secret; fresh token per request.anchor_pair_is_consistentrequiresblock.state_root == state.hash_tree_root()withlatest_block_header.state_rootzeroed. The genesis-seed path computes and stamps the canonical root before callingget_forkchoice_store.parent_hashortimestampdon't match the prior state, with typedInvalidPayloadParentHash/InvalidPayloadTimestamperrors surfaced throughprocess_block.Tests Added / Run
Unit
cargo test -p ethlambda-ethrex-client --lib— JWT, types, client builder, transport errorcargo test -p ethlambda-state-transition— 4 new tests forprocess_execution_payloadcargo test -p ethlambda-blockchain --lib—build_block_embeds_provided_execution_payloadWire smoke (
cargo test -p ethlambda-ethrex-client --test wire_smoke)forkchoice_updated_v3_round_trip— asserts the request body shape (camelCase, JWT, method name) and parses cannedSYNCINGresponserpc_error_surfaces_typed— JSON-RPC error envelope surfaces as typed errorTcpListener::bindtests fail under a sandbox withPermissionDenied; they pass in a normal shell.Spec fixtures
The
BlockBodyschema change (addedexecution_payload) breaks the leanSpec fixture set, which predates this schema. Fixtures are gated behindFIXTURES_AWAIT_M6_REGEN: bool = trueinforkchoice_spectests.rs,stf_spectests.rs,signature_spectests.rs, plus a type-filter inssz_spectests.rs. These will be re-enabled once leanSpec lands the equivalentExecutionPayloadfield (the proposal draft indocs/plans/lean-execution-payload-schema.mdis the upstream artifact for that).Live end-to-end against real ethrex — chain advancing both sides
Setup: ethrex main (
v8.0.0-HEAD-8d83b00) withShanghai/Cancun/Prague/Osaka @0,chainId=9,--syncmode full, JWT shared with ethlambda. Single ethlambda validator (is-aggregator=true).ethrex log (per slot):
…then
block_number=2 ... 3 ... 4 ... 5 ... 6 ... 7 ... 8 ... 9 ..., each FCU'sheadmatches the previous slot's minted payload.ethlambda log (per slot):
Finalization advances slot-by-slot on both sides. Round-trip confirmed: ethrex's FCU
head=matches ethlambda'sblock.body.execution_payload.block_hash, ethlambda's next-slot FCU advances to ethrex's newly-minted block hash.Reproduction
Pre-flight:
/tmp/jwt.hexexists —openssl rand -hex 32 > /tmp/jwt.hex(same file read by both processes)/tmp/ethrex-genesis.jsonexists withShanghai/Cancun/Prague/Osaka @0andchainId: 9ethrexon PATH is main HEAD (V5 support) — not the v8.0.0-tagged releasecargo build --releasein this repoStart T1 first, wait for
Starting Auth-RPC server at 127.0.0.1:8551, then T2.T1 — ethrex
rm -rf /tmp/ethrex-data && \ ethrex --network /tmp/ethrex-genesis.json \ --datadir /tmp/ethrex-data \ --authrpc.addr 127.0.0.1 --authrpc.port 8551 \ --authrpc.jwtsecret /tmp/jwt.hex \ --http.addr 127.0.0.1 --http.port 8545 \ --p2p.disabled \ --syncmode full \ --log.level infoT2 — ethlambda paired
The
--execution-genesis-block-hashvalue is theGenesis Block Hashethrex prints on startup — read it from T1's first few log lines if you regenerate the EL genesis.Related Issues / PRs
docs/plans/lean-execution-payload-schema.md— to be upstreamed; unlocks spec-fixture regeneration which lifts theFIXTURES_AWAIT_M6_REGENgates✅ Verification Checklist
make fmt— cleanmake lint(clippy-D warnings) — cleancargo test --workspace --release— workspace green (spec fixtures intentionally gated; see above)