Skip to content

[codex] add evm-only executor load test harness#3658

Draft
codchen wants to merge 8 commits into
codex/evmonly-staking-dynamic-gasfrom
codex/evm-only-executor-loadtest
Draft

[codex] add evm-only executor load test harness#3658
codchen wants to merge 8 commits into
codex/evmonly-staking-dynamic-gasfrom
codex/evm-only-executor-loadtest

Conversation

@codchen

@codchen codchen commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a standalone evmonly-loadtest command that continuously feeds generated EVM transfer blocks into the EVM-only executor with generated genesis state, configurable result sinks, and Prometheus/stdout throughput metrics.

The harness is intended for optimistic executor saturation testing with one external executor worker and internal OCC workers. It defaults recipients to unique addresses and keeps the workload structure isolated so future contract-style transactions can be added beside the transfer generator.

This also adds --prebuild-blocks, which generates all bounded blocks before starting executor workers. That mode separates block-generation throughput from executor throughput for diagnosing whether builders are competing with execution.

The current branch also freezes generated prebuilt genesis state before execution so the synthetic StateReader can serve immutable state without lock/copy overhead during the executor-only measurement.

Adds OCC diagnostics for future contract-style load tests: block results now report OCC attempts, sequential fallbacks, fallback reasons, and aggregated conflict keys. The loadtest exports Prometheus counters for attempts/fallbacks/conflicts, fallback reasons, and conflict keys labeled by access/kind/address/slot; stdout totals include occ_attempts, occ_fallbacks, and occ_conflicts.

Adds --workload=erc20-transfer, which installs a minimal ERC20-like transfer runtime at --erc20-contract, preloads each generated sender's token balance in the contract's balances mapping, and sends signed transfer(address,uint256) calls to unique recipients by default. The generated runtime performs real contract code execution, SLOAD/SSTORE, and a standard Transfer(address,address,uint256) log.

Adds a lightweight persistent result sink with --result-sink=file --persist-dir=<dir>. This writes append-only RLP records to changesets.rlp and receipts.rlp, with each record framed as block height plus payload length plus RLP payload. --persist-sync can additionally fsync each block for a stricter storage-pressure mode; without it the files are buffered and flushed per block. The sink removes both output files on normal completion, execution errors, and SIGINT/SIGTERM; a hard SIGKILL still cannot run process cleanup.

Validation

  • go test ./giga/evmonly/...
  • Native transfer smoke: go run ./giga/evmonly/cmd/evmonly-loadtest --metrics-addr= --report-interval=0 --prebuild-blocks --blocks=20 --txs-per-block=1000 --builders=16 --workers=1 --executor-workers=12 --gas-price-wei=0 --min-gas-price-wei=0 --queue-size=64
  • ERC20 transfer smoke: go run ./giga/evmonly/cmd/evmonly-loadtest --metrics-addr= --report-interval=0 --prebuild-blocks --blocks=2 --txs-per-block=4 --builders=2 --workers=1 --executor-workers=2 --workload=erc20-transfer --gas-price-wei=0 --min-gas-price-wei=0 --queue-size=2
  • Persistent sink smoke: DIR=$(mktemp -d /tmp/evmonly-persist-smoke.XXXXXX); go run ./giga/evmonly/cmd/evmonly-loadtest --metrics-addr= --report-interval=0 --prebuild-blocks --blocks=2 --txs-per-block=4 --builders=2 --workers=1 --executor-workers=2 --workload=erc20-transfer --gas-price-wei=0 --min-gas-price-wei=0 --result-sink=file --persist-dir="$DIR"; find "$DIR" -maxdepth 1 -type f -print; rm -rf "$DIR"
  • SIGTERM cleanup smoke: build /tmp/evmonly-loadtest-cleanup, run with --blocks=0 --result-sink=file --persist-dir=<tmp>, send SIGTERM after startup, and verify find <tmp> -maxdepth 1 -type f prints no files. Observed interrupted run: input_blocks=156 finished_blocks=146 txs=73000 errors=0, remaining files <none>.

EC2 benchmark

The 199k/202k EC2 benchmark numbers below were run from commit 4ec8da52c on a c8i.48xlarge in us-east-1a with SMT disabled using CoreCount=96,ThreadsPerCore=1, Go 1.25.6, one external executor worker, prebuilt blocks, unique senders/recipients, and zero gas price/min gas price. The current head adds OCC diagnostics, ERC20 transfer load support, and optional persistent result sinks on top of that benchmarked native-transfer path.

Reproduce the 199.3k TPS run

Launch the instance with CPU options equivalent to:

aws ec2 run-instances \
  --instance-type c8i.48xlarge \
  --cpu-options CoreCount=96,ThreadsPerCore=1 \
  --image-id <al2023-x86_64-ami> \
  --key-name <key> \
  --security-group-ids <sg> \
  --subnet-id <subnet> \
  --associate-public-ip-address

Build the loadtest binary on the instance:

curl -fsSL https://go.dev/dl/go1.25.6.linux-amd64.tar.gz -o /tmp/go1.25.6.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf /tmp/go1.25.6.linux-amd64.tar.gz
git clone --branch codex/evm-only-executor-loadtest https://github.com/sei-protocol/sei-chain.git
cd sei-chain
/usr/local/go/bin/go build -o /tmp/evmonly-loadtest ./giga/evmonly/cmd/evmonly-loadtest

Run the benchmark:

GOMAXPROCS=96 GOGC=200 /tmp/evmonly-loadtest \
  --metrics-addr "" \
  --blocks 1000 \
  --txs-per-block 5000 \
  --prebuild-blocks \
  --builders 96 \
  --workers 1 \
  --executor-workers 96 \
  --gas-price-wei 0 \
  --min-gas-price-wei 0 \
  --report-interval 5s

Observed output:

prebuild complete elapsed=9.366s blocks=1000 txs=5000000 build_blocks/s=106.77 build_tx/s=533873.27
complete elapsed=25.09s input_blocks=1000 finished_blocks=1000 txs=5000000 gas=105000000000 errors=0 avg_input_blocks/s=39.86 avg_finished_blocks/s=39.86 avg_tx/s=199284.61 avg_gas/s=4184976805.58

For comparison, the same shape and command with GOGC=400 crossed the 200k target:

prebuild complete elapsed=8.918s blocks=1000 txs=5000000 build_blocks/s=112.14 build_tx/s=560690.94
complete elapsed=24.643s input_blocks=1000 finished_blocks=1000 txs=5000000 gas=105000000000 errors=0 avg_input_blocks/s=40.58 avg_finished_blocks/s=40.58 avg_tx/s=202893.72 avg_gas/s=4260768150.96

Other comparisons from the same testing round:

  • c8i.48xlarge with default SMT topped out around 171k TPS on the 2M-tx sweep.
  • c8i.48xlarge with SMT disabled and default GC reached about 191k TPS over 5M tx.
  • c7a.48xlarge topped out around 161k TPS on the 2M-tx sweep.
  • Larger 10k tx/block blocks regressed to about 184k TPS over 5M tx.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJul 1, 2026, 4:17 AM

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.62312% with 352 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.34%. Comparing base (7c6c44c) to head (78319b4).

Files with missing lines Patch % Lines
giga/evmonly/cmd/evmonly-loadtest/main.go 63.30% 261 Missing and 74 partials ⚠️
giga/evmonly/occ.go 79.26% 16 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@                          Coverage Diff                          @@
##           codex/evmonly-staking-dynamic-gas    #3658      +/-   ##
=====================================================================
+ Coverage                              58.15%   58.34%   +0.19%     
=====================================================================
  Files                                   2185     2179       -6     
  Lines                                 178564   178107     -457     
=====================================================================
+ Hits                                  103836   103912      +76     
+ Misses                                 65413    65035     -378     
+ Partials                                9315     9160     -155     
Flag Coverage Δ
sei-chain-pr 70.75% <64.62%> (+14.45%) ⬆️
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
giga/evmonly/occ.go 79.42% <79.26%> (+5.76%) ⬆️
giga/evmonly/cmd/evmonly-loadtest/main.go 63.30% <63.30%> (ø)

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codchen codchen force-pushed the codex/evmonly-staking-dynamic-gas branch from bd57e4c to 7c6c44c Compare June 30, 2026 06:11
@codchen codchen force-pushed the codex/evm-only-executor-loadtest branch from 8e28929 to 4ec8da5 Compare June 30, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant