Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .ai/TASK_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1130,6 +1130,53 @@ Executes a command with arguments.

---

### run_network_disruption

Applies or heals network disruptions (partitions, isolations, shaping) on a Kurtosis devnet via the [disruptoor](https://github.com/ethpandaops/disruptoor) HTTP API. Waits for the API to report healthy, performs the action, then reads back the applied state. Partition/isolation/shaping entries are passed to disruptoor verbatim (wire format of `PUT /v1/state`); disruptoor validation errors are surfaced in the task failure.

**Config:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `disruptoorUrl` | string | required | Base URL of the disruptoor HTTP API (e.g. `http://disruptoor:7700`) |
| `action` | string | "set" | `set` (replace whole state), `update` (merge entries by name), `clear` (heal everything) |
| `partitions` | array[object] | [] | Partition entries: `name`, `groups` (2+ disjoint selectors), optional `scope`, `symmetric` |
| `isolations` | array[object] | [] | Isolation entries: `name`, `target` selector cut off from the rest of the enclave (multi-container targets are isolated as a group), optional `scope` |
| `shaping` | array[object] | [] | Shaping entries: `name`, `target`, `delay`/`jitter`/`loss`/`bandwidth`, `scope: [include_control]` |
| `removeNames` | array[string] | [] | Entry names to remove before merging (`update` action only) |
| `awaitApiTimeout` | duration | 30s | How long to wait for the API to report healthy before acting (0 = act immediately) |
| `pollInterval` | duration | 2s | Interval between health probes |
| `requestTimeout` | duration | 10s | Timeout for a single HTTP request |

Selectors are ethereum-package label matches (e.g. `{node-index: 1, client-type: beacon}`); `scope` values are `cl_p2p`, `el_p2p`, `include_control` (default `[cl_p2p, el_p2p]`; `include_control` also cuts RPC/engine/metrics/VC-CL traffic).

**Outputs:**
| Variable | Type | Description |
|----------|------|-------------|
| `appliedState` | object | Disruptoor state after the action (reflects applied reality) |
| `partitionCount` | int | Active partitions after the action |
| `isolationCount` | int | Active isolations after the action |
| `shapingCount` | int | Active shaping rules after the action |

**Example:**
```yaml
- name: run_network_disruption
title: "Black out node 1's beacon client"
config:
disruptoorUrl: "http://disruptoor:7700"
isolations:
- name: blackout-target-cl
target: { node-index: 1, client-type: beacon }
scope: [cl_p2p, el_p2p, include_control]

# heal (also use as a cleanupTask):
- name: run_network_disruption
config:
disruptoorUrl: "http://disruptoor:7700"
action: clear
```

---

### run_spamoor_scenario

Runs a spamoor stress testing scenario.
Expand Down
3 changes: 3 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ require (
github.com/sirupsen/logrus v1.9.4
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.11.1
github.com/swaggo/http-swagger v1.3.4
github.com/swaggo/swag v1.16.6
github.com/tyler-smith/go-bip39 v1.1.0
Expand All @@ -60,6 +61,7 @@ require (
github.com/consensys/gnark-crypto v0.20.1 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
github.com/crate-crypto/go-eth-kzg v1.5.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/deckarep/golang-set/v2 v2.6.0 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 // indirect
github.com/emicklei/dot v1.6.4 // indirect
Expand Down Expand Up @@ -94,6 +96,7 @@ require (
github.com/pk910/dynamic-ssz v1.3.2 // indirect
github.com/pk910/hashtree-bindings v0.2.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/protolambda/bls12-381-util v0.1.0 // indirect
github.com/r3labs/sse/v2 v2.10.0 // indirect
Expand Down
166 changes: 166 additions & 0 deletions pkg/tasks/run_network_disruption/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# `run_network_disruption` Task

## Description

The `run_network_disruption` task drives a [disruptoor](https://github.com/ethpandaops/disruptoor) instance to apply or heal network disruptions — hard partitions, single-target isolations, and traffic shaping — on a Kurtosis-launched devnet.

### Task Behavior

- Waits for the disruptoor API to report healthy (configurable timeout), then performs the configured action.
- Partition, isolation, and shaping entries are passed through to disruptoor **verbatim** (the wire format of `PUT /v1/state`, see the [disruptoor JSON schema](https://github.com/ethpandaops/disruptoor/blob/master/schemas/v1-state.json)). Assertoor only checks that every entry carries a unique `name`; everything else is validated by disruptoor, whose error messages are surfaced in the task failure.
- Task outputs are read back via `GET /v1/state` after the action, which reflects the *applied* state.
- The task completes as soon as the disruption is applied (disruptoor applies synchronously); pair it with `check_*` tasks to assert the network effects, and put a `clear` invocation in `cleanupTasks` so an aborted test heals the network.

### Actions

- **`set`** (default): Replace the entire disruptoor state with the configured entries. Anything previously active that is not part of this request is healed.
- **`update`**: Read-merge-write. Entries named in `removeNames` are dropped from the current state, then each configured entry replaces its same-name predecessor or is appended. The write is guarded with `If-Match` and retried when a concurrent writer wins the race. Use this to compose disruptions across tasks (e.g. keep a baseline jitter while toggling a blackout).
- **`clear`**: Heal everything.

## Configuration Parameters

- **`disruptoorUrl`**:\
Base URL of the disruptoor HTTP API (e.g. `http://disruptoor:7700`). Required.

- **`action`**:\
Action to perform: `set`, `update`, or `clear`. Default: `set`.

- **`partitions`**:\
Disruptoor partition entries. Each splits the enclave into 2+ disjoint groups; traffic crossing group boundaries is dropped. Fields: `name`, `groups` (list of selectors), optional `scope`, optional `symmetric`.

- **`isolations`**:\
Disruptoor isolation entries. Each cuts the containers matched by its `target` selector off from **the rest of the enclave** — the counterparty group is computed by disruptoor at apply time, so it never needs to be enumerated and stays correct when the topology changes. A target matching multiple containers is isolated *as a group* (traffic among its members keeps flowing); declare one isolation per container to black out several containers individually. Fields: `name`, `target` (selector), optional `scope`.

- **`shaping`**:\
Disruptoor shaping entries: per-target `delay`/`jitter`/`loss`/`bandwidth` degradation. Requires `scope: [include_control]` acknowledgement (disruptoor v0 shapes all egress traffic).

- **`removeNames`**:\
Entry names to remove from the current state before merging. Only valid with `action: update`.

- **`awaitApiTimeout`**:\
How long to wait for the disruptoor API to report healthy before acting. `0` acts immediately. Default: `30s`.

- **`pollInterval`**:\
Interval between health probes while waiting for the API. Default: `2s`.

- **`requestTimeout`**:\
Timeout for a single HTTP request. Default: `10s`.

### Selectors and scopes

Group and target selectors are label matches against the enclave's containers; keys without a dot get the `com.kurtosistech.custom.ethereum-package.` prefix. Common keys on ethereum-package devnets: `node-index` (1-based participant index) and `client-type` (`beacon`, `execution`, `validator`). Multiple values within a key OR together; multiple keys AND together.

`scope` selects the port classes a disruption bites on: `cl_p2p`, `el_p2p` (the default pair), and `include_control` as an explicit opt-in to also cut RPC/engine/metrics/VC↔CL traffic. Without `include_control`, tests keep their visibility into the disrupted node.

## Outputs

- **`appliedState`**:\
The disruptoor state after the action (object; reflects applied reality).

- **`partitionCount`** / **`isolationCount`** / **`shapingCount`**:\
Number of active entries of each kind after the action.

## Examples

### Fully black out one node's beacon client, then heal

Cuts participant 1's CL off from everything — other participants *and* its own execution/validator client:

```yaml
- name: run_network_disruption
title: "Black out the target beacon node"
config:
disruptoorUrl: "http://disruptoor:7700"
isolations:
- name: blackout-target-cl
target: { node-index: 1, client-type: beacon }
scope: [cl_p2p, el_p2p, include_control]

# ... assert the network effects with check_* tasks ...

- name: run_network_disruption
title: "Heal the blackout"
config:
disruptoorUrl: "http://disruptoor:7700"
action: clear
```

### Isolate a whole participant

Without `client-type`, the target matches the participant's CL, EL, and VC together — they keep talking to *each other* but lose the rest of the network. Useful for "node offline" scenarios where the stack itself stays coherent:

```yaml
- name: run_network_disruption
title: "Take participant 2 off the network"
config:
disruptoorUrl: "http://disruptoor:7700"
isolations:
- name: offline-node-2
target: { node-index: 2 }
```

### Variable-driven targets

Task `config` is static YAML; to build entries from test variables, set the whole field via a `configVars` jq expression:

```yaml
- name: run_network_disruption
title: "Black out the configured participant"
configVars:
disruptoorUrl: "disruptoorApiUrl"
isolations: >-
| [{
name: "assertoor-blackout-target-cl",
target: {"node-index": (.targetParticipantIndex | tonumber), "client-type": "beacon"},
scope: ["cl_p2p", "el_p2p", "include_control"]
}]
config: {}
```

### Two-way network split

```yaml
- name: run_network_disruption
title: "Split the network in half"
config:
disruptoorUrl: "http://disruptoor:7700"
partitions:
- name: fork-split
groups:
- { node-index: [1, 2] }
- { node-index: [3, 4] }
scope: [cl_p2p, el_p2p]
```

### Compose disruptions with update

Keep a baseline jitter active while toggling a blackout on and off:

```yaml
- name: run_network_disruption
title: "Add blackout on top of existing disruptions"
config:
disruptoorUrl: "http://disruptoor:7700"
action: update
isolations:
- name: blackout-node-3
target: { node-index: 3 }

- name: run_network_disruption
title: "Remove only the blackout"
config:
disruptoorUrl: "http://disruptoor:7700"
action: update
removeNames: [blackout-node-3]
```

### Cleanup task

```yaml
cleanupTasks:
- name: run_network_disruption
title: "Heal all network disruptions"
config:
disruptoorUrl: "http://disruptoor:7700"
action: clear
```
140 changes: 140 additions & 0 deletions pkg/tasks/run_network_disruption/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
package runnetworkdisruption

import (
"fmt"
"net/url"
"strings"
"time"

"github.com/ethpandaops/assertoor/pkg/helper"
)

// Action selects what the task does against the disruptoor API.
type Action string

const (
// ActionSet replaces the entire disruptoor state with the configured entries.
ActionSet Action = "set"
// ActionUpdate merges the configured entries into the current state by name.
ActionUpdate Action = "update"
// ActionClear heals all active disruptions.
ActionClear Action = "clear"
)

// Config holds the task configuration for driving a disruptoor instance:
// applying network partitions, isolations, and shaping rules to a
// Kurtosis-launched devnet, or healing them again.
//
// Partition, isolation, and shaping entries are passed through to disruptoor
// verbatim (wire format of PUT /v1/state, see the disruptoor JSON schema).
// Assertoor only checks that every entry carries a name — everything else is
// validated server-side so new disruptoor fields work without assertoor
// changes.
type Config struct {
DisruptoorURL string `yaml:"disruptoorUrl" json:"disruptoorUrl" require:"A" desc:"Base URL of the disruptoor HTTP API (e.g. http://disruptoor:7700)."`
Action Action `yaml:"action" json:"action" desc:"Action to perform: set (replace the whole disruptoor state), update (merge entries by name into the current state), clear (heal everything)."`
Partitions []map[string]any `yaml:"partitions" json:"partitions,omitempty" desc:"Disruptoor partition entries: each splits the enclave into 2+ disjoint groups."`
Isolations []map[string]any `yaml:"isolations" json:"isolations,omitempty" desc:"Disruptoor isolation entries: each cuts the containers matched by its target selector off from the rest of the enclave (the counterparty group is computed by disruptoor; a multi-container target is isolated as a group)."`
Shaping []map[string]any `yaml:"shaping" json:"shaping,omitempty" desc:"Disruptoor shaping entries: per-target delay/jitter/loss/bandwidth degradation."`
RemoveNames []string `yaml:"removeNames" json:"removeNames,omitempty" desc:"Entry names to remove from the current state before merging (update action only)."`
AwaitAPITimeout helper.Duration `yaml:"awaitApiTimeout" json:"awaitApiTimeout" desc:"How long to wait for the disruptoor API to report healthy before acting (0 = act immediately)."`
PollInterval helper.Duration `yaml:"pollInterval" json:"pollInterval" desc:"Interval between health probes while waiting for the API."`
RequestTimeout helper.Duration `yaml:"requestTimeout" json:"requestTimeout" desc:"Timeout for a single HTTP request."`
}

// DefaultConfig returns a Config with default values.
func DefaultConfig() Config {
return Config{
Action: ActionSet,
AwaitAPITimeout: helper.Duration{Duration: 30 * time.Second},
PollInterval: helper.Duration{Duration: 2 * time.Second},
RequestTimeout: helper.Duration{Duration: 10 * time.Second},
}
}

// Validate validates the configuration.
func (c *Config) Validate() error {
if c.DisruptoorURL == "" {
return fmt.Errorf("disruptoorUrl is required")
}

parsed, err := url.Parse(c.DisruptoorURL)
if err != nil {
return fmt.Errorf("invalid disruptoorUrl %q: %w", c.DisruptoorURL, err)
}

if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("invalid disruptoorUrl %q: scheme must be http or https", c.DisruptoorURL)
}

c.DisruptoorURL = strings.TrimRight(c.DisruptoorURL, "/")

c.Action = Action(strings.ToLower(string(c.Action)))

entryCount := len(c.Partitions) + len(c.Isolations) + len(c.Shaping)

switch c.Action {
case ActionSet:
if entryCount == 0 {
return fmt.Errorf("set with no partitions/isolations/shaping would clear everything; use action: clear instead")
}
case ActionUpdate:
if entryCount == 0 && len(c.RemoveNames) == 0 {
return fmt.Errorf("update requires at least one partition/isolation/shaping entry or removeNames")
}
case ActionClear:
if entryCount > 0 {
return fmt.Errorf("clear does not take partition/isolation/shaping entries")
}
default:
return fmt.Errorf("invalid action %q, must be one of: set, update, clear", c.Action)
}

if len(c.RemoveNames) > 0 && c.Action != ActionUpdate {
return fmt.Errorf("removeNames is only valid with action: update")
}

if err := validateEntryNames("partitions", c.Partitions); err != nil {
return err
}

if err := validateEntryNames("isolations", c.Isolations); err != nil {
return err
}

if err := validateEntryNames("shaping", c.Shaping); err != nil {
return err
}

if c.AwaitAPITimeout.Duration > 0 && c.PollInterval.Duration <= 0 {
return fmt.Errorf("pollInterval must be positive")
}

if c.RequestTimeout.Duration <= 0 {
return fmt.Errorf("requestTimeout must be positive")
}

return nil
}

// validateEntryNames checks that every entry in a passthrough list carries a
// unique, non-empty name. Names are what disruptoor keys entries on and what
// the update action merges by, so they must be present client-side.
func validateEntryNames(kind string, entries []map[string]any) error {
seen := make(map[string]bool, len(entries))

for i, entry := range entries {
name, ok := entry["name"].(string)
if !ok || name == "" {
return fmt.Errorf("%s[%d]: name is required", kind, i)
}

if seen[name] {
return fmt.Errorf("%s[%d]: duplicate name %q", kind, i, name)
}

seen[name] = true
}

return nil
}
Loading