attsim - Attestation Simulation

A GossipSub simulation framework for testing Ethereum attestation propagation using the Shadow discrete-event network simulator.

Overview

This repository provides tools to simulate GossipSub-based peer-to-peer networks using Shadow, focusing on realistic network conditions and Ethereum consensus layer parameters.

Key Features:

libp2p GossipSub implementation with Ethereum consensus-specs parameters
Multiple network topology generators (mesh, tree, random-regular)
Shadow discrete-event network simulation
Deterministic peer IDs and reproducible simulations
Client diversity simulation (Prysm/Lighthouse with realistic validation timing)
Slot-based attestation publishing with configurable slot timing
Percentile-based BLS batch verification and validation times

Prerequisites

Required

Go 1.23+ - for building simulation binaries
Shadow - discrete-event network simulator (installation guide)
Python 3.12+ - for configuration generation and visualization
uv - Python package manager (installation guide)

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Python dependencies are automatically managed by uv via pyproject.toml
# No manual installation needed - uv will handle dependencies when running scripts

Verify Installation

# Check Shadow is installed
shadow --version

# Check Go is installed
go version

# Check uv is installed
uv --version

Quick Start

# Run simulation with default parameters (random-regular topology, 10 nodes)
# This runs the simulation, tests results, and generates a propagation plot
make all

# Or with custom parameters
make run-sim NODE_COUNT=20 MESH_NODE_COUNT=12 MESH_ATTESTER_COUNT=4 NON_MESH_ATTESTER_COUNT=4

# Change topology type
make run-sim TOPOLOGY_TYPE=mesh NODE_COUNT=10

Directory Structure

attsim/
├── README.md              # This file
├── Makefile               # Build and run commands
├── network_graph.py       # Shadow network configuration generator
├── shadow.template.yaml   # Shadow configuration template
├── simconfig.yaml         # Simulation configuration (client split, timing, etc.)
├── plot_propagation.py    # Message propagation plotting script
├── test_results.py        # Simulation results validation
├── simconfig/             # Simulation config parsing
│   └── simconfig.go       # Config data structures
├── topology/              # Network topology generation
│   ├── topology.go        # Topology data structures
│   ├── generators.go      # Topology generation algorithms
│   └── gen/               # Topology generator CLI tool
│       └── main.go
├── gossipsub/             # GossipSub simulation
│   ├── main.go            # Simulation entry point
│   └── slot_ticker.go     # Slot timing management
├── plots/                 # Generated plot outputs
└── updated-plots/         # Latest simulation plots

Usage

The Makefile handles the complete workflow: topology generation, building, and running the Shadow simulation.

Run Simulation

# Run with defaults (random-regular topology, 10 nodes)
make all

# Custom node count and node type distribution
make run-sim NODE_COUNT=20 MESH_NODE_COUNT=10 MESH_ATTESTER_COUNT=5 NON_MESH_ATTESTER_COUNT=5

# Different topologies
make run-sim TOPOLOGY_TYPE=mesh NODE_COUNT=10
make run-sim TOPOLOGY_TYPE=tree NODE_COUNT=31 BRANCHING=2
make run-sim TOPOLOGY_TYPE=random-regular NODE_COUNT=20 PEER_COUNT=6

# With progress bar
make run-sim PROGRESS=true

Makefile Targets

make all - Complete workflow (build, generate topology, run simulation, test, plot)
make build - Build the GossipSub binary
make build-topology-gen - Build the topology generator
make generate-topology - Generate topology file
make generate-config - Generate Shadow configuration
make generate-config-only - Regenerate configuration without rebuild
make run-shadow - Run Shadow simulation (assumes config exists)
make run-sim - Run complete simulation (shadow + test + plot)
make test - Test simulation results
make plot - Plot message propagation over time
make clean - Clean up artifacts and results
make help - Show help with all options

Topology Options

The Makefile supports three topology types:

mesh: Full connectivity (all-to-all)
tree: Hierarchical tree with configurable branching factor
random-regular: Random graph where mesh nodes have uniform degree

Configuration Variables

Node Configuration:

NODE_COUNT - Total number of nodes (default: 10)
MESH_NODE_COUNT - Number of mesh subscriber nodes (default: 6)
MESH_ATTESTER_COUNT - Number of mesh attester nodes that publish & subscribe (default: 2)
NON_MESH_ATTESTER_COUNT - Number of non-mesh attester nodes that only publish (default: 2)

Topology Configuration:

TOPOLOGY_TYPE - Topology type: mesh, tree, random-regular (default: random-regular)
PEER_COUNT - Peer count for mesh nodes in random-regular (default: 4)
NON_MESH_NODE_PEER_COUNT - Peer count for non-mesh nodes (default: 4)
BRANCHING - Branching factor for tree topology (default: 2)

Simulation Configuration:

PROGRESS - Show Shadow progress bar (default: false)
LOG_LEVEL - Log level (default: info)

Additional parameters are configured in simconfig.yaml:

client_split - Prysm/Lighthouse client distribution
prysm_validator - Batch interval and verification times
lighthouse_validator - Validation times
slots_to_run - Number of slots to simulate
msg_size - Message size in bytes

GossipSub Parameters

The simulation uses Ethereum consensus-specs parameters for realistic behavior:

Parameter	Value	Description
D	8	Target mesh size (number of peers)
Dlo	6	Lower bound on mesh size
Dhi	12	Upper bound on mesh size
Dlazy	6	Gossip target (lazy push)
Heartbeat	700ms	Heartbeat interval
Fanout TTL	60s	Time-to-live for fanout peers
History Length	6	Message cache windows
History Gossip	3	Windows to gossip about

Parameters from: Ethereum consensus-specs

Simulation Parameters

Node Types

The simulation supports three types of nodes:

Mesh Nodes: Subscribe to the topic and receive messages (do not publish)
Mesh Attester Nodes: Subscribe to the topic AND publish attestation messages
Non-Mesh Attester Nodes: Only publish messages (fanout peers, do not subscribe)

Client Types

Each node is assigned a client type based on the client_split in simconfig.yaml:

Prysm: Uses BLS batch verification with configurable batch intervals and percentile-based verification times
Lighthouse: Uses direct validation with percentile-based validation times

Configuration File (simconfig.yaml)

genesis_time: 946665000        # Unix timestamp for genesis
slot_time: 12                  # Slot duration in seconds
slots_to_run: 30               # Number of slots to simulate
msg_size: 192                  # Message size in bytes

client_split:
  prysm: 44                    # Percentage of Prysm clients
  lighthouse: 56               # Percentage of Lighthouse clients

prysm_validator:
  batch_interval: 5            # Batch interval in milliseconds
  p99_batch_verifier_time: 32500  # Verification times in microseconds
  p95_batch_verifier_time: 8500
  p90_batch_verifier_time: 5000
  p75_batch_verifier_time: 4350
  p50_batch_verifier_time: 3900
  p25_batch_verifier_time: 3900

lighthouse_validator:
  p99_validator_time: 9500     # Validation times in microseconds
  p95_validator_time: 4900
  p90_validator_time: 4650
  p75_validator_time: 3700
  p50_validator_time: 2600
  p25_validator_time: 1300

General Parameters

Node Count: Total number of participating nodes
Topology: Network connection graph (mesh, tree, random-regular)
Log Level: debug, info, warn, error

Development

Building

# Build everything and run simulation
make all

# Or build components separately
make build-topology-gen  # Build topology generator
make build              # Build GossipSub binary

Testing Different Configurations

# Test different node counts and distributions
make run-sim NODE_COUNT=10 MESH_NODE_COUNT=6 MESH_ATTESTER_COUNT=2 NON_MESH_ATTESTER_COUNT=2
make run-sim NODE_COUNT=50 MESH_NODE_COUNT=30 MESH_ATTESTER_COUNT=10 NON_MESH_ATTESTER_COUNT=10
make run-sim NODE_COUNT=100 MESH_NODE_COUNT=60 MESH_ATTESTER_COUNT=20 NON_MESH_ATTESTER_COUNT=20

# Test different topologies
make run-sim TOPOLOGY_TYPE=mesh NODE_COUNT=10
make run-sim TOPOLOGY_TYPE=tree NODE_COUNT=31 BRANCHING=2
make run-sim TOPOLOGY_TYPE=random-regular NODE_COUNT=20 PEER_COUNT=6

# Test different peer counts for mesh vs non-mesh nodes
make run-sim PEER_COUNT=8 NON_MESH_NODE_PEER_COUNT=4

Viewing Results

# View all logs
cat shadow.data/hosts/node*/gossipsub.*.stdout

# View specific node logs
cat shadow.data/hosts/node0/gossipsub.*.stdout
cat shadow.data/hosts/node1/gossipsub.*.stdout

# Check for errors
grep -i error shadow.data/hosts/*/gossipsub.*.stderr

# Plot message propagation
make plot

Clean Up

# Clean build artifacts and simulation results
make clean

Architecture

Simulation Workflow

Topology Generation
- The Makefile automatically generates a topology file based on parameters
- Supported types: mesh (full connectivity), tree (hierarchical), random-regular (uniform degree)
- Topology includes node type assignments (mesh, mesh-attester, non-mesh-attester)
- Client type distribution (Prysm/Lighthouse) based on simconfig.yaml
- Topology saved as JSON with node connections and metadata
Shadow Configuration
- Network graph generated with realistic global latencies (Australia, Europe, Asia, Americas, Africa)
- Node types: supernodes (1024 Mbps) and fullnodes (50 Mbps)
- Nodes distributed across geographic locations
- Reads parameters from both topology and simconfig files
GossipSub Simulation
- Each node runs as a separate process in Shadow
- Nodes use deterministic peer IDs for reproducibility
- All nodes start synchronized at 2000/01/01 00:02:00
- Slot-based message publishing (attester nodes publish once per slot)
- Message validation based on client type (Prysm batch vs Lighthouse direct)

Topology JSON Format

{
  "node_count": 10,
  "mesh_node_ids": [0, 1, 2, 3, 4, 5],
  "mesh_attester_node_ids": [6, 7],
  "non_mesh_attester_node_ids": [8, 9],
  "prysm_node_ids": [0, 2, 4, 6, 8],
  "lighthouse_node_ids": [1, 3, 5, 7, 9],
  "connections": [
    {"from": 0, "to": 1},
    {"from": 1, "to": 2},
    {"from": 2, "to": 3}
  ]
}

Node Behavior

Mesh Nodes (subscribers only):

Join GossipSub topic and subscribe
Receive and log messages from attesters
Participate in gossip protocol mesh

Mesh Attester Nodes (publish & subscribe):

Join GossipSub topic and subscribe
Publish one attestation message per slot
Receive and log messages from other attesters
Validate messages using client-specific validation (Prysm batch or Lighthouse direct)

Non-Mesh Attester Nodes (fanout publishers):

Join GossipSub topic (without subscribing)
Publish one attestation message per slot
Use fanout mechanism to reach mesh peers

Client Validation

Prysm Client:

Uses BLS batch verification queue
Messages are batched at configurable intervals (default: 5ms)
Batch verification time follows percentile distribution from real-world data

Lighthouse Client:

Uses direct per-message validation
Validation time follows percentile distribution from real-world data

Key Features

Deterministic peer IDs for reproducibility
Shadow-aware timing (synchronized start)
Topology-based peering (controlled network structure)
Slot-based message publishing
Realistic client validation timing (Prysm/Lighthouse)
Percentile-based verification time distributions

Plots

the 50ms_batch_interval_plot.png represents the message propagation CDF with the 50ms batch interval.

the 5ms_batch_interval_plot.png represents the message propagation CDF with the 5ms batch interval.

More details on the topology, client_split, batch_verifier times, lighthouse validator time should be self-explanatory in the plot

License

Same as parent project.

Related Projects

eth-ec-broadcast - Parent project with erasure coding improvements
Shadow - Discrete-event network simulator
libp2p - Modular peer-to-peer networking stack
go-libp2p-pubsub - GossipSub implementation

Name		Name	Last commit message	Last commit date
Latest commit History 20 Commits
.github/workflows		.github/workflows
gossipsub		gossipsub
plots		plots
simconfig		simconfig
topology		topology
updated-plots		updated-plots
.gitignore		.gitignore
.python-version		.python-version
CLAUDE.md		CLAUDE.md
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
network_graph.py		network_graph.py
plot_propagation.py		plot_propagation.py
pyproject.toml		pyproject.toml
shadow.template.yaml		shadow.template.yaml
simconfig.yaml		simconfig.yaml
test_results.py		test_results.py
uv.lock		uv.lock

ethp2p/attsim

Folders and files

Latest commit

History

Repository files navigation