Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration Reference

This chapter documents every configuration type in Moonpool with its fields, types, and default values. All values are sourced directly from the codebase.

SimulationBuilder

The builder pattern for configuring and running simulation experiments. Created via SimulationBuilder::new().

MethodParametersDescription
workload(w)impl WorkloadAdd a single workload instance, reused across iterations
workload_with_client_id(cid, w)ClientId, impl WorkloadSingle workload with custom client ID strategy
workloads(count, factory)WorkloadCount, Fn(usize) -> Box<dyn Workload>Add factory-created workload instances
workloads_with_client_id(count, cid, factory)WorkloadCount, ClientId, factoryFactory workloads with custom client IDs
processes(count, factory)impl Into<ProcessCount>, Fn() -> Box<dyn Process>Add server processes (system under test)
tags(dimensions)&[(&str, &[&str])]Attach round-robin tag distribution to processes
attrition(config)AttritionEnable automatic process reboots during chaos phase
invariant(i)impl InvariantAdd an invariant checked after every simulation event
invariant_fn(name, f)String, closureAdd a closure-based invariant
fault(f)impl FaultInjectorAdd a custom fault injector for the chaos phase
chaos_duration(dur)DurationSet the chaos phase duration (faults run concurrently with workloads)
set_iterations(n)usizeRun exactly N iterations (default: 1)
set_iteration_control(ctrl)IterationControlSet the iteration control strategy
set_time_limit(dur)DurationRun for a wall-clock time duration
set_debug_seeds(seeds)Vec<u64>Use specific seeds for deterministic debugging
random_network()Enable randomized NetworkConfiguration per iteration
enable_exploration(config)ExplorationConfigEnable fork-based multiverse exploration
replay_recipe(recipe)BugRecipeReplay a specific bug recipe
run()Execute the simulation, returns SimulationReport

Default state

A freshly created SimulationBuilder::new() has:

  • iteration_control: IterationControl::FixedCount(1)
  • use_random_config: false (uses NetworkConfiguration::default())
  • exploration: disabled
  • seeds: empty (auto-generated)
  • No workloads, processes, invariants, or fault injectors

IterationControl

Controls how many iterations a simulation runs.

VariantTypeDescription
FixedCount(n)usizeRun exactly n iterations
TimeLimit(duration)DurationRun for a wall-clock time duration

Note: The UntilAllSometimesReached(N) pattern mentioned in CLAUDE.md is implemented at the test level by checking assertion coverage, not as a variant of IterationControl.

ProcessCount

Controls how many process instances to spawn per iteration.

VariantTypeDescription
Fixed(n)usizeSpawn exactly n processes every iteration
Range(range)RangeInclusive<usize>Spawn a seeded random count from the inclusive range

Accepts usize or RangeInclusive<usize> via Into<ProcessCount>.

WorkloadCount

Controls how many workload instances to spawn per iteration.

VariantTypeDescription
Fixed(n)usizeSpawn exactly n instances
Random(range)Range<usize>Spawn a seeded random count from the half-open range

ClientId

Strategy for assigning client IDs to workload instances.

VariantTypeDescription
Fixed(base)usizeSequential IDs starting from base: instance 0 gets base, instance 1 gets base + 1, etc.
RandomRange(range)Range<usize>Random ID drawn from [start..end) per instance (not guaranteed unique)

Default: Fixed(0) (sequential starting from 0, matching FoundationDB’s WorkloadContext.clientId).

Attrition

Built-in configuration for automatic process reboots during the chaos phase. Requires .chaos_duration() to be set.

FieldTypeDefaultDescription
max_deadusizeMaximum number of simultaneously dead processes
prob_gracefulf64Weight for graceful reboots (signal + grace period)
prob_crashf64Weight for crash reboots (immediate kill)
prob_wipef64Weight for crash + storage wipe reboots
recovery_delay_msOption<Range<usize>>1000..10000Delay before restarting a killed process (ms)
grace_period_msOption<Range<usize>>2000..5000Time allowed for graceful shutdown before force-kill (ms)

The prob_* fields are weights, not probabilities. They are normalized internally and do not need to sum to 1.0.

RebootKind

The type of reboot chosen based on attrition probabilities:

VariantBehavior
GracefulSignal shutdown token, wait grace period, drain send buffers, then restart
CrashImmediate task cancel, all connections abort, no buffer drain
CrashAndWipeSame as Crash plus immediate storage wipe for the process (scoped by IP)

NetworkConfiguration

Top-level network simulation parameters.

FieldTypeDefault
bind_latencyRange<Duration>50us..150us
accept_latencyRange<Duration>1ms..6ms
connect_latencyRange<Duration>1ms..11ms
read_latencyRange<Duration>10us..60us
write_latencyRange<Duration>100us..600us
chaosChaosConfigurationSee below

Constructor variants

ConstructorDescription
NetworkConfiguration::default()Standard defaults with chaos enabled
NetworkConfiguration::random_for_seed()Randomized per seed for chaos testing
NetworkConfiguration::fast_local()Minimal latencies, all chaos disabled

ChaosConfiguration

All fault injection settings for the simulated network. Part of NetworkConfiguration.

Clogging

FieldTypeDefault
clog_probabilityf640.0
clog_durationRange<Duration>100ms..300ms

Network Partitions

FieldTypeDefault
partition_probabilityf640.0
partition_durationRange<Duration>200ms..2s
partition_strategyPartitionStrategyRandom

PartitionStrategy variants: Random, UniformSize, IsolateSingle.

Bit Flips

FieldTypeDefault
bit_flip_probabilityf640.0001 (0.01%)
bit_flip_min_bitsu321
bit_flip_max_bitsu3232
bit_flip_cooldownDuration0

Partial Writes

FieldTypeDefault
partial_write_max_bytesusize1000

Random Connection Close

FieldTypeDefault
random_close_probabilityf640.00001 (0.001%)
random_close_cooldownDuration5s
random_close_explicit_ratiof640.3 (30% explicit)

Clock Drift

FieldTypeDefault
clock_drift_enabledbooltrue
clock_drift_maxDuration100ms

Buggified Delay

FieldTypeDefault
buggified_delay_enabledbooltrue
buggified_delay_maxDuration100ms
buggified_delay_probabilityf640.25 (25%)

Connection Failures

FieldTypeDefault
connect_failure_modeConnectFailureModeProbabilistic
connect_failure_probabilityf640.5 (50%)

ConnectFailureMode variants: Disabled, AlwaysFail, Probabilistic (50% refused, 50% hang).

Latency Distribution

FieldTypeDefault
latency_distributionLatencyDistributionUniform
slow_latency_probabilityf640.001 (0.1%)
slow_latency_multiplierf6410.0

LatencyDistribution variants: Uniform, Bimodal (99.9% fast, 0.1% slow).

Handshake Delay

FieldTypeDefault
handshake_delay_enabledbooltrue
handshake_delay_maxDuration10ms

PeerConfig

Configuration for peer behavior and automatic reconnection. Part of moonpool-transport.

FieldTypeDefault
initial_reconnect_delayDuration100ms
max_reconnect_delayDuration30s
max_queue_sizeusize1000
connection_timeoutDuration5s
max_connection_failuresOption<u32>None (unlimited)
monitorOption<MonitorConfig>Some(MonitorConfig::default())

Constructor variants

Constructorinitial_reconnect_delaymax_reconnect_delaymax_queue_sizeconnection_timeoutmax_connection_failures
PeerConfig::default()100ms30s10005sNone
PeerConfig::local_network()10ms1s100500msSome(10)
PeerConfig::wan_network()500ms60s500030sNone

MonitorConfig

Ping-based connection health monitoring for peers. Follows FoundationDB’s connectionMonitor pattern.

FieldTypeDefault
ping_intervalDuration1s
ping_timeoutDuration2s
max_tolerated_timeoutsu323

Constructor variants

Constructorping_intervalping_timeoutmax_tolerated_timeouts
MonitorConfig::default()1s2s3
MonitorConfig::local_network()500ms1s2
MonitorConfig::wan_network()5s10s5

ExplorationConfig

Configuration for fork-based multiverse exploration. Passed to SimulationBuilder::enable_exploration().

FieldTypeDescription
max_depthu32Maximum fork depth (0 = no forking)
timelines_per_splitu32Children per splitpoint in fixed-count mode
global_energyi64Total number of fork operations allowed
adaptiveOption<AdaptiveConfig>Adaptive forking config; None = fixed-count mode
parallelismOption<Parallelism>Multi-core exploration; None = sequential

Parallelism

Controls how many forked children run concurrently.

VariantSlot count
MaxCoresAll available CPU cores
HalfCoresHalf of available cores (integer division, min 1)
Cores(n)Exactly n concurrent children
MaxCoresMinus(n)All cores minus n (min 1)

AdaptiveConfig

Configuration for coverage-yield-driven batch forking. Used when ExplorationConfig::adaptive is Some.

FieldTypeDescription
batch_sizeu32Children to fork per batch before checking coverage yield
min_timelinesu32Minimum total forks per mark (even if barren after first batch)
max_timelinesu32Hard cap on total forks per mark
per_mark_energyi64Initial energy budget per assertion mark
warm_min_timelinesOption<u32>Minimum timelines for warm starts (multi-seed); defaults to batch_size if None

How the 3-level energy system works

  1. Global energy (global_energy): hard cap on total timelines across all marks. When this hits 0, all exploration stops.
  2. Per-mark energy (per_mark_energy): initial budget for each assertion mark. When exhausted, the mark draws from the reallocation pool.
  3. Reallocation pool: energy returned by barren marks (marks that stopped producing new coverage). Productive marks can draw from this pool to continue exploring.

A mark is considered barren when a batch of children produces no new coverage bits and the mark has already spawned at least min_timelines (or warm_min_timelines during a warm start). Barren marks return their remaining per-mark energy to the reallocation pool.