This chapter documents every configuration type in Moonpool with its fields, types, and default values. All values are sourced directly from the codebase.
The builder pattern for configuring and running simulation experiments. Created via SimulationBuilder::new().
Method Parameters Description
workload(w)impl WorkloadAdd a single workload instance, reused across iterations
workload_with_client_id(cid, w)ClientId, impl WorkloadSingle workload with custom client ID strategy
workloads(count, factory)WorkloadCount, Fn(usize) -> Box<dyn Workload>Add factory-created workload instances
workloads_with_client_id(count, cid, factory)WorkloadCount, ClientId, factoryFactory workloads with custom client IDs
processes(count, factory)impl Into<ProcessCount>, Fn() -> Box<dyn Process>Add server processes (system under test)
tags(dimensions)&[(&str, &[&str])]Attach round-robin tag distribution to processes
attrition(config)AttritionEnable automatic process reboots during chaos phase
invariant(i)impl InvariantAdd an invariant checked after every simulation event
invariant_fn(name, f)String, closureAdd a closure-based invariant
fault(f)impl FaultInjectorAdd a custom fault injector for the chaos phase
chaos_duration(dur)DurationSet the chaos phase duration (faults run concurrently with workloads)
set_iterations(n)usizeRun exactly N iterations (default: 1)
set_iteration_control(ctrl)IterationControlSet the iteration control strategy
set_time_limit(dur)DurationRun for a wall-clock time duration
set_debug_seeds(seeds)Vec<u64>Use specific seeds for deterministic debugging
random_network()– Enable randomized NetworkConfiguration per iteration
enable_exploration(config)ExplorationConfigEnable fork-based multiverse exploration
replay_recipe(recipe)BugRecipeReplay a specific bug recipe
run()– Execute the simulation, returns SimulationReport
A freshly created SimulationBuilder::new() has:
iteration_control : IterationControl::FixedCount(1)
use_random_config : false (uses NetworkConfiguration::default())
exploration : disabled
seeds : empty (auto-generated)
No workloads, processes, invariants, or fault injectors
Controls how many iterations a simulation runs.
Variant Type Description
FixedCount(n)usizeRun exactly n iterations
TimeLimit(duration)DurationRun for a wall-clock time duration
Note : The UntilAllSometimesReached(N) pattern mentioned in CLAUDE.md is implemented at the test level by checking assertion coverage, not as a variant of IterationControl.
Controls how many process instances to spawn per iteration.
Variant Type Description
Fixed(n)usizeSpawn exactly n processes every iteration
Range(range)RangeInclusive<usize>Spawn a seeded random count from the inclusive range
Accepts usize or RangeInclusive<usize> via Into<ProcessCount>.
Controls how many workload instances to spawn per iteration.
Variant Type Description
Fixed(n)usizeSpawn exactly n instances
Random(range)Range<usize>Spawn a seeded random count from the half-open range
Strategy for assigning client IDs to workload instances.
Variant Type Description
Fixed(base)usizeSequential IDs starting from base: instance 0 gets base, instance 1 gets base + 1, etc.
RandomRange(range)Range<usize>Random ID drawn from [start..end) per instance (not guaranteed unique)
Default : Fixed(0) (sequential starting from 0, matching FoundationDB’s WorkloadContext.clientId).
Built-in configuration for automatic process reboots during the chaos phase. Requires .chaos_duration() to be set.
Field Type Default Description
max_deadusize– Maximum number of simultaneously dead processes
prob_gracefulf64– Weight for graceful reboots (signal + grace period)
prob_crashf64– Weight for crash reboots (immediate kill)
prob_wipef64– Weight for crash + storage wipe reboots
recovery_delay_msOption<Range<usize>>1000..10000Delay before restarting a killed process (ms)
grace_period_msOption<Range<usize>>2000..5000Time allowed for graceful shutdown before force-kill (ms)
The prob_* fields are weights , not probabilities. They are normalized internally and do not need to sum to 1.0.
The type of reboot chosen based on attrition probabilities:
Variant Behavior
GracefulSignal shutdown token, wait grace period, drain send buffers, then restart
CrashImmediate task cancel, all connections abort, no buffer drain
CrashAndWipeSame as Crash plus immediate storage wipe for the process (scoped by IP)
Top-level network simulation parameters.
Field Type Default
bind_latencyRange<Duration>50us..150us
accept_latencyRange<Duration>1ms..6ms
connect_latencyRange<Duration>1ms..11ms
read_latencyRange<Duration>10us..60us
write_latencyRange<Duration>100us..600us
chaosChaosConfigurationSee below
Constructor Description
NetworkConfiguration::default()Standard defaults with chaos enabled
NetworkConfiguration::random_for_seed()Randomized per seed for chaos testing
NetworkConfiguration::fast_local()Minimal latencies, all chaos disabled
All fault injection settings for the simulated network. Part of NetworkConfiguration.
Field Type Default
clog_probabilityf640.0
clog_durationRange<Duration>100ms..300ms
Field Type Default
partition_probabilityf640.0
partition_durationRange<Duration>200ms..2s
partition_strategyPartitionStrategyRandom
PartitionStrategy variants: Random, UniformSize, IsolateSingle.
Field Type Default
bit_flip_probabilityf640.0001 (0.01%)
bit_flip_min_bitsu321
bit_flip_max_bitsu3232
bit_flip_cooldownDuration0
Field Type Default
partial_write_max_bytesusize1000
Field Type Default
random_close_probabilityf640.00001 (0.001%)
random_close_cooldownDuration5s
random_close_explicit_ratiof640.3 (30% explicit)
Field Type Default
clock_drift_enabledbooltrue
clock_drift_maxDuration100ms
Field Type Default
buggified_delay_enabledbooltrue
buggified_delay_maxDuration100ms
buggified_delay_probabilityf640.25 (25%)
Field Type Default
connect_failure_modeConnectFailureModeProbabilistic
connect_failure_probabilityf640.5 (50%)
ConnectFailureMode variants: Disabled, AlwaysFail, Probabilistic (50% refused, 50% hang).
Field Type Default
latency_distributionLatencyDistributionUniform
slow_latency_probabilityf640.001 (0.1%)
slow_latency_multiplierf6410.0
LatencyDistribution variants: Uniform, Bimodal (99.9% fast, 0.1% slow).
Field Type Default
handshake_delay_enabledbooltrue
handshake_delay_maxDuration10ms
Configuration for peer behavior and automatic reconnection. Part of moonpool-transport.
Field Type Default
initial_reconnect_delayDuration100ms
max_reconnect_delayDuration30s
max_queue_sizeusize1000
connection_timeoutDuration5s
max_connection_failuresOption<u32>None (unlimited)
monitorOption<MonitorConfig>Some(MonitorConfig::default())
Constructor initial_reconnect_delaymax_reconnect_delaymax_queue_sizeconnection_timeoutmax_connection_failures
PeerConfig::default()100ms 30s 1000 5s None
PeerConfig::local_network()10ms 1s 100 500ms Some(10)
PeerConfig::wan_network()500ms 60s 5000 30s None
Ping-based connection health monitoring for peers. Follows FoundationDB’s connectionMonitor pattern.
Field Type Default
ping_intervalDuration1s
ping_timeoutDuration2s
max_tolerated_timeoutsu323
Constructor ping_intervalping_timeoutmax_tolerated_timeouts
MonitorConfig::default()1s 2s 3
MonitorConfig::local_network()500ms 1s 2
MonitorConfig::wan_network()5s 10s 5
Configuration for fork-based multiverse exploration. Passed to SimulationBuilder::enable_exploration().
Field Type Description
max_depthu32Maximum fork depth (0 = no forking)
timelines_per_splitu32Children per splitpoint in fixed-count mode
global_energyi64Total number of fork operations allowed
adaptiveOption<AdaptiveConfig>Adaptive forking config; None = fixed-count mode
parallelismOption<Parallelism>Multi-core exploration; None = sequential
Controls how many forked children run concurrently.
Variant Slot count
MaxCoresAll available CPU cores
HalfCoresHalf of available cores (integer division, min 1)
Cores(n)Exactly n concurrent children
MaxCoresMinus(n)All cores minus n (min 1)
Configuration for coverage-yield-driven batch forking. Used when ExplorationConfig::adaptive is Some.
Field Type Description
batch_sizeu32Children to fork per batch before checking coverage yield
min_timelinesu32Minimum total forks per mark (even if barren after first batch)
max_timelinesu32Hard cap on total forks per mark
per_mark_energyi64Initial energy budget per assertion mark
warm_min_timelinesOption<u32>Minimum timelines for warm starts (multi-seed); defaults to batch_size if None
Global energy (global_energy): hard cap on total timelines across all marks. When this hits 0, all exploration stops.
Per-mark energy (per_mark_energy): initial budget for each assertion mark. When exhausted, the mark draws from the reallocation pool.
Reallocation pool : energy returned by barren marks (marks that stopped producing new coverage). Productive marks can draw from this pool to continue exploring.
A mark is considered barren when a batch of children produces no new coverage bits and the mark has already spawned at least min_timelines (or warm_min_timelines during a warm start). Barren marks return their remaining per-mark energy to the reallocation pool.