Configuration Reference

• SimulationBuilder
  • Default state
• IterationControl
• ProcessCount
• WorkloadCount
• Chaos duration and recovery mode
• Attrition
  • AttritionScope
  • RebootKind
• NetworkConfiguration
  • Constructor variants
  • Network fault mask
  • Latency distribution
  • Link latency (distance-based)
• ChaosConfiguration
  • Clogging
  • Per-Pair Permanent Latency
  • Network Partitions
  • Bit Flips
  • Partial Writes
  • Partial Reads
  • Random Connection Close
  • Black Hole
  • Clock Drift
  • Buggified Delay
  • Connection Failures
  • Handshake Delay
• ExplorationConfig

This chapter documents every configuration type in Moonpool with its fields, types, and default values. All values are sourced directly from the codebase.

SimulationBuilder

The builder pattern for configuring and running simulation experiments. Created via SimulationBuilder::new().

Default state

A freshly created SimulationBuilder::new() has:

• iteration_control: IterationControl::UntilCoverageStable { plateau_seeds: 10, max_iterations: 1000 }
• chaos: all surfaces off (network/storage use default(), no attrition)
• swarm_operations: false (workloads see the full operation alphabet)
• check_determinism: false (each seed runs once)
• exploration: disabled
• seeds: empty (auto-generated)
• No workloads, processes, invariants, or fault injectors

IterationControl

Controls how many iterations a simulation runs.

Note: UntilCoverageStable uses real LLVM sancov code coverage when the binary is instrumented (built via cargo xtask sim run) and falls back to assertion-slot coverage otherwise. The report names the signal it used.

ProcessCount

Controls how many process instances to spawn per iteration.

Accepts usize or RangeInclusive<usize> via Into<ProcessCount>.

WorkloadCount

Controls how many workload instances to spawn per iteration.

Chaos duration and recovery mode

.chaos_duration(dur) bounds the period during which Moonpool may inject new faults. When it expires the runner crosses the chaos → recovery boundary in one step: ctx.chaos_shutdown() is cancelled so fault injectors wind down, and SimWorld::enter_recovery_mode() switches off every configuration-driven fault family and heals the partitions the simulator is holding.

The persistent consequences of faults already injected remain part of the simulated state. The cluster is not healthy at the cutoff — only the environment stops generating new faults. Recovering from the damage is the simulated system’s job.

The recovery window is the quiet tail between the cutoff and workload completion, while the processes are still alive. The settle phase that follows is not a recovery window: processes are aborted before it, so it only drains the events left in the scheduler.

Attrition

Built-in configuration for automatic process reboots during the chaos phase. Requires .chaos_duration() to be set.

The prob_* fields are weights, not probabilities. They are normalized internally and do not need to sum to 1.0.

In ChaosMode::Swarm, the recovery range is scaled per seed to 50%-200% of its configured values. Clustered campaigns also swarm among topology-backed scopes whose groups fit max_dead. Both decisions are draws on the simulation stream, taken once at build time.

AttritionScope

Which failure domain a reboot kills. PerMachine, PerZone, and PerDatacenter require a .cluster() topology; without locality they are a no-op.

RebootKind

The type of reboot chosen based on attrition probabilities:

NetworkConfiguration

Top-level network simulation parameters.

Bind, connect, and accept use their latency fields as genuinely delayed operations. Each call remains pending until its targeted scheduler completion fires. Established reads wait on buffered byte delivery or a network waker. Write and link latency apply to ordered byte delivery after the stream accepts the bytes into its send window (see Flow control).

Constructor variants

Network fault mask

SimulationBuilder::network_fault_mask() applies a typed allow-mask after the per-seed Random or Swarm profile and buggify knob perturbations, but before the SimWorld is created. The mask consumes no simulation or configuration RNG draws, so it is compatible with frontier exploration and exact recipe replay. The default is NetworkFaultMask::all(), which leaves existing builders unchanged.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([Chaos::Network(ChaosMode::Swarm)])
    .network_fault_mask(
        NetworkFaultMask::all().without(NetworkFault::BitFlip),
    )
    .enable_exploration(exploration_config)
}

The mask covers Clog, Partition, BitFlip, RandomClose, ConnectFailure, ClockDrift, BuggifiedDelay, PairLatency, and BlackHole. Removing a family can only suppress a fault; it cannot enable a family the sampled profile turned off. Partial reads and writes are TCP/buggify behavior rather than independently sampled fault families and remain active.

Latency distribution

Each per-operation latency field above is a LatencyDistribution, not a plain range. This lets a simulation exercise the heavy P99 tail where timeout cascades and retry storms live. Every variant samples deterministically through the simulation RNG. Reads do not add a separate delay: they wait for bytes whose delivery already includes write and link latency.

default() and fast_local() keep every field Uniform, so behavior is unchanged unless you opt in. random_for_seed() mixes all three shapes per field for chaos seeds. The same LatencyDistribution type configures storage read_latency, write_latency, and sync_latency.

To replace the hand-picked defaults with values measured on a real machine, see Calibrating Against a Real Machine.

Every sampled delay enters the global Scheduler<Event>. Same-time events keep their insertion order through stable schedule IDs, and dropping a delayed network future cancels its live schedule.

Link latency (distance-based)

link_latency is realism, not chaos: it lives on NetworkConfiguration and is applied whatever the chaos mode. Each ordered IP pair is classified through the installed locality topology, samples its class distribution once at first contact, and keeps that value for the run. The result lands in the same per-pair budget as max_pair_latency and is summed with it. A pair where either endpoint has no locality gets nothing.

ChaosConfiguration

All fault injection settings for the simulated network. Part of NetworkConfiguration.

Clogging

Per-Pair Permanent Latency

Each ordered IP pair samples one fixed latency from this range at first contact and adds it to every delivery on that pair for the whole run (FoundationDB’s SimClogging). An all-zero range disables it. FDB’s MAX_CLOGGING_LATENCY * random01() is the ZERO..MAX case.

Network Partitions

PartitionStrategy variants: Random, UniformSize, IsolateSingle, IsolateZone, IsolateDatacenter, AsymmetricSend, AsymmetricRecv. The two Isolate{Zone,Datacenter} arms need a .cluster() topology and degrade to Random selection without one. The two Asymmetric* arms cut a single node one way, using the same primitives as partition_send_from() / partition_recv_to().

Bit Flips

Partial Writes

Partial Reads

Random Connection Close

Black Hole

A direction that delivers nothing, for the rest of the connection’s life: it accepts writes until its send window is full, then blocks; see Black Holes.

Clock Drift

Buggified Delay

Builder campaigns apply this fault only to sleeps scheduled inside .chaos_duration(). Setup and the post-chaos quiet tail do not consume its RNG draws or receive extra delay (recovery mode also clears the flag outright at the cutoff). A directly constructed SimWorld has no campaign window and applies the configured fault for its whole lifetime, unless you call SimWorld::enter_recovery_mode() yourself.

Connection Failures

ConnectFailureMode variants: Disabled, AlwaysFail, Probabilistic (50% refused, 50% hang).

Handshake Delay

ExplorationConfig

Configuration for frontier-based exploration. Passed to SimulationBuilder::enable_exploration().

Live processes are bounded by 1 + workers regardless of exploration depth. max_runs_per_seed is a ceiling, not a quota: a seed whose root run discovers nothing globally new stops after a single timeline.