Configuring the SimulationBuilder

• The Minimal Builder
• Adding Processes
• Iteration Control
• Invariants
• Chaos
• Every Builder Method
• Putting It All Together

The SimulationBuilder is the glue. It takes your Process, your Workload, your invariants and your chaos configuration, and wires them into a runnable campaign. This chapter walks the calls our key-value simulation needs, then lists every public method with the chapter that explains it.

The Minimal Builder

The smallest simulation has one workload and no processes.

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .workload(KvWorkload::new(100))
    .run()?;
}

This places the workload at 10.0.0.1, runs it under the default iteration control, and returns a SimulationReport. run() returns Err(SimulationError::InvalidConfiguration) when the builder cannot run as written: a fault injector or Chaos::Attrition without chaos_duration, or an instance workload under exploration or check_determinism. In that case no seed ran. Failing seeds are never an Err. They are in the report.

Adding Processes

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(100))
    .run()?;
}

Three servers at 10.0.1.1 to 10.0.1.3, one client at 10.0.0.1. The workload finds the servers through ctx.topology().all_process_ips(). Call .processes() once per role. Each call is a group on its own 10.0.{g+1}.x range, and Process Groups, Tags and Topology covers groups, .cluster() and .tags().

Iteration Control

One seed is one schedule. Bugs hide in rare interleavings, so a campaign runs many seeds. IterationControl has two modes.

UntilCoverageStable is the default, with plateau_seeds: 10 and max_iterations: 1000. It stops once every observed assert_sometimes! and assert_reachable! has fired and coverage has not grown for plateau_seeds consecutive seeds. Tune it with:

#![allow(unused)]
fn main() {
.until_coverage_stable(10, 5_000) // 10 quiet seeds, 5,000 safety cap
}

FixedCount runs exactly n seeds:

#![allow(unused)]
fn main() {
.set_iterations(100)
}

Generated seeds derive from a wall-clock base seed, so two campaigns explore different seeds. To replay, pin them:

#![allow(unused)]
fn main() {
.set_debug_seeds(vec![42, 7891])
.set_iterations(2)
}

set_debug_seeds supplies the first seeds. set_iterations bounds the count. Without the second call the default control keeps drawing fresh seeds after 42 and 7891. Reproducing with FixedCount is the debugging recipe.

Invariants

Invariants run after every simulation step and read the captured tracing timeline. They catch cross-workload properties while the run is still going.

#![allow(unused)]
fn main() {
use std::cell::Cell;
use moonpool_sim::{Invariant, TraceQuery, assert_always};

struct SingleLeader { cursor: Cell<usize> }

impl Invariant for SingleLeader {
    fn name(&self) -> &str { "single_leader" }

    fn observe(&self, q: &dyn TraceQuery, _sim_time_ms: u64) {
        // A cursor scan: only the events since the last step, never the whole timeline.
        for e in q.since("leader_elected", &self.cursor) {
            assert_always!(e.u64("term").is_some(), "leader_elected carries a term");
        }
    }

    fn reset(&mut self) { self.cursor.set(0); }
}

// On the builder:
.invariant(SingleLeader { cursor: Cell::new(0) })
.invariant_fn("kv_commits_name_a_key", |q, _t| {
    for e in q.snapshot("kv_set_committed") {
        assert_always!(e.str("key").is_some(), "kv_set_committed carries a key");
    }
})
}

Events and Invariants covers emission, capture and the query API.

Chaos

enable_chaos takes a list of surfaces, each with a ChaosMode. Random samples the full surface every seed. Swarm enables a random subset of its fault families per seed and turns the rest fully off.

#![allow(unused)]
fn main() {
use moonpool_sim::{Attrition, AttritionScope, AttritionVictims, Chaos, ChaosMode};

SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(200))
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition {
            config: Attrition {
                max_dead: 1,
                prob_graceful: 0.3,
                prob_crash: 0.5,
                prob_wipe: 0.2,
                recovery_delay_ms: None,
                grace_period_ms: None,
                scope: AttritionScope::PerProcess,
                victims: AttritionVictims::Any,
                wipe_budget: None,
            },
            mode: ChaosMode::Swarm,
        },
    ])
    .set_iterations(100)
    .run()?;
}

chaos_duration opens the window in which new faults may be injected. When it closes, the runner cancels the injectors and enters recovery mode: every configuration-driven fault family switches off and held partitions heal. Damage already done stays done. The quiet tail between the cutoff and workload completion is where the system under test recovers, and Chaos in Moonpool describes that lifecycle. Each Chaos variant has its own chapter:

Every Builder Method

Three groups of methods consume no randomness: the masks, the deployment-shape settings (tcp_send_window_bytes, accept_backlog_capacity, link_latency) and trace_level. Adding them to a campaign changes no seed’s schedule. Every Chaos surface, every ranged count and swarm_operations draw from the simulation stream in a fixed order, so adding one shifts every seed. That is intended, and Seed-Driven Reproducibility explains the one-stream rule behind it.

Putting It All Together

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .tags(&[("role", &["primary", "replica"])])?
    .workloads(WorkloadCount::Random(1..4), |_| Box::new(KvWorkload::new(200)))
    .invariant(SingleLeader { cursor: Cell::new(0) })
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::BuggifyKnobs,
    ])
    .swarm_operations()
    .until_coverage_stable(10, 2_000)
    .run()?;
}

The builder creates the simulated world, assigns IPs, seeds the stream, runs the orchestration loop, collects assertions and metrics, and produces the report. Running and Observing reads that report.