Seed-Driven Reproducibility
One number. A single u64. That is all moonpool needs to fully determine a simulation: which connections fail, when packets arrive, whether BUGGIFY triggers, how long disk writes take, what order events process. Same seed, same execution, same bugs. This is the property that makes everything else work.
The RNG Core
Moonpool uses ChaCha8Rng from the rand_chacha crate, seeded from the u64 value. ChaCha8 is fast, produces high-quality randomness, and is deterministic across platforms. The RNG lives in thread-local storage, which is correct because each simulation runs on a single thread.
#![allow(unused)]
fn main() {
// At the start of every simulation run
set_sim_seed(seed);
// Inside the simulation engine, every decision uses the thread-local RNG:
let latency: u64 = sim_random_range(1..50); // network delay in ms
let should_fail: bool = sim_random_f64() < 0.25; // 25% fault probability
let value: f64 = sim_random(); // general-purpose random value
}These are framework-internal functions used by the simulation engine itself. Your application code should use providers.random().random() and providers.random().random_range() instead, which route through the same underlying RNG but go through the provider abstraction.
There is exactly one stream. sim_random(), sim_random_range(), sim_random_f64(), the RandomProvider your code sees, the executor’s choice of which task to poll next, select! branch offsets, the per-seed swarm subsets, buggify activation and firing, and every network, storage and block-device fault coin all draw from the same thread-local RNG. Moonpool keeps no private stream for its own decisions: the framework draws its randomness the way the code it simulates does. This means the order of calls matters. Adding a new sim_random() call anywhere in the simulation shifts every subsequent random value. This is intentional. It means small code changes produce different simulation trajectories, naturally exploring new parts of the state space.
Call Count Tracking
Every RNG call increments a thread-local counter. You can read it with get_rng_call_count(). This sounds mundane, but it is one of the most useful debugging tools in the framework.
When a seed produces a bug, you can narrow down exactly where the execution diverges from expected behavior by watching the call count. “The bug triggers after RNG call 847” tells you precisely which decision in the simulation started the chain of events leading to failure. Combined with breakpoints and logging, this turns a mysterious distributed failure into a step-through debugging session.
The explorer framework takes this further: it records call counts at fork points, creating a “recipe” of count@seed transitions that can replay an exact exploration path.
The Determinism Canary
Everything above assumes nothing escapes the seed. Something always tries to: a HashMap iterated in its randomized order, a static that survives from one run into the next, a wall-clock timestamp, an OS random number pulled in by a dependency. Each one makes the simulation draw something different, or draw at a different moment, and the seed stops reproducing anything. moonpool ships a canary for exactly this, the mechanism madsim’s Runtime::check_determinism uses:
#![allow(unused)]
fn main() {
SimulationBuilder::new()
.check_determinism()
.workload_factory(|| Box::new(MyWorkload::new()))
.run();
}Every seed now runs twice. During the first run, each draw on the simulation stream records a 64-bit fingerprint: a probe of the generator’s state after the draw, taken on a clone so the check consumes no randomness, mixed with the logical clock. During the second run the same hook compares each draw’s fingerprint against the record, and when the run ends the whole record must have been consumed. Because task scheduling, select! offsets, swarm masks and every fault coin are draws on that one stream, any uncontrolled difference in scheduling, ordering, draw count or simulated timing shows up as a fingerprint that does not match, and the seed fails with the always-assertion determinism canary: replay matched the recorded draw sequence, naming the first diverging draw. A replay that matches a prefix and then stops early fails too: the leftover record is the evidence.
Two things to know. It is a canary, not a trace: it tells you that the runs diverged and at which draw, and the call-count tools above take it from there. And it doubles the cost of every seed, so it belongs in a dedicated CI job or a debugging session, not in the default sweep. Like exploration it needs factory workloads; an instance workload would carry its state into the second run and diverge by construction.
Multi-Seed Testing
A single seed tests one execution path. To build confidence, you need many paths. Moonpool’s builder supports two modes:
FixedCount runs a set number of iterations, each with a different random seed:
#![allow(unused)]
fn main() {
SimulationBuilder::new()
.set_iterations(100) // 100 different seeds
// ... workloads ...
.run();
}UntilCoverageStable (the default) keeps drawing seeds until every observed assert_sometimes! / assert_reachable! has fired and code coverage has stopped growing, capped at a safety limit:
#![allow(unused)]
fn main() {
SimulationBuilder::new()
.until_coverage_stable(10, 5_000) // 10 quiet seeds, 5_000 safety cap
// ... workloads ...
.run();
}The power of seed-driven testing compounds over time. Run 1,000 seeds in CI on every commit. Run 100,000 overnight. Each seed explores a different combination of timing, faults, and ordering. Bugs that require three independent unlikely events to coincide will surface within a few thousand seeds because the simulation amplifies failure probability through BUGGIFY.
Debugging a Failing Seed
When CI reports a failure, the output includes the seed:
FAILED seed=17429853261 — connection timeout during leader election
Pin that seed and run it locally with logging enabled:
#![allow(unused)]
fn main() {
SimulationBuilder::new()
.set_iterations(1)
.set_debug_seeds(vec![17429853261])
// ... same workloads ...
.run();
}The simulation replays the exact same execution. Set a breakpoint. Step through. The bug is deterministic now.