The Sim Book

A guide to building and testing distributed systems with moonpool, a deterministic simulation framework for Rust.

Moonpool takes ideas from FoundationDB, TigerBeetle and Antithesis and puts them in one set of Rust crates. You write your system once against provider traits. In production the traits are backed by TokioProviders: real TCP, a real disk, a real clock. In tests the same code runs inside a simulator that is deliberately worse than production: partitions, torn writes, slow machines, reboots, all driven by one seed. No #[cfg(test)]. No mocks. No Sim* copies of your types.

The book has six parts and an appendix:

• Part I explains why simulation testing exists and what moonpool offers. You can read it without any code.
• Part II covers the foundations: determinism, the provider pattern, and the split between the system under test and the test driver.
• Part III builds a simulation step by step: processes, workloads, the SimulationBuilder, chaos, assertions, events, metrics and debugging.
• Part IV wires existing applications (axum, tonic, hyper) into the simulator and takes the same code to production and to the browser.
• Part V covers the raw network layer and the typed RPC crate built on it.
• Part VI covers multiverse exploration and the crash-aware journal built on top of the simulator.
• The Appendix holds the reference tables: assertions, crates, configuration, faults, the calibration tool and the xtask runner.

If you are new, start with The Case for Simulation. If you already know why you want simulation, jump to Why Moonpool Exists for the feature tour, then to Your First Simulation. AI agents working on a moonpool codebase should read the Guide for AI Agents.

Index

• Quick Start Routes
• Foreword
• Part I: Why Simulation Testing
• Part II: Foundations
• Part III: Building Simulations
• Part IV: Simulating Existing Applications
• Part V: Raw Network Simulation
• Part VI: Building on Top
• Appendix

This page is a map of the book. Each entry links to one chapter and says what it covers.

Quick Start Routes

Foreword

• State of Moonpool: the crates, what is stable, what is new, what is missing.

Part I: Why Simulation Testing

• The Case for Simulation: the gap between localhost and production, and why coverage numbers lie.
• Prevention vs Discovery: regression tests guard what we know; simulation finds what we do not know.
• From Mocks to Simulation: why mocks and #[cfg(test)] fail at scale, and how traits replace them.
• A Brief History: FoundationDB, Antithesis, TigerBeetle and what moonpool takes from each.
• Why Moonpool Exists: one short paragraph for each moonpool feature, with links.

Part II: Foundations

• Determinism as a Foundation: threads, I/O and randomness, and how moonpool removes each one.
• The Single-Core Constraint: one thread gives one legal order; Send-bounded traits; no tokio runtime in a simulation.
• Seed-Driven Reproducibility: one u64 seed controls the run; RNG call counts; the opt-in determinism canary.
• The Deterministic Executor: seeded task scheduling, task owners, the CPU gate, kill-on-drop.
• The Provider Pattern: the Providers bundle; TokioProviders and SimProviders.
• Quick Start: Swapping Implementations: one generic function, run on Tokio and in the simulator; the forbidden calls.
• Deep Dive: Why Providers Exist: why not mocks, “never more permissive”, static dispatch.
• The Six Providers: Time, Task, Network, Random, Storage and System, plus Resolver and select!.
• Same Code in Production and Simulation: no Sim* wrappers; hint!, buggify! and cpu_charge are no-ops in production; lean release builds.
• System Under Test vs Test Driver: why Process and Workload are two traits.
• Process: Your Server: the factory, reboots, the context, crash_self.
• Workload: Your Test Driver: setup, run, check; the operation alphabet; slowness and system samples.

Part III: Building Simulations

• Your First Simulation: the plan and the full target program.
• Defining a Process: accept loop, select! on shutdown, ranged counts.
• Writing a Workload: topology lookups, a reference model, why run() must finish.
• Configuring the SimulationBuilder: every builder method in one table, with links.
• Running and Observing: nextest vs xtask, the report, stop conditions, stall guards.
• Process Groups, Tags and Topology: several roles, IP ranges, tags, .cluster() failure domains.
• Chaos Testing vs Simulation: two schools of breaking things, and where faults enter.
• Chaos in Moonpool: every Chaos surface, ChaosMode::Random and Swarm, the chaos window.
• Buggify: Fault Injection: buggify!, buggify_with_prob!, buggify_named!, buggify_knob!, two-phase activation.
• hint!: Naming Fault Sites: the code names a moment, the simulator decides; vetoes; the journal example.
• Swarm Testing: Masks and Budgets: fault masks, budgets, swarm_operations, one workload with many behaviours.
• Attrition: Process Reboots: reboot kinds, max_dead, victims per group, failure domains.
• Correlated Outages and Self-Crashes: Chaos::Outage, OutageVeto, SimContext::crash_self.
• Network Faults: the nine families, partitions, black holes, network masks.
• Storage Faults: storage layering, the barrier-bounded crash model, the lost-synced-write oracle.
• Aiming Storage Faults: LayoutRegion, fault focus, targeted injection, replicated fault patterns.
• CPU Time and Gray Failures: the CPU gate, cpu_charge, SystemProvider, Chaos::GrayFailure, slow machines.
• Custom Fault Injectors: FaultInjector, fault_factory, the FaultContext methods.
• Assertions: Finding Bugs: the fifteen macros, where counts live, how assertions decide when to stop.
• Invariants vs Discovery vs Guidance: three jobs, and what each kind does in the runner and the explorer.
• Always and Sometimes: the five boolean macros.
• Numeric Assertions: the eight numeric macros and watermarks.
• Compound Assertions: assert_sometimes_all! and assert_sometimes_each!.
• Events and Invariants: plain tracing events, the Invariant trait, the fault timeline.
• Application Metrics: per-node Prometheus metrics, metric queries, a metastable failure.
• Designing Workloads That Find Bugs: operation alphabets, swarm, invariant patterns.
• Debugging a Failing Seed: the workflow.
• Reproducing with FixedCount: pin the seed, turn up the logs.
• Reading the Event Trace: event kinds, faults in the timeline, RNG call counts.
• Common Pitfalls: seventeen mistakes and their fixes.
• Discovering Properties: eight attention focuses to find what to assert.

Part IV: Simulating Existing Applications

• Using moonpool-sim Standalone: real HTTP over simulated TCP.
• Where to Draw the Line: fakes vs containers, per dependency.
• Wiring a Web Service: an axum service step by step.
• The hyper Stack: gRPC and HTTP: H2Server, ReconnectingChannel, tonic.
• What You’re Testing (and What You’re Not): the limits of the model.
• Using Providers in Production: TokioProviders, the lean dependency tree, inert fault sites.
• Migrating Existing Code to Providers: the call mapping and the gotchas.
• Simulation in the Browser: the wasm build and the live demo.

Part V: Raw Network Simulation

• Simulating the Network: TCP, not packets; who owns what.
• Typed RPC with moonpool-rpc: driver, endpoints, delivery modes, reply streams, failure monitor.
• What moonpool-rpc Promises: the promises and the seven campaigns that check them.
• Admission, Security and Derive: admission queues, buggify points, TLS, JWT, the derive macro.

Part VI: Building on Top

• Multiverse Exploration: fork at discovery.
• The Exploration Problem: the sequential luck problem.
• The Frontier Controller: recipes, the discovery journal, bug recipes.
• Bounded Workers: a fixed pool of forked workers.
• Exemplars and Continuations: semantic states and the under-explored frontier.
• Multi-Seed Exploration: exploration across many seeds.
• Exploring a Consensus Protocol: a worked procedure.
• A Crash-Aware Journal: moonpool-journal, its commit hints, and its tests under Chaos::Storage.

Appendix

• Assertion Reference: every macro in one table.
• Crate Map: every crate, its features and dependencies.
• Configuration Reference: every config struct with defaults.
• Fault Reference: every fault family with defaults.
• Glossary: the terms.
• Sim Compatibility Checklist: ten checks before code runs in a simulation.
• Calibrating Against a Real Machine: moonpool-calibrate measures a disk and a network.
• Running Simulation Binaries with xtask: cargo xtask sim.

Guide for AI Agents

• The Rules
• Crates and Features
• Providers
• Process, Workload, Context
• SimulationBuilder
• SimulationReport
• Assertions
• Invariants and Traces
• Buggify and Hints
• Chaos Configuration
• FaultContext (custom injectors)
• Exploration
• Low-Level SimWorld
• Staged Adoption
• Code to Read First

This page is a dense map of moonpool for a coding agent that adds deterministic simulation to a system. Every name on it exists in the current source. When this page and the code disagree, the code wins: read the source and update this page with the change. The narrative chapters explain each mechanism; this page shows how to combine them.

The Rules

1. Production code is generic over P: Providers. It calls time.sleep, task.spawn_task, network.connect, storage.open, random.random_range, system.sample. Never tokio::spawn, tokio::time, tokio::select!, std::fs, std::net, thread_rng, Instant::now.
2. The same code runs in production over TokioProviders and in simulation over SimProviders. No Sim* wrapper in application code.
3. Process is the server, rebuilt by its factory on every reboot. Workload is the test driver and survives reboots. Never merge them.
4. Facts go out as tracing events. Cross-process safety goes in an Invariant. Local checks go in assert_always!. Rare paths get assert_sometimes!. Dangerous moments get buggify! or hint!.
5. Every random decision is a counted draw on one seeded stream. BTreeMap, not HashMap. Futures are Send + 'static. No unwrap().
6. Replay before diagnosis: set_debug_seeds(vec![seed]).set_iterations(1) or replay_timeline(seed, recipe).
7. Never delete an assertion that catches a bug. Fix the bug.
8. Validate with nix develop --command cargo fmt, cargo clippy, cargo nextest run, and mdbook build book/ when a public API changes.

Crates and Features

Production: moonpool = { default-features = false, features = ["tokio"] }. Simulation harness: moonpool-sim in dev-dependencies or a separate binary crate. wasm: moonpool-sim with default-features = false.

Providers

OpenOptions::direct_io(DirectIo::{Disabled, Optional, Required}), BlockFile<F> for block arithmetic over one open file (owned *_blocks_aligned forms for zero copy), AlignedBuf (zeroed, for_constraints, truncate, is_aligned_to), LayoutRegion { path, bytes, kind, stripe, shard } to name byte ranges. moonpool_sim::select! replaces tokio::select! with the same grammar. moonpool_sim::executor::spawn(name, fut) is the sim-only raw spawn.

Process, Workload, Context

#[async_trait]
impl Process for Node {
    fn name(&self) -> &str { "node" }
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> { .. }
}

#[async_trait]
impl Workload for Driver {
    fn name(&self) -> &str { "driver" }
    async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> { Ok(()) }  // default
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> { .. }
    async fn check(&mut self, ctx: &SimContext) -> SimulationResult<()> { Ok(()) }  // default, after chaos
}

SimContext methods:

SimulationBuilder

let report = SimulationBuilder::new()
    .processes(3, || Box::new(Node::default()))              // group 0 on 10.0.1.{1..3}
    .tags(&[("role", &["leader", "follower"])])?             // round-robin over the last group
    .workloads(WorkloadCount::Fixed(2), |i| Box::new(Driver::new(i)))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition { config: attrition, mode: ChaosMode::Swarm },
    ])
    .chaos_duration(Duration::from_secs(60))
    .invariant(NoSplitBrain::new())
    .run()?;
if report.is_failure() { report.eprint(); std::process::exit(1); }

No builder method accepts a NetworkConfiguration or StorageConfiguration. Without Chaos::Network the network still has FoundationDB’s always-on faults (random close, clock drift, buggified delay, probabilistic connect failure). Without Chaos::Storage storage has no faults.

SimulationReport

report.eprint() prints it. success_rate(), average_wall_time(), average_simulated_time().

Assertions

Limits: 2048 sites, 256 buckets. Only sometimes-style assertions create discoveries. Full table: Assertion Reference.

Invariants and Traces

tracing::info!(target: "raft", term, leader = %my_ip, "leader_elected");   // emit, in process code

impl Invariant for NoSplitBrain {
    fn name(&self) -> &'static str { "no_split_brain" }
    fn observe(&self, q: &dyn TraceQuery, _sim_time_ms: u64) {
        for e in q.since("leader_elected", &self.cursor) {       // cursor: Cell<usize>
            let term = e.u64("term"); let leader = e.str("leader");
            assert_always!(.., "one leader per term");
        }
    }
    fn reset(&mut self) { .. }                                    // per seed
}

Capture rule: INFO and above, constant message, inside a process or workload task. TraceEvent { seq, time_ms, source, target, level, name, fields } with u64, i64, f64, bool, str accessors. TraceQuery { len(name), since(name, &cursor), snapshot(name) }. Sim faults appear as "sim_fault" events with a kind field. init_sim_tracing(Level) in a binary’s main.

Buggify and Hints

All are no-ops in production. moonpool-journal shows the pattern: four hint! moments per batched commit (one after a failed sync writes the batch again) (an ordered commit’s two durable writes are named inside the simulated write_at_durable, “storage: durable write landed, not synced”), and two per roll onto a recycled spare segment.

Chaos Configuration

Every default and range: Configuration Reference. Every fault: Fault Reference.

FaultContext (custom injectors)

#[async_trait]
impl FaultInjector for Partitioner {
    fn name(&self) -> &str { "partitioner" }
    async fn inject(&mut self, ctx: &FaultContext) {
        while !ctx.chaos_shutdown().is_cancelled() {
            if let Some(ip) = ctx.reboot_random(RebootKind::Crash).ok().flatten() { .. }
            ctx.time().sleep(Duration::from_secs(5)).await.ok();
        }
    }
}

Exploration

.workload_factory(|| Box::new(Driver::default()))
.enable_exploration(ExplorationConfig { workers: 0, max_runs_per_seed: 2400, branching_factor: 4, max_frontier: 512, max_recipe_len: 32 })

workers: 0 is deterministic and fork-free: use it in tests. workers > 0 only in a standalone binary run through cargo xtask sim run <name>. Bugs land in report.exploration.bug_recipes as BugRecipe { seed, recipe }; replay with replay_timeline(seed, recipe). format_timeline / parse_timeline convert "151@8837201 -> 80@1293847". Guidance comes from sometimes-style assertions only. See Multiverse Exploration.

Low-Level SimWorld

For provider tests without the runner: SimWorld::new_with_seed(seed), new_with_network_config(cfg), set_storage_config, set_process_storage_config(ip, cfg), storage_provider(ip), network_provider(ip), time_provider(), step(), has_pending_events(), run_until_empty(), take_faults(), partition_pair(from, to, d), corrupt_file(path, sectors), fail_file_with_eio(path, sectors, EioTarget), simulate_crash_for_process(ip, close_files), set_fault_focus, publish_layout. Drive futures with moonpool_sim::executor::Executor::new(seed).block_on(..) and step the world on Pending (the drive helper in the root AGENTS.md).

Staged Adoption

1. Boundary. Make the code generic over P: Providers. Replace every forbidden call (Sim Compatibility Checklist). Run one seed with no chaos. Add check_determinism() and keep it green.
2. Driver and oracle. Write the Workload with a small reference model (BTreeMap of expected state, not a copy of the server). Reconcile ambiguous results: a timed-out write may have committed.
3. Facts and invariants. Emit tracing events for every protocol decision. Write cursor-based invariants for agreement, validity, persistence, prefix, client semantics.
4. Guidance. Add assert_sometimes! on every retry, timeout and recovery path. Add watermarks for depth. Keep bucket vocabularies coarse.
5. Chaos. Enable Chaos::Network(Swarm), then Storage(Swarm), then Attrition, then Outage, Cpu, GrayFailure. Add hint! moments in storage code. Add buggify! at dangerous choices. Mask what the harness cannot observe.
6. Campaign. Keep the default UntilCoverageStable. Register the binary in xtask (Running Simulation Binaries with xtask). Add exploration when a bug needs a sequence of rare events.
7. Debug. Pin the seed with set_debug_seeds, set_iterations(1), trace_level(LevelFilter::ERROR). Read the event trace. Fix the root cause. Re-enable chaos.

Code to Read First

State of Moonpool

• The crates
• What is stable
• What is new
• What is missing
• How to read this book

Moonpool is a hobby project under active development. It is not production-ready. There is no backward-compatibility policy: when an API changes, the old one is deleted, not deprecated. The workspace is at version 0.8.0. This book documents the framework as it exists today and marks what is new.

The crates

The workspace holds sixteen crates. Ten are published to crates.io. Six are tools and examples that stay in the repository. The Crate Map has the dependency graph.

What is stable

The simulation engine is the oldest and most exercised part. One OS thread, one seeded random stream, logical time that jumps when every task is blocked. The same seed gives the same execution, and SimulationBuilder::check_determinism runs every seed twice to check that claim. The engine runs the example simulations and the RPC campaigns in CI on every commit.

Chaos covers the network (partitions, latency, close, corruption, partial writes), storage (corruption, torn writes, misdirection, sync failures, stalls, the barrier-bounded crash model) and process attrition (graceful reboots and crashes). ChaosMode::Swarm enables a random subset of fault families per seed. buggify_knob! pushes configuration knobs to extremes.

The assertion suite has the 15 macros of the Antithesis taxonomy: always, sometimes, reachable, unreachable, eight numeric forms, assert_sometimes_all! and assert_sometimes_each!. The explorer forks a child at every discovery and keeps the accounting in shared memory.

Events and invariants are plain tracing events. SimulationLayer captures them, and invariants read them through TraceQuery after every step.

Integration runs real HTTP/2, axum and tonic traffic through moonpool-hyper, and the simulator runs in the browser through wasm.

What is new

These landed in the last cycle. Each has a chapter.

• hint!: shipped code names a moment, and the simulator decides what goes wrong there. HintVeto lets a harness cap how many hints strike. See hint!: Naming Fault Sites.
• buggify_named! and set_activation: per-seed scenarios driven by named sites. See Buggify.
• Chaos::Outage and OutageVeto: a correlated outage takes a whole process group down at one instant, and a harness can defer it. SimContext::crash_self and SelfCrash let a workload crash its own process. A .cluster() group may draw zero processes. See Correlated Outages and Self-Crashes.
• CPU time and gray failures: Chaos::Cpu, Chaos::GrayFailure, cpu_charge and SimContext::slowness. A machine stays alive but slow. SystemProvider exposes CPU and disk counters to the code under test. See CPU Time and Gray Failures.
• Aimed storage faults: storage_fault_mask, owner-scoped and byte-range targeted injection, LayoutRegion, FaultContext::durable_bytes, and the replicated patterns Minority, Striped and Rolling per process group. See Aiming Storage Faults.
• moonpool-journal v2: a position-keyed CLSTORE journal with Batch::fits and fits_another, Journal::peek_meta, an Ambiguous tail and hints inside a commit. See A Crash-Aware Journal.
• moonpool-rpc: admission queues per endpoint, the derive, tls and jwt features, and the moonpool-rpc-sim campaigns. See Typed RPC with moonpool-rpc.

What is missing

Multi-process parallel exploration across CPU cores is not implemented. The explorer runs one controller and a bounded pool of short-lived forked workers. There is no deterministic hypervisor: code that bypasses the providers is not simulated.

How to read this book

Part I is motivation. You can read it without touching code, and the ideas apply to any simulation framework. Parts II through VI follow the current API. Every example is adapted from a test or an example in the repository. Part VI describes the most novel piece, fork-based exploration. If you are evaluating whether exploration matters for your system, start there.

The Case for Simulation

• The gap
• The combinatorial problem
• Why coverage numbers lie
• The structural argument
• What simulation needs

You deploy on a Friday afternoon. The tests are green. Code review was thorough. You ran the integration suite twice. You go home.

At 2am your phone lights up. A network partition isolated two nodes for eleven seconds. During recovery, a message arrived out of order. A retry collided with a timeout. The system entered a state nobody on the team had imagined. Data was lost.

The code was correct for the world your tests described. It was not correct for the world production delivered.

The gap

Development environments are clean. The network is localhost. Disks never fail. Clocks agree. Messages arrive in order, exactly once. We know this is fiction. Our test environments reproduce the fiction anyway. We test against a world that does not exist, then we are surprised when the real world finds the bugs we missed.

Production is different. A study of 198 failures in Cassandra, HBase, HDFS, MapReduce and Redis found that 74% were deterministic. Most could be reproduced on three nodes or fewer. 77% could have been caught by a unit test that checked the error path. The bugs were not exotic. They were ordinary mistakes in error-handling code that nobody tested. A later study of network partition failures showed that 80% of catastrophic failures in distributed systems involve a partition, and 27% of those lose data. These are not edge cases. They are Tuesday.

The gap between development and production is not a small oversight. More careful tests do not close it. It is structural.

The combinatorial problem

Take a modest system: three nodes, a leader election, replicated state. Now list what can go wrong. Any node can crash and restart. The network between any pair of nodes can partition. Messages can be delayed, reordered or duplicated. Disk writes can be torn or lost. Clocks can drift.

One scenario looks like this: “Node B crashes during an election while Node A has a write in flight and the link between A and C drops for two seconds.” That is one scenario. How many are there?

With coarse modeling, three nodes, five failure types and ten time steps give thousands of distinct failure histories. Five nodes with realistic granularity give millions. That is before application state: which data was in flight, which transactions were uncommitted, which clients were retrying.

The same problem appears in ordinary services. An e-commerce API with six dimensions (user type, payment method, delivery option, promotion, inventory status, currency) needs 648 combinations for basic coverage. One more option per dimension pushes it past 4,000. A real system has hundreds of dimensions. On one real project, a 300-line feature needed a 10,000-line test PR to keep combinatorial coverage.

This is not a tooling problem. No test framework makes 10,000 hand-written tests sustainable. The state space of a distributed system grows faster than any team can write tests for it.

Why coverage numbers lie

A coverage report says 85% line coverage and 70% branch coverage. These numbers measure how much code the tests execute. They say nothing about how much state space the tests explore.

A distributed system can run the same lines in thousands of orderings with thousands of timing relationships. Line coverage treats all of them as one. Branch coverage is a little better but still blind to interleaving. You can have 100% branch coverage and never test what happens when an election overlaps with a partition during a compaction.

Will Wilson put it precisely: humans cannot enumerate all control-flow paths, which is why we need tests. The same limit makes it impossible for humans to write complete tests. This is not a failure of discipline. Manual tests verify what developers imagined. They cannot verify what developers did not imagine. Bugs live in the places nobody imagined.

The structural argument

Here is the argument in its shortest form.

1. Distributed systems fail in ways that depend on the order of concurrent events.
2. The number of orderings grows combinatorially with system size.
3. Humans write tests for scenarios they can imagine.
4. The scenarios that cause bugs are the ones nobody imagined.

So complete manual testing of a distributed system is not hard. It is impossible.

This does not mean we stop testing. It means we need a different kind of test. Instead of writing scenarios by hand, we generate them. Instead of a clean environment, we test in one that is worse than production. Instead of hoping we covered the important cases, we build infrastructure that explores the failure space for us.

That is what simulation gives us. Not better tests. A different kind of testing.

What simulation needs

Three properties make this kind of testing work. The rest of Part I and all of Part II come back to them.

Determinism. The whole system runs on one thread with one seeded random source. The same seed gives the same execution, so a failure is a number you can replay. See Determinism as a Foundation.

Control of the environment. Time, tasks, network, disk and randomness go through traits with two implementations: the real one for production, the simulated one for tests. The simulated one is where faults are injected. See The Provider Pattern.

Properties, not expectations. Tests state what must always hold and what must sometimes happen. The infrastructure generates the scenarios and checks the properties. See Prevention vs Discovery.

Prevention vs Discovery

• Prevention: guarding what you know
• Discovery: finding what you do not know
• Three principles that enable discovery
• The feedback loop
• Guidance: a third job for assertions
• A spectrum, not a binary

Most teams treat testing as one activity. Write tests, run tests, read the results. But two different activities hide under that one word, and mixing them up is one of the most expensive mistakes in software engineering.

Prevention: guarding what you know

The first kind of testing is prevention. You ship a feature. A user reports a bug. You fix it and write a test that would have caught it. The next time someone touches that code, the test fails before the bug can return.

This is regression testing. It is valuable, well understood and nearly universal. CI pipelines run thousands of these tests on every commit. The mental model is defensive: we build a wall around known-good behavior, one brick per bug report.

Prevention answers one question: did we break what already worked?

Discovery: finding what you do not know

The second kind of testing is discovery. Discovery does not guard known behavior. It searches for failure modes nobody has found yet. It asks: what else is broken?

Prevention tests encode scenarios a developer imagined. Discovery tests generate scenarios no developer imagined. Prevention is a wall. Discovery is a search party sent into unmapped territory.

Most teams have a portfolio that is 100% prevention and 0% discovery. Every test answers a requirement or a bug report. No test was written to find bugs the team does not know about. This is not a criticism. Until recently, discovery at scale was available only to teams that could spend years on infrastructure. The tools have caught up.

Three principles that enable discovery

What does it take to move from prevention to discovery? Three principles, each built on the one before.

Deterministic simulation. Discovery generates huge numbers of scenarios and must replay any that fail. If a bug needs a specific order of network events, we need that exact order again. Determinism makes every execution reproducible from one seed. A failing seed is a bug report anyone on the team can replay. Without determinism, discovery produces ghosts.

Controlled fault injection. Discovery must exercise the paths production hits and development does not: partitions, disk corruption, reordering, clock skew, a crash in the middle of recovery. These are the normal operating conditions of a system at scale. Controlled injection means the simulator decides when and how faults occur, from the same seed, so every fault is reproducible and tunable.

Sometimes assertions. This is the subtle one. Prevention uses assertions that say “this must always be true.” Discovery needs a second kind: “this should sometimes be true.” A sometimes assertion on a retry path does not check that retries work every time. It checks that the simulation reached the retry path at all. Without it, you can run a million simulations and never know whether they explored the interesting parts of the state space or repeated the happy path a million times.

In moonpool these are assert_always! and assert_sometimes!, two of the 15 macros in Assertions: Finding Bugs.

The feedback loop

Put the three principles together. Simulation generates scenarios. Fault injection pushes them into failure territory. Always assertions catch violations. Sometimes assertions tell us whether we explored deeply enough.

When a sometimes assertion never fires, the simulation is not reaching some region of the state space. That is a signal to change something: enable another fault family, change the workload, add concurrent operations. When every sometimes assertion fires and every always assertion holds, we have evidence (not proof) that the system handles the failure modes we care about.

Moonpool turns this loop into the default stop condition. IterationControl::UntilCoverageStable runs seed after seed until every sometimes assertion has fired and code coverage has stopped growing for plateau_seeds seeds in a row. See Running and Observing.

The developer’s job changes. Instead of writing one test per scenario, you define correctness properties and read the results. The infrastructure does the exploration.

Lawrie Green of Antithesis described assertions as memos to two audiences. They tell the computer what to check. They tell the developer what they believe about their system. When an assertion fails on correct code, the developer’s mental model was wrong. That is a finding too.

Guidance: a third job for assertions

Discovery has one more use for assertions. When a simulation reaches a state it has never seen before (a sometimes assertion fires for the first time, a numeric watermark improves), that moment is worth more exploration. Moonpool’s explorer forks the process at exactly that point and continues from the new state with fresh randomness. The assertion is no longer only a check. It is a signal that guides the search. See Invariants vs Discovery vs Guidance and Multiverse Exploration.

A spectrum, not a binary

Prevention and discovery do not compete. Every team needs prevention. Adding even a small amount of discovery changes what you find.

You do not have to jump from zero to full deterministic simulation. Random delays in an integration test are discovery. Property-based tests with random inputs are discovery. One sometimes assertion in an existing simulation is discovery.

The question is not whether to do discovery testing. The question is how far along the spectrum to go. Each step finds bugs that prevention, by construction, never will.

From Mocks to Simulation

• Why mocks fail at scale
• The #[cfg(test)] trap
• The alternative: trait-based simulation
• The fidelity spectrum
• Injected conditions over expected calls
• Where the code names a moment
• Moonpool’s provider pattern

Every experienced developer has a mock story. You spend a day writing mocks for a database client. You specify which method returns what, in which order. The tests pass. Then someone refactors the internal call sequence without changing any external behavior, and every mock breaks. The mocks were not testing correctness. They were testing implementation details.

This is not the fault of any mocking library. It is a structural problem with how mocks work.

Why mocks fail at scale

A mock replaces a dependency with a fake that returns pre-programmed responses. To program those responses, you must know which methods your code calls, in which order, with which arguments. The test author carries a mental model of the whole call stack between the code under test and the mocked dependency.

For a unit test of one function, that is manageable. For an integration test of a distributed protocol with concurrent operations, retries, timeouts and failure handling, it is a maintenance nightmare. Every internal refactor breaks mocks that tested behavior. Every new failure path needs new mock responses. The mock setup grows until it rivals the system it tests.

Mocks also skip layers. Production code talks to a TCP socket. The mock replaces the database client. Between those two layers live connection pooling, serialization, retry logic, timeout handling and error translation. None of that code runs in a mock-based test. You test a different system from the one you deploy.

The #[cfg(test)] trap

Rust developers often reach for conditional compilation. #[cfg(test)] swaps in a test-only implementation with no runtime cost and no trait indirection. But the binary you test is now a different binary from the one you deploy. Different code paths, different struct fields, different behavior.

Suppose a bug lives in the interaction between your retry logic and your connection pool. Your test build replaces the pool with an in-memory stub under #[cfg(test)]. The bug is invisible to the test suite. You have not tested the system. You have tested a system-shaped thing that shares some code with it.

The same trap has a second form: a SimMachine next to Machine, a SimDisk next to Disk, a #[cfg(test)] hook that lets the test reach inside. Each one is a second implementation that the tests exercise and production never runs.

The alternative: trait-based simulation

There is a different approach. Define a trait that describes the interface your code needs. Implement it once for production (real TCP, real disk, real clock). Implement it once for simulation (simulated network, simulated disk, simulated clock). Application code depends on the trait. The simulated implementation runs real logic, not canned responses, so every path production exercises, simulation exercises too.

#![allow(unused)]
fn main() {
pub trait TimeProvider: Clone + Send + Sync + 'static {
    fn sleep(&self, duration: Duration)
        -> impl Future<Output = Result<(), TimeError>> + Send;

    fn now(&self) -> Duration;
}
}

In production, sleep calls tokio::time::sleep. In simulation, sleep registers a timer with the simulated scheduler, and logical time jumps forward when every task is blocked. The application code is identical. No #[cfg(test)]. No conditional compilation.

This is the provider pattern. FoundationDB made the same decision with its INetwork interface: one interface, two implementations (Net2 for production, Sim2 for simulation), zero conditional logic in the database.

The fidelity spectrum

Not every simulated implementation needs full fidelity. There is a spectrum.

No-op. sleep returns at once, send discards the message. Useful for pure logic that happens to call I/O.

In-memory. Messages go into a queue, disk writes into a BTreeMap. Fast and deterministic, but no timing, no failures, no reordering.

Controlled simulation. Messages are delayed by random amounts, connections drop on a fault schedule, disk writes can be torn or corrupted. This is where bugs hide, because the system must handle every way the real world deviates from the happy path. These implementations scale: you write each one once, and every component that uses the trait gets simulation for free. No per-test setup.

Full simulation. A whole cluster of processes with simulated topology, coordinated fault injection and time compression. FoundationDB runs hundreds of simulated processes on one thread and compresses hours of cluster behavior into seconds.

Pick the level by what you test. A serialization function needs no-op I/O. A retry loop needs controlled failure injection. A consensus protocol needs full cluster simulation.

Injected conditions over expected calls

The deepest difference between mocks and simulation is the direction of control.

A mock specifies outputs: “when this method is called with these arguments, return this value.” The author must predict every call. If the code takes another path, the mock panics.

A simulation injects conditions: “connections close with this probability, connects take 1 to 11 ms, a sector written since the last sync may be lost after a crash.” The simulation does not care which methods run or in which order. It cares whether the system recovers correctly whatever happens.

Mock-based tests verify that code follows a specific path. Simulation-based tests verify that code produces correct results on every path the simulation explores. One tests implementation. The other tests behavior.

Where the code names a moment

There is one place where the code under test does talk to the simulator, and it is worth being precise about it. A trait swap cannot know that “the batch is durable but its messages are not sent yet” is an interesting instant. Only the code knows that.

Moonpool lets shipped code name such a moment with hint!("batch durable, not sent").await. The code never names a fault. In production the hint is a no-op. In simulation the simulator decides whether to strike there and how, from the fault families the seed enabled. The code’s own rare-but-legal choices use buggify!: an early timeout, a small buffer, a slow path. Both are inert outside a simulation, so the shipped binary carries them at no cost. See hint!: Naming Fault Sites and Buggify.

Moonpool’s provider pattern

This is what moonpool implements. Every interaction with the outside world goes through one of six traits in moonpool-core: TimeProvider for clocks and timers, TaskProvider for spawning, NetworkProvider for connections, StorageProvider for files, RandomProvider for randomness, SystemProvider for CPU and disk counters. The Providers trait bundles them, and TokioProviders is the production bundle.

Your code calls time.sleep() instead of tokio::time::sleep(), and task.spawn_task() instead of tokio::spawn(). In production these call the real runtime. In simulation they feed a deterministic scheduler where every timer, message and disk operation is controlled, reproducible and open to fault injection.

No mocks to maintain. No #[cfg(test)] to split your binary. The same code, in a world that is deliberately worse than reality. Part II covers the traits in The Six Providers and the rule behind them in Same Code in Production and Simulation.

A Brief History

• FoundationDB’s bet (2009-2015)
• Antithesis generalizes it (2020 to now)
• The adoption wave (2020 to now)
• Where moonpool draws from
• The unifying philosophy

Simulation-driven development did not appear from nowhere. It grew over fifteen years across three eras. Each era widened who could use it and how.

FoundationDB’s bet (2009-2015)

In 2009 a small team set out to build a distributed transactional database. Before they wrote the database, they spent about two years on a simulator. No storage engine, no query layer, no client protocol. Just the simulation.

The thesis was simple and, to most observers, insane. Distributed systems fail in ways that depend on the order of concurrent events. The only way to test those orders systematically is to control them completely.

They built Flow, a C++ extension that compiles actor-style code to single-threaded callbacks. Every connection, disk operation and timer goes through one interface, INetwork, with two implementations: Net2 for production (real TCP over Boost.ASIO) and Sim2 for simulation (in-memory buffers with deterministic delays). One seeded random number generator controls everything. Same seed, same execution, every time.

Two techniques made the simulation unusually effective. BUGGIFY marks points in the code where a rare but legal behavior can be forced: a packet sent out of order, a truncated disk write, a skipped timeout, a shrunk buffer. Each site is activated at random once per run and then fires with a configurable probability. BUGGIFY widens the contract under test far beyond what any specification documents. It tests not what the system is supposed to handle but what it can handle.

Correlated failures came from a Hurst-exponent model. In a real datacenter, a failing drive makes its rack neighbors more likely to fail. Independent-failure models make cascades astronomically rare. FoundationDB turned the correlation up so that failure patterns a datacenter sees once in years appear in simulation in milliseconds.

The scale was large. Tens of thousands of simulations every night, each simulating minutes to hours of cluster time under heavy fault injection. A physical cluster called Sinkhole, real motherboards wired to programmable power switches that toggled continuously, served as the reality check. Sinkhole never found a database bug that the simulator had missed. It found bugs in other software and in hardware.

The cost was equally large: a custom language, a simulation-first architecture and two years of infrastructure before the product. For more than a decade FoundationDB’s simulation was a benchmark others admired but could not copy.

Antithesis generalizes it (2020 to now)

Will Wilson, one of the original FoundationDB engineers, co-founded Antithesis around a different question: can the same techniques apply to any software, without a rewrite?

The answer was a deterministic hypervisor. Instead of building simulation into the system from day one, Antithesis intercepts nondeterminism at the virtual machine. Any program in any language becomes a pure function from an input byte stream to a sequence of states. Save, restore, fork, replay. No Flow. No custom language. No architectural prerequisite.

On top of the hypervisor sits a guided exploration engine. The SDK provides declarative assertions that say what to explore without saying how: Sometimes (reach this condition at least once), Always (this invariant holds), SometimesAll (drive the search across a frontier of sub-goals). The platform combines these signals with coverage-guided forking and adaptive search.

The results validated the approach. Customers with mature, well-tested systems found new bugs within two to three weeks of onboarding: deep concurrency issues, subtle data corruption, recovery defects that survived years of conventional testing. The engine needs little domain knowledge. In one demonstration Antithesis beat the game Gradius by watching three bytes of memory and maximizing time alive.

The adoption wave (2020 to now)

The third era is happening now. Simulation-driven development is spreading beyond the teams that invented it.

TigerBeetle, a financial transactions database, adopted deterministic simulation from day one. Its simulator adds storage faults drawn from real hardware failure modes: misdirected reads and writes, phantom writes, torn writes, read corruption. Dropbox used simulation to validate its sync engine rewrite. WarpStream applied deterministic simulation to its whole SaaS platform. In Rust, the madsim crate showed that a tokio-compatible deterministic runtime could drive existing async code.

Clever Cloud uses simulation-driven development to build Materia, a distributed multi-model multi-tenant database. Thirty minutes of simulation covers roughly 24 hours of equivalent chaos testing.

What took an elite team two years of custom infrastructure in 2009 is becoming available to small teams on ordinary languages and frameworks. The investment is shrinking. The bugs found per engineering hour are growing.

Where moonpool draws from

Moonpool takes one idea from each era, and the rest of the book names the source whenever it matters.

From FoundationDB: the single-threaded simulator with one random stream, INetwork as the provider pattern, BUGGIFY as buggify! and hint!, the simulated worker’s please_reboot as attrition, and fdbrpc as the model for moonpool-rpc.

From TigerBeetle: the storage fault model, from torn and misdirected writes to phantom writes and sync failures, and the discipline of aiming faults so a replica set keeps a clean copy.

From Antithesis: the assertion taxonomy and the idea that a discovery is a point worth forking from.

From the paper behind CLSTORE (Alagappan et al., FAST ’18): a journal that tells a crash from corruption and names what was damaged, built on the simulator’s storage faults.

The unifying philosophy

Across all three eras the philosophy is the same. Make execution deterministic so bugs are reproducible. Inject faults that are worse than production so a system that survives is overqualified for the real world. Explore systematically so bugs are found by infrastructure, not by imagination.

The tools change. The languages change. The accessibility changes. The philosophy does not.

Why Moonpool Exists

• What already existed
• Where moonpool sits
• Same code in production and simulation
• What moonpool offers today
• What makes it different

After ten years of operating distributed systems, one pattern keeps repeating: the bugs that hurt most are the ones nobody imagined.

You build a system. You test the scenarios you can think of. You deploy. Then a partition overlaps with a retry storm during a rolling upgrade, and the system enters a state nobody on the team anticipated. The code was not wrong. It was never tested against that combination of events.

The combinatorial problem from The Case for Simulation makes this inevitable. You cannot test what you do not know. Manual tests give regression coverage, not absence of bugs. That realization led to moonpool: the testing toolbox I wish I had when I started building distributed systems.

What already existed

The answer was scattered across several projects and two decades of work. FoundationDB showed that a whole cluster can run on one thread with one seeded random stream, and that fault injection on top of that finds more bugs in a night than production finds in a year. TigerBeetle took the same idea to the disk, with faults drawn from real hardware failure modes. Antithesis turned assertions into a declarative SDK and used them to guide a search that forks at every discovery. A Brief History tells that story.

Nobody had put these pieces together as a Rust library you add with one line in Cargo.toml. That is the gap moonpool fills.

Where moonpool sits

Moonpool is a library-level simulation framework. There is no hypervisor, no custom virtual machine, no special runtime. You add a dependency, write your code against six provider traits, and your system becomes simulatable. Simulations run as ordinary cargo test invocations in your CI pipeline.

This is a deliberate trade. Antithesis’s hypervisor simulates any software without modification. Moonpool asks that your I/O go through the providers, which means designing for simulation from the start or refactoring existing code (Migrating Existing Code to Providers). In exchange you get zero-cost abstractions in production, full control over fault injection, and no service to call.

Same code in production and simulation

One rule shapes every API in moonpool: the code you test is the code you ship. Shipped code is never wrapped in a Sim* variant. There is no SimMachine, no SimDisk, no test-only hook. Code depends on provider traits. TokioProviders backs them in production, and the simulator’s providers back them in tests.

When shipped code knows that an instant is interesting, it names the moment with hint!("label").await. In production that is a no-op. In simulation the simulator decides whether to strike there and how, under the seed’s chaos, its swarm masks and its budgets. The code’s own rare-but-legal choices use buggify!, buggify_with_prob! or buggify_named!, inert outside a simulation. The rule and its consequences are in Same Code in Production and Simulation.

What moonpool offers today

Every feature below has a chapter. This is the map.

Six provider traits. TimeProvider, TaskProvider, NetworkProvider, RandomProvider, StorageProvider and SystemProvider in moonpool-core, bundled by the Providers trait. TokioProviders is the production bundle. The traits are Send + Sync, so your code keeps its Arc<RwLock<_>> and DashMap. See The Provider Pattern and The Six Providers.

A deterministic executor. One OS thread, seeded task scheduling, logical time that jumps when every task is blocked. moonpool_sim::select! replaces tokio::select! so branch order comes from the seed, not from process entropy. See The Deterministic Executor.

Process and Workload. A Process is the system under test. The simulator creates a fresh one on every boot. A Workload is the test driver. It survives reboots, drives requests and checks results. See System Under Test vs Test Driver.

The SimulationBuilder. .processes() and .cluster() register process groups on their own subnets, .tags() spreads roles, .enable_chaos() picks surfaces, .chaos_duration() bounds the window, .invariant() and .metrics_factory() add observers, .until_coverage_stable() and .set_debug_seeds() control iterations, and .run() returns a SimulationReport. See Configuring the SimulationBuilder and Process Groups, Tags and Topology.

Chaos surfaces. Chaos::Network, Chaos::Storage, Chaos::Attrition, Chaos::Outage, Chaos::BuggifyKnobs, Chaos::Cpu and Chaos::GrayFailure, each with a ChaosMode. ChaosMode::Random turns everything on. ChaosMode::Swarm enables a random subset per seed. See Chaos in Moonpool.

Buggify and hints. buggify! sites are activated once per run and fire with a probability. buggify_named! with set_activation builds per-seed scenarios. buggify_knob! pushes a configuration knob to an extreme. hint! names a moment and leaves the fault to the simulator, with HintVeto as a harness budget. See Buggify and hint!.

Swarm testing. One workload, many behaviors. network_fault_mask, storage_fault_mask and swarm_operations give each seed its own subset of fault families and operations, so a seed with few faults reaches deep states and a seed with many faults stresses recovery. See Swarm Testing.

Attrition, outages and self-crashes. Attrition reboots processes gracefully or by crash under a regime with a max_dead budget. Chaos::Outage takes a whole group down at one instant, and OutageVeto lets the harness defer it. SimContext::crash_self lets a workload crash its own process and get a SelfCrash handle. See Attrition and Correlated Outages and Self-Crashes.

Network faults. Connect delays, random close, bit flips, partial writes, packet loss and partitions, all on real byte streams through NetworkProvider. See Network Faults and Simulating the Network.

Storage faults. One StorageProvider, one StorageFile, OpenOptions::direct_io for database-shaped needs and BlockFile for block arithmetic. Faults follow TigerBeetle and FoundationDB: corruption, torn and misdirected writes, phantom writes, sync failures, stalls, and the barrier-bounded crash model where every sector written since the last sync resolves on its own. A lost-synced-write oracle fails any run where a sector a sync called durable changes across a crash. See Storage Faults.

Aimed storage faults. Injection scoped to an owner and a byte range, LayoutRegion as a format-neutral map, and ReplicatedFaults patterns (Minority, Striped, Rolling) that damage replicas while a clean copy survives somewhere. See Aiming Storage Faults.

CPU time and gray failures. Chaos::Cpu charges every poll to the process’s one core. cpu_charge lets shipped code declare a cost. Chaos::GrayFailure keeps a machine alive and slow, and SimContext::slowness tells a workload about it. SystemProvider exposes the counters your code reads to detect it. See CPU Time and Gray Failures.

Custom fault injectors. fault_factory builds a fresh FaultInjector per timeline, with a FaultContext that can crash, restart, read durable_bytes and read workload state. See Custom Fault Injectors.

Fifteen assertion macros. assert_always!, assert_sometimes!, assert_reachable!, assert_unreachable!, assert_always_or_unreachable!, eight numeric forms, assert_sometimes_all! and assert_sometimes_each!. See Assertions: Finding Bugs and the Assertion Reference.

Events and invariants. Correctness facts are plain tracing events. SimulationLayer captures them into a timeline, and an Invariant reads them through TraceQuery after every step. The same events feed production observability. See Events and Invariants.

Metrics. moonpool-prometheus scrapes the counters, gauges and histograms your code already keeps into the report, per node and across seeds. See Application Metrics.

Debugging. A failing seed is a number. set_debug_seeds with set_iterations(1) replays it, and the event trace reads backwards from the first violation. See Debugging a Failing Seed.

HTTP and gRPC. moonpool-hyper supplies HyperExecutor, HyperTimer and HyperIo, so axum, tonic and plain hyper run unchanged on the providers. See The hyper Stack and Wiring a Web Service.

Typed RPC. moonpool-rpc is FoundationDB’s fdbrpc without its wire format: interfaces as data, delivery modes, streams, per-endpoint EndpointQueue admission, buggify sites inside the runtime, and derive, tls and jwt features. moonpool-rpc-sim runs six campaigns in CI. See Typed RPC with moonpool-rpc and Admission, Security and Derive.

Production and the browser. The lean build default-features = false, features = ["tokio"] keeps the simulator out of your release binary. The simulator itself compiles to wasm32-unknown-unknown, and the book embeds a browser demo. See Using Providers in Production and Simulation in the Browser.

Multiverse exploration. moonpool-explorer forks the process at every discovery, records a replay recipe of (rng_call_count, seed) breakpoints, and schedules bounded continuations through a pool of short-lived workers. Coverage comes from sancov edges under cargo xtask sim run. See Multiverse Exploration.

A crash-aware journal. moonpool-journal implements CLSTORE over the providers: positions in any order, a crash told apart from corruption, damage named for repair from peers, and hint! inside every commit. See A Crash-Aware Journal.

Tooling. cargo xtask sim list | run | run-all runs the simulation binaries, and moonpool-calibrate measures a real host to produce latency constants. See Running Simulation Binaries with xtask and Calibrating Against a Real Machine.

What makes it different

Many frameworks stop at network-level fault injection. Moonpool adds three things.

First, storage simulation at the same fidelity as the network, aimed so that replicated data keeps a clean copy and the system has something to repair from.

Second, fault sites in shipped code. hint! lets the code name the moment and the simulator pick the fault, so the production binary carries its own test plan at no cost.

Third, fork-based exploration. The sequential luck problem (a bug that needs n unlikely events in a row has probability p^n) is the central bottleneck of simulation testing. Instead of hoping one timeline walks through every door, moonpool forks at each door and explores both sides.

Moonpool is not only a simulation engine. It is a simulation engine with a search strategy for the state spaces where bugs need a sequence of unlikely events to appear.

Determinism as a Foundation

• The Three Enemies
• Moonpool’s Answer
• Where Time Lives
• What This Gives You

A distributed system dropped writes for 47 seconds. The logs show three nodes that disagreed about leadership. The window has passed. The cluster healed itself. You cannot reproduce the failure on your laptop. You cannot reproduce it in staging. You spend three days in the logs, you form a theory, you ship a fix you are 60% sure about, and you wait for it to happen again.

This is debugging without determinism. The bug is real, but the conditions that caused it are gone. You cannot replay the execution. You cannot step through it. You cannot even confirm that your fix addresses the right cause.

The Three Enemies

A Rust program that uses async networking has three sources of non-determinism: thread scheduling, real I/O and random numbers.

Thread scheduling. The OS decides when a thread runs. Two threads that race on a shared counter increment it correctly 999,999 times, then corrupt it once. The interleaving that triggers the bug depends on CPU load, on thermal throttling, on how many browser tabs are open. A multi-threaded tokio runtime moves tasks across a thread pool. The order in which tasks resume after an .await belongs to the scheduler. Run the program twice and you get two execution orders.

Real I/O. The outside world injects randomness. A TCP packet arrives 2 ms late. A DNS lookup takes 800 ms instead of 5 ms. A disk write returns ENOSPC because another process filled the partition. Your code is a deterministic function, but its inputs are chaos.

Random numbers. rand::rng() pulls entropy from the OS. Two runs with identical inputs make different random choices: different retry delays, different leader elections, different shard assignments.

A distributed-systems bug usually needs all three at once: a specific interleaving, during a specific network delay, with a specific disk latency. To reproduce the bug you must reproduce all three. That is close to impossible.

Moonpool’s Answer

Moonpool brings all three sources under one seed.

One thread, one executor. A simulation runs on moonpool_sim::executor::Executor, a single-threaded executor that moonpool owns. There is no tokio runtime inside a simulation. When several tasks are ready, the executor picks the next one with a seeded draw. The Single-Core Constraint and The Deterministic Executor explain this.

Providers instead of raw I/O. Every system call your code makes goes through a trait: TimeProvider, TaskProvider, NetworkProvider, RandomProvider, StorageProvider and SystemProvider. In production the trait calls tokio. In simulation the trait calls the simulated world. The code in between is identical. The Provider Pattern covers this.

One random stream. The simulation has exactly one ChaCha8Rng, seeded from a u64. The executor’s scheduling draws, the select! branch offsets, the network and storage fault coins, buggify activation, the per-seed swarm masks and every RandomProvider call your code makes come from that one stream. Moonpool keeps no private RNG for its own decisions. Seed-Driven Reproducibility covers this.

With all three under control, a simulation is a pure function of its seed. The same seed gives the same execution, bit for bit. A different seed explores a different interleaving, a different fault schedule, a different set of random choices.

Where Time Lives

The event side of the guarantee is explicit. One Scheduler<Event> owns logical time, cancellation and a monotonic sequence number. Events with the same timestamp run in sequence order, so same-time events are FIFO without any dependence on heap internals. Scheduling into the past clamps to the current time.

The scheduler does not own the simulated resources. NetworkSimulation owns connections, topology, network faults and network wakers. StorageEngine owns files, open handles, disk episodes and storage wakers. Each component consumes its own events and returns ordered effects for the coordinator, SimWorld, to schedule. Random decisions happen inside the component that owns the state. Global ordering happens in one scheduler. Wakers run only after the world lock is released. This split is what makes replay exact: there is one place where “what happens next” is decided.

What This Gives You

A failing seed turns a production incident into a local debugging session. Instead of “it happened once between nodes 7 and 12 under load”, the report ends with:

  Faulty seeds: [8839214571]

You pin that seed with set_debug_seeds(vec![8839214571]), run one iteration, and set a breakpoint. The bug is there, every time. You fix it. You run 10,000 more seeds. They pass.

This is not aspirational. FoundationDB ran 5 to 10 million simulation iterations per night, each with a different seed. Their physical validation cluster, real servers with programmable power switches, never found a database bug that simulation had missed. Determinism made that possible.

Everything in moonpool builds on this foundation. Chaos, assertion coverage, swarm testing, multiverse exploration: all of it rests on one guarantee. Given the same seed, the simulation produces the same result. The rest of this part explains how moonpool keeps that guarantee.

The Single-Core Constraint

• One Thread, One Order
• How We Get There
• Send-Bounded Traits
• What This Costs

Moonpool runs every simulation on one OS thread. This is not a limitation we accept with regret. It is the first design decision, and every other decision follows from it.

One Thread, One Order

A multi-threaded tokio runtime uses a work-stealing thread pool. When a task yields at an .await, any thread in the pool can pick it up. Two tasks that resume “at the same time” run in either order, and the order changes between runs. This is fine for production throughput. It is fatal for a simulation that must replay.

With one thread there is exactly one execution order for a given set of ready tasks. Task A runs before task B, or B before A, and our scheduler makes the choice, not the OS. The RNG picks the order. The seed determines the RNG. Done.

How We Get There

Every simulation runs on the moonpool deterministic executor:

moonpool_sim::reset_sim_rng();
moonpool_sim::set_sim_seed(seed);

let mut executor = moonpool_sim::executor::Executor::new(seed);
executor.block_on(async move {
    // the entire simulation runs here
});

One OS thread drives the whole simulation. There is no work-stealing pool and no preemption between yield points. When a future resumes, it resumes on the thread that suspended it. Which ready task runs next is a seeded draw from the ready queue. The same seed gives the same order, bit for bit. A different seed explores a different interleaving. A FIFO scheduler would run ready tasks in registration order on every seed, forever, and would hide every race between them. The Deterministic Executor covers the mechanism.

There is no tokio runtime inside a simulation. A bare tokio::spawn panics at runtime, because there is no runtime to spawn onto. Spawn through ctx.task().spawn_task() or moonpool_sim::executor::spawn(). Sleep through ctx.time().sleep(). Select with moonpool_sim::select!.

Send-Bounded Traits

Here moonpool diverges from earlier single-threaded designs. A single-threaded runtime could drop Send bounds and let you put Rc<RefCell<T>> across .await points. We deliberately do not.

Provider traits use native async-fn-in-trait with Send + Sync + 'static supertraits, and every async method returns impl Future<…> + Send:

pub trait TimeProvider: Clone + Send + Sync + 'static {
    fn sleep(&self, duration: Duration) -> impl Future<Output = Result<(), TimeError>> + Send;
    fn now(&self) -> Duration;
    // ...
}

The traits the runner stores as dyn objects (Process, Workload, FaultInjector) use #[async_trait] without ?Send, with the same Send + Sync + 'static supertraits.

We pay for Send to get interoperability, not parallelism.

The runtime still runs on one thread. No future ever moves between threads at runtime. But the type system commits to Send + Sync at every trait boundary, so your code can use the ecosystem it already knows:

• Arc<RwLock<State>> for shared state
• DashMap for concurrent maps
• Arc<AtomicBool> for flags
• Send-bounded spawning: ctx.task().spawn_task("name", fut) in the sim, tokio::spawn in production

A team that adopts moonpool to test an existing service should not rewrite its type hierarchy. If the production code uses Arc<RwLock<…>> (it does), moonpool runs it as is. The simulation drives those types on one thread, but the types look exactly like production.

There is no LocalSet, no spawn_local and no build_local anywhere in moonpool. Spawned futures are Send + 'static, as tokio::spawn requires.

What This Costs

Send + Sync rules out a few patterns: raw Rc, and non-Send futures from external crates. In practice the patterns that matter (locks, channels, spawned tasks) are already Send-friendly.

The other cost is CPU parallelism inside one run. A simulation of 20 nodes runs on one core, not 20. This rarely matters. Simulated time is compressed: one core simulates hours of cluster time in seconds of wall-clock time, because the scheduler jumps straight to the next event. And we run many simulations in parallel across cores, each with its own seed. A 16-core machine runs 16 independent simulations at once, each deterministic on its own thread.

Production code is unaffected. The same code that runs single-threaded in the simulation runs on a multi-threaded tokio runtime in production. The provider pattern, covered next, makes that swap invisible to the code.

Seed-Driven Reproducibility

• The RNG Core
• Call Count Tracking
• The Determinism Canary
• Many Seeds
• Debugging a Failing Seed

One number. A single u64. That is all moonpool needs to determine a simulation in full: which connections fail, when packets arrive, which buggify sites fire, how long disk writes take, which task polls next. Same seed, same execution, same bugs. This property makes everything else work.

The RNG Core

Moonpool uses ChaCha8Rng from the rand_chacha crate, seeded from the u64. ChaCha8 is fast, produces good randomness and is deterministic across platforms. The generator lives in thread-local storage. That is correct because each simulation runs on one thread, and parallel test threads never share a stream.

// The runner does this at the start of every iteration
moonpool_sim::reset_sim_rng();
moonpool_sim::set_sim_seed(seed);

// Inside the engine, every decision draws from the thread-local stream
let latency: u64 = sim_random_range(1..50);   // a message delay
let fail: bool = sim_random_bool(0.25);       // a fault coin, always one draw
let ratio: f64 = sim_random_f64();            // FDB's random01()

sim_random, sim_random_range, sim_random_f64 and sim_random_bool are the framework’s own entry points. Your application code uses providers.random().random() and providers.random().random_range() instead. The simulation’s SimRandomProvider routes those calls to the same stream.

There is exactly one stream. The executor’s choice of the next task, the select! start offset, the per-seed swarm masks, buggify activation and firing, every network, storage and CPU fault coin, and every RandomProvider call your code makes all draw from the same generator. Moonpool keeps no private stream for its own decisions. So the order of draws matters. Add one sim_random() call anywhere and every later value shifts. This is intended: a small code change gives a different trajectory and explores new state.

Call Count Tracking

Every draw increments a thread-local counter, readable with rng_call_count(). This sounds mundane. It is one of the most useful debugging tools in the framework.

When a seed produces a bug, “the bug triggers after draw 847” tells you which decision started the chain of events. set_rng_breakpoints(vec![(count, new_seed)]) reseeds the stream when the count passes count. The explorer records these count@seed pairs at each discovery and replays an exact path with SimulationBuilder::replay_timeline(seed, recipe). Because scheduling and select! offsets are draws on the same stream, a recipe replays the task order along with the faults. Multiverse Exploration builds on this.

The Determinism Canary

Everything above assumes nothing escapes the seed. Something always tries to: a HashMap iterated in its randomized order, a static that survives from one iteration into the next, a wall-clock timestamp, an OS random number from a dependency. Each one makes the simulation draw something different, or draw at a different moment, and the seed stops reproducing. Moonpool ships a canary for exactly this, the mechanism madsim’s Runtime::check_determinism uses:

SimulationBuilder::new()
    .check_determinism()
    .workload_factory(|| Box::new(MyWorkload::new()))
    .run()?;

Every seed now runs twice. During the first run, each draw records a 64-bit fingerprint: a probe of the generator’s state after the draw, taken on a clone so the check consumes no randomness, mixed with the logical clock. During the second run the same hook compares each draw against the record. At the end the whole record must have been consumed. Any uncontrolled difference in scheduling, draw count or simulated timing shows up as a mismatch, and the seed fails with the always-assertion determinism canary: replay matched the recorded draw sequence, naming the first diverging draw. A replay that stops early fails too. The replay is also judged as a full run: a workload error, an always-violation or a deadlock fails the seed even when every fingerprint matched.

Two things to know. It is a canary, not a trace: it says that the runs diverged and at which draw. The call-count tools take it from there. And it doubles the cost of every seed, so it belongs in a dedicated CI job or a debugging session. Like exploration it needs factory workloads (workload_factory or workloads), because an instance workload would carry its state into the second run.

The end-to-end tripwire for moonpool itself is crates/moonpool-sim/tests/determinism.rs: two full SimulationBuilder runs of racing workloads on the same seed must produce byte-identical traces.

Many Seeds

One seed tests one path. Confidence needs many. The builder has two iteration modes.

UntilCoverageStable is the default: plateau_seeds: 10, max_iterations: 1000. The run keeps drawing seeds until every observed assert_sometimes! and assert_reachable! has fired and code coverage has not grown for 10 consecutive seeds, capped at 1,000. Under cargo xtask sim run the coverage signal is real sancov edge coverage. Otherwise it is assertion-slot coverage. Tune it with:

SimulationBuilder::new()
    .until_coverage_stable(10, 5_000)   // 10 quiet seeds, 5,000 cap

FixedCount runs a set number of seeds:

SimulationBuilder::new()
    .set_iterations(100)

Each seed also draws its own swarm: with ChaosMode::Swarm a seed enables a random subset of fault families, and swarm_operations() enables a random subset of each workload’s operations. Two seeds therefore differ in which behaviours are even possible, not only in timing. Swarm Testing explains why this finds more bugs than every fault on at once.

Seed-driven testing compounds. Run 1,000 seeds in CI on every commit. Run 100,000 overnight. A bug that needs three unlikely events to coincide surfaces within a few thousand seeds, because buggify amplifies the probability of each event.

Debugging a Failing Seed

When a run fails, the report lists the seeds:

  Faulty seeds: [17429853261]

Pin that seed and run it alone:

SimulationBuilder::new()
    .set_iterations(1)
    .set_debug_seeds(vec![17429853261])
    .trace_level(tracing::level_filters::LevelFilter::DEBUG)
    // ... the same processes and workloads ...
    .run()?;

The simulation replays the exact same execution. trace_level changes what the timeline captures and never changes scheduling or randomness, so the seed replays identically at any level. Set a breakpoint. Step through. The bug is deterministic now. Debugging a Failing Seed walks through the full procedure.

The Deterministic Executor

• Why Not tokio?
• Randomized and Deterministic
• Where Is the Reactor?
• The Drive Loop
• Task Owners and the CPU Gate
• Coming from tokio
• Kill-on-Drop
• Exploration Process Safety
• Verifying It Yourself

Moonpool simulations do not run on tokio. They run on moonpool_sim::executor::Executor, a single-threaded executor whose every scheduling decision derives from the iteration seed. This chapter explains why it exists, how it works, and what changes if you come from tokio.

Why Not tokio?

Two reasons, one practical and one fundamental.

The practical one: seeding tokio’s runtime randomness needs the unstable RngSeed API, which forces --cfg tokio_unstable onto every crate in every consumer’s workspace. For a project pinned to stable Rust that is a hard adoption barrier.

The fundamental one: tokio’s current-thread scheduler is a FIFO queue, and FIFO is the wrong policy for a bug-finding simulator. When one event makes two tasks runnable, FIFO runs them in registration order, on every seed, forever. Any race between those two tasks is structurally unexplorable. No number of seeds will ever try the other order. A simulation executor must treat “which runnable task goes next” as a decision to explore, like a message latency or a fault timing.

Randomized and Deterministic

These two properties sound contradictory. They are not. Deterministic means “same seed, same execution, bit for bit”, not “one fixed schedule”. The executor keeps its ready tasks in a Vec and picks the next one with swap_remove at a seeded-random index. madsim uses the same distribution, and FoundationDB’s sim2 randomizes the same way:

seed 42:  task-3, task-0, task-2, task-1, ...
seed 42:  task-3, task-0, task-2, task-1, ...   (always)
seed 43:  task-1, task-3, task-0, task-2, ...

Each seed replays exactly. The population of seeds covers the ordering space. Ordering bugs, the kind that survive code review because both orders look fine, live exactly there.

The scheduling index is a draw on the simulation’s one random stream, the same sim_random_range a process reaches through ctx.random(). There is no private scheduling RNG. Which task runs next is counted like every other decision, so an exploration recipe replays the schedule along with the faults. The executor draws nothing when exactly one task is runnable: that is a forced choice, not a decision, and the dominant one-event-wakes-one-task path leaves the stream untouched.

Where Is the Reactor?

There is none. A production runtime pairs its executor with a reactor (epoll, a timer wheel) that turns OS events into waker calls. In the simulation, SimWorld and its Scheduler<Event> play that role. The scheduler owns logical time, stable same-time ordering and cancellation. It dispatches events to NetworkSimulation and StorageEngine, which own their resource state and their waker registries. The wake batches they return are invoked after the world lock is released. The executor answers one question only: of the tasks runnable right now, which one runs next?

The Drive Loop

Executor::block_on(main) pins the orchestrator future on the stack as the driver. It never enters the ready queue and is never scheduled randomly:

loop {
    poll driver ── Ready? ─────────────► return
    run_until_stalled()   // poll ready tasks in seeded-random order
                          // until none is runnable
    (driver not woken AND queue empty) ► panic: deadlock (with the seed)
}

The orchestrator’s step loop interleaves the world with the task pool:

sim.step();                              // process one virtual event
crate::executor::until_stalled().await;  // let every woken task run

The driver is re-polled only after a complete drain. So when until_stalled().await returns, every task that was runnable has been polled to Pending or to completion. Under tokio this was approximated with yield_now and FIFO ordering. Under randomized scheduling “the back of the queue” does not exist, so the executor provides the contract directly.

Inside a task, use executor::yield_now() instead. It reschedules the task at a seeded-random position. until_stalled() is driver-only and asserts it in every build profile: a task that awaits “run until nothing is runnable” waits for itself.

One consequence deserves a warning sign. A task must not busy-wait on progress that only the driver can make. A loop { yield_now().await } that polls a flag set by a simulation event keeps the ready queue non-empty forever, so the drain never finishes and sim.step() never runs. The executor kills such a loop after 1,000,000 polls in one drain, with a panic that names the task and the seed. Park on a real wake instead: a sleep, a Notify, a channel.

Task Owners and the CPU Gate

Every task carries metadata: an id, a name, and an owner. The owner is the IP of the simulated process whose root task spawned it, inherited by every task spawned while one of its tasks polls. The driver, the workloads and the fault injectors have no owner.

The owner serves two features. A hint!("label").await in shipped code finds the process to reboot through the owner of the task being polled, without being handed one. And with the CPU model on (Chaos::Cpu), a process’s task polls only when its one CPU slot is free. Otherwise the world parks the runnable and schedules a CpuFree event. A parked task is not runnable, so until_stalled keeps its contract. Tasks without an owner never pay. CPU Time and Gray Failures covers the model.

Coming from tokio

The task API mirrors what the orchestrator, and your process or workload code through TaskProvider, expected from tokio:

Spawned futures stay Send + 'static. Execution is one OS thread, but the bounds let application code use Arc<RwLock<…>> and DashMap without contortion.

Panic handling has two layers. A task spawned with executor::spawn records its panic in its own JoinHandle. A task spawned through ctx.task().spawn_task() carries one more contract: the runner records an unobserved child panic with its process or workload identity, as an unobserved_task_panic event, and marks the seed as failed. If the caller awaits the handle and handles Err(JoinError::Panicked), the panic is observed and the seed can recover. Dropping or detaching the handle leaves the panic unobserved. A process’s tasks are also bound to the scope of its boot: when the process is killed, every task it spawned, however deep, is dropped with it. That drop is cancellation, not a panic.

time.sleep(Duration::ZERO) schedules a real same-time timer, so a burst of immediate work keeps the scheduler’s FIFO order. The runner allows the fan-out, then requests shutdown after 100,000 consecutive events at one logical instant. If the tasks ignore shutdown, the seed fails as a deadlock. Time advancing resets the count, so long timer-driven runs keep working.

Two behaviours are deliberately better than tokio’s:

• A genuine deadlock (driver not woken, nothing runnable) panics with the seed in the message instead of parking the thread forever.
• A task that stays runnable forever, a busy-yield loop or a select! over a source it re-readies synchronously, trips the poll bound in every build profile instead of hanging the drain.

Kill-on-Drop

Simulation iterations must be hermetic: a task leaked by seed N must never run during seed N+1. Dropping the per-iteration tokio runtime used to guarantee that. The executor replicates it with a waker registry, the pattern async-executor uses. Every live task registers one waker. Drop wakes them all in task-id order, which schedules every parked task, then pops and drops Runnables until the queue is empty. Runnables the CPU model parked are dropped too. Dropping a Runnable cancels its task and drops its future. Any task that drop wakes lands in the same queue and the same loop consumes it.

Exploration Process Safety

The explorer forks only between complete timelines, after the executor and all of its tasks have been dropped. Assertion macros record discovery coordinates. They are not fork points. Forked workers are still opt-in, because unrelated host threads may hold locks that the child inherits. The default workers: 0 mode runs continuations in-process and is the right choice for CI.

Verifying It Yourself

crates/moonpool-sim/tests/executor.rs enforces the contracts: join, abort, detach, panic isolation, kill-on-drop, same-seed replay and cross-seed diversity. crates/moonpool-sim/tests/determinism.rs is the end-to-end tripwire: two full SimulationBuilder runs of racing workloads on the same seed must produce byte-identical traces. If any component, the executor, select! or a provider, ever consults an unseeded source, that test fails.

The Provider Pattern

• The Core Idea
• The Providers Bundle
• One Line Changes Everything
• Thin Providers, Stateful Engines
• What Stays Outside the Bundle

Every distributed system does five things with the outside world: it reads the clock, it spawns concurrent tasks, it talks over the network, it draws random numbers, and it reads and writes files. That is the whole surface where non-determinism leaks in. Moonpool seals all five behind traits, and adds a sixth trait that reads how busy the process and its disk are, so that measurement runs in simulation too.

The Core Idea

Define a trait for each category of I/O. Implement the trait twice: once over real tokio calls for production, once over the deterministic world for testing. Your application code is generic over the trait. It never knows which implementation it runs against.

                  Application Code
             (generic over P: Providers)
                        |
            +-----------+-----------+
            |                       |
      TokioProviders          SimProviders
     (real TCP, wall         (simulated TCP,
      clock, real disk,       logical clock,
      OS entropy)             fault-injected disk,
                              seeded RNG)

This is interface swapping, the technique FoundationDB uses with its INetwork interface: Net2 in production, Sim2 in simulation. The difference is that Rust enforces it at compile time. If your code compiles with P: Providers, it works with both implementations.

The Providers Bundle

Six type parameters through every signature would be painful:

// Nobody wants to write this
fn run_server<N, T, K, R, S, Y>(net: N, time: T, task: K, rand: R, storage: S, sys: Y)
where
    N: NetworkProvider, T: TimeProvider, K: TaskProvider,
    R: RandomProvider, S: StorageProvider, Y: SystemProvider,
{ /* ... */ }

Moonpool bundles them in one trait, Providers:

pub trait Providers: Clone + Send + Sync + 'static {
    type Network: NetworkProvider;
    type Time: TimeProvider;
    type Task: TaskProvider;
    type Random: RandomProvider;
    type Storage: StorageProvider;
    type System: SystemProvider;

    fn network(&self) -> &Self::Network;
    fn time(&self) -> &Self::Time;
    fn task(&self) -> &Self::Task;
    fn random(&self) -> &Self::Random;
    fn storage(&self) -> &Self::Storage;
    fn system(&self) -> &Self::System;
}

Now your code carries one type parameter:

fn run_server<P: Providers>(providers: P) {
    let time = providers.time().clone();
    let net = providers.network().clone();
    // use them like tokio
}

Two implementations exist. TokioProviders::new() in moonpool-core is production. SimProviders::new(sim, ip) in moonpool-sim is simulation: it takes a weak handle on the SimWorld and the IP of the process it serves, so its network, storage and system providers are scoped to that process’s machine. Your application code imports neither. It sees P: Providers and nothing else.

Each provider in the bundle is Clone. Clone the one you need and move it into a spawned task, the way you would clone an Arc.

One Line Changes Everything

The swap between production and simulation happens at the call site, not inside the logic:

// Production
let providers = TokioProviders::new();
run_server(providers);

// Simulation: inside a Process or Workload, the runner hands out SimProviders
let providers = ctx.providers().clone();
run_server(providers);

Same run_server. Same code path. The only difference is which Providers is plugged in. This is the foundation that everything else in moonpool stands on. Chaos, assertion coverage, hints, exploration: all of it rests on the guarantee that the shipped code runs unmodified inside the simulator. Same Code in Production and Simulation states that rule in full.

Thin Providers, Stateful Engines

On the simulation side each provider is deliberately thin. SimTimeProvider::sleep schedules a timer in the global Scheduler<Event>. SimNetworkProvider::connect asks NetworkSimulation for a delayed connect operation. SimStorageProvider::open asks StorageEngine. The engines own the state, the exact pending results, the faults and the wakers. They return scheduling effects to SimWorld, which puts them on the one timeline. The application-facing trait stays familiar, and the simulator never collapses into one global state object.

What Stays Outside the Bundle

Name resolution is a separate trait, Resolver, with TokioResolver in production and ScriptedResolver in simulation. Most code never resolves a name, and code that does picks its resolver next to its providers. moonpool_core::select! is a macro, not a provider: it is tokio’s own select! expansion with a seeded start offset once moonpool-sim is in the build graph. The Six Providers lists every trait, method by method.

Quick Start: Swapping Implementations

• A Function Generic Over Providers
• The Forbidden List
• Running in Production
• Running in Simulation

Here is what providers look like in practice. We write one function that uses the time and random providers, run it on tokio, then run the same function under simulation. The code is adapted from crates/moonpool/examples/retrying_worker.rs, which you can run both ways.

A Function Generic Over Providers

use moonpool::prelude::*;
use std::time::Duration;

/// Retry with exponential backoff and jitter, through providers only.
async fn fetch_with_retry<P: Providers>(providers: &P, max_attempts: u32) -> Result<u32, String> {
    let time = providers.time();
    let random = providers.random();

    for attempt in 0..max_attempts {
        // Stand-in for a fallible network call: succeeds 60% of the time.
        // In the sim this coin comes from the seeded stream.
        if random.random_bool(0.6) {
            return Ok(attempt);
        }

        // Every sleep goes through the time provider, so the simulation
        // advances logical time instead of waiting.
        let base_ms = 10u64 << attempt;
        let jitter_ms = random.random_range(0..base_ms.max(1));
        time.sleep(Duration::from_millis(base_ms + jitter_ms))
            .await
            .map_err(|e| e.to_string())?;
    }

    Err(format!("gave up after {max_attempts} attempts"))
}

Notice what is not in this code: no tokio::time::sleep(), no rand::rng(). The function uses providers.time().sleep() and providers.random().random_bool(). That is the whole discipline. moonpool::prelude::* brings the provider traits into scope, which you need because their async fns are only callable with the trait imported.

The Forbidden List

These direct calls break determinism. Never use them in code that will run under simulation:

A direct tokio call is a hole in the simulation. It uses real I/O or real time, and the simulation has no control over it. The result is two different runs for one seed. Inside a simulation a bare tokio::spawn does not even run: there is no tokio runtime, so it panics.

tokio::select! is the subtle one. Its fair mode draws a start offset from tokio’s thread-local RNG, which is seeded from process entropy outside a tokio runtime. moonpool::select! is tokio’s own macro expansion with the offset drawn from the simulation stream, so the grammar is identical and only the number source changes. Outside a simulation it behaves like tokio’s. The Six Providers has the details.

Running in Production

use moonpool::prelude::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let providers = TokioProviders::new();
    let runtime = tokio::runtime::Builder::new_current_thread()
        .enable_all()
        .build()?;
    let result = runtime.block_on(fetch_with_retry(&providers, 5));
    println!("production run finished: {result:?}");
    Ok(())
}

TokioProviders bundles TokioTimeProvider, TokioTaskProvider, TokioNetworkProvider, TokioRandomProvider, TokioStorageProvider and TokioSystemProvider. Each one delegates to tokio, rand or the kernel. Real sleeps, real sockets, OS entropy.

Running in Simulation

A workload pulls SimProviders from its context and hands it to the same function:

use async_trait::async_trait;
use moonpool::{SimContext, SimulationBuilder, SimulationResult, Workload};

struct RetryWorkload;

#[async_trait]
impl Workload for RetryWorkload {
    fn name(&self) -> &'static str {
        "retry-client"
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        // Either outcome is fine: we assert the worker never panics or hangs
        // under simulated time, across many seeds.
        let _ = fetch_with_retry(ctx.providers(), 5).await;
        Ok(())
    }
}

#[test]
fn survives_chaos() {
    let report = SimulationBuilder::new()
        .set_iterations(50)
        .workload(RetryWorkload)
        .run()
        .expect("simulation configuration is valid");
    assert_eq!(report.failed_runs, 0, "worker should survive every seed");
}

Same function call. Now sleep() advances simulation time, so 50 seeds with their backoff schedules finish in milliseconds. random_bool(0.6) draws from the seeded stream, so a failing seed replays the same sequence of outcomes. Run both from the example:

cargo run --example retrying_worker    # production, on tokio
cargo test --example retrying_worker   # simulation, 50 seeds

That is the whole workflow. Write code generic over P: Providers, call provider methods instead of tokio, and let the runner pick the implementation.

Deep Dive: Why Providers Exist

• Why Not #[cfg(test)]
• Why Not Mocks
• Never More Permissive
• The Pattern
• Why Static Dispatch
• What You Get

Providers are not the first idea a team reaches for. Most teams start with #[cfg(test)] or a mock framework. Both have a fundamental problem that providers solve.

Why Not #[cfg(test)]

Conditional compilation is tempting. Keep the real call in production, swap a fake in during tests:

#[cfg(not(test))]
async fn connect(addr: &str) -> io::Result<TcpStream> {
    TcpStream::connect(addr).await
}

#[cfg(test)]
async fn connect(addr: &str) -> io::Result<TcpStream> {
    FakeStream::new()  // returns at once, no network
}

The problem: you no longer test your production code. The test binary compiles a different function. Every #[cfg(test)] block is a fork where production and test behaviour can diverge in silence. Oxide’s engineers documented this across five large repositories: #[cfg(test)] blocks “prevent testing real code paths”. The tested code is not the shipped code.

Providers remove the fork. There is one connect() in your application. It calls providers.network().connect(). In production that dispatches to tokio. In simulation it dispatches to the simulator. The application code is identical, down to the compiler output for the generic function.

Why Not Mocks

A mock framework like mockall records expectations: “this method is called with these arguments and returns this value”. A mock tests that your code calls the right method in the right order. It does not test what happens when the network does something unexpected.

A mock for TCP says: “when connect("10.0.1.1:9000") is called, return Ok(stream)”. That says nothing about a connection that takes 3 seconds, a connection that succeeds and then resets on the first read, or a peer that rebooted between two requests and lost its state.

Providers are full implementations, not recorded expectations. The simulated network provider keeps connection state, buffers bytes, injects delays, models TCP half-close, and drops a connection on partition. It is a complete in-memory TCP. When your code connects through it, it gets a real (simulated) session with real (simulated) failure modes.

That fidelity includes asynchronous establishment. Simulated bind, connect and accept do not hand back a ready socket. They park the calling future, schedule a network operation with a delay, and wake the future when the delay expires. Storage follows the same contract for read_at, write_at, sync_all and set_len: each returns Pending and completes through a scheduled storage event. A provider is an adapter into a stateful engine, not a shortcut around the scheduler.

Oxide’s codebases show a fidelity spectrum: no-op fakes that return Ok(()), in-memory implementations that hold real data without real I/O, and full protocol simulation. Providers sit at the top of that spectrum, because a simulation must hit the edge cases production hits.

Never More Permissive

A simulated provider has one more duty: it must never accept what the production provider refuses. If the simulator accepted an open that a real filesystem rejects, simulated code would come to depend on it, and the run would stay green until it reached a real disk.

Two examples from the storage provider. OpenOptions::validate applies std::fs::OpenOptions’ own rules on both backends: truncate, create and create_new need write or append, and append with truncate contradict each other. The simulator refuses those with InvalidInput before it touches a file. And DirectIo::Required opens an existing file. A Required open that would have to create the file is refused by both backends. crates/moonpool-sim/tests/storage/parity.rs runs the same scenarios through TokioStorageProvider and SimStorageProvider and asserts that they answer alike. The simulated list_dir is checked against tokio’s for the same tree.

The Pattern

The recipe is short. Oxide’s teams converged on it independently, with almost no mock-framework use:

1. Define a trait for the external dependency.
2. Implement it for production with real tokio calls.
3. Implement it for simulation, deterministic and controllable.
4. Inject it through generics: compile-time dispatch, zero runtime overhead.

// Step 1: the trait (native async fn in trait, no #[async_trait])
pub trait TimeProvider: Clone + Send + Sync + 'static {
    fn sleep(&self, duration: Duration) -> impl Future<Output = Result<(), TimeError>> + Send;
    fn now(&self) -> Duration;
    // ...
}

// Step 2: production
impl TimeProvider for TokioTimeProvider {
    async fn sleep(&self, duration: Duration) -> Result<(), TimeError> {
        tokio::time::sleep(duration).await;  // a real wall-clock delay
        Ok(())
    }
    // ...
}

// Step 3: simulation (moonpool-sim). sleep() schedules a timer event and
// parks the future; the scheduler advances logical time to the timer.

// Step 4: inject through generics
async fn heartbeat_loop<T: TimeProvider>(time: T) {
    loop {
        send_heartbeat().await;
        time.sleep(Duration::from_secs(5)).await.ok();
    }
}

If heartbeat_loop compiles with T: TimeProvider, it works with both TokioTimeProvider and SimTimeProvider. No runtime dispatch, no dynamic casts, no “forgot to wire the mock” surprise.

Why Static Dispatch

The provider traits are not object-safe, on purpose: the Clone supertrait, the generic random<T>(), the impl Future return types and the associated stream types all rule out dyn Providers. Simulation is test-only, so no production code ever needs to pick an implementation at runtime. Where generics hurt, erase at an application boundary with a narrow trait of your own. moonpool-rpc and moonpool-hyper do exactly this: they are generic over P: Providers all the way down and ship no providers of their own.

What You Get

Same code path. Your production code is your test code. No divergence, no conditional compilation.

Full behaviour, not call verification. A simulated network partitions, delays, reorders and resets. A simulated disk tears writes, corrupts sectors and lies about syncs. A mock records calls.

Resource ownership stays local. The network and storage engines own their state, pending results, faults and wakers. SimWorld coordinates their effects through one scheduler. Cancellation and shutdown are explicit: a crash completes pending work with errors, and a missing result is never mistaken for success.

A compile-time boundary. If you call tokio directly, the code still compiles and runs, but it is non-deterministic. Moonpool’s conventions, enforced in review, catch these: no tokio::time::sleep(), no tokio::spawn(), no tokio::net::*, no tokio::select!. Always go through the provider. Sim Compatibility Checklist is the full list.

The Six Providers

• TimeProvider
• TaskProvider
• NetworkProvider
• RandomProvider
• StorageProvider and StorageFile
• SystemProvider
• The Providers Bundle
• Local Providers
• Resolver
• select!

Moonpool puts every interaction between your code and the outside world behind six provider traits: TimeProvider, TaskProvider, NetworkProvider, RandomProvider, StorageProvider and SystemProvider. Each covers one category of I/O. Together they form a complete boundary around your application. A seventh trait, Resolver, sits outside the bundle for code that resolves names. The select! macro completes the set.

Every provider trait is Clone + Send + Sync + 'static. The simulation runs on one thread, but the types are Send-bounded so your code stays ordinary: Arc<RwLock<…>>, DashMap and Send task spawning all work. The async methods use native async-fn-in-trait, declared as -> impl Future<…> + Send so the bound propagates. There is no #[async_trait] and no ?Send in the provider layer.

TimeProvider

Every timeout, backoff, heartbeat and lease check goes through TimeProvider.

pub trait TimeProvider: Clone + Send + Sync + 'static {
    fn sleep(&self, duration: Duration) -> impl Future<Output = Result<(), TimeError>> + Send;
    fn now(&self) -> Duration;
    fn timer(&self) -> Duration { self.now() }
    fn timeout<F, T>(&self, duration: Duration, future: F)
        -> impl Future<Output = Result<T, TimeError>> + Send
    where F: Future<Output = T> + Send, T: Send;
}

The now() and timer() split comes from FoundationDB’s sim2. now() is the exact clock. timer() is a default method that returns now(), which is what production keeps. The simulation overrides it: timer() can run up to clock_drift_max (default 100 ms) ahead of now(). Use now() for scheduling. Use timer() for lease expiry and heartbeat deadlines, so clock skew between processes is part of every seed. sleep returns TimeError::Shutdown when the simulation is gone, and timeout returns TimeError::Elapsed.

Production: TokioTimeProvider delegates to tokio::time::sleep and tokio::time::timeout. now() is the elapsed time since the provider was created.

Simulation: SimTimeProvider::sleep schedules a timer in the Scheduler<Event>. When no task is runnable, the simulator pops the next event and jumps the clock to it. Hours of cluster time pass in seconds. timeout is a select! { biased; … } over the future and a timer, so the event order decides the winner and the race draws nothing.

TaskProvider

pub trait TaskProvider: Clone + Send + Sync + 'static {
    type JoinHandle: Future<Output = Result<(), JoinError>> + Detach + Send + 'static;

    fn spawn_task<F>(&self, name: &str, future: F) -> Self::JoinHandle
    where F: Future<Output = ()> + Send + 'static;

    fn yield_now(&self) -> impl Future<Output = ()> + Send;
}

Spawned futures are Send + 'static, as tokio::spawn requires. The handle resolves with Err(JoinError::Cancelled) or Err(JoinError::Panicked) when the task did not complete normally. Detach::detach() is explicit fire-and-forget: dropping the handle does the same, but detach() states the intent and satisfies the must_use lint.

Production: TokioTaskProvider calls tokio::spawn. The name feeds two tracing::trace! events, at start and at completion, and nothing else.

Simulation: SimTaskProvider spawns onto the deterministic executor, which stores the name and traces every poll under it. A process’s provider carries the scope of its boot: when the process is killed, every task spawned through it, and every task those spawned, is dropped. A workload’s provider has no scope, because a workload survives everything. A spawned task runs inside the span current at spawn time, so its tracing events keep their process or workload attribution.

NetworkProvider

pub trait NetworkProvider: Clone + Send + Sync + 'static {
    type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;
    type TcpListener: TcpListenerTrait<TcpStream = Self::TcpStream> + 'static;

    fn bind(&self, addr: &str) -> impl Future<Output = io::Result<Self::TcpListener>> + Send;
    fn connect(&self, addr: &str) -> impl Future<Output = io::Result<Self::TcpStream>> + Send;
}

pub trait TcpListenerTrait: Send + Sync + 'static {
    type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;

    fn accept(&self) -> impl Future<Output = io::Result<(Self::TcpStream, String)>> + Send;
    fn local_addr(&self) -> io::Result<String>;
}

The stream types are futures::io::AsyncRead + AsyncWrite, the runtime-agnostic traits, so one codec or framing layer works on both backends. bind, connect and accept behave like their tokio counterparts.

Production: TokioNetworkProvider wraps tokio::net through tokio_util::compat. Every connected and accepted stream has Nagle’s algorithm disabled (TCP_NODELAY, best-effort), as FoundationDB’s Net2 does, because request-reply protocols write a small frame and wait for the answer.

Simulation: NetworkSimulation owns listeners, connections, topology, faults, pending results and wakers. bind, connect and accept park until their scheduled latency expires. Established streams use in-memory buffers with deterministic delivery delays, TCP half-close and fault injection: connection drops, partitions, corruption. Binding port zero on a numeric address assigns a distinct dynamic port, and local_addr() returns it. A bare process IP is also a valid address. Dropping a stream with unread bytes resets the peer, as TCP does. AsyncWrite::close shuts down the write half only. Each listener has a bounded accept backlog, 128 by default, set per run with SimulationBuilder::accept_backlog_capacity. Network Faults lists the fault families.

RandomProvider

pub trait RandomProvider: Clone + Send + Sync + 'static {
    fn random<T>(&self) -> T where StandardUniform: Distribution<T>;
    fn random_range<T>(&self, range: Range<T>) -> T where T: SampleUniform + PartialOrd;
    fn random_ratio(&self) -> f64 { self.random() }
    fn random_bool(&self, probability: f64) -> bool { self.random_ratio() < probability }
}

Only random and random_range are required. The trait is fully synchronous: random numbers never need to suspend.

Production: TokioRandomProvider uses rand::rng(), thread-local OS entropy.

Simulation: SimRandomProvider draws from the one seeded ChaCha8Rng stream. Every call is counted, so it is part of the replay and of exploration recipes.

StorageProvider and StorageFile

pub trait StorageProvider: Clone + Send + Sync + 'static {
    type File: StorageFile + 'static;

    fn open(&self, path: &str, options: OpenOptions) -> impl Future<Output = io::Result<Self::File>> + Send;
    fn exists(&self, path: &str) -> impl Future<Output = io::Result<bool>> + Send;
    fn delete(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
    fn rename(&self, from: &str, to: &str) -> impl Future<Output = io::Result<()>> + Send;
    fn create_dir_all(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
    fn list_dir(&self, path: &str) -> impl Future<Output = io::Result<Vec<String>>> + Send;
    fn sync_dir(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
}

pub trait StorageFile: AsyncRead + AsyncWrite + AsyncSeek + Unpin + Send + Sync + 'static {
    fn sync_all(&self) -> impl Future<Output = io::Result<()>> + Send;
    fn sync_data(&self) -> impl Future<Output = io::Result<()>> + Send;
    fn constraints(&self) -> IoConstraints;
    fn is_direct_io(&self) -> bool;
    fn read_at(&self, offset: u64, buf: &mut [u8]) -> impl Future<Output = io::Result<usize>> + Send;
    fn write_at(&self, offset: u64, buf: &[u8]) -> impl Future<Output = io::Result<usize>> + Send;
    fn write_at_durable(&self, offset: u64, buf: &[u8]) -> impl Future<Output = io::Result<usize>> + Send {
        // default: write_at, then sync_data
    }
    // The zero-copy forms: the buffer moves in and comes back with the count.
    fn read_at_aligned(&self, offset: u64, buf: AlignedBuf) -> impl Future<Output = io::Result<(usize, AlignedBuf)>> + Send {
        // default: read_at on the buffer's bytes
    }
    fn write_at_aligned(&self, offset: u64, buf: AlignedBuf) -> impl Future<Output = io::Result<(usize, AlignedBuf)>> + Send {
        // default: write_at on the buffer's bytes
    }
    fn write_at_durable_aligned(&self, offset: u64, buf: AlignedBuf) -> impl Future<Output = io::Result<(usize, AlignedBuf)>> + Send {
        // default: write_at_durable on the buffer's bytes
    }
    fn size(&self) -> impl Future<Output = io::Result<u64>> + Send;
    fn set_len(&self, size: u64) -> impl Future<Output = io::Result<()>> + Send;
    fn allocate(&self, len: u64) -> impl Future<Output = io::Result<()>> + Send;
}

The split is the rule. The provider owns the namespace: open, exists, delete, rename, directory creation, listing and sync_dir. The file owns already-open bytes. A delete or a rename over a name removes the name and nothing else. A file opened before it keeps reading and writing the same bytes, as on Unix, until the last name and the last handle are both gone. The namespace is per process: two processes that open data.db open two files on two disks.

Three distinctions the trait keeps apart, because production keeps them apart:

• A completed write is visible, not durable. Only sync_all or sync_data makes it survive a crash. write_at_durable is the one write that is durable on return, and only for its own bytes: on Linux the Tokio backend issues one pwritev2(RWF_DSYNC) (a FUA write on a direct-I/O file whose blocks are allocated), which flushes nothing outside its range. The default, and the simulator, run write_at then sync_data.
• File durability is not directory-entry durability. After open(create), write and sync_all, a crash can still leave no name pointing at the file. sync_dir on the parent is the call that makes the name durable, as SQLite and PostgreSQL do after a create, delete or rename.
• set_len grows the readable range. It does not allocate, initialize or make the length durable. allocate is the call that reserves space: fallocate(2) mode 0 on Linux, growing the file to at least len. The range reads as zeros, a full disk fails there and not on a later write, and the length is still durable only after sync_all. The simulator already counts a grown range against the disk’s capacity, so there it is the same as growing.

OpenOptions mirrors std::fs::OpenOptions (read, write, create, create_new, truncate, append) and adds direct_io(DirectIo::{Disabled, Optional, Required}). Direct I/O is an option on opening an ordinary file, not a second provider: there is no BlockProvider. A direct-I/O file reports its alignment through constraints() and refuses the stream API with stream_io_unsupported(), because a shared cursor cannot honour an alignment. read_at and write_at are positioned, take &self, and may transfer fewer bytes than asked. Each positioned transfer also has an owned form over an AlignedBuf (read_at_aligned, write_at_aligned, write_at_durable_aligned): the same transfer and contract, with the buffer moved in and handed back beside the count (dropped on error). It exists for zero copy. A backend that runs I/O on another thread cannot move a borrowed slice there, so the borrowed form pays a copy into its own aligned buffer; the owned form moves the caller’s buffer instead. A caller that already holds its bytes in an AlignedBuf uses it, any other caller keeps the borrowed form, and neither is a different kind of file. BlockFile<F> wraps one open file to add block arithmetic and the loops that turn partial transfers into whole ones. It holds no path and opens nothing. LayoutRegion lets a format describe where its bytes live, so a fault injector can aim at them. Storage Faults and Aiming Storage Faults continue there.

Production: TokioStorageProvider wraps tokio::fs. Positioned I/O runs on the blocking pool, and O_DIRECT comes from libc. The owned transfers move the caller’s AlignedBuf into the blocking task and back, so pwrite, pwritev2 and pread reach the caller’s own memory; the borrowed ones copy through a bounce buffer, as a slice cannot cross into a 'static task.

Simulation: the owned transfers keep the trait’s defaults, the borrowed transfer on the buffer’s bytes, so every fault applies to them unchanged. StorageEngine keeps one in-memory image per file. A positioned write is visible to a stream read, and a block write to both. Open handles have independent cursors and pending operations. Every operation returns Pending and completes through a scheduled event, with an exact result, so a crash or shutdown completes pending work with errors instead of silence. Each SimStorageProvider is scoped to a process IP, which selects that process’s disk configuration and degradation episodes. The fault model follows TigerBeetle and FoundationDB: read and write corruption, EIO, the barrier-bounded crash model, misdirected I/O, short transfers, sync lies, unsynced directory-entry loss, and IOPS and bandwidth timing.

SystemProvider

How busy is this process? FoundationDB reads cumulative counters every few seconds: CPU seconds from getrusage, machine CPU from /proc/stat, the disk’s busy time from /proc/diskstats. Two readings and a division give cpu.usage_cores and disk.busy. SystemProvider is that reading:

pub trait SystemProvider: Clone + Send + Sync + 'static {
    // Cumulative counters now. `data_dir` selects the disk device.
    fn sample(&self, data_dir: &str) -> io::Result<SystemSample>;
}

A SystemSample carries the instant at, process_cpu, the usable cores, machine_busy and machine_total, the run loop’s busy time, and the DiskCounters of the device that holds data_dir: io_ticks (time with at least one I/O in flight), in_flight, completed reads and writes, their bytes and their times. space is the DiskSpace of the filesystem that holds data_dir (statvfs: total_bytes and the free_bytes an unprivileged writer may use), or, in simulation, of a disk given a capacity. The provider does no arithmetic. Every counter only grows, so a counter that goes back tells the caller the window is not valid.

Production: TokioSystemProvider reads getrusage(RUSAGE_SELF), /proc/stat and the /proc/diskstats line whose major and minor numbers match stat(data_dir). On btrfs, overlayfs and some containers no line matches, so disk is None. The run loop’s busy time is the busiest tokio worker’s. Remember that io_ticks is iostat’s %util: on flash and RAID it reaches 100% long before the device runs out of throughput.

Simulation: FoundationDB switches its system monitor off in simulation. Moonpool does the opposite. StorageEngine counts, per process, the time its disk has at least one operation in flight, parked operations on a failed disk included, so a stall or throttle episode makes io_ticks rise. SimWorld::disk_counters(ip) returns the same numbers, so a harness can check a process’s report against ground truth. With the CPU model on, every poll of the process’s tasks costs simulated time on its one core, and process_cpu, machine_busy and run_loop_busy are that core’s busy time. With the model off they stay at zero. The world remembers the last 64 samples per process (SYSTEM_SAMPLES_KEPT), readable with ctx.system_samples(ip). A sample draws nothing from the seed.

The Providers Bundle

pub trait Providers: Clone + Send + Sync + 'static {
    type Network: NetworkProvider;
    type Time: TimeProvider;
    type Task: TaskProvider;
    type Random: RandomProvider;
    type Storage: StorageProvider;
    type System: SystemProvider;

    fn network(&self) -> &Self::Network;
    fn time(&self) -> &Self::Time;
    fn task(&self) -> &Self::Task;
    fn random(&self) -> &Self::Random;
    fn storage(&self) -> &Self::Storage;
    fn system(&self) -> &Self::System;
}

TokioProviders::new() bundles the six production implementations. SimProviders::new(sim, ip) bundles the six simulation implementations, scoped to one process IP. Your application code sees P: Providers and nothing else.

Local Providers

A thread-per-core runtime (compio, monoio) pins every task, file and socket to one thread. Its files hold an Rc to their descriptor and its futures hold the thread’s proactor, so none of them is Send, and it cannot implement the traits above. Each Send trait has a local form: the same methods and the same contract, with no Send or Sync bound on the trait, its associated types or its futures.

pub trait LocalStorageFile: AsyncRead + AsyncWrite + AsyncSeek + Unpin + 'static {
    fn read_at(&self, offset: u64, buf: &mut [u8]) -> impl Future<Output = io::Result<usize>>;
    // ... every StorageFile method, no `+ Send`
}
impl<F: StorageFile> LocalStorageFile for F { /* forwards every method */ }

Every Send backend implements the local traits through the blanket impls, so the Tokio and simulated providers get them for free. A thread-per-core backend implements only the local traits. Code written once against the local traits runs on every backend, and when the backend is Send the futures it builds are Send too: the auto traits of the concrete future leak through the generic code, so it can still go to tokio::spawn.

Tasks have no blanket impl in either direction. A TaskProvider cannot spawn a !Send future, and a local provider pins its tasks to one thread. A backend that can do both implements both. CompioTaskProvider is the production local one. SimTaskProvider implements both traits, because the executor runs every task on one thread anyway: a local task inherits the owner and the boot scope, dies with its process, reports its panics, and is scheduled by the same seeded executor. So local code runs under simulation unchanged, Rc<RefCell<..>> across awaits included. A Process or Workload future stays Send: it spawns the local code with ctx.task().spawn_local_task(..) and awaits or detaches the handle, which is Send (crates/moonpool-sim/tests/local_workload.rs). ctx.run_local(name, |ctx| async move { .. }) does this for a body that returns an answer: it runs the body as a local task of the process, with its own clone of the context, and returns the answer over a channel.

LocalProviders leaves the task slot out so that it can be a blanket impl over Providers: every existing bundle is a LocalProviders. Code that spawns local tasks takes a LocalTaskProvider beside the bundle, or a ThreadProviders bundle: a LocalProviders with a local task slot (local_task()). SimProviders is one (through a blanket impl over every Providers whose task provider is also local) and CompioProviders is one. TokioProviders is not. Random and system keep their Send traits: their methods are synchronous, so a thread-per-core runtime backs them with the same types Tokio uses.

The compio backend. The compio-* features (umbrella compio-providers, off by default) add a production backend on compio, a completion-based, thread-per-core runtime: io_uring on Linux (polling where the kernel refuses io_uring), IOCP on Windows. It implements only the local traits:

Every call needs a compio runtime on the current thread. Storage opens follow the Tokio backend’s direct-I/O policy, from the same code: DirectIo::Optional upgrades to O_DIRECT where the filesystem allows it, DirectIo::Required fails rather than fall back and never creates a file, and a direct file advertises the alignment the kernel reports. The owned *_aligned transfers hand the caller’s AlignedBuf straight to io_uring. On Linux an append file’s stream writes go through an O_APPEND descriptor, so they are atomic against other writers, and positioned I/O uses a second descriptor of the same file without the flag. Elsewhere pwrite honours its offset under O_APPEND, so the stream appends at the size it reads first, which is not atomic against other writers.

Libraries on the local traits. moonpool-journal and moonpool-rpc are written against the local traits, so the same code runs on Tokio, on compio and under simulation. The journal takes a LocalStorageProvider. The RPC runtime takes an RpcProviders bundle: a LocalProviders whose values (not its futures or streams) are Send + Sync, which every bundle in moonpool is. Over a Send backend their futures stay Send, and tests/send.rs in each crate checks this. On compio, poll the RPC driver and its callers on one thread, for example with compio::runtime::spawn(driver.run()). Each crate has a compio-providers feature that runs its compio tests and its backends example, which compares the two backends (see docs/benchmarks/).

io_uring in containers. Docker’s default seccomp profile blocks the io_uring_setup, io_uring_enter and io_uring_register syscalls, and some hosts turn io_uring off with the kernel.io_uring_disabled sysctl. There, compio falls back to its polling driver: the code still works, but without the batched submissions that make the backend worth it. To get io_uring in a container, run it with a seccomp profile that allows those three syscalls.

Method-name ambiguity. A concrete Tokio or sim type has both traits’ methods under the same names. With both traits in scope, file.read_at(..) is ambiguous: import one trait per module, or call through the trait (LocalStorageFile::read_at(&file, ..)). Generic code bounded by one trait is not affected. The local traits are not in the prelude.

Resolver

Name resolution is not in the bundle. Most code never resolves a name, and code that does picks its resolver next to its providers.

pub trait Resolver: Clone + Send + Sync + 'static {
    fn resolve(&self, target: &str) -> impl Future<Output = io::Result<Vec<SocketAddr>>> + Send;
}

The target is the same host:port string connect takes. A numeric target resolves to itself. An Ok answer is never empty: a name with no address is a NotFound error, so a caller can tell a failed lookup from a failed connection. Caching and time-to-live belong to the caller.

Production: TokioResolver (feature tokio-net) calls tokio::net::lookup_host. CompioResolver (feature compio-net) runs the operating system’s lookup on a helper thread and answers over a channel, so its future is Send and runs on any executor.

Simulation: ScriptedResolver answers from a shared table that a fault injector or workload edits while the run goes. Answers are immediate and draw no randomness.

select!

moonpool_core::select!, re-exported as moonpool_sim::select! and moonpool::select!, is tokio’s own select! macro with one change. tokio’s fair mode polls branches starting at a random index drawn from its thread-local RNG, which is seeded from process entropy when there is no tokio runtime. That is a hole in the replay. Moonpool’s macro calls tokio’s internal expansion with the start offset drawn from select_support::select_offset, which moonpool-sim points at the seeded stream for each iteration. Grammar, limits, if preconditions and the 64-branch cap are tokio’s, byte for byte.

select! { biased; … } forwards verbatim to tokio: source-order polling is already deterministic. Use biased; when branches have a natural priority, such as a shutdown or timeout guard. Use the default form when branches are peers racing for the same wake, so the seeds explore both winners.

Which macro you get is decided by cargo feature unification. With core’s select feature alone, the re-export is tokio’s macro verbatim. The moment moonpool-sim enters the build graph, even as a dev-dependency, its deterministic-select feature replaces it for every crate in that graph. Same Code in Production and Simulation explains what that means for a crate’s manifest.

Same Code in Production and Simulation

• No Sim* Wrappers
• Three Mechanisms
• Inert in Production
• Keeping Sim Out of the Release Binary
• The select! Asymmetry
• The Checklist

One rule sits under every other design choice in moonpool: the code you ship is the code you simulate. Not a copy of it. Not a wrapper around it. The same compiled function, with a different Providers type plugged in. This chapter states the rule, shows the three mechanisms that make it hold, and shows how a crate keeps the simulator out of its release binary.

No Sim* Wrappers

The tempting shortcut is a test-only twin of each component: a SimMachine next to Machine, a SimDisk next to Disk, a #[cfg(test)] hook that lets the test reach inside. Each twin is a place where the tested behaviour and the shipped behaviour drift apart. After a year, the twin tests itself.

Moonpool has none of these in shipped code. There is no SimMachine, no SimDisk, no test-only hook inside a journal or an RPC runtime. The only Sim* types in the workspace are the providers themselves: SimTimeProvider, SimNetworkProvider, SimStorageProvider and their siblings in moonpool-sim. They implement the same traits TokioTimeProvider and TokioStorageProvider implement. Everything above the trait line is one implementation.

The same rule applies to your code. A process in a simulation is your server, built by a factory, not a simplified model of it. moonpool-journal is a worked example: one Journal<P: LocalStorageProvider> runs as Journal<TokioStorageProvider> in its accept_log example, as Journal<CompioStorageProvider> on a thread-per-core runtime, and as Journal<SimStorageProvider> in its tests, and the crash tests exercise the exact commit path production uses.

Three Mechanisms

Three things let one implementation serve both worlds.

Providers carry the I/O. The code depends on P: Providers, never on tokio. In production main builds TokioProviders::new(). In a simulation the runner hands a SimProviders to each process and workload through ctx.providers(). The Provider Pattern covers this. moonpool-rpc and moonpool-hyper follow the same rule: both are generic over P: Providers, ship no providers, and never depend on moonpool-sim.

The code names moments, the simulator strikes. Some faults are best placed by the code that knows the moment. A journal knows the instant where its entries are written but not synced. The code names that moment with hint! from moonpool-buggify:

segment.write(batch.entry_at, &batch.entries).await?;
// A power loss here tears entries no record points at yet.
moonpool_buggify::hint!("journal commit: entries written, not synced", COMMIT_HINT_PROB).await;
segment.sync().await?;

The code never names a fault. It says “this is interesting”. In production the point draws nothing, asks no one, and the future resolves at once. In a simulation the point is a disruptive buggify site: activated once per run, firing at POINT_PROB (5%) per call or at the rate the site gives. When it fires, moonpool-sim’s HintSink finds the calling process through the owner of the task being polled and asks the first attrition regime whose victim filter admits that process. The regime’s max_dead budget, reboot kinds and recovery delays apply, as for its own timed reboots. The harness can add a HintVeto for a budget the simulator cannot see. A seed whose swarm drew the never-reboot regime never reboots on a hint. hint!: Naming Fault Sites covers the vetoes.

The code’s own choices use buggify. When the code itself picks a rare but legal behaviour, it uses the inline macros: buggify!() fires at 25% while its site is active, buggify_with_prob!(p) at a site rate, buggify_fault_with_prob!(p) for a site that makes an operation fail and goes silent in the recovery tail, buggify_named!("label", p) for a site a harness can switch per seed with set_activation, and buggify_pick! and buggify_range! for a site that overrides a choice or stretches a value. moonpool-rpc uses these inside its shipped transport: a session cut right after a frame, a request refused as Overloaded, a deferred stream acknowledgement. cpu_charge(duration) is the same idea for CPU time: shipped code reports that it did a known amount of work, and a simulator with a CPU model makes that work take time.

Inert in Production

All of this costs nothing outside a simulation. moonpool-buggify is a pure-std crate with zero dependencies. Its state is thread-local and disabled by default. Outside a simulation:

• every buggify!, buggify_with_prob!, buggify_fault_with_prob! and buggify_named! site evaluates to false with no draw and no side effect
• buggify_pick! and buggify_range! return None
• hint! returns a future that resolves at once, without asking any sink
• cpu_charge does one thread-local read and returns
• is_simulated() returns false, so code can tilt a rate or a check in simulation without changing an outcome a caller can observe

A simulation turns the crate on for one run: moonpool-sim installs its seeded draw with set_random_source, calls buggify_init(0.5) so each site is active with 50% probability, installs its hint and CPU sinks, and resets everything when the run ends. The same thread-local state serves moonpool_sim::buggify! and moonpool_buggify::buggify!, so a library and the harness that tests it share one set of activation decisions.

Keeping Sim Out of the Release Binary

Determinism costs nothing at runtime in production, and the build must cost nothing either. Three manifest rules keep the simulator, the explorer and libc/mmap out of a release.

A library depends on the traits and the macros only. moonpool-journal’s manifest is the pattern:

[dependencies]
moonpool-core = { version = "0.8.0", default-features = false }
moonpool-buggify = "0.8.0"

[dev-dependencies]
moonpool-core = { version = "0.8.0", default-features = false, features = ["tokio-fs"] }
moonpool-sim = "0.8.0"

default-features = false on moonpool-core drops the tokio providers and the select! passthrough, so the library compiles with no tokio at all. The simulator enters only as a dev-dependency. The tests get SimStorageProvider; the release gets nothing.

A binary enables the production providers it needs. moonpool-core’s features are granular: tokio-task, tokio-time, tokio-net, tokio-fs, tokio-random, tokio-system, with tokio-providers as the umbrella that gives TokioProviders. A consumer that only needs files enables tokio-fs. With every tokio feature off, moonpool-core is wasm-clean.

The facade has a lean configuration. The moonpool crate’s default features are ["sim", "tokio"], the full framework. A production build opts out:

moonpool = { version = "0.8.0", default-features = false, features = ["tokio"] }

tokio enables core’s tokio-providers and select, so TokioProviders and moonpool::select! are there. sim is off, so moonpool-sim and moonpool-explorer are absent from the dependency tree. hyper, prometheus, rpc, rpc-derive, rpc-tls and rpc-jwt are opt-in on top of either configuration.

The select! Asymmetry

One thing to know when you read a lock file. moonpool_core::select! is tokio’s macro verbatim with core’s select feature, and tokio’s expansion with a seeded offset with core’s deterministic-select feature. moonpool-sim enables deterministic-select. Cargo unifies features across a build graph, so the moment moonpool-sim is in the graph, even as a dev-dependency of one crate, every crate in that graph gets the seeded macro, including your library compiled for tests. Both forms are tokio’s expansion with identical grammar. The only change is where the start offset comes from, and outside a simulation that offset is entropy-seeded, as tokio’s is. A release build without moonpool-sim keeps tokio’s macro byte for byte.

The Checklist

When you review a change against this rule, ask four questions:

1. Does the code reach tokio, std::time, rand or the filesystem without a provider? Move it behind P: Providers.
2. Does a test reach inside the component through a hook that production does not use? Replace the hook with a hint! or a buggify_named! site the simulator can drive.
3. Does the component have a Sim* twin? Delete the twin and run the real one under SimProviders.
4. Does the library’s manifest depend on moonpool-sim outside [dev-dependencies]? Move it.

Using Providers in Production shows the production side of the wiring, and Migrating Existing Code to Providers walks through an existing service.

System Under Test vs Test Driver

• Two Perspectives on One System
• Process: The Thing You Build
• Workload: The Thing That Tests
• Different Lifecycles
• Addresses
• Why Not One Trait?
• What Comes Next

Every moonpool simulation has two roles. Understanding this split is the one concept to have before you write a line of simulation code.

Two Perspectives on One System

Think about how you test a web server. You have the server: handling requests, managing connections, storing data. And you have something else: a client, a load generator, a harness that sends requests and checks the answers.

These two have different jobs. The server is the system. The driver exercises the system.

Moonpool makes the split explicit with two traits: Process and Workload.

Process: The Thing You Build

A Process is your server, your node, your participant in the distributed system. It is the system under test. The code inside a Process is the code you ship. There is no simulation-only copy of it.

Processes have a hard life in simulation:

• They crash. The simulation kills them without warning, at a timed reboot, at a correlated outage, or at a hint! the process named itself.
• They reboot. After a kill, the runner calls your factory for a fresh instance.
• They lose state. Every in-memory field is gone on reboot. Only what reached the simulated disk survives, and only the part that was synced.
• They get partitioned, delayed and slowed. Connections break, packets arrive late, a gray failure makes one machine’s CPU, disk or network crawl.

This is the point. The simulation throws chaos at your server and watches what breaks.

In FoundationDB’s simulation the equivalent is the fdbserver process: the same code as production, inside a world where the simulator decides when clocks advance, when packets arrive and when machines die.

Workload: The Thing That Tests

A Workload is your test driver. It lives outside the system under test. It sends requests, observes answers, and judges whether the system behaved.

Workloads have an easy life:

• They never crash. The simulation does not reboot workloads, and a crash_self from a workload is an error.
• They survive everything. Processes come and go. Workloads keep running.
• They see the whole picture. A workload knows every process IP, every group, every tag, and can read the simulator’s ground truth: a process’s disk counters, its system samples, its current slowness.
• They judge correctness. After the run, each workload’s check() validates the final state, and its verdict counts.

In FoundationDB the equivalent is tester.actor.cpp. The tester knows the cluster, drives operations against it, and validates the result.

Different Lifecycles

The two lifecycles are different in shape.

A Process lifecycle is a loop: the factory builds one, it runs, it is killed, the factory builds another, it runs, it is killed. Each incarnation starts with empty memory. Its ctx.providers() is fresh each boot, and every task it spawned dies with it.

A Workload lifecycle is a line: setup() once at the start, run() for the test, check() at the end. One life from start to finish.

The key point: a Process does not know when it will die, and a Workload does not care when Processes die. The Process does its job correctly or it does not. The Workload keeps sending requests and tracking what happened.

Addresses

The runner gives each role its own IP range, so a log line tells you at a glance who speaks:

Each .processes() or .cluster() call registers one group, named after its Process::name(), with its own factory and its own per-seed count. Process Groups, Tags and Topology covers multi-role systems.

Why Not One Trait?

Why not “participants” that do both? Because mixing the two concerns makes a mess. If server code also tracks test state, it cannot be rebooted cleanly. If driver code also runs server logic, it cannot survive a crash.

The split also matches production. You deploy servers. You run integration tests against them. Different code, different lifecycles, different concerns. And it matches the rule in Same Code in Production and Simulation: the Process is the shipped code, and the Workload is the only place where test logic lives.

What Comes Next

The next two chapters cover each trait in detail: Process with its factory, its reboots and its context, then Workload with its three phases and its operation alphabet. After that, Part III builds a complete simulation from scratch.

Process: Your Server

• The Process Trait
• The Factory
• State and Reboots
• The Context
• Tags for Roles
• Three Kinds of Reboot
• Crashing Yourself
• A Concrete Example

A Process is the system under test. It is the code you ship, running inside a simulation where the framework controls time, the network, the disk and failure.

The Process Trait

The trait is minimal by design:

#[async_trait]
pub trait Process: Send + Sync + 'static {
    fn name(&self) -> &str;
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()>;
}

Two methods. name() identifies the process type, and it names the process group the builder registers. run() holds your server logic. Every Process is Send + Sync + 'static, so it composes with Arc<RwLock<…>> and DashMap like ordinary tokio code. The runtime is still one thread.

When run() returns Ok(()) or an Err, the process has exited on its own. When the simulation kills the process, the future is cancelled and run() never returns. So is every task the process spawned through ctx.task().spawn_task(), however deep: a crash ends the whole process, not only its root future, so a background worker never survives to run beside the rebooted process. When run() panics, the seed fails. The panic is caught and recorded against the process’s IP, and the iteration is reported as failed even if every workload returned Ok.

The Factory

You never construct a Process once. You give the builder a factory:

SimulationBuilder::new()
    .processes(3, || Box::new(MyServer::new()))

Why a factory? Reboots. When the simulation kills a process and restarts it, the runner calls the factory for a new instance. Each boot starts from a clean slate, as a restarted server does. Three processes with two reboots each means nine factory calls.

The count can be a range. .processes(3..=7, …) draws a count per seed, so cluster size is one more thing the seeds explore.

State and Reboots

This is the rule to internalize: all in-memory state is lost on reboot.

If your process keeps a BTreeMap of client sessions in a field, that map is gone after a reboot. The new instance starts empty. Only data written to the simulated disk survives, and of that, only what was synced. A write that completed but was not synced resolves by crash physics: kept, lost, or torn, sector by sector. Storage Faults describes the crash model.

This matches reality. A crashed server does not recover its heap. It reads persistent state from disk and rebuilds.

To carry a fact across boots for the test’s sake, use ctx.state(), the shared state handle. It belongs to the run, not to the process, so a boot counter published there is visible to the next boot and to every workload and invariant. Use it for test bookkeeping only. Shipped code must not depend on it.

The Context

run() receives a SimContext. The parts a process uses:

The providers are fresh each boot. Bind listeners, open files and connect to peers inside run(), not in the factory.

Tags for Roles

Many systems need nodes with different roles: leader and follower, primary and secondary, several data centres. Tags handle this:

SimulationBuilder::new()
    .processes(5, || Box::new(MyNode::new()))
    .tags(&[
        ("role", &["leader", "follower"]),
        ("dc", &["east", "west", "eu"]),
    ])?

tags() applies to the last registered group and returns a Result, because it is an error to call it before processes(). Tags distribute round-robin:

Read your tags from the topology:

async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    match ctx.topology().my_tags().get("role") {
        Some("leader") => run_leader(ctx).await,
        Some("follower") => run_follower(ctx).await,
        _ => Ok(()),
    }
}

ctx.topology().ips_tagged("role", "leader") finds the others. For a second role with its own process type, register a second group instead. Process Groups, Tags and Topology covers both.

Three Kinds of Reboot

RebootKind has three variants:

Graceful. The runner cancels ctx.shutdown(). The process has a grace period to finish in-flight work and close connections. Send buffers drain, so peers see a clean FIN. If the process does not return in time, it is force-killed.

Crash. Instant death. The task is cancelled at the crash instant. Connections reset, so peers see ConnectionReset. Unsynced storage data may be lost.

CrashAndWipe. A crash that also deletes every file the process owns. This simulates a lost disk or a fresh node joining the cluster. The wipe is scoped to the process’s IP.

Handle a graceful shutdown by racing your work against the token. The biased; form polls the branches in order and draws nothing, which is right when one branch is a guard:

loop {
    let result = moonpool_sim::select! {
        biased;
        r = listener.accept() => r,
        () = ctx.shutdown().cancelled() => return Ok(()),
    };
    // ...
}

You do not handle a crash. There is nothing to handle. The simulation cancels your future and moves on.

Crashing Yourself

Sometimes a crash must land at an exact point of your own protocol: between a write and its sync, in the middle of an append. ctx.crash_self(RebootKind::Crash, Some(Duration::from_millis(10))) schedules the same force kill a fault injector would, one scheduler tick from now, and restarts the process after the delay. With None the process stays down until an injector restarts it. ctx.self_crash() returns a SelfCrash handle that is Clone + Send + Sync, for code that holds no context. A graceful kind is refused: a process shuts down gracefully by returning. This is a test instrument. Shipped code names the moment with hint! instead and lets the simulator decide, as Same Code in Production and Simulation explains.

A Concrete Example

An echo server as a Process, from moonpool’s own reboot tests:

use futures::io::{AsyncReadExt, AsyncWriteExt};

struct EchoProcess;

#[async_trait]
impl Process for EchoProcess {
    fn name(&self) -> &'static str {
        "echo"
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        loop {
            let result = moonpool_sim::select! {
                biased;
                r = listener.accept() => r,
                () = ctx.shutdown().cancelled() => return Ok(()),
            };

            match result {
                Ok((mut stream, _peer)) => {
                    let mut buf = [0u8; 64];
                    if let Ok(n) = stream.read(&mut buf).await {
                        if n > 0 {
                            let _ = stream.write_all(&buf[..n]).await;
                        }
                    }
                }
                // Under chaos an accept can fail; the server keeps serving.
                Err(e) => tracing::debug!("accept error: {e}"),
            }
        }
    }
}

The patterns: ctx.network() for sockets, ctx.shutdown() for a graceful stop, moonpool_sim::select! for the race. Never call tokio directly. In a real service the body of run() would call a function generic over P: Providers with ctx.providers(), and main would call the same function with TokioProviders.

Workload: Your Test Driver

• The Workload Trait
• The Three Phases
  • Setup: Prepare the Ground
  • Run: Drive the System
  • Check: Judge the Outcome
• Workloads Survive Everything
• The Context
• The Operation Alphabet
• Multiple Instances
• What a Good Workload Looks Like

A Workload exercises the system under test and judges its correctness. Processes are the code you ship. Workloads are the code that finds your bugs.

The Workload Trait

#[async_trait]
pub trait Workload: Send + Sync + 'static {
    fn name(&self) -> &str;

    async fn setup(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
        Ok(())
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()>;

    async fn check(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
        Ok(())
    }
}

Four methods, two with defaults. The lifecycle is strict: setup(), then run(), then check(). Each phase has its own rules.

The Three Phases

Setup: Prepare the Ground

setup() runs sequentially across workloads. Workload A’s setup completes before workload B’s starts.

Use it to find your servers, open connections and build data structures. When run() starts, every workload is ready.

async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    let server = ctx
        .topology()
        .all_process_ips()
        .first()
        .cloned()
        .ok_or_else(|| SimulationError::InvalidState("no servers".into()))?;
    self.server = Some(server);
    Ok(())
}

Do not hold a connection across setup if chaos is on. The server can reboot before run() starts. Connect inside run() and reconnect on error.

Run: Drive the System

run() is where the action is. All workloads run concurrently: several clients hitting the same servers, competing for resources, triggering races.

Send requests, observe responses, keep a reference model, and assert on every answer. The phase ends when every workload returns or when the runner cancels ctx.shutdown().

async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    for _ in 0..self.operations {
        if ctx.shutdown().is_cancelled() {
            break;
        }
        match self.random_op(ctx.random()) {
            Op::Write { key, value } => {
                self.send_write(ctx, &key, &value).await?;
                self.model.insert(key, value);
            }
            Op::Read { key } => {
                let got = self.send_read(ctx, &key).await?;
                let expected = self.model.get(&key).cloned();
                assert_always!(got == expected, "read returns the last committed write");
            }
        }
    }
    Ok(())
}

Check: Judge the Outcome

check() runs sequentially after every run() has finished and the event queue has drained. The system is quiescent. No message is in flight, no timer is pending.

Use it for final validation. Did the conservation law hold? Did every acknowledged write survive the reboots? The verdict counts: a check() that returns Err or panics fails the seed, exactly as a failing run() does, even when every run() returned Ok.

async fn check(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
    let total = self.model.values().sum::<i64>();
    assert_always!(total == self.deposited - self.withdrawn, "money is conserved");
    Ok(())
}

Workloads Survive Everything

This is the difference from processes. When the simulation kills a server, your workload keeps running. When a connection breaks, your workload reconnects. When the network partitions, your workload sees the errors and adapts.

Your workload remembers the whole run, across every reboot. That is how you verify recovery: the workload sent a write before the crash, the server rebooted, and the workload reads the value back.

A workload spawns tasks too, through ctx.task().spawn_task(). Those tasks carry no process scope, so they live as long as the workload does. A panic in a detached task is recorded against the workload and fails the seed unless the handle is awaited and the error handled.

The Context

Every phase receives a SimContext. The parts a workload uses:

The last three let a workload compare what a process reports about itself against what the simulator knows. A process that says “my disk is 90% busy” can be checked against ctx.disk_counters(ip). A workload that sees a slow answer can ask ctx.slowness(ip) whether the machine is in a gray failure, and assert that a healthy one answers fast. CPU Time and Gray Failures builds on this.

The Operation Alphabet

Strong workloads define an operation alphabet: the set of actions they can take. Deposit, withdraw, read, transfer, small delay. Each has a weight:

fn random_op(&self, random: &SimRandomProvider) -> Op {
    match random.random_range(0..100) {
        0..30 => Op::Deposit { .. },
        30..50 => Op::Withdraw { .. },
        50..70 => Op::Read { .. },
        70..90 => Op::Transfer { .. },
        _ => Op::SmallDelay,
    }
}

The weights matter. Too many reads and you never test write conflicts. Too many writes and you never test read-after-crash. Cover normal operations, adversarial inputs, and delays that let background work run.

Then let the seeds vary the alphabet. With SimulationBuilder::swarm_operations(), each seed enables a random subset of up to 256 operation ids, and swarm_op_enabled(op_id) tells the workload which ones:

let enabled: Vec<Op> = ALPHABET
    .iter()
    .copied()
    .filter(|op| swarm_op_enabled(op.id()))
    .collect();

Why? Some bugs only appear when an operation is absent. A compaction that is never triggered, a read that never races a write. A single workload with a swarmed alphabet finds those across seeds. Many separate workloads, one per scenario, do not. Prefer one workload, many behaviours. Swarm Testing has the evidence, and Designing Workloads That Find Bugs has the patterns.

Multiple Instances

One client is rarely enough. workloads() builds several from a factory:

SimulationBuilder::new()
    .workloads(WorkloadCount::Fixed(3), |i| Box::new(ClientWorkload::new(i)))

Each instance gets its own client_id() and its own IP. WorkloadCount::Random(1..6) draws a different count per seed from the simulation stream, so it stays deterministic per seed. The factory receives the index, and the workload is responsible for a unique name(), such as format!("client-{i}").

Factory workloads are also what check_determinism() and exploration need: an instance workload would carry its state from one timeline into the next.

What a Good Workload Looks Like

1. Define an operation alphabet with weights, and swarm it.
2. Keep a reference model of the expected state.
3. Assert on every response with assert_always!.
4. Mark interesting paths with assert_sometimes!, so the run stops only when they fired.
5. Validate the final state in check() against the model.
6. Publish facts through ctx.state() and plain tracing events, so invariants can check cross-workload properties.

Your First Simulation builds one from scratch.

Your First Simulation

• What We Are Building
• The Destination
• Prerequisites
• The Plan
• Why This Order

Part II covered the theory: deterministic execution, providers, and the split between Process and Workload. Now we build a complete simulation and run it.

What We Are Building

Over the next five chapters we write a simulation test for a small key-value server. At the end you have:

• A Process: the key-value server. It accepts TCP connections and answers get and set requests. The simulator can crash it and reboot it.
• A Workload: the test driver. It sends random operations, keeps a reference model of what the server should contain, and checks every answer.
• Assertions inside both, so the report tells us what the run found and what it reached.
• A SimulationBuilder configuration that runs the pair under chaos for many seeds.

The server is deliberately small. The interesting part is not its logic. It is how the simulation wraps around it and finds bugs that unit tests never reach.

The Destination

This is the main we are working towards. Every call in it gets its own section in the next chapters.

use std::time::Duration;

use moonpool_sim::{
    Attrition, AttritionScope, AttritionVictims, Chaos, ChaosMode, SimulationBuilder,
    SimulationError,
};

fn main() -> Result<(), SimulationError> {
    // WARN keeps a 100-seed campaign quiet; raise it when one seed misbehaves.
    moonpool_sim::init_sim_tracing(tracing::Level::WARN);

    let report = SimulationBuilder::new()
        .processes(3, || Box::new(KvServer))
        .workload(KvWorkload::new(200))
        .enable_chaos([
            Chaos::Network(ChaosMode::Swarm),
            Chaos::Attrition {
                config: Attrition {
                    max_dead: 1,
                    prob_graceful: 0.3,
                    prob_crash: 0.5,
                    prob_wipe: 0.2,
                    recovery_delay_ms: None,
                    grace_period_ms: None,
                    scope: AttritionScope::PerProcess,
                    victims: AttritionVictims::Any,
                    wipe_budget: None,
                },
                mode: ChaosMode::Swarm,
            },
        ])
        .chaos_duration(Duration::from_secs(30))
        .set_iterations(100)
        .run()?;

    report.eprint();
    if report.is_failure() {
        std::process::exit(1);
    }
    Ok(())
}

Three servers, one client, network faults and process reboots for 30 simulated seconds, 100 seeds. run() returns the report. is_failure() is true when a seed failed, an always-assertion was violated, or the assertion table overflowed.

Prerequisites

The repository builds inside the Nix shell. Every cargo command runs through nix develop:

nix develop --command cargo build

Simulation binaries live in crates/moonpool-sim-examples/src/bin/sim/ and are managed by xtask:

cargo xtask sim list          # what exists
cargo xtask sim run tonic     # run binaries whose name matches "tonic"
cargo xtask sim run-all       # run every binary

Your own simulation can be a plain #[test] or a binary. Both call the same builder. Running and Observing covers both routes and the report they print.

The Plan

1. Defining a Process: implement the Process trait. Bind a listener, parse requests, honour the shutdown token, lose all memory on reboot.
2. Writing a Workload: implement the Workload trait. Pick random operations, keep a reference model, assert on every answer, finish in bounded simulated time.
3. Configuring the SimulationBuilder: wire the pair together. Every public builder method, with a link to the chapter that explains it.
4. Running and Observing: run the campaign, read the report, know when it succeeded, and know what stops it.
5. Process Groups, Tags and Topology: grow from one role to many. Groups, tags, failure domains, and how a workload finds each of them.

Why This Order

We build bottom-up. The Process comes first because it is the system under test. The Workload comes second because it exercises the Process. The builder ties them together. Running comes last because the report only makes sense once every piece is in place.

Each chapter produces code that the next one uses. After Running and Observing you have a working simulation. The chaos chapters that follow, starting with Chaos in Moonpool, extend it with more fault surfaces.

One rule holds through all of it: the server code is the code you ship. It depends on provider traits, never on tokio directly, so the same KvServer runs under TokioProviders in production and under the simulator’s providers here. Same Code in Production and Simulation explains why this is the foundation everything else rests on.

Defining a Process

• The Process Trait
• The Process Struct
• Implementing run()
• Handling Requests
• Registering with the Builder
• What a Process Knows About Itself
• What the Shutdown Token Gives You
• Key Takeaways

Our key-value server is a Process. It listens for connections, answers get and set requests, and respects the shutdown token. Everything it does goes through providers, never through raw tokio calls.

The Process Trait

The trait has two methods. name() identifies the process type. run() is the main loop, called on every boot.

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Process: Send + Sync + 'static {
    fn name(&self) -> &str;
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()>;
}
}

run() returns when the process exits on its own, or never returns because the simulator cancelled it. Both are normal. The trait is Send + Sync + 'static and uses plain #[async_trait], so the process can share state through Arc<RwLock<..>> like any production server.

The Process Struct

A Process is created from a factory on every boot, so the struct starts empty.

#![allow(unused)]
fn main() {
use async_trait::async_trait;
use moonpool_sim::{Process, SimContext, SimulationResult};

struct KvServer;
}

No fields. Each boot gets a fresh KvServer. Everything it accumulates lives until the next crash. A server that must survive reboots writes through ctx.storage(), and Storage Faults shows what that costs.

Implementing run()

The main loop binds a listener, then races accept() against the shutdown token.

#![allow(unused)]
fn main() {
use std::collections::BTreeMap;

use moonpool_sim::{NetworkProvider, TcpListenerTrait};

#[async_trait]
impl Process for KvServer {
    fn name(&self) -> &str {
        "kv"
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        // In memory on purpose: a crash loses it, and the workload's
        // reference model must cope with that.
        let mut store: BTreeMap<String, Vec<u8>> = BTreeMap::new();

        loop {
            let accepted = moonpool_sim::select! {
                biased;
                accepted = listener.accept() => accepted,
                () = ctx.shutdown().cancelled() => return Ok(()),
            };
            if let Ok((stream, _addr)) = accepted {
                handle_connection(stream, &mut store).await;
            }
        }
    }
}
}

Binding the listener. ctx.network().bind(ctx.my_ip()) registers this process’s IP for incoming connections. No real port opens. The NetworkProvider and TcpListenerTrait traits must be in scope for bind() and accept().

The store. A BTreeMap on the stack. When the process crashes, the map is gone. The ordered map also keeps iteration deterministic, which HashMap does not.

The select. moonpool_sim::select! is the deterministic twin of tokio::select!. With biased; it polls the branches in order and draws no randomness. Without biased; the branch offset comes from the seed, so the same seed still replays the same choice. tokio::select! is forbidden in simulated code: its branch offset comes from entropy the seed does not control.

The shutdown token. During a graceful reboot the simulator cancels ctx.shutdown() and waits a grace period. The return Ok(()) branch lets the process exit cleanly. During a crash reboot the simulator aborts the task at once, so this branch never runs.

Handling Requests

The connection handler parses a tiny wire protocol: G<key> for get and S<key_len><key><value> for set. A production system keeps its own framing or hands the stream to hyper, as The hyper Stack shows. A raw protocol shows the provider boundary directly.

#![allow(unused)]
fn main() {
use futures::io::{AsyncReadExt, AsyncWriteExt};
use moonpool_sim::network::sim::SimTcpStream;

async fn handle_connection(mut stream: SimTcpStream, store: &mut BTreeMap<String, Vec<u8>>) {
    let mut buf = vec![0u8; 4096];
    loop {
        let n = match stream.read(&mut buf).await {
            Ok(0) | Err(_) => return, // closed, or a fault closed it for us
            Ok(n) => n,
        };
        let request = &buf[..n];
        let response = match request[0] {
            b'G' => {
                let key = String::from_utf8_lossy(&request[1..]);
                store.get(key.as_ref()).cloned().unwrap_or_default()
            }
            b'S' => {
                let key_len = usize::from(request[1]);
                let key = String::from_utf8_lossy(&request[2..2 + key_len]).to_string();
                let value = request[2 + key_len..].to_vec();
                store.insert(key, value.clone());
                value
            }
            _ => Vec::new(),
        };
        let _ = stream.write_all(&response).await;
    }
}
}

SimTcpStream implements futures::io::AsyncRead and AsyncWrite. The reads and writes are buffer-driven: a read completes on the current poll when bytes are waiting, and otherwise parks until a Delivery event lands them. Under network chaos a read can return an error or zero bytes at any point. The handler treats both as the end of the connection and returns.

Registering with the Builder

.processes() takes a count and a factory. The factory runs on every boot.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
}

This creates three servers at 10.0.1.1, 10.0.1.2 and 10.0.1.3. Each runs independently. Each can be rebooted independently.

A range makes the cluster size part of the seed:

#![allow(unused)]
fn main() {
.processes(3..=7, || Box::new(KvServer))
}

Every seed draws a count between 3 and 7 from the simulation stream. The same seed draws the same count. One .processes() call is one group, named after name(). A system with several roles registers one group per role, and Process Groups, Tags and Topology covers how they share the address plan.

What a Process Knows About Itself

ctx.my_ip() is its address. ctx.client_id() is its index within its group and ctx.client_count() is the group’s size, so three servers number themselves 0, 1 and 2. ctx.topology() answers the rest: its tags, its group, the other process IPs. A process that must crash itself at an exact point of its own protocol calls ctx.crash_self(RebootKind::Crash, None), which Correlated Outages and Self-Crashes describes.

What the Shutdown Token Gives You

Checking ctx.shutdown() is optional. If the process ignores it, a graceful reboot still works: the simulator force-kills the task when the grace period ends. Checking it tests a different code path from a hard crash. The process can:

• Finish in-flight requests.
• Flush write buffers.
• Close connections so peers see EOF instead of a reset.

Attrition: Process Reboots covers the three reboot kinds and the grace period.

Key Takeaways

Every Process follows the same pattern:

1. Bind a listener with ctx.network().
2. Race accept() against ctx.shutdown() with moonpool_sim::select!.
3. Use ctx.time() for every sleep and timeout.
4. Keep state in memory and expect to lose it, or write it through ctx.storage().
5. Return Ok(()) on shutdown.

The factory produces a blank instance. The simulator manages the lifecycle. Your job is the server logic.

Writing a Workload

• The Workload Trait
• The Workload Struct
• Setup: Finding the Servers
• Run: The Operation Loop
• Why run() Must Finish
• Check: Final Validation
• Events for Invariants
• One Instance or Many
• Patterns That Find Bugs

The workload drives our key-value server and checks that it behaves. It sends requests, keeps a reference model of the expected state, and asserts on every answer while the run is still going.

The Workload Trait

A workload has three lifecycle phases. The runner calls them in order.

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Workload: Send + Sync + 'static {
    fn name(&self) -> &str;
    async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> { Ok(()) }
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()>;
    async fn check(&mut self, ctx: &SimContext) -> SimulationResult<()> { Ok(()) }
}
}

• setup runs for every workload before any run() starts. Bind listeners, find peers, clear state.
• run is the main logic. All workloads run concurrently, under chaos when it is enabled.
• check runs after every run() has returned and the world has settled. An Err here fails the seed exactly like a failing run().

Unlike a Process, a workload survives reboots. The servers die and come back. The test driver keeps running and keeps its model.

The Workload Struct

#![allow(unused)]
fn main() {
use std::collections::BTreeMap;
use std::time::Duration;

use async_trait::async_trait;
use futures::io::{AsyncReadExt, AsyncWriteExt};
use moonpool_sim::{
    NetworkProvider, RandomProvider, SimContext, SimulationError, SimulationResult,
    TimeProvider, Workload, assert_always, assert_sometimes,
};

struct KvWorkload {
    /// Operations per seed. Finite, so the run ends.
    num_ops: usize,
    /// What the servers should contain, as far as this client knows.
    model: BTreeMap<String, Vec<u8>>,
    keys: Vec<String>,
}

impl KvWorkload {
    fn new(num_ops: usize) -> Self {
        Self {
            num_ops,
            model: BTreeMap::new(),
            keys: (0..16).map(|i| format!("k{i}")).collect(),
        }
    }
}
}

The model is the important field. Every write goes to the server and to the model. Every read is compared against the model.

Setup: Finding the Servers

ctx.topology() knows every participant: which IPs are processes, which groups and tags they carry, which are other workloads.

#![allow(unused)]
fn main() {
#[async_trait]
impl Workload for KvWorkload {
    fn name(&self) -> &str {
        "kv-client"
    }

    async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        if ctx.topology().all_process_ips().is_empty() {
            return Err(SimulationError::InvalidState("no kv servers".into()));
        }
        // A `.workload()` instance is reused across seeds: start each one clean.
        self.model.clear();
        Ok(())
    }
}

Three lookups cover most cases:

#![allow(unused)]
fn main() {
let servers = ctx.topology().all_process_ips();          // every process IP
let one = ctx.peer("kv");                                 // first process named "kv"
let leaders = ctx.topology().ips_tagged("role", "leader"); // by tag
let acceptors = ctx.topology().ips_in_group("acceptor");   // by group
}

Run: The Operation Loop

The loop picks a server, picks an operation, and checks the answer. Every random choice comes from ctx.random(), so it replays with the seed.

#![allow(unused)]
fn main() {
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let servers = ctx.topology().all_process_ips().to_vec();

        for i in 0..self.num_ops {
            if ctx.shutdown().is_cancelled() {
                break;
            }
            let server = &servers[ctx.random().random_range(0..servers.len())];
            let key = self.keys[ctx.random().random_range(0..self.keys.len())].clone();

            match ctx.random().random_range(0..100) {
                0..40 => {
                    let value = format!("v{i}").into_bytes();
                    match self.send_set(ctx, server, &key, &value).await {
                        Ok(()) => {
                            self.model.insert(key, value);
                            assert_sometimes!(true, "kv: set succeeded");
                        }
                        // A fault is an outcome, not a failure. Reaching it is coverage.
                        Err(_) => assert_sometimes!(true, "kv: set failed under chaos"),
                    }
                }
                40..80 => {
                    if let Ok(value) = self.send_get(ctx, server, &key).await {
                        let expected = self.model.get(&key).cloned().unwrap_or_default();
                        assert_always!(
                            value == expected,
                            "kv: read matches the model",
                            { "key" => key, "server" => server }
                        );
                    }
                }
                _ => {
                    let _ = ctx.time().sleep(Duration::from_millis(10)).await;
                }
            }
        }
        Ok(())
    }
}

Two assertion kinds work together here.

assert_always! guards an invariant. A get that disagrees with the model is a definite bug. The optional { key => value } block prints with the seed when it fails, so the report names the key and the server.

assert_sometimes! marks a path that must fire at least once across the campaign. If "kv: set failed under chaos" never fires, the chaos is too weak. If "kv: set succeeded" never fires, the client cannot reach the servers at all.

The request helpers are ordinary provider code. Each one is bounded by a simulated timeout, because a connect to a server mid-reboot can hang until the network heals:

#![allow(unused)]
fn main() {
impl KvWorkload {
    async fn send_get(&self, ctx: &SimContext, server: &str, key: &str) -> SimulationResult<Vec<u8>> {
        let io = async {
            let mut stream = ctx.network().connect(server).await?;
            stream.write_all(format!("G{key}").as_bytes()).await?;
            let mut buf = vec![0u8; 4096];
            let n = stream.read(&mut buf).await?;
            Ok::<_, std::io::Error>(buf[..n].to_vec())
        };
        match ctx.time().timeout(Duration::from_secs(2), io).await {
            Ok(result) => Ok(result?),
            Err(_) => Err(SimulationError::InvalidState("get timed out".into())),
        }
    }
}
}

send_set is the same shape with the S framing from Defining a Process.

Why run() Must Finish

A workload returns when it is done. The runner takes the first Ok from any workload as the signal to cancel ctx.shutdown() for every process and every other workload (except under chaos_until_run_ends(), where the runner waits for the last one). A workload that loops forever keeps the seed alive until the stall guards declare a deadlock. Count operations, or sleep for a fixed simulated duration, then return. Running and Observing covers the guards.

A model that only knows what this client wrote is wrong under crashes: a Crash reboot loses the server’s in-memory map, so a later get returns empty where the model expects a value. That is the first bug this workload finds, and it is a real one. The fix belongs in the server, through ctx.storage(), not in the assertion.

Check: Final Validation

check() runs after the world is quiet. No messages in flight, no faults, processes aborted. (With chaos_until_run_ends() it runs on the live processes instead, after the cutoff: the quiet tail where the workload reads the final state back. See Chaos in Moonpool.) It is the place for whole-run properties:

#![allow(unused)]
fn main() {
    async fn check(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
        assert_always!(
            self.model.len() <= self.keys.len(),
            "kv: model never holds more keys than the alphabet"
        );
        Ok(())
    }
}
}

check() runs inside the event loop, so network RPCs still work. A workload can also read the captured timeline here through ctx.observability(), which implements TraceQuery.

Events for Invariants

Per-call checks live in the workload. Cross-workload properties live in invariants, which run after every simulation step and read tracing events. The workload emits plain production-style events:

#![allow(unused)]
fn main() {
// After a successful set: the same line an OpenTelemetry subscriber would see.
tracing::info!(key = %key, server = %server, "kv_set_committed");
}

An invariant registered on the builder scans "kv_set_committed" events with a cursor. Events and Invariants covers the capture rule, TraceQuery, and the Invariant trait. For shared values that are not events, ctx.state().publish("key", value) and ctx.state().get::<T>("key") give workloads and fault injectors a typed blackboard.

One Instance or Many

The builder offers three registrations:

• .workload(KvWorkload::new(200)) keeps one instance across every seed. It gets client_id 0. Clear its state in setup().
• .workloads(WorkloadCount::Random(1..6), |i| Box::new(KvWorkload::new(200))) builds fresh instances each seed, 1 to 5 of them, each with its own client_id and client_count.
• .workload_factory(|| Box::new(KvWorkload::new(200))) builds one fresh instance each seed.

Exploration and check_determinism() run a seed more than once and require the factory forms. Designing Workloads That Find Bugs explains why one workload with a wide operation alphabet beats several narrow ones.

Patterns That Find Bugs

Reference model. Track expected state. Compare every answer. Any divergence is a bug.

Weighted operation alphabet. Mix writes, reads and pauses, and let swarm testing vary the mix per seed.

Both assertion kinds. assert_always! for what must hold on every call. assert_sometimes! for what the campaign must reach.

Errors are outcomes. A network error under chaos is expected. Log it, count it, move on. The bug is a wrong answer, never an absent one.

Configuring the SimulationBuilder

• The Minimal Builder
• Adding Processes
• Iteration Control
• Invariants
• Chaos
• Every Builder Method
• Putting It All Together

The SimulationBuilder is the glue. It takes your Process, your Workload, your invariants and your chaos configuration, and wires them into a runnable campaign. This chapter walks the calls our key-value simulation needs, then lists every public method with the chapter that explains it.

The Minimal Builder

The smallest simulation has one workload and no processes.

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .workload(KvWorkload::new(100))
    .run()?;
}

This places the workload at 10.0.0.1, runs it under the default iteration control, and returns a SimulationReport. run() returns Err(SimulationError::InvalidConfiguration) when the builder cannot run as written: a fault injector or Chaos::Attrition without chaos_duration, or an instance workload under exploration or check_determinism. In that case no seed ran. Failing seeds are never an Err. They are in the report.

Adding Processes

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(100))
    .run()?;
}

Three servers at 10.0.1.1 to 10.0.1.3, one client at 10.0.0.1. The workload finds the servers through ctx.topology().all_process_ips(). Call .processes() once per role. Each call is a group on its own 10.0.{g+1}.x range, and Process Groups, Tags and Topology covers groups, .cluster() and .tags().

Iteration Control

One seed is one schedule. Bugs hide in rare interleavings, so a campaign runs many seeds. IterationControl has two modes.

UntilCoverageStable is the default, with plateau_seeds: 10 and max_iterations: 1000. It stops once every observed assert_sometimes! and assert_reachable! has fired and coverage has not grown for plateau_seeds consecutive seeds. Tune it with:

#![allow(unused)]
fn main() {
.until_coverage_stable(10, 5_000) // 10 quiet seeds, 5,000 safety cap
}

FixedCount runs exactly n seeds:

#![allow(unused)]
fn main() {
.set_iterations(100)
}

Generated seeds derive from a wall-clock base seed, so two campaigns explore different seeds. To replay, pin them:

#![allow(unused)]
fn main() {
.set_debug_seeds(vec![42, 7891])
.set_iterations(2)
}

set_debug_seeds supplies the first seeds. set_iterations bounds the count. Without the second call the default control keeps drawing fresh seeds after 42 and 7891. Reproducing with FixedCount is the debugging recipe.

Invariants

Invariants run after every simulation step and read the captured tracing timeline. They catch cross-workload properties while the run is still going.

#![allow(unused)]
fn main() {
use std::cell::Cell;
use moonpool_sim::{Invariant, TraceQuery, assert_always};

struct SingleLeader { cursor: Cell<usize> }

impl Invariant for SingleLeader {
    fn name(&self) -> &str { "single_leader" }

    fn observe(&self, q: &dyn TraceQuery, _sim_time_ms: u64) {
        // A cursor scan: only the events since the last step, never the whole timeline.
        for e in q.since("leader_elected", &self.cursor) {
            assert_always!(e.u64("term").is_some(), "leader_elected carries a term");
        }
    }

    fn reset(&mut self) { self.cursor.set(0); }
}

// On the builder:
.invariant(SingleLeader { cursor: Cell::new(0) })
.invariant_fn("kv_commits_name_a_key", |q, _t| {
    for e in q.snapshot("kv_set_committed") {
        assert_always!(e.str("key").is_some(), "kv_set_committed carries a key");
    }
})
}

Events and Invariants covers emission, capture and the query API.

Chaos

enable_chaos takes a list of surfaces, each with a ChaosMode. Random samples the full surface every seed. Swarm enables a random subset of its fault families per seed and turns the rest fully off.

#![allow(unused)]
fn main() {
use moonpool_sim::{Attrition, AttritionScope, AttritionVictims, Chaos, ChaosMode};

SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(200))
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition {
            config: Attrition {
                max_dead: 1,
                prob_graceful: 0.3,
                prob_crash: 0.5,
                prob_wipe: 0.2,
                recovery_delay_ms: None,
                grace_period_ms: None,
                scope: AttritionScope::PerProcess,
                victims: AttritionVictims::Any,
                wipe_budget: None,
            },
            mode: ChaosMode::Swarm,
        },
    ])
    .set_iterations(100)
    .run()?;
}

chaos_duration opens the window in which new faults may be injected. When it closes, the runner cancels the injectors and enters recovery mode: every configuration-driven fault family switches off and held partitions heal. Damage already done stays done. The quiet tail between the cutoff and workload completion is where the system under test recovers, and Chaos in Moonpool describes that lifecycle. Each Chaos variant has its own chapter:

Every Builder Method

Three groups of methods consume no randomness: the masks, the deployment-shape settings (tcp_send_window_bytes, accept_backlog_capacity, link_latency) and trace_level. Adding them to a campaign changes no seed’s schedule. Every Chaos surface, every ranged count and swarm_operations draw from the simulation stream in a fixed order, so adding one shifts every seed. That is intended, and Seed-Driven Reproducibility explains the one-stream rule behind it.

Putting It All Together

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .tags(&[("role", &["primary", "replica"])])?
    .workloads(WorkloadCount::Random(1..4), |_| Box::new(KvWorkload::new(200)))
    .invariant(SingleLeader { cursor: Cell::new(0) })
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::BuggifyKnobs,
    ])
    .swarm_operations()
    .until_coverage_stable(10, 2_000)
    .run()?;
}

The builder creates the simulated world, assigns IPs, seeds the stream, runs the orchestration loop, collects assertions and metrics, and produces the report. Running and Observing reads that report.

Running and Observing

• Two Ways to Run
• Reading the Report
• What Success Means
• When Things Fail
• Stop Conditions
• Stall Detection
• Checking Determinism
• Exit Codes
• The Feedback Loop

You have a Process, a Workload and a builder. Now we run the campaign, read the report, and learn what stops a run.

Two Ways to Run

A simulation is ordinary Rust that calls SimulationBuilder::run(). There is no runtime to build: run() drives each seed on a fresh deterministic executor on one OS thread. A bare tokio::spawn inside it panics, because there is no tokio runtime.

As a test. Put the builder in a #[test] and run it with nextest:

nix develop --command cargo nextest run -p moonpool-sim process_groups --nocapture

Fast, coverage-blind, and the way every integration test under crates/moonpool-sim/tests/ runs.

As a binary. The examples in crates/moonpool-sim-examples/src/bin/sim/ are the pattern. xtask builds them with sancov instrumentation and runs them:

cargo xtask sim list            # every simulation binary
cargo xtask sim run tonic       # binaries whose name matches
cargo xtask sim run-all         # all of them

Seeds and iteration counts are builder calls, not CLI flags. A binary’s main is short:

fn main() {
    moonpool_sim::init_sim_tracing(tracing::Level::WARN);

    let report = SimulationBuilder::new()
        .processes(3, || Box::new(KvServer))
        .workload(KvWorkload::new(200))
        .enable_chaos([Chaos::Network(ChaosMode::Swarm)])
        .set_iterations(100)
        .run()
        .expect("simulation configuration is valid");

    report.eprint();
    if report.is_failure() {
        std::process::exit(1);
    }
}

init_sim_tracing installs a stdout tracing_subscriber::fmt at the given level. There is no RUST_LOG plumbing. expect is right here: an Err from run() is a configuration mistake, not a test result. Running Simulation Binaries with xtask covers registering a new binary.

Reading the Report

report.eprint() prints to stderr, coloured when stderr is a terminal and NO_COLOR is unset. format!("{report}") is the same text without colour. A healthy campaign looks like this:

━━━ Simulation Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  100 iterations   100 passed   0 failed   ✓ 100.0%

  Wall Time    12ms avg        1.20s total
  Sim Time     45.23s avg
  Events       8,432 avg

━━━ Assertions (4) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓  always       "kv: read matches the model"   12,847 pass  0 fail
  ✓  always       "kv: model never holds more..." 100 pass  0 fail
  ✓  sometimes    "kv: set failed under chaos"   412 / 6,514  (6.3%)
  ✓  sometimes    "kv: set succeeded"            6,102 / 6,514  (93.7%)

The lines that matter:

• ✓ 100.0%: no seed returned an error, panicked or deadlocked.
• 0 fail on an always row: no invariant violation.
• ✓ on a sometimes row: the path fired at least once. A ○ marks one that never did.

Four more sections appear when there is something to show. Violations lists every failed always-assertion. Coverage Gaps lists every sometimes or reachable assertion that never fired. Saturated or Saturation FAILED explains how an UntilCoverageStable run stopped. Seeds lists each seed with its wall time, sim time and event count, failing seeds in red.

What Success Means

A campaign succeeds when two conditions hold at once:

1. No always-assertion violated. Every assert_always! passed on every evaluation, on every seed.
2. Every sometimes-assertion fired. Every assert_sometimes! and assert_reachable! was true at least once across the campaign.

Both matter. A run that never violates an invariant but never reaches an error path is not testing enough. A run that reaches every path but tolerates wrong answers is not checking enough.

report.is_failure() covers the first condition and more: it is true when seeds_failing is non-empty, when assertion_violations is non-empty, or when dropped_assertion_allocations is above zero, which means the assertion table overflowed and some evaluations were never counted. Coverage is deliberately not part of it. Check coverage_violations and convergence_timeout yourself when CI should fail on a coverage gap.

When Things Fail

━━━ Simulation Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  100 iterations   98 passed   2 failed   ✗ 98.0%

  Wall Time    12ms avg        1.20s total
  Sim Time     45.23s avg
  Events       8,432 avg

  Faulty seeds: [7891, 42033]

━━━ Assertions (4) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✗  always       "kv: read matches the model"   12,800 pass  47 fail
  ...

━━━ Violations ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✗  assert_always!('kv: read matches the model') failed 47 times out of 12,847

The report says which seeds failed, which assertion broke, and how rare it was. An assertion written with a detail block also printed [ASSERTION FAILED] kv: read matches the model (seed=7891) | key=k3 server=10.0.1.2 on stderr the moment it failed, which is why the workload chapter attached the key and the server. Debugging a Failing Seed takes it from here: pin seed 7891 with set_debug_seeds(vec![7891]) and set_iterations(1), raise the trace level, read backwards from the first violation.

Stop Conditions

How long should a campaign run? Any fixed count is wrong: too small misses bugs, too large burns time. The right answer is to stop on a signal, not a count.

UntilCoverageStable { plateau_seeds, max_iterations } is the default. It stops when every observed sometimes, numeric-sometimes and reachable assertion has fired and coverage has not grown for plateau_seeds consecutive seeds. Under cargo xtask sim run the binary is sancov-instrumented, so the signal is real code coverage, the count of distinct edges exercised. Under cargo nextest run there is no instrumentation and it falls back to assertion coverage, the set of coverage messages that fired. The Saturated section names which signal it used. A campaign with no coverage assertion stops on the plateau alone and logs a warning. max_iterations is a safety cap: hitting it sets report.convergence_timeout = true and prints Saturation FAILED, so CI can fail loudly instead of treating “we ran out of seeds” as success. No fork happens unless enable_exploration is set.

FixedCount(n) runs exactly n seeds. Use it to replay, and when a budget must be predictable.

Stall Detection

A seed must end. Four deterministic guards make sure it does, each a pure function of the simulated schedule, so a replay hits the same guard at the same step.

• No progress. More than three consecutive loop passes with an empty event queue and no workload finished trigger a shutdown request. The same count again after the request declares a deadlock.
• Same-time loop. 100,000 consecutive events at one logical time, such as a re-armed zero-duration sleep, trigger a shutdown request, then a deadlock.
• Run-time budget. Simulated time past run_time_budget (one simulated hour by default) with workloads still running triggers a shutdown request. Another full budget past that declares a deadlock. This catches a detached task that re-arms a timer every tick while the workload waits on it, which the first guard cannot see. Setup and check phases use the same guard.
• Settle timeout. After every workload returns, processes are aborted and the orchestrator drains the queue without yielding. Five simulated minutes of events still pending fails the seed with SimulationError::SettleTimeout, which surfaces leaked timers and unclosed connections.

A deadlocked seed stops the whole campaign. The report still carries everything earlier seeds gathered, with the deadlocked seed in seeds_failing and convergence_timeout left false.

Two more things end a campaign early. A workload task that panics or is cancelled before it returns its Box<dyn Workload> stops the run, because reusing the survivors would hand one workload another’s identity. An ordinary Err from a workload keeps the instance and fails only that seed. A process that panics fails its seed as well.

Checking Determinism

check_determinism() runs every seed twice and compares a fingerprint after every draw on the simulation stream. The replay is a full run, judged like one. Any uncontrolled difference, a HashMap iterated in random order, a wall-clock read, a static that survives a run, fails the seed with the always-assertion "determinism canary: replay matched the recorded draw sequence" and names the first diverging draw. It doubles the cost and needs factory workloads. Turn it on when a seed stops reproducing, and Reproducing with FixedCount shows how to bisect from there.

Exit Codes

A binary exits 0 on success and 1 on failure. Gate on is_failure():

#![allow(unused)]
fn main() {
if report.is_failure() {
    eprintln!(
        "ERROR: {} seeds failed {:?}, {} violations, {} dropped allocations",
        report.seeds_failing.len(),
        report.seeds_failing,
        report.assertion_violations.len(),
        report.dropped_assertion_allocations
    );
    std::process::exit(1);
}
}

Gating on seeds_failing alone lets a run with partial assertion accounting pass. The examples’ finish_or_exit_on_failure helper in crates/moonpool-sim-examples/src/support.rs is exactly this.

The Feedback Loop

Write a workload, run it, find a bug, fix the bug, add an assertion so it stays fixed, run again. The assertion table is the scoreboard. Every row ✓ with high counts means the test is thorough and the system is correct. A ○ means a path the chaos is not reaching. A ✗ means a real bug.

Process Groups, Tags and Topology

• Groups
  • IP allocation
  • Asking the topology
  • One fault regime per group
• Tags
• .cluster(): failure domains
  • Zero-process draws
  • Locality queries
  • What a cluster unlocks
• Choosing Between Them

Our key-value simulation has one kind of server. Real systems have several: a consensus tier and a pool of spares, acceptors and the matchmakers that reconfigure them, a storage tier spread over three datacenters. The builder models all of that with three orthogonal tools. Groups say which role a process plays. Tags label processes inside a group. Failure domains say which processes share fate.

Groups

Every .processes() call registers one group, named after its process type’s name(). Each group has its own factory and its own per-seed count.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3..=5, || Box::new(Acceptor))   // name() == "acceptor"
    .processes(0..=3, || Box::new(Matchmaker)) // name() == "matchmaker"
}

The counts are drawn from the simulation stream in registration order. Two things follow. A seed replays the same shape. And each role’s size varies independently of the others’, so one campaign covers three acceptors with no matchmaker and five acceptors with three.

Registering the same name twice panics: the name is the group’s identity, so each role needs its own process type or its own name().

IP allocation

Every group owns its own /24. The g-th group registered, counting from zero, lives on 10.0.{g+1}.{1..=N}. Workloads live on 10.0.0.x.

A group’s members are contiguous and recognisable at a glance, and a single-group builder keeps the 10.0.1.x addresses it always had. The plan holds 255 groups of 255 processes. A group that draws zero processes this seed keeps its range: later groups do not shift down.

Asking the topology

ctx.topology() answers group questions from a process or a workload:

Every answer is read live: a process a workload spawns mid-run shows up once it boots, and a retired one is gone at once (see Changing the Deployment Mid-Run).

Inside a process, ctx.client_id() and ctx.client_count() are its index within and the size of its own group, now. An acceptor numbers itself 0..n without knowing how many matchmakers this seed drew. The integration test crates/moonpool-sim/tests/process_groups.rs pins all of this:

#![allow(unused)]
fn main() {
async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    let topo = ctx.topology();
    let members = topo.ips_in_group(self.role);
    assert_always!(
        topo.my_group().as_deref() == Some(self.role),
        "process_groups: a process knows its own group"
    );
    assert_always!(
        ctx.client_count() == members.len(),
        "process_groups: client_count is the group size"
    );
    assert_always!(
        members.get(ctx.client_id()).is_some_and(|ip| ip == ctx.my_ip()),
        "process_groups: client_id indexes the group's members"
    );
    // ...
}
}

A workload tells “no matchmakers this seed” from “this system has no matchmakers” by checking groups() against ips_in_group("matchmaker").

One fault regime per group

Groups are what role-aware faults aim at. Each Chaos::Attrition entry adds one injector, and AttritionVictims::group(..) scopes its draw and its max_dead budget to one pool:

#![allow(unused)]
fn main() {
.enable_chaos([
    Chaos::Attrition { config: regime(1, AttritionVictims::group("acceptor")), mode: ChaosMode::Random },
    Chaos::Attrition { config: regime(2, AttritionVictims::group("matchmaker")), mode: ChaosMode::Random },
])
.chaos_duration(Duration::from_secs(10))
}

At most one dead acceptor and at most two dead matchmakers, drawn independently. Chaos::Outage names groups the same way: Outage::groups(["acceptor"]) crashes every acceptor at one instant. ReplicatedFaults::new(DomainLevel::Zone).group("node") spreads storage damage over one group’s domains. A custom injector reads ctx.ips_in_group(..) on its FaultContext. Attrition, Correlated Outages and Aiming Storage Faults cover each.

Tags

Tags label processes inside a group. .tags() attaches dimensions to the last .processes() or .cluster() call. Each dimension is distributed round-robin, independently, and each group’s round-robin starts over.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(5, || Box::new(ConsensusNode))
    .tags(&[
        ("role", &["leader", "follower"]),
        ("dc", &["east", "west", "eu"]),
    ])?
}

Process i gets values[i % values.len()] for each dimension: process 0 is role=leader, dc=east, process 1 is role=follower, dc=west, process 2 is role=leader, dc=eu, and so on. .tags() returns Result<Self, SimulationError> and fails with InvalidState when no group precedes it, so the call takes a ? or an expect.

A process reads its own tags with ctx.topology().my_tags().get("role"). ProcessTags::matches("role", "leader") is the boolean form. The TagRegistry behind ctx.topology() maps every process IP to its tags:

#![allow(unused)]
fn main() {
let leaders = ctx.topology().ips_tagged("role", "leader");   // Vec<String>
let tags = ctx.topology().tags_for("10.0.1.2");               // Option<&ProcessTags>
}

The same registry serves fault injectors: FaultContext::reboot_tagged("role", "leader", kind) reboots every match, and AttritionVictims::tagged("role", "voter") scopes a built-in reboot regime by tag instead of by group. Tags are orthogonal to groups: a group is which factory booted you, a tag is a label the builder handed you. Use a group when the roles are different code, and a tag when one process type behaves differently by configuration.

.cluster(): failure domains

.processes() places processes with no notion of where they run. .cluster() lays a group out across a FoundationDB-style hierarchy of datacenter, zone and machine, so collocated processes share fate.

#![allow(unused)]
fn main() {
use moonpool_sim::LocalityConfig;

SimulationBuilder::new()
    // 3 datacenters x 3 zones x 3 machines x 2 processes = 54 processes
    .cluster(LocalityConfig::new(3, 3, 3, 2), || Box::new(TopologyProcess))
}

The LocalityConfig is the spawn spec: the product of its four dimensions is the process count. Each dimension takes a fixed count or a range sampled per seed, so LocalityConfig::new(1..=3, 3, 3, 1) varies the datacenter count. Consecutive process indices share a machine, which is what makes a machine-scoped reboot meaningful. A .cluster() call is a group like any other: it gets its own IP range, its own name, and .tags() may still follow it.

Zero-process draws

A dimension may draw zero, as in LocalityConfig::new(0..=2, 1, 1, 1). The group is then empty for that seed: an optional role the deployment runs without. It gets no IPs. ips_in_group(name) is empty while groups() still lists it. Attrition finds no victim in it. A ReplicatedFaults scoped to it draws the Spared pattern. Every dimension is drawn in order even when an earlier one drew zero, so the shape never changes the draw schedule. The same holds for a plain .processes(0..=3, ..) group.

Locality queries

With a cluster the topology answers where things are:

The topology example in crates/moonpool-sim-examples/src/topology.rs checks these on every boot:

#![allow(unused)]
fn main() {
let collocated = topo.peers_on_my_machine();
assert_always!(
    collocated.len() == PROCESSES_PER_MACHINE - 1,
    format!("expected {} collocated peer(s), saw {}", PROCESSES_PER_MACHINE - 1, collocated.len())
);
let in_dc = topo.ips_in_domain(DomainLevel::Datacenter, locality.datacenter());
assert_always!(in_dc.len() == PROCESSES_PER_DATACENTER, "every datacenter holds the same count");
}

Domain ids are dc1, dc1-z2, dc1-z2-m3: globally unique, so two datacenters never share a machine id.

What a cluster unlocks

• Correlated reboots. AttritionScope::PerMachine, PerZone and PerDatacenter reboot every collocated process together, within the max_dead budget. Under ChaosMode::Swarm the regime picks a scope that fits the budget per seed.
• Distance-dependent latency. .link_latency(LinkLatencyConfig::default()) keeps a cross-datacenter hop slow on a healthy seed. Pairs without locality, workload clients included, are unaffected.
• Replicated fault patterns. .replicated_storage_faults(ReplicatedFaults::new(DomainLevel::Zone)) damages a replicated record in a minority of zones, in a stripe rotation, or one domain at a time, never everywhere.

.processes() and .cluster() mix on one builder, so a consensus tier can be clustered while its clients stay flat.

Choosing Between Them

All three are visible to processes, workloads and fault injectors through the same registries, so a workload that targets “a follower in dc2 that shares a machine with a leader” composes ips_tagged, ips_in_domain and peers_on_my_machine without any new API.

Changing the Deployment Mid-Run

• The handle
• The topology is live
• Chaos sees new members
• Determinism
• Limits and the report
• Example

The builder draws each group’s boot shape once per seed. Real systems do not keep that shape. An operator racks new machines, decommissions old ones, and starts a new cell, all while clients run. A bug that only shows up while membership changes needs a simulation that changes membership.

Moonpool has no operator actor for this. Every workload gets one handle, ctx.deployment(), and an operator is an ordinary workload that uses it.

The handle

#![allow(unused)]
fn main() {
let deployment = ctx.deployment();

// One new process in "server", booted from the group's factory.
let ip = deployment.spawn("server")?;                 // "10.0.1.4"

// The same, placed in a failure domain of a .cluster() group.
let ip = deployment.spawn_in("node", DomainSpec::NewZoneIn {
    datacenter: "dc1".to_string(),
})?;

// A whole new group on the next /24, ranks 0..3.
let ips = deployment.spawn_group("spare", 3, || Box::new(Spare))?;

// Stop a process for good.
deployment.retire("10.0.1.2")?;

// +1 on every spawn that boots and every retire.
let version = deployment.version();
}

A process gets a handle too, but every call returns an error: only a client changes the deployment. A hand-built SimContext carries a detached handle that refuses as well.

The topology is live

WorkloadTopology and FaultContext read one shared Membership per iteration. They do not hold a copy taken at boot.

• ips_in_group, all_process_ips, ips_tagged, ips_in_domain, group_for, tags_for and locality_for include a spawned process once it boots and drop a retired one at once.
• A process’s client_id() and client_count() are its rank and its group’s size now. A spawned process takes the next rank. A retire moves the ranks above it down.
• peer_by_name and peers() list the entities present at boot only. Find spawned processes by group.

Chaos sees new members

• Attrition draws its victim from the live pool at each decision, and max_dead counts against the live pool. A spawned process can be rebooted once it booted, never before.
• Outages pick their groups’ live members when they fire.
• Gray failure keeps the windows it drew at the start of the chaos window. Once the deployment changed, a window that starts picks its victim from the live pool (one draw), so a spawned machine can go slow and a retired one never does.
• A fault aimed at a retired IP (reboot, crash, restart) is dropped.

Determinism

Spawn and retire are simulation events (ProcessSpawn, ProcessRetire), handled by the orchestrator next to the restarts. The IP is reserved at the call. No randomness is drawn. So a seed spawns at the same time with the same IP on every replay, in the explorer’s forks and under check_determinism(). The timeline records process_spawn and process_retire sim faults.

Limits and the report

SimulationBuilder::max_spawns(n) caps the processes spawned per run (default 16). A spawn past the cap returns an error, so a buggy operator cannot fill a /24.

Each iteration’s SimulationMetrics::deployment lists the spawned and retired IPs per group. The report prints a summary line:

  Deployment   rack_node +140 -93 (spawned, retired)

Example

crates/moonpool-sim-examples/src/operator.rs (cargo xtask sim run operator): an operator workload racks and decommissions nodes of a .cluster() group while a client pings the nodes the topology lists now, under attrition. crates/moonpool-sim/tests/deployment.rs pins each behavior.

Chaos Testing vs Simulation

• Two Schools of Breaking Things
• Chaos Engineering: Learning from Production
• Simulation: Breaking Things Before They Exist
• Simulation Subsumes Chaos
• Complementary, Not Competing
• Moonpool’s Approach

Two Schools of Breaking Things

There are two common ways to test a distributed system under failure. Chaos engineering takes a running system and injects faults into it: kill a VM, cut a network link, fill a disk. Simulation testing builds a fake world and runs the system inside it. The faults come from the world itself. Both break things on purpose. They break things in different ways, and the difference decides how we use each one.

Chaos Engineering: Learning from Production

Chaos engineering, made popular by Netflix’s Chaos Monkey, uses production or a production-like staging environment as the laboratory. You pick an experiment (“what happens if we kill this database node?”). You form a hypothesis (“traffic reroutes within 5 seconds”). You run the experiment against real infrastructure and observe.

This approach has real strengths. It tests the actual system: real configuration, real dependencies, real network stacks. It finds problems no pre-production test can find: a misconfigured load balancer, a stale DNS cache, a gap in monitoring. When chaos engineering finds something, it matters, because it happened in the real world.

But chaos engineering has structural limits:

• Non-deterministic. You cannot reproduce the exact sequence of events that caused a failure. The bug happened once, under conditions you cannot rebuild.
• Reactive. You find problems after they exist in production, not before.
• Slow. Each experiment takes real time on real machines. One scenario takes minutes. A million scenarios is not practical.
• Blast radius. Real users can be affected. An experiment that goes wrong is an outage.

Chaos engineering finds symptoms. The database went down and something bad happened. But why? A race? A missing retry? A lock held too long during recovery? Getting from symptom to cause is forensic work after the fact, often with no way to replay the failure.

Simulation: Breaking Things Before They Exist

Simulation testing takes the opposite path. Instead of injecting faults into a real system, you build a simulated world where faults are a first-class feature. The network drops a connection because the configuration says so. The disk tears a write because the crash model says so. A clock drifts because the simulator makes it drift.

Everything runs in one process on one OS thread, driven by a seeded random number generator. The same seed produces the same execution, every time. A failing test is not a flaky signal. It is a reproducible bug with an exact replay.

The advantages follow directly:

• Deterministic. Every failure replays. A failing seed is a permanent regression test.
• Proactive. You find bugs before the code reaches production, before staging, often before it leaves your machine.
• Fast. No real I/O, no real time. Moonpool simulates hundreds of seconds of cluster time in single-digit real seconds. FoundationDB ran 5 to 10 million simulation runs per night.
• Wide. Different seeds explore different fault combinations. Run enough seeds and you cover failure scenarios no one would write a test for by hand.

Simulation Subsumes Chaos

Here is the key point: simulation subsumes chaos engineering. Everything chaos engineering does, simulation does too, with replay, speed and breadth added on top.

Chaos engineering injects a random partition? Simulation injects a random partition, and you can replay the exact moment with the exact timing. Chaos engineering kills a process? Simulation kills it, restarts it, and verifies the recovery path, in milliseconds of real time.

Will Wilson made this concrete with FoundationDB’s Sinkhole: a rack of real servers wired to programmable power switches, toggled continuously to validate the real system. Sinkhole never found a database bug that simulation had missed. It found bugs in other software and in hardware. The simulation was a stricter, more adversarial environment than reality.

Complementary, Not Competing

This does not make chaos engineering useless. The two are complementary:

Simulation is for development. It runs on every commit, explores millions of fault scenarios, and catches bugs before they ship. It tests the logic of your system against the worst possible world.

Chaos engineering is for production validation. It verifies that your real deployment, with its real configuration and dependencies, behaves as expected. It catches what simulation cannot model: kernel behaviour, cloud provider quirks, misconfigured infrastructure.

Think of simulation as the wind tunnel and chaos engineering as the flight test. You do not skip the wind tunnel because you plan to fly the plane. You do not skip the flight test because the wind tunnel looked good.

Moonpool’s Approach

In moonpool, chaos is a tool inside the simulation, not an alternative to it. Process reboots, correlated outages, network faults, storage faults, slow machines and code-path perturbation are all chaos techniques. They run inside a deterministic simulator, so every fault replays and every failure can be debugged.

Faults enter from two sides. The environment side is the builder: enable_chaos lists the fault surfaces a campaign turns on, and each seed draws its own subset and intensities. The code side is the shipped program: buggify! takes a rare-but-legal path, and hint! names a moment where a fault would be interesting, so the simulator can strike there. Production never sees either. The same binary runs in both worlds, which is the rule Same Code in Production and Simulation explains.

Chaos in Moonpool maps every surface. The chapters after it take one surface each.

Chaos in Moonpool

• Make Rare Bugs Common
• Two Sides of a Fault
• Every Chaos Surface
• ChaosMode: Random or Swarm
• The Chaos Window
  • Chaos for the Whole Run
• Determinism Is Not Negotiable

Make Rare Bugs Common

Real distributed systems fail in ways that are individually rare and collectively certain. A partition happens once a month. A disk tears a write once a year. A process dies during recovery once in a thousand deployments. Multiply enough small probabilities across enough nodes and enough time, and the question is not if but when.

The principle behind chaos in moonpool is simple: amplify failure probabilities so that rare bugs become common. A monthly partition happens every few seconds in simulation. A yearly torn write happens every hundred writes. A crash at the worst possible moment happens on the seed that chose that moment.

This is not reckless. Because every fault is drawn from the seed, the bug found at high probability is the same bug that would appear at low probability in production, found a thousand times faster, with a replay attached.

Two Sides of a Fault

Faults enter the simulation from two sides, and the split matters.

The environment side is the builder. SimulationBuilder::enable_chaos takes a list of Chaos surfaces. The runner samples each one per seed, runs it during the chaos window, and switches it off when the window closes. The code under test never knows a surface exists.

The code side is the shipped program. buggify! makes a rare-but-legal decision of the code’s own (a tiny batch, a skipped resend). hint! names a moment where an environmental fault would be interesting (a batch is durable but not yet sent), and the simulator decides whether to strike. Both are no-ops in production, so the code that ships is the code that runs under chaos.

Every Chaos Surface

enable_chaos accepts any mix of these variants. Each one is sampled by a ChaosMode where it has one.

Two surfaces are not in the list because they are not sampled per seed. Custom fault injectors are registered with fault_factory and script whatever they want through FaultContext, see Custom Fault Injectors. Replicated fault patterns are a shaping of Chaos::Storage, registered with replicated_storage_faults, see Aiming Storage Faults.

#![allow(unused)]
fn main() {
use moonpool_sim::{Attrition, AttritionScope, AttritionVictims, Chaos, ChaosMode, SimulationBuilder};

SimulationBuilder::new()
    .processes(3, || Box::new(MyProcess))
    .workload(MyWorkload)
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition {
            config: Attrition {
                max_dead: 1,
                prob_graceful: 0.3,
                prob_crash: 0.5,
                prob_wipe: 0.2,
                recovery_delay_ms: None,
                grace_period_ms: None,
                scope: AttritionScope::PerProcess,
                victims: AttritionVictims::Any,
                wipe_budget: None,
            },
            mode: ChaosMode::Swarm,
        },
        Chaos::BuggifyKnobs,
    ])
    .chaos_duration(Duration::from_secs(30))
    .run()?;
}

A surface absent from the list stays off. Its configuration is the default: low latency, no faults.

ChaosMode: Random or Swarm

Every sampled surface takes one of two modes.

ChaosMode::Random turns on the whole surface and draws its intensities per seed. Every network family is a little bit on, on every seed. For attrition and outages, Random runs the configured regime as written.

ChaosMode::Swarm turns on a random subset of the surface’s fault families per seed and fully disables the rest. One seed is clog-only. Another is partition plus bit flip. Another is the all-off baseline. For attrition, Swarm draws the reboot regime itself, including the never-reboot regime. For outages, Swarm drops the outage on half the seeds.

Swarm is the mode we recommend. Swarm Testing: Masks and Budgets explains why, lists every mask and budget, and shows how one workload under swarm covers more than many workloads without it.

The Chaos Window

Every surface runs inside the window chaos_duration sets. The builder refuses a campaign that registers attrition, an outage, a gray failure or a fault_factory without one: run() returns SimulationError::InvalidConfiguration instead of never running the injector.

When the window closes, the runner ends chaos in one step:

1. It cancels FaultContext::chaos_shutdown, so injector loops wind down.
2. It calls SimWorld::enter_recovery_mode, which turns off every configuration-driven network, storage and block-device fault family and heals the partitions it holds.
3. It calls moonpool_buggify::buggify_enter_recovery, so disruptive buggify sites and every hint! fall silent.

The cutoff bounds fault generation, nothing else. Corrupted sectors stay corrupted. Lost writes stay lost. A process already killed restarts on its own schedule. The cluster is not healthy at the cutoff. The environment has stopped making it worse, and recovering is now the system’s job. The quiet tail between the cutoff and workload completion is where a rebooted process rejoins and a cluster reconverges. Give the workload enough simulated time there, then assert on convergence.

Chaos for the Whole Run

A fixed window has a weakness: the workload’s operations and the faults meet only while both happen to run. If the window is short, most operations run in the quiet tail and never meet a fault. chaos_until_run_ends() uses FoundationDB’s test shape instead. The fault surfaces run for the whole run phase, beside the client workload, and the checks run on a quiet, live cluster:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([Chaos::Network(ChaosMode::Swarm)])
    .chaos_duration(Duration::from_secs(300)) // the cap
    .chaos_until_run_ends()
}

In this mode:

• Chaos ends when the last workload’s run() returns, or at chaos_duration, which is now a cap. The run’s length is the workload’s choice: it can draw one per seed.
• A run() that returns early does not shut its siblings down.
• check() runs on the live processes after the cutoff. It is the quiet tail: there the system recovers, and the workload reads its final state back and judges convergence. Killed processes come back on their usual schedule, and no new fault starts.
• Fault injectors keep running until check() ends, with their chaos_shutdown cancelled. The processes are aborted only after check().
• chaos_duration is also the reference length of a gray failure’s window. Set it near the longest run the workload draws.

moonpool_sim::chaos_ended_at() gives the cutoff time (and in_chaos() whether it passed), so code that times a step from the cutoff can find it.

Determinism Is Not Negotiable

Every chaos decision is a draw on the one simulation stream. The same seed produces the same faults in the same order at the same times. So:

• A failing seed is a permanent, reproducible bug report.
• You debug a failure by replaying it with tracing turned up.
• Fixing a bug and re-running the seed verifies the fix.
• Different seeds explore different fault combinations on their own.

The draw order is a contract. Network and storage profiles are drawn before the world reseeds the stream. The operation-alphabet mask comes next, then attrition regimes in registration order, then outages, then the CPU model and gray failures. Moving one draw shifts every seed, which is why the runner pins the order and why a surface you leave off costs no draw. The Fault Reference lists every family, its default rate and its mask bit.

Buggify: Fault Injection

• The Idea
• Two-Phase Activation
• Four Injection Patterns
• Disruptive Points and the Recovery Tail
• Named Points a Harness Can Force
• Spiking Config Knobs
• Rate Calibration
• Tilting Behaviour in Simulation
• CPU Charges
• Production Safety

The Idea

Most distributed system bugs do not live on the happy path. They hide in error handlers, timeout branches, retry loops and recovery code. Those paths run only when something goes wrong, so they are the least tested code in the system and the most critical when a failure comes.

FoundationDB solved this with BUGGIFY: scatter conditional fault points through the codebase, active only in simulation. Each point perturbs the code in a small way. Return an error instead of success. Add a delay. Shrink a buffer. Run enough seeds and the perturbations push the code through every error path, every retry, every recovery sequence.

Moonpool implements this as the buggify! family of macros in the zero-dependency moonpool-buggify crate. moonpool-sim re-exports them, so moonpool_sim::buggify! and moonpool_buggify::buggify! share one state.

Two-Phase Activation

A naive design fires every point on every call. That is chaos, but not useful chaos. If every operation fails, the system never makes progress, and you never test the interactions between partial failures.

Moonpool uses FoundationDB’s two-phase model:

Phase 1: Activation. The first time a run reaches a buggify location, the location is activated or deactivated by one draw. The decision holds for the whole run. A deactivated location never fires, however often it is reached. The runner activates each location with a fixed probability of 50% (buggify_init(0.5)).

Phase 2: Firing. Each time an active location is reached, it fires with a per-site probability. buggify! fires at 25%. buggify_with_prob! takes its own rate.

#![allow(unused)]
fn main() {
use moonpool_buggify::{buggify, buggify_with_prob};

// Each call site is one location, keyed by file:line.
// First encounter: a 50% activation coin. Then 25% per call while active.
if buggify!() {
    return Err(Error::timeout("simulated timeout"));
}

// Fire at 50% instead of the default 25%.
if buggify_with_prob!(0.5) {
    buffer_size = 1; // force single-byte reads
}
}

Neither rate is a builder setting. The two phases mean each seed tests a different combination of active points. Seed 42 activates a request timeout and deactivates a storage error. Seed 43 does the reverse. Run enough seeds and you cover the combinations.

Four Injection Patterns

FoundationDB’s codebase uses BUGGIFY in a few recurring shapes. All of them carry over.

Error injection on a success path. After an operation succeeds, sometimes return an error anyway:

#![allow(unused)]
fn main() {
let result = connection.send(message).await;
if result.is_ok() && buggify!() {
    // Force the caller through its error handling.
    return Err(Error::io("buggified send failure"));
}
}

Artificial delay. Widen a race window:

#![allow(unused)]
fn main() {
if buggify!() {
    time.sleep(Duration::from_millis(100)).await?;
}
}

Parameter randomization. buggify_pick! and buggify_range! draw a value at an active point, with no rand dependency. buggify_pick!(prob, n) returns Some(i) with i in 0..n. buggify_range!(prob, lo..hi) returns Some(v) in the range. Both return None when the point does not fire, and always outside a simulation:

#![allow(unused)]
fn main() {
use moonpool_buggify::buggify_range;

// Most runs use the configured batch size. Some runs use a tiny one.
let batch_size = buggify_range!(0.25, 1..3).unwrap_or(DEFAULT_BATCH_SIZE);
}

Alternative code path. Take a rare path more often:

#![allow(unused)]
fn main() {
let should_compact = needs_compaction() || buggify_with_prob!(0.1);
}

Each of these is a decision of the code. The code needs nothing from the environment to take the path. A crash, a failed sync or a cut link is a different thing. The code cannot decide those, so it names the moment with hint! and the simulator decides. hint!: Naming Fault Sites draws that line.

Disruptive Points and the Recovery Tail

A point that makes an operation fail (a cut session, a refused request) is different from a point that only takes a rare path. If it keeps firing after the chaos window closes, the liveness checks a workload makes in its quiet tail fail on injection that should have stopped. Mark those points with buggify_fault_with_prob!:

#![allow(unused)]
fn main() {
use moonpool_buggify::buggify_fault_with_prob;

if buggify_fault_with_prob!(0.02) {
    return Err(Error::overloaded());
}
}

During the chaos window it behaves like buggify_with_prob!. When the runner closes the window, it calls moonpool_buggify::buggify_enter_recovery, and from then until the next run every fault point evaluates to false without a draw. Ordinary buggify! and buggify_with_prob! points keep firing through the tail: a rare-but-harmless path is still worth taking while the system recovers. FoundationDB draws the same line by gating its disruptive points on speedUpSimulation.

Named Points a Harness Can Force

Some decisions are made once per seed: hold one journal for the whole window, withhold a whole class of requests. The plain macros key a location by its file:line, and each activation is an independent coin. A harness that builds a scenario from several such ingredients waits for the product of their coins. buggify_named! keys the location by a stable label instead, so the harness decides it together with the other ingredients:

#![allow(unused)]
fn main() {
use moonpool_buggify::buggify_named;

// Shipped code. With probability 1.0 the location's activation is the
// per-seed decision. Disruptive, so silent in the recovery tail.
if buggify_named!("withhold gc requests", 1.0) {
    return;
}

// Harness, once it drew the scenario for this seed.
moonpool_sim::set_activation("withhold gc requests", scenario);
}

set_activation(label, active) replaces the location’s own coin. It draws nothing. Call it before the location’s first encounter, or later to override. buggify_init clears it with every other decision at the start of a run. A point the harness never sets keeps its own coin. Every label must be unique: two points with one label share one activation. Swarm Testing shows how a harness uses this as one more swarm axis.

Spiking Config Knobs

The patterns above scatter buggify! through your own code. The simulation has knobs of its own: network latencies, disk IOPS, fault probabilities. FoundationDB randomizes them the same way, with one line at config construction:

if (randomize && BUGGIFY) KNOB = deterministicRandom()->random(lo, hi);

Moonpool has buggify_knob!(default, lo..hi). It returns default on most seeds. When its call site activates and fires it returns a value drawn inside the range:

#![allow(unused)]
fn main() {
// Most seeds keep the sampled IOPS. A few push the disk to a crawl.
self.iops = moonpool_sim::buggify_knob!(self.iops, 100..5_000);
}

You opt in through the same enable_chaos list as everything else:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::BuggifyKnobs,
    ])
    .workload(MyWorkload)
    .run()?;
}

Chaos::BuggifyKnobs is a modifier, not a surface. It perturbs knobs only on surfaces you already enabled, so it never switches on a family you left off. On the storage surface the spiked knobs are IOPS, bandwidth, the sync-failure rate, stall and throttle rates and durations, and the disk-failure rate. On the network surface they are the clog, partition, random-close and black-hole rates, and the bit-flip rate, which a seed can push from a sampled 0.001% to 0.02% of sends to around 1%, enough for an integrity check above the transport to meet corruption within a bounded run. Bit flips are spiked only on seeds that enabled them.

This is a different coverage axis from swarm testing. Swarm decides which fault families are on. Knobs decide how hard an enabled knob gets pushed. A seed can run storage-only with a disk throttled to 100 IOPS instead of 25,000. The two compose. buggify_knob! lives in moonpool-sim, not in moonpool-buggify: it is simulation configuration, not application fault injection.

Rate Calibration

Not every point should fire at one rate. FoundationDB uses three tiers:

The default 25% works for most points. Avoid buggify_with_prob!(0.9) on a fault path: a system that fails 90% of attempts never reaches the interactions between partial success and partial failure.

Tilting Behaviour in Simulation

moonpool_buggify::is_simulated() reports whether this thread runs a simulation (buggify is enabled and a random source is installed). It is FoundationDB’s g_network->isSimulated(). Use it to tilt a rate, a cadence or an extra check in simulation. Never let it change an outcome a caller can observe.

CPU Charges

A simulated poll costs a small base cost when the CPU model is on. The base cost does not know that one poll decoded a large batch and another bumped a counter. cpu_charge lets the code say so:

#![allow(unused)]
fn main() {
use std::time::Duration;

fn fold_batch(records: &[Vec<u8>]) {
    // ... the real work ...
    // A profile says one record costs about 200 ns to fold.
    moonpool_buggify::cpu_charge(Duration::from_nanos(200) * records.len() as u32);
}
}

Inside a simulation with the CPU model on, the process’s clock moves forward by the cost times its slowness. What the process does next (a send, a write, a timer) happens after the work. Everywhere else the call costs one thread-local read. Put a charge only on a hot path a profile shows: an encode, a checksum, a fold.

Production Safety

Buggify is gated on simulation state. Outside a run, every macro evaluates to false. The cost is a thread-local read. You leave the call sites in production code, as FoundationDB leaves BUGGIFY in fdbserver.

[dependencies]
moonpool-buggify = "0.8"

The crate is pure std with zero dependencies, so sans-I/O and production crates depend on it directly, and it compiles for wasm32-unknown-unknown. It owns the disabled-by-default state, the macros, hint! and cpu_charge. When a run starts, moonpool-sim installs its seeded random source with set_random_source, calls buggify_init, and installs the hint and CPU sinks. When the run ends, buggify_reset makes everything inert again.

Two rules keep replay exact. Never mix non-deterministic entropy into a buggified branch: rand::random() breaks the seed. And do not use buggify for business logic. A branch that changes application semantics belongs in the application, not behind a fault coin.

hint!: Naming Fault Sites

• The Code Names a Moment, the Simulator Decides
• A No-Op in Production
• What a Strike Looks Like
• Three Vetoes and a Fourth
• Where to Put a Hint
• A Worked Example: moonpool-journal
• What a Hint Is Not

The Code Names a Moment, the Simulator Decides

Attrition reboots a process wherever a task happens to yield. Volume alone rarely lands a reboot on the one await that matters: the batch is synced, the acknowledgements are not sent yet. That edge is one await wide. The code knows where it is. The simulator does not.

FoundationDB’s server code solves this with if (buggify()) throw please_reboot(), and its simulated worker turns the throw into a reboot. The site is precise, and the site chooses the fault. Moonpool keeps the precision and moves the choice: the code names a moment. It never names a fault.

#![allow(unused)]
fn main() {
use moonpool_buggify::hint;

storage.sync().await?;
hint!("batch durable, not sent").await;
send(messages).await;
}

The label is a human string and a coverage bucket. It carries no semantics the simulator acts on. The simulator strikes the calling process if the seed’s chaos allows it. Today every strike is a reboot, the one fault that tests a storage moment and a network moment alike: it loses the unsynced bytes and the unsent messages at once.

A No-Op in Production

hint! expands to a disruptive buggify site (buggify_fault_internal) with a label. Outside a simulation, buggify is disabled, so the site draws nothing, asks no one, and the returned future is ready at once. The cost is one thread-local read. moonpool-buggify has zero dependencies, so the same code ships to production and to wasm32. This is the rule from Same Code in Production and Simulation: no SimMachine, no test-only hook, one binary.

What a Strike Looks Like

hint! returns a Hinted future. The decision is taken when hint! is called, not when the future is polled. Hinted::strike() reports it as a Strike:

On Killed the kill is scheduled one scheduler tick away through the same ProcessForceKill event a fault injector uses. The process’s tasks abort, its connections reset, and every write not yet synced resolves by the disk’s crash physics. Nothing after the .await runs. That is why the macro is #[must_use]: a hint you do not await lets the code after the moment run on a process that is already dead.

The simulator finds the process through the task being polled: every task carries the process that spawned it, across spawn_task. A hint from a workload, an injector or the driver has no owner, so the sink answers None and the hint resolves.

Three Vetoes and a Fourth

Three vetoes stand between a hint and a reboot. None of them commands.

1. The site. A hint is a buggify location. It is activated once per run at the runner’s 50%, and it fires per call at a low rate: hint::POINT_PROB (5%) for hint!("label"), or a literal at the site for hint!("label", 0.2). A site at rate 1.0 is a per-seed scenario: activation is the draw. Once the chaos window closes the site is silent, like every buggify_fault_with_prob! point, and the runner removes the sink. The rate is a literal next to the moment it describes, never a knob.

2. The seed’s chaos. The HintSink hands the decision to the first Chaos::Attrition regime whose victims filter admits the process. No regime covers the process? No strike. The regime’s reboot-kind weights pick graceful, crash or crash-and-wipe. Its recovery delay applies. A seed whose swarm drew the never-reboot regime (max_dead = 0) never reboots on a hint either. Register attrition with ChaosMode::Swarm and the hints follow the same per-seed subset as the timed reboots.

3. The budget. The regime’s max_dead counts dead processes among its victim pool. A hint that would exceed it is refused, exactly as a timed reboot would be. The hinting process alone is rebooted, whatever the regime’s failure scope.

4. The harness’s budget. Some budgets the simulator cannot see. A replicated store can lose copies at only so many replicas before a crash stops being survivable, and only the harness keeps that count. It publishes a HintVeto in the run’s state. The sink asks it last, once a reboot fits the regime, so a true answer is a reboot and the harness can count it as spent:

#![allow(unused)]
fn main() {
use moonpool_sim::HintVeto;

// From a process at boot, or from a fault injector at the window's start.
let ledger = ledger.clone();
HintVeto::new(move |ip, _label| ledger.permit_cut(ip)).publish(ctx.state());
}

The veto runs inside the hint’s call, so it must not draw randomness. HINT_VETO_KEY is the state key, and a later publish replaces an earlier veto.

The same veto guards the regime’s own timed reboots of one process. A timed reboot lands at any instant, inside a commit too, so a budget that only hints asked would leak. Before each one, a per-process regime asks the veto with the label TIMED_REBOOT_LABEL. A false answer skips that round. A per-domain regime’s group reboot does not ask it.

The chain costs nothing until a site fires. Every draw (activation, firing, kind, delay) goes through the one simulation stream from inside the call, so a hinted reboot replays exactly. crates/moonpool-sim/tests/hint.rs runs the chain under check_determinism and shows a refused veto leaving the process alive.

Where to Put a Hint

A hint is worth placing where an environmental fault changes what the next boot finds. Two shapes cover most of them:

• After a sync, before a send. The data is durable, and no peer knows. A reboot here leaves the peers unaware of what this process holds.
• After a write, before its sync. The data is staged, not durable. A reboot here tears the write, and recovery must cope with a partial record.

Other good moments: a promise committed before its acknowledgement leaves, a prepare sent before its reply is handled, a metainfo update staged behind a synced batch. A bad moment is a hot path with no state at stake: the site fires often and finds nothing.

A hint is not for a decision of the code. Skipping a resend, shrinking a batch, taking a rare branch: those need no environment, so they are plain buggify! sites. Nothing is both.

A Worked Example: moonpool-journal

moonpool-journal ships three hints inside its batched commit protocol (crates/moonpool-journal/src/journal.rs). The journal depends on moonpool-core and moonpool-buggify only, so every user of the journal gets these power cuts with no code of its own.

#![allow(unused)]
fn main() {
segment.write(batch.entry_at, &batch.entries).await?;
// A power loss here tears entries no record points at yet.
moonpool_buggify::hint!("journal commit: entries written, not synced", COMMIT_HINT_PROB).await;
segment.write(batch.persist_at, &batch.records).await?;
// A power loss here may tear the records, rebuild them from their
// entries, or leave the last batch ambiguous.
moonpool_buggify::hint!("journal commit: records written, not synced", COMMIT_HINT_PROB).await;
segment.sync().await?;
if let Some(meta) = meta {
    // The batch is durable and the metainfo still names the old one.
    moonpool_buggify::hint!("journal commit: batch synced, metainfo stale", COMMIT_HINT_PROB).await;
    meta_file.update(meta).await?;
}
}

COMMIT_HINT_PROB is 0.25, the buggify default, not the 5% of a hot-path point: a commit is short, and a cut inside it is the case its recovery exists for. Each of the three moments maps to a recovery path the journal must take: a torn record, a record rebuilt from its entry, an ambiguous last batch. The ordered protocol writes each log with one durable write (write_at_durable), so its two write-then-sync windows sit inside one call each. The simulated storage names them itself, with hint!("storage: durable write landed, not synced", 0.25) between the write and the sync it models. A harness that counts the replicas it can afford to lose publishes a HintVeto, and the journal never knows. A Crash-Aware Journal follows those recoveries.

What a Hint Is Not

A hint cannot aim at another process. The fleet killer, the rack outage and the scripted crash stay on the harness side, as outages and custom injectors. A process that must crash at an exact point of its own protocol, with no chaos regime in the way, uses SimContext::crash_self instead. That is a command, not a hint, and it is covered with the outages.

The design note behind this API weighed a named fault (“reboot me”), a weakness taxonomy and a resource-scoped statement, and rejected all three: each tells the simulator something it already owns, or lets production code choose a fault. A labelled moment is the smallest thing the code must say.

Swarm Testing: Masks and Budgets

• Passive Suppression
• ChaosMode::Swarm, Surface by Surface
• Caller Masks
• Budgets
• Swarm the Operation Alphabet
• Named Sites as a Swarm Axis
• One Workload, Many Behaviours

Passive Suppression

ChaosMode::Random sets every fault family of a surface to a random non-zero rate. Clogging is a little on, partitions are a little on, bit flips are a little on, all at once, on every seed. That sounds thorough. It is the opposite.

The problem is called passive suppression, and it comes from “Swarm Testing” by Groce, Zhang, Eide, Chen and Regehr (ISSTA 2012). When every feature is always slightly active, the features crowd each other out. To find a clogging bug you often need clogging cranked hard, for a sustained stretch, with nothing else interfering. If partitions keep tearing down the connection you were trying to clog, you never drive clogging deep enough. The all-on configuration explores a narrow band in the middle of the space and never visits the extremes where bugs live. In the paper, swarm testing found 42% more distinct compiler crashes than a hand-tuned default configuration.

The fix is counterintuitive. For each run, turn off a random subset of features entirely. One seed is clog-only. Another is partition plus bit flip. Another is the all-off baseline. Across many seeds you reach the single-family extremes the all-on config never reaches. Will Wilson’s Antithesis talks made the same point for distributed systems.

ChaosMode::Swarm, Surface by Surface

ChaosMode::Swarm applies the paper per surface. Each surface starts from its Random profile, then draws one coin per family and switches the losers fully off. The all-off subset is allowed on purpose: a no-fault run is a valid configuration, and it is where slow leaks show.

Durations, cooldowns and strategies stay as sampled. They are inert once their family is off.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition { config: regime, mode: ChaosMode::Swarm },
    ])
}

Every coin comes from the one simulation stream, seeded for the iteration. Network draws first, then storage, both before the world reseeds the stream. The operation-alphabet mask comes next, then attrition regimes in registration order, then outages, the CPU model and gray failures. A coin is drawn whether or not the family wins, so the count per surface is fixed. Same seed, same subset, every time: a replay rebuilds the subset before the counted run starts, and a fork-explorer recipe never lands on a configuration draw.

Caller Masks

Swarm picks a subset the seed likes. Sometimes you need a subset you like: the system under test does not handle a family yet, or you are hunting one family and want the rest silent. network_fault_mask and storage_fault_mask retain only the families you name:

#![allow(unused)]
fn main() {
use moonpool_sim::{NetworkFault, NetworkFaultMask, StorageFault, StorageFaultMask};

SimulationBuilder::new()
    .enable_chaos([Chaos::Network(ChaosMode::Swarm), Chaos::Storage(ChaosMode::Swarm)])
    // Hunting a replication bug: no corrupted frames, no lying syncs.
    .network_fault_mask(NetworkFaultMask::all().without(NetworkFault::BitFlip))
    .storage_fault_mask(
        StorageFaultMask::all()
            .without(StorageFault::SyncFailure)
            .without(StorageFault::DiskFailure),
    )
}

Both masks have the same shape: all(), none(), with(fault), without(fault) and contains(fault). The default is all(). A mask is applied after profile sampling and after Chaos::BuggifyKnobs, right before the world is built, and it consumes no randomness. Adding or removing a mask shifts no draw, so recipes and replays stay valid. A family a mask removes is never chosen, whatever the seed’s coin said.

The NetworkFault families are Clog, Partition, BitFlip, RandomClose, ConnectFailure, ClockDrift, BuggifiedDelay, PairLatency and BlackHole. The StorageFault families are Corruption, Eio, Misdirect, PhantomWrite, SyncFailure, ShortTransfer, DirEntryLoss, CrashDamage, Degradation and DiskFailure. Partial reads and writes are TCP behaviour driven by buggify, not a family, so no mask touches them.

Budgets

A mask says which families may strike. A budget says how much damage may stand at once. Every built-in injector carries one:

Budgets keep chaos survivable. A system with replication factor 3 tolerates one failure, so max_dead: 1. A budget the simulator cannot see belongs to the harness: a HintVeto refuses reboot hints and an OutageVeto holds an outage back, both published in the run’s state. hint! and Correlated Outages show each one.

Swarm the Operation Alphabet

The same suppression happens inside a workload. A bounded queue driven by a 50/50 push and pop mix is a random walk that takes tens of millions of steps to fill. pop keeps undoing what push does, and the overflow bug is never reached. The fix is the same: enable a random subset of operations per seed.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .workload_factory(|| Box::new(QueueWorkload::new()))
    .swarm_operations()
}

With .swarm_operations() the runner draws a 256-bit mask right after seeding, one bit per u8 operation id, four draws. moonpool_sim::swarm_op_enabled(op_id) reads one bit, consumes no randomness, and gives the same answer every time. Without the call every operation reports enabled, so a non-swarm run is byte-identical to one with no swarm code.

#![allow(unused)]
fn main() {
// Cache the subset once per run. Fall back to the full alphabet if the
// seed disabled everything, so the workload always has something to do.
let enabled: Vec<u8> = (0..NUM_OPS).filter(|&i| swarm_op_enabled(i)).collect();
let enabled = if enabled.is_empty() { (0..NUM_OPS).collect() } else { enabled };

// One draw per step, folded into the subset. A resample loop would make
// the draw count vary, and the explorer keys replay on that count.
let raw = moonpool_sim::sim_random::<u64>();
let op = enabled[(raw % enabled.len() as u64) as usize];
}

The dungeon example in crates/moonpool-sim-examples/src/dungeon.rs is the reference, and crates/moonpool-sim/tests/swarm_op_alphabet.rs shows an assertion that only fires under the swarm. Designing Workloads That Find Bugs has the full pattern.

Named Sites as a Swarm Axis

Buggify is a swarm too: every buggify! location is on for half the seeds and off for the rest. The coins are independent, which is what you want for scattered points and not what you want for a scenario. A harness that needs “hold the journal and withhold GC requests” on the same seed waits for the product of two coins.

buggify_named! keys a site by a label, and set_activation decides it for the run:

#![allow(unused)]
fn main() {
// Shipped code.
if buggify_named!("hold journal", 1.0) { /* ... */ }
if buggify_named!("withhold gc requests", 1.0) { /* ... */ }

// Harness, at the start of the seed: one scenario draw decides both.
let scenario = ctx.random().random_bool(0.5);
moonpool_sim::set_activation("hold journal", scenario);
moonpool_sim::set_activation("withhold gc requests", scenario);
}

A site at rate 1.0 turns activation into the per-seed decision. A site the harness never sets keeps its own coin. buggify_init clears every decision at the next seed. Buggify has the rules.

One Workload, Many Behaviours

A common reflex is to write one workload per scenario: a partition workload, a crash workload, a slow-disk workload, a no-fault workload. Each one is a file to maintain, a report to read, and a fixed configuration that explores one point in the space.

Swarm turns the reflex around. Write one workload that exercises the whole operation alphabet and checks every invariant. Enable every surface with Swarm. Now each seed is a scenario the harness never had to name: seed 7 is partitions and crash-only attrition with pop disabled, seed 8 is a no-fault baseline with the CPU model on, seed 9 is slow disks and a rack outage. The combinations multiply across seeds, and the ones that find bugs are the ones no one would have written.

The per-seed decisions are visible, so a failing seed explains itself. Each attrition injector carries its filter in its name (attrition[group=acceptor]). Every fault the engine injects lands in the timeline as a sim_fault event with a kind field. The assert_sometimes_each! buckets under attrition_reboot_kind and hint_struck say which regime was live. Pin the seed with set_debug_seeds, and the swarm rebuilds the same subset from the seed before the first event runs.

Separate workloads still earn their place when the invariants differ, not when the faults do. A workload that checks linearizability and a workload that checks a compaction schedule are two workloads. A workload that checks linearizability under partitions and another under crashes is one workload under swarm.

Attrition: Process Reboots

• Why Processes Need to Die
• Three Kinds of Reboot
• The Attrition Configuration
• Using Attrition
• Random or Swarm
• The max_dead Constraint
• Rolling Wipes: the Wipe Budget
• Role-Aware Victims: One Regime per Group
• Failure Domains: Correlated Reboots

Why Processes Need to Die

A distributed system that only works while every node is healthy is a single point of failure with extra network hops. Real clusters lose nodes all the time. A rolling deployment restarts a process. A kernel panic kills one. A power failure wipes its storage. Your system must handle all three while it keeps serving requests.

Attrition is moonpool’s built-in injector for that drizzle of failures. During the chaos window it waits 1 to 5 simulated seconds, picks a live victim, kills it in one of three ways, and restarts it after a recovery delay. Then it waits again. The goal is to prove the system tolerates node loss without anyone writing a test per failure.

Three Kinds of Reboot

RebootKind models three real failures.

Graceful is the controlled shutdown. The process’s ctx.shutdown() token fires. The process gets a grace period to drain buffers, close connections and flush writes. If it has not returned when the period ends, it is force-killed. Send buffers drain during the grace period (FIN semantics). Then a fresh instance boots. This models rolling deployments and well-behaved process managers.

Crash is sudden death. The process task is cancelled at the crash instant, before its connections abort, so nothing the crash disturbs can wake it again. Connections abort with no drain. Peers see connection resets. Unsynced storage resolves by the disk’s crash physics. The process runs no work until its restart. This models kernel panics, OOM kills and hardware faults.

CrashAndWipe is the worst case. Same as Crash, and every file the process owns is deleted. The process restarts as a brand-new node. The wipe is scoped to the process’s IP, so other processes keep their disks. This models a replaced machine or a lost disk.

The Attrition Configuration

#![allow(unused)]
fn main() {
use moonpool_sim::{Attrition, AttritionScope, AttritionVictims};

Attrition {
    max_dead: 1,
    prob_graceful: 0.3,
    prob_crash: 0.5,
    prob_wipe: 0.2,
    recovery_delay_ms: Some(1000..10000),
    grace_period_ms: Some(2000..5000),
    scope: AttritionScope::PerProcess,
    victims: AttritionVictims::Any,
    wipe_budget: None,
}
}

max_dead caps the processes dead at the same time among this injector’s victims. A process counts as dead from the instant its reboot is decided, graceful or crash, until its restart runs. Victims are drawn only from live processes, and a reboot aimed at a dead process is skipped, so one outage never schedules two restarts.

prob_graceful, prob_crash, prob_wipe are weights, not probabilities. They need not sum to 1.0. The injector normalizes them and draws a kind. prob_wipe: 0.0 disables wipes. All three at zero means always crash.

recovery_delay_ms is the range a dead process stays dead, drawn per reboot. None means 1000..10000.

grace_period_ms is the range a graceful shutdown may take, drawn per reboot. None means 2000..5000.

scope is the failure domain a reboot targets. PerProcess kills one process. PerMachine, PerZone and PerDatacenter kill every collocated process together and need a .cluster() topology.

victims is who may die at all. AttritionVictims::Any draws over every server process. AttritionVictims::group("acceptor") and AttritionVictims::tagged("role", "voter") narrow the pool.

wipe_budget bounds the wiped processes of the pool that are not yet healed. None (the default) leaves wipes bounded by max_dead alone. Rolling Wipes explains it.

Using Attrition

#![allow(unused)]
fn main() {
use std::time::Duration;
use moonpool_sim::{Attrition, AttritionScope, AttritionVictims, Chaos, ChaosMode, SimulationBuilder};

SimulationBuilder::new()
    .processes(3, || Box::new(MyProcess::new()))
    .enable_chaos([Chaos::Attrition {
        config: Attrition {
            max_dead: 1,
            prob_graceful: 0.3,
            prob_crash: 0.5,
            prob_wipe: 0.2,
            recovery_delay_ms: None,
            grace_period_ms: None,
            scope: AttritionScope::PerProcess,
            victims: AttritionVictims::Any,
            wipe_budget: None,
        },
        mode: ChaosMode::Random,
    }])
    .chaos_duration(Duration::from_secs(60))
    .workload(MyWorkload::new())
    .run()?;
}

chaos_duration is required. run() returns SimulationError::InvalidConfiguration for a builder that registers attrition without it, instead of dropping the injector unrun. When the window closes, no new reboot is scheduled. A reboot already decided still completes: its restart is a scheduled event, so no victim is stranded down into the recovery tail. The tail is where the rebooted process rejoins and the cluster reconverges, so give the workload enough time there.

Random or Swarm

ChaosMode::Random runs the regime as written on every seed. ChaosMode::Swarm draws the regime per seed, in six draws on the simulation stream:

1. max_dead becomes 0 on half the seeds. That is the never-reboot regime, which finds the slow leaks and timer bugs that constant restarting hides.
2. Each reboot kind is masked off on half the seeds. One kind left is a single-mode regime, such as always-crash or graceful-only.
3. The recovery-delay range is scaled to between 50% and 200% of the configured one.
4. With .cluster(), the scope is drawn among PerProcess and the correlated scopes whose domains all fit max_dead. A flat .processes() campaign keeps its scope.

The injector still makes its usual draws per reboot for kind, delay and victim. The regime costs a fixed footprint at a fixed position, so it replays with the seed. Regimes are drawn in registration order after the stream is reseeded. Swarm Testing explains why the never-reboot seed is as valuable as the crash-heavy one.

The max_dead Constraint

max_dead is the bridge between chaos and correctness. Without it, attrition could kill every node at once, which is chaos but not useful chaos. Set it to what your system claims to survive. Replication factor 3 tolerates one failure, so max_dead: 1. A system that needs 3 of 5 nodes uses max_dead: 2. Attrition then tests failures the system should survive, not failures that no design survives.

The budget is per victim pool. With AttritionVictims::Any it is the whole cluster. With a group or tag filter it is that pool alone, so “at most one dead acceptor” means exactly that, whatever happens to the matchmakers. A reboot hint from the code (see hint!) spends the same budget: the first regime whose filter admits the hinting process decides, and a full budget refuses the hint.

Rolling Wipes: the Wipe Budget

max_dead counts processes that are dead. A wiped process is dead only during its recovery delay. It then restarts, alive but empty, long before the system has rebuilt what it lost. So two wipes a few seconds apart each pass max_dead: 1, and together they can erase two of three replicas. A Paxos deployment with three matchmakers lost two of them that way, 6 s apart.

A wipe budget makes wipes roll through the pool, one healed process at a time:

#![allow(unused)]
fn main() {
use moonpool_sim::{WipeHeal, WipeLedger};

let regime = Attrition {
    max_dead: 1,
    victims: AttritionVictims::group("matchmaker"),
    ..base
}
.wipe_budget(1, WipeHeal::Signal); // at most one unhealed wipe

// In the system under test, once a wiped node has caught up:
WipeLedger::heal(ctx.state(), ctx.my_ip());
}

A wiped process is unhealed from the instant its wipe is decided until it heals. While the pool holds max_unhealed unhealed processes, a drawn CrashAndWipe becomes a plain Crash. A group wipe (a machine, zone or datacenter scope) counts every process of the group. A victim that is itself unhealed does not count twice, since wiping it again loses nothing more. The downgrade applies on every path the regime reboots through: timed reboots, failure-domain reboots, and hints.

Two ways to heal:

• WipeHeal::Signal: the harness declares a process healed with WipeLedger::heal(state, ip). It returns whether the process was unhealed, so calling it on every catch-up is safe. Only the system knows when a replaced node has caught up. A process never signaled stays unhealed for the rest of the run, so every later wipe in its pool is a crash.
• WipeHeal::After(duration): healed that long in simulated time after the wiped process restarts. A signal heals it earlier.

WipeLedger::unhealed(state) lists the processes wiped and not yet signaled healed.

The budget costs no randomness. The kind is drawn as before, then judged against the ledger, so a seed draws the same values with or without a budget. Swarm mode copies wipe_budget through and draws nothing for it. Every wipe enters the ledger, including a scripted FaultContext::reboot(ip, RebootKind::CrashAndWipe) and a process that crashes itself with a wipe. A scripted wipe is never downgraded, but it counts against a budgeted pool it lands in.

Role-Aware Victims: One Regime per Group

A cluster rarely has one kind of node. Once you register process groups, uniform attrition draws its victim over every process. On a seed with three acceptors and five spares, most kills land on the spares, and the consensus tier you meant to stress barely notices. Tell attrition who it may kill:

#![allow(unused)]
fn main() {
Attrition {
    max_dead: 1,
    victims: AttritionVictims::group("acceptor"),   // never a spare
    ..base
}
}

The filter costs no randomness. The victim is drawn uniformly over the eligible processes, so a filter that excludes nothing leaves every seed’s draw schedule untouched, and a filter that leaves nothing eligible skips the round without a draw. It applies to every scope: a machine-scoped reboot only picks machines that host an eligible process and only kills the eligible processes on it.

Because the budget is per pool, each group can have its own regime. Every Chaos::Attrition entry adds one independent injector:

#![allow(unused)]
fn main() {
.enable_chaos([
    Chaos::Attrition {
        // Acceptors: at most one down at a time, regime swarmed per seed.
        config: Attrition { max_dead: 1, victims: AttritionVictims::group("acceptor"), ..base.clone() },
        mode: ChaosMode::Swarm,
    },
    Chaos::Attrition {
        // Matchmakers: hammered independently, up to three down at once.
        config: Attrition { max_dead: 3, victims: AttritionVictims::group("matchmaker"), ..base },
        mode: ChaosMode::Random,
    },
])
}

The two injectors run side by side and never block each other. In the report each carries its filter in its name (attrition[group=acceptor], attrition[tag=role=voter]), so a failing seed says which regime was live.

Failure Domains: Correlated Reboots

max_dead: 1 protects you from two random deaths at once. Real outages are rarely random and rarely one node at a time. A host reboots and every process pinned to it dies together. These are correlated failures, and they break quorum logic because the nodes that die were never independent.

Flat IPs cannot express this, so moonpool uses FoundationDB’s hierarchy: datacenter, zone, machine, process. You describe it with .cluster() and a LocalityConfig, which Process Groups, Tags and Topology covers. With a topology in place, scope earns its keep:

#![allow(unused)]
fn main() {
.enable_chaos([Chaos::Attrition {
    config: Attrition {
        max_dead: 2,  // one machine's worth of processes
        scope: AttritionScope::PerMachine,
        ..base
    },
    mode: ChaosMode::Random,
}])
}

PerMachine picks a machine and reboots every process on it in the same instant, as a host kernel panic would. PerZone does the same one level up, and PerDatacenter takes a region. max_dead still counts processes, and a group reboot is atomic against it: the injector only kills a domain when the whole group fits the remaining budget. With two processes per machine, max_dead: 2 lets exactly one machine be down and never leaves a machine half-dead. A domain with no live eligible process is never drawn. Without a topology the correlated scopes do nothing.

A machine reboot and a rack power loss look alike but differ in one way: attrition stays under max_dead, so a quorum always survives. A fault that takes a whole group at once, past any budget, is Chaos::Outage, in Correlated Outages and Self-Crashes. A scripted sequence (“crash node X, wait for the workload, bring X back”) is a custom injector, in Custom Fault Injectors.

Correlated Outages and Self-Crashes

• The Whole Rack Goes Dark
• Chaos::Outage
• Why the Fault Is Trustworthy
• OutageLanded: Acting at the Instant No Copy Lives
• OutageVeto: A Budget the Simulator Cannot See
• Self-Crashes: A Process Chooses the Instant

The Whole Rack Goes Dark

Attrition never takes more than max_dead processes at once, and that is the point: it models the steady drizzle of failures a healthy cluster rides through. A rack-wide power loss is a different fault. Every process of a group loses power at the same instant. Every memory is gone. What survives is exactly what the disks hold, and no peer can repair a copy from memory, because no peer is awake. Then the machines come back one by one, each on its own schedule, and one straggler takes much longer than the rest.

FoundationDB’s simulator models this as killDataCenter in sim2.actor.cpp: it kills every machine of a datacenter in one loop and lets each machine’s own reboot delay bring it back. A quorum protocol that never faces it has never had its recovery path tested from cold disks alone.

Chaos::Outage

#![allow(unused)]
fn main() {
use std::time::Duration;
use moonpool_sim::{Chaos, ChaosMode, Outage, SimulationBuilder};

SimulationBuilder::new()
    .processes(3, || Box::new(Acceptor::new()))
    .processes(2, || Box::new(Matchmaker::new()))
    .enable_chaos([Chaos::Outage {
        config: Outage {
            start: Duration::from_secs(10)..Duration::from_secs(25),
            down: Duration::from_secs(1)..Duration::from_secs(5),
            straggler: Some(Duration::from_secs(20)..Duration::from_secs(40)),
            ..Outage::groups(["acceptor"])
        },
        mode: ChaosMode::Swarm,
    }])
    .chaos_duration(Duration::from_secs(30))
}

Outage::groups names the process groups that lose power together (their Process::name). It sets probability to 1.0, start to the first ten seconds of the window, down to one to ten seconds, and no straggler. Override the fields you care about.

The builder validates the regime before the first seed. An unregistered group, a probability outside 0..=1, a range that ends before it starts, or a straggler range inside down returns SimulationError::InvalidConfiguration from run(). So does an outage registered without chaos_duration.

At most once per run, with probability, at a time drawn from start, every live process of the named groups crashes in the same tick. Each one restarts after its own draw from down. With a straggler range, one victim, picked at random, is held down for a draw from that range instead. A process already dead at the outage instant is not a victim: it comes back on its own schedule. A .cluster() group whose per-seed count drew zero has no live victim, so the outage lands on no one and the run goes on.

ChaosMode::Random keeps the outage on every seed, still subject to probability. ChaosMode::Swarm drops the outage on half the seeds and the straggler on half the rest, in two draws made whether or not the outage survives.

Why the Fault Is Trustworthy

Two properties make the outage safe to build on.

Every draw comes from the simulation stream. The probability and start are drawn when the window opens. The delays are drawn at the outage instant, in ascending IP order. A seed or an exploration recipe replays the same kill and restart schedule.

Each restart is a scheduled ProcessRestart event, not a step of the injector. A restart lands even after the chaos window closes and the injector is gone, so a victim is never stranded down into the recovery tail. Victims count toward dead_count, so attrition holds off while they are down. Give the workload enough time after the window for the last straggler to come back.

OutageLanded: Acting at the Instant No Copy Lives

The injector tells you when it has landed. Once every victim’s kill has run, it publishes an OutageLanded under OUTAGE_STATE_KEY in the run’s StateHandle:

#![allow(unused)]
fn main() {
use moonpool_sim::{OUTAGE_STATE_KEY, OutageLanded};

if let Some(landed) = ctx.state().get::<OutageLanded>(OUTAGE_STATE_KEY) {
    // landed.at: the instant the kills were decided.
    // landed.victims: every victim, ascending IP order.
    // landed.down: each victim's time down, parallel to victims.
    // landed.straggler: the one held longest, if any.
}
}

A custom injector that waits for that key can plan latent disk damage at the one moment no copy lives in memory anywhere. A process can read it after its own reboot to learn it came back from a group-wide cut.

OutageVeto: A Budget the Simulator Cannot See

The harness can hold an outage back. A replicated store with a commit in flight would be left ambiguous by a cut, and only the harness knows a commit is in flight. Publish an OutageVeto under OUTAGE_VETO_KEY:

#![allow(unused)]
fn main() {
use moonpool_sim::OutageVeto;

// From a fault injector at the start of the window.
let ledger = ledger.clone();
OutageVeto::new(move |victims| ledger.safe_to_cut(victims)).publish(ctx.state());
}

At the outage instant, before it draws anything, the injector asks the veto with every victim’s IP in ascending order. A true answer is an outage: every victim crashes at once, so the harness may record it as done. A false answer leaves every victim alive, and the injector asks again VETO_RETRY (1 ms) later, with the victims live at that moment, until the veto permits the outage or the chaos window closes. The veto must not draw randomness, because its answer must not move the draw schedule.

It is the outage’s form of HintVeto: one answer for the whole victim set, so a budget that counts victims together is judged once. crates/moonpool-sim/tests/outage.rs shows a veto that holds the outage until a chosen instant and records every set it was asked about.

Self-Crashes: A Process Chooses the Instant

An outage and attrition are the environment’s decisions. A hint! is a request the simulator may refuse. Sometimes a test needs a crash at an exact point of a process’s own protocol, with no regime in the way: mid-append, between two syncs, right after a lease is written. SimContext::crash_self is that command.

#![allow(unused)]
fn main() {
use std::time::Duration;
use moonpool_sim::RebootKind;

file.write_at(0, &new_sectors).await?;
// The kill lands one tick from now, while the sync is in flight.
ctx.crash_self(RebootKind::Crash, Some(Duration::from_millis(50)))?;
file.sync_all().await?; // never completes
}

The call schedules the same force kill a fault injector’s reboot or crash does, one scheduler tick away. The process’s tasks abort, its connections reset, and its storage goes through crash resolution, so every write not yet synced, including the one whose sync is in flight, resolves by crash physics. RebootKind::CrashAndWipe also wipes the disk. Call it, then keep awaiting the work it interrupts: the kill lands before that work’s next storage completion.

With Some(delay) the process restarts that long after the kill. With None it stays down until a fault injector restarts it. No randomness is drawn, and the process counts as dead until it restarts. Two calls are errors: RebootKind::Graceful (a process shuts down gracefully by returning) and a call from a workload (only a process can crash itself).

Code that holds no SimContext, such as a driver’s hooks generic over Providers, takes a handle at boot:

#![allow(unused)]
fn main() {
// A SelfCrash is Clone + Send + Sync + 'static, so it rides into a hook.
let handle = ctx.self_crash()?;
handle.crash(RebootKind::Crash, Some(Duration::from_millis(10)))?;
}

crates/moonpool-sim/tests/crash_self.rs crashes a process between a write and its sync and checks, on the next boot, that every sector resolved to one of the two values the barrier-bounded crash model allows. Use a self-crash when the test is the point. Use a hint! when the shipped code should name the moment and the seed should decide.

Network Faults

• Connections, Not Packets
• Turning Faults On
• The Nine Families
• Partitions
• Black Holes
• Closing a Connection
• Deployment Shape
• Masks and Recovery

Connections, Not Packets

Moonpool injects network faults at the TCP connection level. FoundationDB made the same choice. A distributed system does not see packets. It sees connections that drop, stall, or report success on one side and failure on the other. Those are the faults that reach application code: reconnection, retries, leader election on disconnect, and reconciliation after a partition.

TigerBeetle simulates packets, which is the right tool for testing a network stack. For an application on top of TCP, connection faults reach the code paths that fail in production.

Turning Faults On

Every network fault lives in NetworkConfiguration, under its chaos field (a ChaosConfiguration). The builder picks a configuration per seed from the Chaos::Network entry:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(Node::new()))
    .workload(Client)
    // Each seed enables a random subset of the nine fault families.
    .enable_chaos([Chaos::Network(ChaosMode::Swarm), Chaos::BuggifyKnobs])
    .chaos_duration(Duration::from_secs(30))
}

Chaos::BuggifyKnobs composes on top. For a seed where a buggify_knob! site fires, one knob jumps to an extreme: 2 to 5 clogs per second, 0.5 to 2 partitions per second, random closes and black holes at 1% to 10%. A knob only spikes a family the seed already has. It never turns on a family the swarm turned off. See Swarm Testing for why “a little of everything” finds fewer bugs than “a lot of a few”.

NetworkConfiguration::fast_local() is the no-chaos profile for unit tests, with microsecond latencies and ChaosConfiguration::disabled().

The Nine Families

Each family is one NetworkFault variant, one bit of the swarm mask, and a few knobs on ChaosConfiguration.

Bit flips are off in the default profile, because TCP delivers what was written: only a protocol that checksums its frames can tell a flip from data. FoundationDB flips bits only where FlowTransport verifies a packet checksum. If your protocol has no checksum (HTTP/2, gRPC), mask the family out.

Two shapes are always on and are not families: short writes and short reads. A write delivers a random prefix of up to partial_write_max_bytes (1000), and a read returns a random prefix of what is buffered, up to partial_read_max_bytes, always at least one byte, because a zero-byte read means EOF. Code that calls read once and expects a whole message breaks here first.

A fault coin is only rolled when an operation can make progress. A re-poll of a read with no data, or a write under backpressure, returns Pending and draws nothing, so runtime-specific spurious polls do not shift the seed.

Partitions

A partition cuts one direction between two IPs. Random partitions arrive as a Poisson process in simulated time, at partition_rate per second. The count does not grow with traffic: a busy run and a quiet run see the same number. At most one random partition is in force at a time. The next one is drawn from the moment the current one heals, so every partition is followed by a healed gap (TigerBeetle’s partition stability). Clogs arrive the same way, at clog_rate per second for the whole network, and may overlap as in FoundationDB.

partition_strategy decides who is cut when a partition arrives:

IsolateZone and IsolateDatacenter read the .cluster() topology. Without one they fall back to Random, so every seed can draw them. The asymmetric pair finds the split brain where a node hears every heartbeat but nobody hears its answers. Each arm records its own fault event (SendPartitionCreated, RecvPartitionCreated) on the timeline.

A partition stalls bytes. It never drops them. A chunk queued or in flight on the cut direction freezes where it is and is re-timed by the length of the cut when the partition heals, with no new random draw. The peer reads the original bytes in order, or sees the connection fail. It never reads a later chunk in place of a missing one, which would corrupt the next frame of h2 on the same connection. A FIN is a chunk like any other, so the peer sees the last bytes, then EOF, never EOF first.

A scripted injector cuts and heals by hand:

#![allow(unused)]
fn main() {
// Inside a FaultInjector: isolate the leader for two seconds.
ctx.partition(&leader, &follower)?;
if ctx.time().sleep(Duration::from_secs(2)).await.is_err() {
    return Ok(()); // the run is shutting down
}
ctx.heal_partition(&leader, &follower)?;
}

FaultContext::partition cuts both directions. FaultContext::heal_partition and is_partitioned complete the set. On SimWorld, partition_pair(from, to, duration) adds one directed entry, restore_partition(from, to) removes both, and partition_send_from(ip, duration) / partition_recv_to(ip, duration) are the one-way cuts.

Black Holes

A silent close ends in EOF. A black hole never ends. When the coin fires, one direction (this side’s sends, the peer’s, or both) starts delivering nothing. Bytes are accepted into the send window and vanish. The peer’s reads stay Pending with no data and no EOF. Both ends look alive. Only the caller’s own deadline can notice, which is the point: a request without a timeout hangs here forever.

#![allow(unused)]
fn main() {
let mut config = NetworkConfiguration::fast_local();
config.chaos.black_hole_probability = 0.0001; // per I/O, like random close
config.chaos.black_hole_cooldown = Duration::from_secs(5);
}

A black hole is permanent for its connection. The only way out is a new connection. A process kill still resets its connections, so a hung peer is never mistaken for a dead one. SimWorld::black_hole_connection(id, hole_send, hole_recv) is the scripted form.

Closing a Connection

Moonpool models three ways a connection ends, and your protocol must survive all three:

• Graceful close: the closer’s FIN goes behind every queued byte. The peer reads everything, then EOF. Dropping a drained stream does this.
• Write shutdown: AsyncWrite::close sends the FIN, further writes fail with BrokenPipe, and the read half stays open. The reply to an EOF-delimited request still arrives.
• Abort: no FIN, no drain. The peer gets a reset on its next read or write. A process crash does this, and so does dropping a stream with unread bytes.

HTTP/1.1 chunked bodies and gRPC trailers both depend on reading after the peer’s shutdown. A simulator with only aborts never finds those bugs.

Deployment Shape

Three builder settings describe the network rather than break it. They apply on every seed and draw no randomness:

Flow control is entropy-neutral: a Pending on a full window and the wake after a read draw nothing. Congestion control and retransmission are out of scope. The model is a reliable ordered byte stream with a bounded number of bytes outstanding.

Masks and Recovery

A NetworkFaultMask removes a family after the seed’s profile is sampled. It draws nothing, so the seed’s other values, its RNG call count and any exploration recipe stay the same:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([Chaos::Network(ChaosMode::Swarm)])
    // gRPC over h2 has no frame checksum: a flipped bit is just data.
    .network_fault_mask(NetworkFaultMask::all().without(NetworkFault::BitFlip))
}

A mask only removes. NetworkFaultMask::all() is the default and changes nothing.

At the chaos_duration cutoff the runner calls SimWorld::enter_recovery_mode(). It zeroes every fault probability, heals every partition and makes every machine healthy. It does not repair damage: flipped bytes stay flipped, a closed connection stays closed, a black hole stays open, and a pair latency already sampled stays slow. The quiet tail is where the reconnect has to happen. The Configuration Reference lists the full boundary contract, and the Fault Reference lists every knob.

One FoundationDB result is worth keeping in mind when you write a custom injector: restoring connections in reverse order of disconnection finds more bugs than restoring them in order. Will Wilson calls it swizzling, and nobody reasoned their way to it. The simulator tried both and the data decided.

Storage Faults

• Disks Lie
• The Storage Layering Rules
  • Zero copy: owned buffers
• Turning Faults On
• Timing, Episodes and Dead Disks
• The Barrier-Bounded Crash Model
• Directory Entries Crash Too
• Failed Writeback (fsyncgate)
• The Lost-Synced-Write Oracle
• Per-Process Disks

Disks Lie

write() returns success and the bytes never reach the platter. fsync() completes and the firmware never flushed its cache. A firmware bug sends a write to the wrong sector. TigerBeetle’s documentation lists these with real incidents behind them, including studies where 8.5% of SATA drives developed silent corruption. Moonpool’s storage faults follow TigerBeetle’s taxonomy and FoundationDB’s crash model, so your integrity code meets these failures in simulation, deterministically.

The Storage Layering Rules

Before the faults, the API they land on. Moonpool has one provider, one file abstraction, one simulated file image. Every API (stream, positioned, block) reads and writes the same bytes.

Database needs are options on opening an ordinary file, never a second provider stack. There is no BlockProvider, no DirectIoProvider and no region-addressed device:

#![allow(unused)]
fn main() {
// A pager opens an existing file and demands uncached I/O.
let pages = provider
    .open("db/pages", OpenOptions::read_write().direct_io(DirectIo::Required))
    .await?;
let blocks = BlockFile::new(pages, 4096)?;
let mut page = blocks.buffer(1)?; // aligned for this file
blocks.read_blocks(7, page.as_mut_slice()).await?;
}

DirectIo has three policies. Disabled is buffered I/O. Optional asks for uncached I/O and accepts a fallback, and file.is_direct_io() tells you which one you got. Required fails the open instead of downgrading, and it opens an existing file only. Creating a file and publishing its name is a namespace protocol that belongs to the format, so a journal bootstraps its own file: create it with an ordinary open, sync_all it, sync_dir its parent, then reopen it with Required.

Keep four distinctions, because code hides behind their opposites:

1. Direct I/O is not durability. An uncached write is visible, not durable, until sync_all or sync_data completes, or unless it was a write_at_durable. A durable write covers its own range only, like O_DSYNC: it does not make an earlier unsynced write elsewhere in the file durable.
2. The caller’s block size is not the file’s I/O alignment, and neither is a crash-atomicity unit. file.constraints() reports offset, length and memory alignment separately. A two-block write tears across a crash like any other.
3. Growing a file is not preallocation.
4. File durability is not directory-entry durability. sync_all makes the bytes and the length survive. Only StorageProvider::sync_dir(parent) makes the name survive.

A file with alignment constraints refuses the stream API with ErrorKind::Unsupported, on both backends, because a shared cursor cannot stay aligned. read_at and write_at are what such a file offers. Both return the count moved, which may be short. BlockFile exists to loop over short transfers so a pager does not.

Zero copy: owned buffers

Every positioned transfer has a borrowed form (&[u8], &mut [u8]) and an owned form over an AlignedBuf (read_at_aligned, write_at_aligned, write_at_durable_aligned; on BlockFile, read_blocks_aligned, write_blocks_aligned, write_blocks_durable_aligned). They are one transfer with one contract. The owned form moves the buffer in and hands it back with the count, so the caller reuses it; an error drops it.

The difference is a copy. The Tokio backend runs positioned I/O on the blocking pool, and a borrowed slice cannot cross into a 'static task: the borrowed form copies the bytes into an aligned bounce buffer before a write and out of it after a read. The owned form moves the caller’s buffer into the task instead, so the kernel transfers straight from and into it. A storage engine that encodes its pages or log blocks directly into an AlignedBuf never copies them on the way, which is what moonpool-journal does.

#![allow(unused)]
fn main() {
let mut page = blocks.buffer(1)?;          // aligned for this file
encode_page(page.as_mut_slice());          // built in place
let page = blocks.write_blocks_durable_aligned(7, page).await?; // no copy
}

BlockFile’s owned methods are forgiving where the borrowed ones refuse: a buffer that misses the file’s memory alignment is copied once into one that meets it, and that one comes back. Only the first request moves the buffer; the rare short transfer is resumed through the borrowed form. AlignedBuf::truncate shortens a buffer in place (the last, partial chunk of a fill), and is_aligned_to checks an address. The simulator keeps the trait’s defaults, the borrowed transfer on the buffer’s bytes, so every fault, hint and oracle applies to the owned form unchanged.

write_at_durable is a write and a data sync in one call. The Tokio backend issues pwritev2(RWF_DSYNC) on Linux and falls back to pwrite plus fdatasync on a kernel that refuses the flag and on other platforms. The simulator models it as exactly write_at then sync_data: the write lands dirty, a hint! (“storage: durable write landed, not synced”) names the window before the sync, and the sync commits it. Sync failures, lying syncs, the barrier-bounded crash model and the lost-synced-write oracle all apply unchanged. The simulated sync is file-wide, which is stronger than the contract: code that relies on a durable write covering an earlier unsynced write passes in simulation and loses data on Linux, so a durable write belongs where its range is the file’s only unsynced write.

Turning Faults On

StorageConfiguration holds the timing and every fault probability. All faults are off by default, and a family at zero draws no randomness. The builder picks a profile per seed:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(Replica::new()))
    .enable_chaos([Chaos::Storage(ChaosMode::Swarm), Chaos::BuggifyKnobs])
    // A phantom write makes a replica forget a vote it cast: out of contract.
    .storage_fault_mask(StorageFaultMask::all().without(StorageFault::PhantomWrite))
    .chaos_duration(Duration::from_secs(30))
}

ChaosMode::Random uses StorageConfiguration::random_for_seed(): fault rates between 0.001% and 0.1%, IOPS from 10K to 100K, bandwidth from 50 to 500 MB/s. ChaosMode::Swarm then switches each family off with 50% probability. Chaos::BuggifyKnobs spikes a knob on some seeds: IOPS down to 100 to 5,000, sync failures up to 5% to 20%, stalls at 10% to 50%. The StorageFaultMask is applied last and draws nothing, so masked and unmasked seeds sample identically.

Timing, Episodes and Dead Disks

Steady state is a SATA SSD by default: 25,000 IOPS, 150 MB/s, reads at 50 to 200 µs, writes at 100 to 500 µs, syncs at 1 to 5 ms. fast_local() is 1M IOPS, 1 GB/s and 1 µs everywhere, with no faults. Calibration replaces the defaults with your machine’s numbers.

Real disks degrade in episodes, which FoundationDB models with its DiskFailureInjector. A stall freezes a disk for disk_stall_duration. A throttle divides IOPS and bandwidth by the two multipliers for disk_throttle_duration. An episode is keyed by the owning process, not the file, so one garbage-collection pause stalls every file the machine owns at once, and the burst of completions when it lifts is the backpressure spike that overflows real queues.

A failed disk never answers. Every read, write, sync or set_len after the failure stays Pending for the rest of the run, like FoundationDB’s failedDisk. The stall detector then reports a deadlock, which is the verdict: a process that cannot survive a hung disk has a bug. At most one disk is failed at a time, so a quorum never loses two members this way. A crash or wipe of the owner replaces the disk and fails the parked operations with OperationInterrupted. SimWorld::fail_disk_for_process(ip) is the scripted form.

What happens to a write when its future is dropped? Code wraps a write in a timeout or a select! and moves on. On a real backend the write does not stop. Tokio’s spawn_blocking task keeps running, and on io_uring the kernel owns the operation and the runtime holds its buffer until it completes. So the simulator does the same. A dropped write, positioned write, sync or set_len keeps its scheduled completion. The effect lands at that time, with the same fault rolls as an awaited operation, and nobody is woken. Closing a handle with a stream write in flight follows the same rule. A code path that assumes “dropped means not written” fails here, not in production. The dropped_mutation_discard_probability coin, drawn once at drop time, models the write that never reached the device: it is discarded whole. A dropped mutation is never half applied. A crash, a wipe or shutdown fails a dropped mutation still in flight like any other in-flight operation, and a mutation parked on a failed disk never lands. A dropped read is cancelled, because it has no effect to land.

A full disk refuses growth. SimWorld::set_disk_capacity(ip, Some(DiskCapacity::new(bytes))) gives one process’s disk a size: the sum of its files’ lengths, an unlinked file still open included. A write past the end of a file, or a set_len that grows one, fails with io::ErrorKind::StorageFull (ENOSPC) when the growth does not fit, and none of its bytes land. A write inside a file’s length needs no space, so a preallocated file stays writable on a full disk. Deleting or shrinking a file frees space at once. DiskCapacity::amplified(n) counts each byte n times, so an ordinary workload fills a disk during a short run. The capacity is the harness’s choice, not a rate: it draws nothing, holds across a crash or a wipe, and recovery mode lifts every capacity. SystemSample::space reports the disk’s DiskSpace { total_bytes, free_bytes } (statvfs in production), so the code under test can see the disk fill before it refuses.

The Barrier-Bounded Crash Model

A sync is the only barrier a file has. Everything written since the last one is up for grabs.

Every simulated file keeps two images: the durable bytes as of the last successful sync, and the visible bytes reads observe. A write dirties sectors (512 bytes, SECTOR_SIZE) in the visible image. A sync commits them. On a crash each dirty sector resolves independently:

Three knobs shape the whole crash. clean_crash_probability (10%, FoundationDB’s number) keeps every unsynced write. correlated_rollback_probability (25%) rolls back a contiguous run of up to correlated_rollback_max_run sectors together, the erase-block damage Zheng measured at FAST’13. length_survives_crash_probability (50%) decides whether an unsynced set_len holds. This is what makes “the later record landed while the earlier one is partial” reachable, which is the case journal recovery exists for.

A lost sector reads the file’s fill pattern. garbage_fill_probability draws it per file: zeros on some, garbage on others (50/50 under random_for_seed). Zeros are the dangerous real case (SATA RZAT, NVMe DLFEAT, unwritten extents), so recovery that infers “never written” from “reads as zero” has to be driven red on both.

A reboot runs the crash through SimWorld::simulate_crash_for_process(ip, close_files). Every pending operation of that process completes with an interrupted error. wipe_storage_for_process(ip) deletes everything the process owns, which is what RebootKind::CrashAndWipe does after the crash. Each crash leaves a FileCrashReport (take_storage_crash_reports()), and every injected fault a StorageFaultRecord (take_storage_fault_records()).

Directory Entries Crash Too

create_dir_all("db")   // the directory is visible
sync_dir(".")          // the db name is durable
open("db/wal", create) // a directory entry
write(..)              // the file fills
sync_all(file)         // the bytes are durable

After that sequence a crash may still leave no db/wal. The bytes were durable, the name was not. SQLite, PostgreSQL and LMDB follow every create, delete and rename with a directory sync, and sync_dir is that call. The simulator keeps a visible namespace and a durable one. Each divergence resolves independently at crash time under unsynced_dir_entry_loss_probability: a created name may be missing, a deleted one may be back. list_dir shows the visible namespace, which is what readdir sees.

Failed Writeback (fsyncgate)

sync_failure_probability is the kind failure: the sync fails and the sectors stay dirty, so a retry really makes them durable. Linux is not that kind. When the background writeback of a page fails, the kernel marks the page clean and reports the error once. A retried fsync then returns success for bytes that never reached the disk. PostgreSQL found this in 2018 (“fsyncgate”), and Rebello et al. (ATC ’20) found it on ext4, XFS and Btrfs.

With writeback_error_probability > 0, a write can make the background writeback of the file’s dirty sectors fail. These sectors become stranded: no later sync flushes them, but they stay volatile, so a crash can still lose them. A write to a stranded sector dirties it again. writeback_error_revert_probability also drops the new bytes from the cache, so reads return the durable bytes at once. Otherwise reads return the new bytes until a crash. The oracle never stamps a stranded sector, so this loss is not a simulator bug.

writeback_error_reporting picks which handles see the error on their next sync:

The safe answer is the one PostgreSQL chose: treat a sync error as data loss and crash, then recover from the log. Never retry the sync.

The Lost-Synced-Write Oracle

At every sync, each committed sector is stamped with a CRC of the bytes the caller was told are durable. That is FoundationDB’s AsyncFileWriteChecker. After a crash, a stamped sector that no longer matches is a simulator bug, and the run fails with an assertion.

Unless the barrier-violation family is armed. With barrier_violation_probability > 0, a sync occasionally lies about a sector, and the oracle flips to must-detect mode: the loss is recorded as an expected StorageFaultKind::LostSyncedWrite. This family is how a consumer proves that cluster-level recovery heals a single lying disk, the fsyncgate class of bug. SimWorld::corrupt_durable_out_of_band(path, sector) plants a deliberate violation to test the oracle itself.

The stamp covers the synced prefix of a file’s last partial sector. Growing the file makes the rest of the sector addressable and unsynced, and leaves the stamped prefix alone. A shrink inside a sector narrows the stamp to the bytes that survived.

Per-Process Disks

Every file is owned by a process IP, and faults, timing and episodes resolve against the owner’s profile. One node can have a flaky SSD while the rest are healthy:

#![allow(unused)]
fn main() {
let degraded = StorageConfiguration {
    read_corruption_probability: 0.01,
    ..StorageConfiguration::default()
};
sim.set_process_storage_config("10.0.1.2".parse()?, degraded);
}

Random faults are layout-blind: they hit whatever sector an operation touches, and most of a preallocated file is zeros nobody reads. Aiming Storage Faults shows how a format moves the damage onto the bytes recovery has to judge, and how a replicated system keeps one clean copy of every record by construction.

Aiming Storage Faults

• Most of a Disk Is Nobody’s Business
• A Map the Format Draws
• The Eligibility Mask
• Fault Focus
• Targeted Injection
• Replicated Fault Patterns
• Which One to Reach For

Most of a Disk Is Nobody’s Business

Random storage faults are layout-blind. A corruption coin lands on whatever sector an operation touches, and a crash resolves every dirty sector alike. Most of a preallocated journal is zeros that nobody reads. A uniform fault mostly lands there, the run stays green, and the sixty-four bytes that recovery has to judge, the identifiers of the last batch, are never hit.

Moonpool keeps its fault coordinates simple: a file plus a flat sector offset. It does not know what a slot, a page or a header is. The format does. So the format tells the simulator where its bytes live, and three mechanisms use that map: an eligibility mask, a fault focus and a replicated fault pattern. A fourth, targeted injection, needs no map at all.

A Map the Format Draws

LayoutRegion in moonpool-core is plain data: one named byte range of one file.

#![allow(unused)]
fn main() {
pub struct LayoutRegion {
    pub path: String,         // as the format opens it
    pub bytes: Range<u64>,
    pub kind: &'static str,   // the format's own label: "slot", "header"...
    pub stripe: Option<u64>,  // the replicated record these bytes hold
}
}

kind lets a harness weigh kinds differently. stripe is the key every replica’s copy of a record shares (a Paxos slot, a page number), or None for bytes only this node has, such as a header. Nothing in it depends on a simulator, so a format lists its regions without a cfg(test).

moonpool-journal is one such format. Journal::regions() returns every region a fault could hit: each live position’s persist record and entry, striped by position, plus the two metainfo copies and every segment’s two header copies with no stripe. Journal::layout(position) returns the Layout of one position. The kinds are Layout::RECORD, Layout::ENTRY, Layout::META and Layout::HEADER. See A Crash-Aware Journal.

The Eligibility Mask

The coarsest control is a veto: (path, sector) -> bool. SimWorld::set_storage_eligibility_mask(mask) installs one, and clear_storage_eligibility_mask() removes it. The random families and the damaging crash outcomes (Lost, LatentFault, Shorn) consult it. An ineligible sector falls back to the always-legal old-or-new resolution.

#![allow(unused)]
fn main() {
// Never damage sector 0 of any file: the superblock stays readable.
sim.set_storage_eligibility_mask(Arc::new(|_path: &str, sector: u64| sector != 0));
}

This is TigerBeetle’s ClusterFaultAtlas pattern, used by a harness that keeps its own ledger. Coins are rolled before the mask is consulted, so installing one never shifts the random stream.

Fault Focus

A mask says whether. A FaultFocus says how likely. It gives a weight to byte ranges of a process’s files and a background weight to every other sector of that disk. Every family rolled per sector or per operation (read and write corruption, read and write EIO, phantom writes, sync lies, and the damaging crash outcomes) has its probability multiplied by the weight of the sectors it would hit:

#![allow(unused)]
fn main() {
// Inside a process: aim this disk's faults at the journal's live regions.
let focus = FaultFocus::new()
    .background(0.1) // the zeros, barely
    .layout(&journal.regions(), |region| {
        if region.kind == Layout::RECORD { 8.0 } else { 4.0 }
    });
ctx.storage().focus_faults(focus)?;
}

spot(path, bytes, weight) adds one range by hand. Where spots overlap, the heaviest wins. A weight of 0 makes a sector immune. A weight never revives a family the configuration leaves at zero, and a misdirected write honours only the immunity. The rolls are drawn whatever the weights, so a focus changes which faults fire, never which draws decide them.

SimStorageProvider::focus_faults is reachable from process code and scopes the focus to that process’s disk. clear_fault_focus() removes it. A hand-driven test calls SimWorld::set_fault_focus(ip, focus). A focus survives its process’s crash and is cleared by a wipe of its storage.

Targeted Injection

A directed test does not want probabilities. It wants this sector broken, now. Four SimWorld methods are path-global, reaching every process’s file at that path:

To damage one copy of one record on one replica, aim at one owner and exact bytes:

A scripted FaultInjector holds the same four on FaultContext: durable_bytes, corrupt_bytes, fail_file_with_eio and clear_file_eio, each taking the process IP as a string. Reading the durable bytes takes no simulated time, rolls no fault coin and draws no randomness, so aiming never perturbs the run:

#![allow(unused)]
fn main() {
// Inside a FaultInjector: damage three bytes of the first replica only.
let first = &ctx.process_ips()[0];
let image = ctx.durable_bytes(first, "data")?;
let record = find_last_record(&image); // the format's own parsing
ctx.corrupt_bytes(first, "data", record.start..record.start + 3)?;
}

The other replicas’ files at "data" are untouched. corrupt_bytes refuses a range past the durable end with StorageError::OutOfRange, and a missing file with NotFound.

Replicated Fault Patterns

A replicated system survives disk damage only while some replica keeps a clean copy of every record. Random faults drawn independently per disk eventually hit every copy of one record, and a valid run becomes unwinnable. A harness could keep a ledger. Moonpool prevents it by construction, because it knows the topology:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .cluster(LocalityConfig::new(1, 3, 1..=2, 1), make_node)
    .enable_chaos([Chaos::Storage(ChaosMode::Swarm)])
    .replicated_storage_faults(ReplicatedFaults::new(DomainLevel::Zone).group("node"))
}

Each process publishes what its disk holds with ctx.storage().publish_layout(&regions). The stripe of each region is the record’s key, shared by every replica’s copy. A ReplicatedFaults covers one process group (.group(name), the processes’ Process::name), or the whole topology without one. Call replicated_storage_faults once per replicated group, so acceptors and matchmakers each draw their own pattern. Processes outside every covered group keep plain storage chaos.

Each seed draws one FaultPattern per group over the failure domains at the chosen DomainLevel (Machine, Zone or Datacenter):

tolerance(n) (default 1) is how many domains may hold damage to one record at once. It is clamped so one domain always keeps every record. Under a stripe rotation, node-local bytes (stripe None, or bytes never published) are damaged only in the tolerance - 1 local domains, and those domains may damage every stripe. A format may answer a damaged header by giving up the whole replica, so node-local damage counts against every stripe. At the default tolerance no node-local byte is damaged under this pattern.

A rolling turn is repaired when every damaged sector has been rewritten, truncated away or deleted, or is no longer inside a region its process publishes. A domain holding no damage passes the turn as soon as another domain’s disk is up for a fault. A domain that never repairs holds the turn, and a run stuck on one turn is the evidence. SimWorld::fault_turn(ip) names the domains holding the turn, and SimStorageProvider::fault_pattern() tells a process which pattern its group drew. SimWorld::draw_replicated_faults(config, &localities) draws a pattern for a hand-driven test.

Failure domains come from the .cluster() topology. A group registered with .processes() has none and is always spared. The pattern shapes the same families a focus weighs. It is a weight of zero where damage may not land, and it composes with a focus and the mask. Disk failure, wipes and lost directory entries are outside it: they take a whole disk, or only unsynced state.

Which One to Reach For

All four compose. The pattern and the mask veto first, the focus weighs what remains, and a targeted injection ignores all three because the test owns its budget.

CPU Time and Gray Failures

• A Poll Should Cost Something
• The CPU Gate
• Charging for Real Work
• Reading the Counters
• Gray Failures
• What the Gate Does Not See

A Poll Should Cost Something

Without a CPU model, a task poll takes no simulated time. Time moves only between events. A process that encodes a megabyte, folds a thousand records and sends the result does all of it at one instant. That hides a whole family of bugs:

• a busy machine answers late, and a peer’s timeout fires first,
• a burst of requests queues behind one core, and the tail latency grows,
• a busyness metric that a load balancer or a leader election reads is always zero, so the code that reads it is never tested.

FoundationDB’s simulator charges a small delay to a task as it runs. Moonpool does the same, with one core per process.

The CPU Gate

The CPU model is a per-seed chaos surface:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(5, || Box::new(Server::new()))
    .workload(Client)
    .enable_chaos([Chaos::Cpu(ChaosMode::Swarm)])
    .chaos_duration(Duration::from_secs(30))
}

ChaosMode::Random turns the model on for every seed. ChaosMode::Swarm turns it on for half of them. A seed with the model off runs exactly as it would without the entry, apart from the draw that decided it. A seed with the model on draws a CpuModel:

A low-level test sets a model with SimWorld::set_cpu_model(Some(CpuModel::new(..))) and reads the ground truth with SimWorld::cpu_busy(ip). A fault injector reads the seed’s model with FaultContext::cpu_model().

How a poll pays:

1. Every process has one CPU slot, free at some instant. Before the executor polls a process’s task, it asks the world. If the slot is busy, the task is parked and a CpuFree event is scheduled for the instant the slot frees. The stall detector counts that event as progress.
2. When the poll starts, the process’s local clock is now + poll_cost × slowness. Every byte sent or written, and every explicit cpu_charge, moves the local clock on. When the poll ends, the slot is free at the local clock.
3. While a poll runs, the process sees its local clock. time().now(), a timer, a storage submit and a sent chunk all happen after the work.
4. The network engine keeps one clock for both ends of a connection. The world shifts every network event a poll schedules by local − now, and a sent chunk leaves no earlier than its writer’s local clock.

Tasks without an owner (the workload, a fault injector, the driver) are never gated and cost nothing. Network and storage completions are world events, not tasks, so they cost nothing either. Tasks a process spawns inherit its owner, so moonpool’s own RPC tasks pay: they are shipped code, and they use the machine’s CPU in production too.

A kill or a crash drops the parked tasks of the process and frees its slot. The new incarnation starts on an idle core.

Charging for Real Work

A poll costs a small base amount. The explicit charges carry the signal. moonpool_buggify::cpu_charge lets shipped code say it just did a known amount of work:

#![allow(unused)]
fn main() {
fn fold_batch(records: &[Vec<u8>]) {
    // ... the real work ...
    moonpool_buggify::cpu_charge(Duration::from_nanos(200) * records.len() as u32);
}
}

Put a charge only on a hot path a profile shows: an encode or decode of a batch, a checksum, a fold. Like hint!, a charge is inert outside a simulation. No sink is installed, so the call costs one thread-local read and does nothing. The simulator installs its sink only while its CPU model is on, and then the calling process’s clock moves forward by the cost times its slowness. The same code runs in production and in simulation. See Same Code in Production and Simulation.

Reading the Counters

SystemProvider is the sixth provider. It reads cumulative CPU and disk counters the way the kernel exposes them (getrusage, /proc/stat, /proc/diskstats) and does no arithmetic. A caller takes two SystemSamples and divides the differences by the elapsed time, as FoundationDB’s getSystemStatistics does:

#![allow(unused)]
fn main() {
// Inside a process: sample every 100 ms, as a load reporter would.
if let Ok(sample) = ctx.providers().system().sample("") {
    samples.push(sample);
}
}

A SystemSample carries at, process_cpu, cores, machine_busy, machine_total, run_loop_busy, disk: Option<DiskCounters> and space: Option<DiskSpace>. In production, TokioSystemProvider and CompioSystemProvider read the kernel. In simulation, with the model on, process_cpu, machine_busy and run_loop_busy are the time the core was busy up to the sample’s instant, never past it, so process_cpu ≤ at always holds. One process per machine and one run loop per process make the three equal. With the model off they stay at zero. The disk counters are the simulated disk’s own: io_ticks is the time with at least one operation in flight, so a stall or throttle episode makes it rise. FoundationDB turns its system monitor off in simulation. Moonpool does not, so the code that reads the counters runs under simulation too.

A workload checks what a process reported against the samples it was really given with SimContext::system_samples(ip) and SimContext::disk_counters(ip).

Gray Failures

A crash is easy to see: the peer times out and moves on. A slow machine is harder. It answers, late. Its heartbeats arrive, a leader on it keeps its lease, and it drags the cluster down. Chaos::GrayFailure slows one or more machines for a long window and keeps them alive:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(5, || Box::new(Acceptor::new()))
    .enable_chaos([
        Chaos::Cpu(ChaosMode::Swarm),
        Chaos::GrayFailure {
            config: GrayFailure::new(2).victims(AttritionVictims::group("acceptor")),
            mode: ChaosMode::Swarm,
        },
    ])
    .chaos_duration(Duration::from_secs(30))
}

When the chaos window opens, the seed draws how many machines are slow (one to max_slow, among the processes victims admits), which ones, and for each one a window and three factors:

Each window is 30% to 100% of the chaos window, placed at a random start inside it. GrayFailure::new(max_slow) sets the defaults, and the window, cpu, disk, network and kinds fields are public. max_slow is the budget: keep it at the number of machines your system tolerates losing, so a quorum can still form. ChaosMode::Swarm masks each kind per seed, so some seeds slow only the disk and some only the network. The reach gates gray: cpu slow, gray: disk slow and gray: network slow show which kinds a campaign reached. A GrayFailure that names an unregistered group fails the run with InvalidConfiguration.

Recovery mode makes every machine healthy again, so the quiet tail runs at full speed.

A custom injector does the same by hand with FaultContext::set_slowness(ip, Slowness { cpu, disk, network }) and reads it back with FaultContext::slowness(ip). Slowness::HEALTHY is all ones. A workload reads a machine’s factors with SimContext::slowness(ip), for example to check that a slow process reports itself as slow:

#![allow(unused)]
fn main() {
// Inside a workload: a slow acceptor must not hold the lease.
if !ctx.slowness(acceptor_ip).is_healthy() {
    assert_sometimes!(leader != acceptor_ip, "a slow machine gave up the lease");
}
}

What the Gate Does Not See

The gate stamps everything that goes through the world: timers, storage operations, network events, sent chunks and the wakes they cause. A few paths reach another owner directly. Each is bounded by one poll’s CPU time, or is not part of the system under test:

If your harness shares state between processes through an Arc<Mutex<..>>, it bypasses the gate the same way. Keep such boards for observation only.

Custom Fault Injectors

• When the Built-Ins Are Not Enough
• Registering a Factory
• The FaultContext
• Scripted Lifecycle Faults
• Publishing Vetoes and Reading Landings
• Rules for an Injector

When the Built-Ins Are Not Enough

Attrition, outages, network and storage chaos are sampled per seed. They are the right tool for “break things at random and see what survives”. A scripted sequence is a different need: crash node X, let the workload make progress, bring X back, and check that X catches up. Or: wait until every replica has written its segment, then corrupt three bytes on one of them. For that you implement FaultInjector.

#![allow(unused)]
fn main() {
use std::time::Duration;
use async_trait::async_trait;
use moonpool_sim::{FaultContext, FaultInjector, RebootKind, SimulationError, SimulationResult, TimeProvider};

struct RollingRestart;

#[async_trait]
impl FaultInjector for RollingRestart {
    fn name(&self) -> &str { "rolling_restart" }

    async fn inject(&mut self, ctx: &FaultContext) -> SimulationResult<()> {
        // Restart each process in order, with time to recover between each.
        for ip in ctx.process_ips() {
            ctx.reboot(&ip, RebootKind::Graceful)?;
            ctx.time().sleep(Duration::from_secs(15)).await
                .map_err(|e| SimulationError::InvalidState(e.to_string()))?;
            if ctx.chaos_shutdown().is_cancelled() {
                break;
            }
        }
        Ok(())
    }
}
}

name() labels the injector in the report. inject() runs as one task for the chaos window. It must watch ctx.chaos_shutdown() and return once the token is cancelled.

Registering a Factory

An injector is registered through a factory, never as an instance:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(MyProcess))
    .workload_factory(|| Box::new(MyWorkload))
    .fault_factory(|| Box::new(RollingRestart))
    .chaos_duration(Duration::from_secs(30))
}

The runner builds a fresh injector from the factory for every root seed and every explored continuation. A mutated instance could not be rewound for a second timeline, so scripted sequences replay exactly from the seed plus recipe, with in-process exploration and forked workers alike. An injector that must report back (an observation log) captures an Arc in the closure.

A fault factory requires chaos_duration. run() returns SimulationError::InvalidConfiguration for a builder that registers one without it. Several factories run side by side, together with the built-in injectors enable_chaos adds.

The FaultContext

FaultContext is what an injector sees. Unlike SimContext, it exposes the fault primitives a workload must not touch.