The Sim Book

A guide to building and testing distributed systems with moonpool, a deterministic simulation framework for Rust.

Moonpool brings together ideas from FoundationDB, TigerBeetle, and Antithesis into a single library-level framework. Write your system once with provider traits, then run it against a simulated world that is deliberately worse than production. Same code, different wiring. No #[cfg(test)]. No mocks.

This book covers the philosophy, the architecture, and the practical details of building simulation-tested distributed systems.

Index

• Quick Start Routes
• Part I: Why Simulation Testing
• Part II: Foundations
• Part III: Building Simulations
• Part IV: Simulating Existing Applications
• Part V: Raw Network Simulation
• Part VI: Building on Top
• Appendix

A sitemap of every chapter in the Moonpool book. Each entry links to a chapter with a summary of what it covers.

Quick Start Routes

• “I am a coding agent adding a full simulation harness” — Guide for AI Agents
• “What is Moonpool?” — The Case for Simulation, then Why Moonpool Exists
• “How do I write my first simulation?” — Your First Simulation and its sub-chapters
• “How do providers work?” — The Provider Pattern
• “How do I add chaos/faults?” — Chaos in Moonpool
• “How do I use assertions?” — Assertions: Finding Bugs
• “How do I simulate raw TCP?” — Simulating the Network
• “How do I test an existing app (e.g. axum)?” — Using moonpool-sim Standalone
• “How do I ship this to production?” — Using Providers in Production
• “How do I migrate my existing app onto providers?” — Migrating Existing Code to Providers
• “Can the simulation run in a browser?” — Simulation in the Browser
• “How does multiverse exploration work?” — Multiverse Exploration
• “What assertions are available?” — Assertion Reference
• “What configuration options exist?” — Configuration Reference

Part I: Why Simulation Testing

• The Case for Simulation — Why distributed systems need simulation; the gap between localhost and production; failure statistics
• Prevention vs Discovery — Two testing philosophies: regression (prevention) vs generative (discovery)
• From Mocks to Simulation — Why mocks break at scale; the #[cfg(test)] trap; maintenance cost
• A Brief History — FoundationDB simulator origins, TigerBeetle storage faults, Antithesis assertions
• Why Moonpool Exists — Synthesizing ideas from FDB, TigerBeetle, and Antithesis into one framework

Part II: Foundations

• Determinism as a Foundation — Three non-determinism sources: threads, I/O, randomness; why reproducibility matters
• The Single-Core Constraint — Single-threaded execution guarantees one legal ordering; the moonpool executor
• Seed-Driven Reproducibility — One u64 seed controls entire simulation; ChaCha8Rng; cross-platform determinism
• The Deterministic Executor — Moonpool’s own single-threaded executor; seeded-random task scheduling; no reactor needed
• The Provider Pattern — Five traits (Time, Network, Task, Random, Storage) abstract all I/O; swap real vs simulated
• Quick Start: Swapping Implementations — Practical example: generic function running against TokioProviders or SimProviders
• Deep Dive: Why Providers Exist — Problems with #[cfg(test)] and mocks; providers eliminate both
• The Five Providers — TimeProvider, NetworkProvider, TaskProvider, RandomProvider, StorageProvider details
• System Under Test vs Test Driver — Process (server code) vs Workload (test driver); two distinct roles
• Process: Your Server — Process trait: name(), run(); recreated fresh on every boot from factory
• Workload: Your Test Driver — Workload trait: setup(), run(), check(); survives reboots; drives and validates

Part III: Building Simulations

• Your First Simulation — End-to-end walkthrough: KV server process, workload, assertions, builder
• Defining a Process — KvServer implementing Process trait; handling TCP; respecting shutdown
• Writing a Workload — KvWorkload tracking state; sending requests; validating responses
• Configuring the SimulationBuilder — Builder pattern: .workload(), .processes(), chaos config, iterations
• Running and Observing — cargo xtask sim run; reading reports; simulation binary structure
• Chaos Testing vs Simulation — Chaos engineering (production, reactive) vs simulation (deterministic, proactive)
• Chaos in Moonpool — Four fault dimensions: buggify, attrition, network faults, storage faults
• Buggify: Fault Injection — Two-phase activation; testing error paths; FoundationDB-inspired
• Attrition: Process Reboots — Graceful, crash, wipe reboot types; randomized kills; recovery delay
• Network Faults — Delayed connection establishment, latency, partitions, drops, reordering, and clogging
• Storage Faults — Exact delayed operations, independent handles, corruption, misdirected I/O, crash errors, and per-process disk behavior
• Assertions: Finding Bugs — Record and continue (Antithesis principle); cascade discovery
• Invariants vs Discovery vs Guidance — Three assertion categories: invariants, sometimes, numeric
• Always and Sometimes — assert_always! (must hold) vs assert_sometimes! (exploration guidance)
• Numeric Assertions — assert_always_less_than!; watermark tracking; explorer optimizes bounds
• Compound Assertions — assert_sometimes_all! for simultaneous sub-goals; frontier tracking
• Events and Invariants — plain tracing events as the timeline, the Invariant trait runs after every step, the auto-recorded fault timeline, SimTime formatter for sim-time log prefixes
• Designing Workloads That Find Bugs — Targeted adversarial design vs white noise; strategy matters
• Debugging a Failing Seed — Five-step workflow: reproduce, isolate, understand, fix, verify
• Reproducing with FixedCount — Pin seed with set_debug_seeds() + set_iterations(1); exact replay
• Reading the Event Trace — Scheduler ordering, targeted component events, and causal chain reconstruction
• Common Pitfalls — Delayed-operation driving, executor drains, provider-only calls, and component boundaries
• Discovering Properties — Systematic property discovery using attention focuses; finding where to place assertions and buggify

Part IV: Simulating Existing Applications

• Using moonpool-sim Standalone — Standalone simulation engine for existing code (axum, Postgres, etc.)
• Where to Draw the Line — Fakes vs test containers; binary failure limitations
• Wiring a Web Service — Worked example: axum service in simulation with Store trait fake, chaos, assertions
• The hyper Stack: gRPC and HTTP: moonpool-hyper’s runtime adapters, the reconnecting h2 channel, the serve helper, and why h2 keepalive and backoff stay deterministic
• What You’re Testing (and What You’re Not) — Tests handler logic and HTTP under chaos; doesn’t test TLS, proxies, startup code
• Using Providers in Production — Deploying the same code on TokioProviders; the lean dependency stanza; feature + platform matrices; the now() gotcha
• Migrating Existing Code to Providers — Routing existing tokio/rand/fs calls through providers; the call mapping; the futures::io Compat gotcha; verifying with the conformance suite
• Simulation in the Browser — Why the sim compiles to wasm; whether block_on parks; building a wasm-able crate; the moonpool-wasm-demo example; the portability CI gate

Part V: Raw Network Simulation

• Simulating the Network — Provider-backed TCP streams, connection-level faults, partial I/O, and the boundary between simulation and the kernel

Part VI: Building on Top

• Multiverse Exploration — Deterministic replay turns executions into accumulated exploration knowledge
• The Exploration Problem — Sequential Luck Problem: N unlikely events need exponential trials without branching
• The Frontier Controller — Recipes, discovery journals, one expansion per productive run
• Bounded Workers — fork() as snapshot optimization; 1 + workers live processes, any depth
• Exemplars and Continuations — Bounded exemplars per semantic state; depth-weighted continuation scheduling
• Multi-Seed Exploration — Cumulative novelty across seeds; barren seeds stop after one run

Appendix

• Assertion Reference — Complete table of 15 assertion macros with behavior and parameters
• Crate Map — Workspace crate diagram and dependency hierarchy
• Configuration Reference — SimulationBuilder methods, ChaosConfiguration, AttritionConfiguration, exploration
• Fault Reference — Every fault by category with config fields and defaults
• Glossary — Alphabetical definitions: adaptive forking, always assertion, attrition, buggify, coverage bitmap, etc.
• Sim Compatibility Checklist — Reference checklist for bringing existing Rust code into a moonpool simulation: forbidden APIs vs the provider seam
• Calibrating Against a Real Machine — Measuring real storage and network latency with raw std I/O, then generating LatencyDistribution constants

Guide for AI Agents

• The Target Architecture
• Stage 0: Map the System Before Writing Code
• Stage 1: Establish the Deterministic Boundary
• Stage 2: Put the System Under Test in Process
• Stage 3: Build a Finite, Stateful Workload
• Stage 4: Start with a Reproducible Smoke Runner
• Stage 5: Add the Correctness Oracle Before More Chaos
  • Local safety
  • Global safety through tracing
• Stage 6: Choose Guidance Macros Deliberately
  • Stable messages are property IDs
  • Boolean guidance
  • Numeric guidance
  • Bucketed guidance
• Stage 7: Add Buggify at High-Level Danger Points
• Stage 8: Layer Runner Chaos One Axis at a Time
  • Swarm the operation alphabet
• Stage 9: Configure Seeds and Stop Conditions for the Job
• Stage 10: Enable Exploration Last
• Stage 11: Debug in a Narrow Loop
• Agent Definition of Done
• Tested Examples to Read Next
• External Design Sources

This is the end-to-end implementation guide for a coding agent adding deterministic simulation to a system with Moonpool. It is deliberately gradual: make one layer trustworthy before adding the next. The specialist chapters explain each mechanism in depth; this page explains how to combine them without losing determinism, correctness, or useful search guidance.

If this page disagrees with the current Rust API, inspect the source and update the page with the code change. The most important API anchors are SimulationBuilder, Process, Workload, SimContext, Invariant, the assertion macros, and the Buggify macros.

The Target Architecture

Keep the system, driver, oracle, and search policy separate:

production logic generic over Providers
                |
       Process factory (system under test)
                |
 simulated network / time / task / random / storage
                |
  Workload factory (operations + reference model)
                |
 tracing facts ──────────────> Invariant (global safety)
 assertion macros ───────────> coverage + exploration guidance
 Buggify / Chaos ─────────────> dangerous but valid situations
                |
      SimulationBuilder (seeds, budget, replay, report)

The Process is the server code being tested. It is killed and recreated on a reboot. The Workload is the test driver. It issues operations, remembers expected outcomes, and survives process reboots within one timeline. An Invariant observes facts from all actors after every simulation step. The builder creates many fresh timelines with different deterministic seeds.

Do not put all four responsibilities in one workload. Doing so bypasses the reboot model and makes bugs much harder to localize.

Stage 0: Map the System Before Writing Code

First write down the following map. An agent should not start implementing the harness until every row has an answer.

Prefer a small abstract model. For a key-value service, use a BTreeMap of expected keys rather than copying the server implementation. For consensus, track proposed and chosen values rather than reproducing Paxos inside the driver. A reference model that shares the implementation’s algorithm can share the same bug.

Before proceeding, identify how ambiguous results are reconciled. A timed-out write might have committed. The workload must query the system, use idempotency keys, or model both legal outcomes; it must not blindly assume that an error means no state change.

Stage 1: Establish the Deterministic Boundary

All behavior that can affect a result must use Moonpool providers.

Keep the production dependency lean (moonpool with production provider features, without sim) and put moonpool-sim in a separate harness package or simulation-only target. This keeps the deterministic engine and explorer out of production artifacts. Follow the repository’s simulation-binary layout so cargo xtask sim list discovers the harness.

A standalone simulation binary needs no Tokio runtime:

fn main() {
    moonpool_sim::init_sim_tracing(tracing::Level::WARN);
    let report = build_simulation().run();
    report.eprint();
}

The executor uses one OS thread but futures must still be Send + 'static. Use Arc<RwLock<_>>, atomics, or another Send structure when state is shared between tasks. Do not use LocalSet, spawn_local, or direct Tokio runtime construction.

Behavior-affecting iteration over an unordered collection is also a source of nondeterminism. Prefer BTreeMap/BTreeSet, sort keys before choosing, or make the choice through the seeded random provider. Logging order alone is less important; protocol decisions are not.

External services, real clocks, environment changes, and global mutable state are outside recipe replay. Move them behind providers or keep them outside the simulated decision path.

Exit criterion: the same seed must produce the same observable trace twice.

Stage 2: Put the System Under Test in Process

A process factory creates a fresh in-memory instance on first boot and every reboot. Only simulated storage survives.

use async_trait::async_trait;
use moonpool_sim::{
    NetworkProvider, Process, SimContext, SimulationResult, TcpListenerTrait,
};

#[derive(Default)]
struct Node;

#[async_trait]
impl Process for Node {
    fn name(&self) -> &str {
        "node"
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        loop {
            moonpool_sim::select! {
                biased;
                accepted = listener.accept() => {
                    let (stream, peer) = accepted?;
                    // Spawn connection handling through ctx.task().
                    // Pass ctx.providers().clone() into generic production code.
                    // The handler should serve at least one healthy request.
                    let _ = (stream, peer);
                }
                () = ctx.shutdown().cancelled() => return Ok(()),
            }
        }
    }
}

Use ctx.my_ip() to bind the node. Read role tags and locality through ctx.topology(). Handle ctx.shutdown() for graceful reboot, but do not rely on it for crash reboot: a crash cancels the task immediately. Reopen listeners, connections, and files on every boot.

Keep the production algorithm generic over P: Providers when possible. The Process should be a thin simulation entry point that supplies ctx.providers() to the same logic used with production providers.

A Process::run error ends that process and is logged, but is not by itself a client-test failure. The workload or an invariant must observe and validate the effect that the process exit has on the protocol.

Exit criterion: one node can boot, serve a healthy request, and stop without leaking a task or pending event.

Stage 3: Build a Finite, Stateful Workload

Workloads have three phases:

1. setup: setup tasks may run concurrently, followed by a barrier before any workload run method starts.
2. run: all workload instances execute concurrently. The first workload to finish triggers the shared shutdown token.
3. check: check tasks run after processes have been aborted and the timeline has settled.

The workload should be finite. Background server loops belong in Process and end through shutdown. A useful workload owns a reference model and chooses from a small, explicit operation alphabet.

use std::collections::BTreeMap;

use async_trait::async_trait;
use moonpool_sim::{SimContext, SimulationError, SimulationResult, Workload};

struct ClientWorkload {
    expected: BTreeMap<String, Vec<u8>>,
    operations: usize,
}

impl ClientWorkload {
    fn new(operations: usize) -> Self {
        Self {
            expected: BTreeMap::new(),
            operations,
        }
    }
}

#[async_trait]
impl Workload for ClientWorkload {
    fn name(&self) -> &str {
        "client"
    }

    async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        if ctx.topology().all_process_ips().is_empty() {
            return Err(SimulationError::InvalidState("no server processes".into()));
        }
        Ok(())
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        for _ in 0..self.operations {
            // Choose an enabled operation with ctx.random(), execute it through
            // ctx.network(), and update `expected` only after reconciling
            // success, failure, and ambiguous timeout outcomes.
        }
        Ok(())
    }

    async fn check(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        // Validate the retained model and captured timeline. Processes have
        // already stopped, so do not attempt a live RPC here.
        let _ = ctx;
        Ok(())
    }
}

Choose registration by lifecycle, not convenience:

Prefer a factory from the beginning. It prevents one seed’s driver state from contaminating another and keeps later exploration/replay migration simple.

For multiple clients, use ctx.client_id() and ctx.client_count() to divide work without overlap. Names should be useful in reports. Use ctx.topology().all_process_ips() or tag queries instead of hard-coding IPs. Coordinate their termination: because the first completed workload triggers global shutdown, one short driver can cancel longer peers. A single workload that spawns coordinated client tasks is often simpler when all clients must finish together.

Perform any final live query during run, before returning. Use check for the retained reference model and ctx.observability() timeline. A check error or panic is currently logged rather than reliably becoming a workload result, so record check contracts with Moonpool always assertions as well.

Exit criterion: the happy-path reference model agrees with the system for one fixed seed.

Stage 4: Start with a Reproducible Smoke Runner

Begin with one known seed and one iteration. Do not add every fault surface at once.

use moonpool_sim::SimulationBuilder;

let report = SimulationBuilder::new()
    .processes(3, || Box::new(Node))
    .workload_factory(|| Box::new(ClientWorkload::new(20)))
    .set_debug_seeds(vec![1])
    .set_iterations(1)
    .run();

report.eprint();
assert_eq!(report.failed_runs, 0, "smoke seed failed: {report}");
assert!(
    report.assertion_violations.is_empty(),
    "safety property failed: {report}"
);

SimulationBuilder::run() is synchronous; do not add .await or create a Tokio runtime around it.

No explicit Chaos entry means the runner uses its baseline network and storage configurations. This is not the same as a perfectly inert network: the current baseline network includes seeded timing variation, probabilistic connect failure, partial I/O, rare corruption/close, clock drift, and buggified delay, while baseline storage faults are off. Chaos::Network and Chaos::Storage select per-seed randomized or swarm configurations.

If a process topology matters, replace .processes with .cluster and add .link_latency. If the system has several kinds of server (acceptors and matchmakers, a tier and a spare pool), call .processes once per role: each call is an independent group named after its process type, with its own per-seed count and its own 10.0.{group}.x IP range, queried with ctx.topology().ips_in_group("name"); a process’s client_id / client_count are its index within and the size of its own group. If roles inside one group matter, call .tags(...) after that group’s registration; remember that .tags returns a Result.

Exit criterion: the seed succeeds repeatedly and the report has no safety violations.

Stage 5: Add the Correctness Oracle Before More Chaos

Use four complementary oracles:

1. Return SimulationError when an operation cannot complete its test contract.
2. Use always-type macros for local facts.
3. Emit stable tracing facts and use Invariant for global, cross-process, cross-time safety.
4. Make the final live query in Workload::run, then use Workload::check for retained model/timeline assertions after processes stop.

Local safety

moonpool_sim::assert_always!(
    applied_index <= committed_index,
    "replica: applied index never exceeds commit index",
    { "committed" => committed_index, "applied" => applied_index }
);

Moonpool’s assertion macros record failures; they do not immediately panic. The simulation report is the authority. Never delete or weaken a failing assertion to make a run pass.

Global safety through tracing

Emit plain INFO-or-higher tracing events inside a Process or Workload task:

tracing::info!(
    target: "protocol",
    slot,
    ballot,
    value = %value_hash,
    "value_chosen"
);

The event message must be a non-empty constant name. Use % for string values. The run’s subscriber is floored at INFO by default, so DEBUG/TRACE spans and events (#[tracing::instrument(level = "trace")] on hot paths) cost one level compare and are never allocated; .trace_level(LevelFilter::DEBUG) on the builder lowers the floor to debug one seed, capturing those events too. Do not add simulated time manually; the observability layer stamps it. Actor spans provide source attribution automatically.

Built-in network, storage, and process faults join the same timeline as "sim_fault" events (SIM_FAULT_EVENT_NAME), with source = "sim" and a kind field. An invariant can use a separate cursor for that event stream to correlate a timeout, reboot, corruption, or partition with later application facts. This is usually more reliable than reconstructing fault history from formatted logs.

An invariant reads only new events using a cursor and resets its own state for the next timeline:

use std::cell::{Cell, RefCell};
use std::collections::BTreeMap;

use moonpool_sim::{Invariant, TraceQuery};

struct Agreement {
    cursor: Cell<usize>,
    chosen: RefCell<BTreeMap<u64, String>>,
}

impl Agreement {
    fn new() -> Self {
        Self {
            cursor: Cell::new(0),
            chosen: RefCell::new(BTreeMap::new()),
        }
    }
}

impl Invariant for Agreement {
    fn name(&self) -> &str {
        "agreement"
    }

    fn observe(&self, query: &dyn TraceQuery, _sim_time_ms: u64) {
        let mut chosen = self.chosen.borrow_mut();
        for event in query.since("value_chosen", &self.cursor) {
            let Some(slot) = event.u64("slot") else {
                moonpool_sim::assert_unreachable!(
                    "protocol: value_chosen event missing slot"
                );
                continue;
            };
            let Some(value) = event.str("value") else {
                moonpool_sim::assert_unreachable!(
                    "protocol: value_chosen event missing value"
                );
                continue;
            };
            if let Some(previous) = chosen.get(&slot) {
                moonpool_sim::assert_always!(
                    previous == value,
                    "protocol: one chosen value per slot"
                );
            } else {
                chosen.insert(slot, value.to_owned());
            }
        }
    }

    fn reset(&mut self) {
        self.cursor.set(0);
        self.chosen.get_mut().clear();
    }
}

Register it with .invariant(Agreement { ... }). Invariants run after every simulation step, so keep them cursor-based and inexpensive. Treat observe as read-only; do not emit new tracing facts or mutate the system from it.

Exit criterion: the harness can detect a deliberately planted wrong result. Remove the planted bug, not the oracle.

Stage 6: Choose Guidance Macros Deliberately

Antithesis describes assertions as both properties and clues for search. Its guidance is useful here too: line coverage says a location ran, while a sometimes assertion describes a meaningful situation. Moonpool uses the successful encounters, watermarks, and frontiers below as semantic replay anchors.

The exact numeric safety macros are assert_always_greater_than!, assert_always_greater_than_or_equal_to!, assert_always_less_than!, and assert_always_less_than_or_equal_to!. The exact numeric guidance macros use the same four suffixes after assert_sometimes_. Do not invent shortened spellings.

Stable messages are property IDs

Assertion slots are keyed by the message hash, not by file and line. Use one short, stable, human-readable message for one property. Do not interpolate a node ID, ballot, slot, key, or seed into the message. Do not reuse one message for different macro kinds. Messages are stored in fixed-size shared slots, so put dynamic context in supported detail fields or tracing events instead. Detail maps are supported by boolean always/optional-always/unreachable and numeric always macros; the sometimes guidance macros do not accept them.

Unlike Antithesis’s build-time assertion catalog, Moonpool allocates an assertion slot only when the macro executes. Consequently:

• assert_reachable!("retry entered") is an anchor once the retry path runs; it cannot report a path that was never executed because no slot exists yet.
• To prove a condition from an always-executed location, use assert_sometimes!(retry_count > 0, "client: retry occurred").
• A sometimes site that never executes is likewise unknown. Put important coverage checks at a stable observation point, not only inside the rare arm.
• An adaptive run with no observed boolean sometimes/reachable slot cannot satisfy its “all reached” gate and will run to the iteration cap. Add at least one repeatedly evaluated boolean coverage contract to a campaign.

Boolean guidance

Use assert_sometimes! for a single meaningful state:

moonpool_sim::assert_sometimes!(
    retry_count > 0,
    "client: request completed after retry"
);

Use assert_sometimes_all! when the hard part is making several facts true at the same time. Name each proposition; all expressions are evaluated.

moonpool_sim::assert_sometimes_all!("protocol: failover completed", [
    ("old leader unavailable", old_leader_unavailable),
    ("new quorum formed", new_quorum_formed),
    ("client completed", client_completed),
]);

This macro guides on frontier improvement: a run with two of three facts is more useful than a run with one. It also distinguishes bounded partial truth combinations, so “only old leader unavailable” and “only new quorum formed” can both become replayable states even though each has frontier one. The final report and adaptive coverage gate remain incomplete until all propositions are true together; do not duplicate it with a separate boolean completion check.

Partial combinations use a 64-bit per-site bitmap. This is bounded guidance, not exhaustive subset accounting: collisions may merge optional hints. Use assert_sometimes_each! with a deliberately small identity vocabulary when exact per-value reporting matters.

Numeric guidance

Use numeric sometimes macros for depth. A boolean such as depth > 3 produces one discovery; a maximizing watermark can keep producing useful discoveries as depth improves.

moonpool_sim::assert_sometimes_greater_than!(
    decided_slots,
    32,
    "protocol: decided slot watermark"
);

moonpool_sim::assert_sometimes_less_than!(
    healthy_replicas,
    3,
    "protocol: low healthy replica watermark"
);

Choose direction carefully. Greater/greater-or-equal guidance maximizes left - right; less/less-or-equal guidance minimizes it. The human report still displays the best observed left operand. Tracking the distance means a dynamic threshold cannot report false progress merely because both operands grew. Values are converted to i64, so avoid relying on larger unsigned ranges.

The first numeric evaluation establishes a watermark discovery even when the comparison has not passed yet. Later improvements can therefore guide a search gradually from 1 toward a target of 100.

Bucketed guidance

Use assert_sometimes_each! to explore a small semantic vocabulary:

moonpool_sim::assert_sometimes_each!(
    "protocol state",
    [("phase", phase_code), ("fault regime", fault_regime)],
    [("decided slots", decided_slots)]
);

Identity values, in their supplied order, decide which bucket this is; key names are descriptive and do not change identity. Quality keys rank better exemplars inside that bucket. Keep identity coarse: phase, role, recovery mode, quorum shape, or fault regime. Put progress such as decided slots, inverse lag, or remaining health in quality.

The shared tables are bounded: 512 assertion sites and 256 assert_sometimes_each! buckets. Supply no more than six identity keys; extra values affect the bucket hash but are not retained for display. Never bucket unbounded keys, request IDs, ballots, or log slots. Bucket them into a finite range first. Once a shared table is full, additional sites or buckets are not tracked, so table exhaustion silently destroys guidance.

Quality always maximizes, supports at most four values, and packs the low 16 bits of each value in order. Normalize values into 0..=32767, put the most important value first, and invert a cost when lower is better.

Bucket guidance has no final “all expected identities were visited” contract. Inspect report.bucket_summaries or add stable outer assert_sometimes! contracts for states that the campaign must reach.

Exit criterion: every important rare state has a stable semantic signal, and the signals describe situations rather than implementation lines.

Stage 7: Add Buggify at High-Level Danger Points

Random network and disk faults naturally exercise low-level failure handling. They may take an enormous number of trials to create a high-level dangerous sequence such as two minimal-quorum elections followed by a retry. Buggify lets the application cooperate with the simulator by making those valid but rare choices directly.

Moonpool’s model has two decisions:

1. On first encounter, each source file:line Buggify location is activated or disabled for the whole timeline. SimulationBuilder currently uses 50% activation.
2. An active location fires on each call with the macro probability: 25% for buggify!() and buggify_knob!(), or the argument to buggify_with_prob!(p).

The first encounter of a default point therefore has a 12.5% marginal firing chance. If the point is activated, later calls each have a 25% chance; if it is disabled, it never fires in that timeline. Keep custom probabilities in 0.0..=1.0.

Buggify is initialized for every builder iteration independently of .enable_chaos(...). There is no builder switch for ordinary call-site Buggify. Outside an active simulation iteration it returns false. Moving a Buggify call to a different line changes its identity and seeded behavior. Adding, removing, or reordering Buggify points also changes simulation RNG consumption and can invalidate exact recipes against the changed binary. Do not call buggify_init or buggify_reset from a builder-managed workload. Two macro expansions on the same file:line share one activation identity.

Start with one point and pair it with a coverage signal and a safety oracle:

let use_minimal_quorum = moonpool_sim::buggify!();
moonpool_sim::assert_sometimes!(
    use_minimal_quorum,
    "protocol: minimal quorum path selected"
);

let responders = if use_minimal_quorum {
    quorum_size
} else {
    all_replicas
};

Buggify itself is not an explorer discovery. The paired sometimes assertion, reachability marker, watermark, frontier, or bucket creates the semantic anchor after the injected behavior becomes meaningful.

Good Buggify candidates are:

• perform only the minimum correct amount of optional work;
• substitute a retryable error after a lower layer succeeded, forcing ambiguous-result handling;
• add or lengthen a delay through ctx.time() at a concurrency-sensitive boundary;
• choose a valid edge configuration, such as a batch size of one;
• activate an alternate supported path that defaults rarely or never.

Use buggify_with_prob!(p) when a frequently evaluated point would otherwise dominate the run. Use buggify_knob!(default, lo..hi) for an application knob:

let batch_size = moonpool_sim::buggify_knob!(128_usize, 1_usize..16_usize);

Always supply a valid, non-empty half-open range. A fired invalid range falls back to range.start, not to the default value.

Chaos::BuggifyKnobs is different. It asks the builder to spike selected built-in network/storage configuration values, and it only modifies a surface that is also explicitly enabled:

.enable_chaos([
    Chaos::Network(ChaosMode::Swarm),
    Chaos::Storage(ChaosMode::Swarm),
    Chaos::BuggifyKnobs,
])

Do not use Buggify to invent impossible states, violate the protocol’s own preconditions, or simulate a process crash. Use built-in attrition for graceful, crash, and crash-and-wipe lifecycle behavior. Keep injection code in simulation-specific adapters or behind the application’s simulation feature so production builds can omit moonpool-sim and the explorer.

The practical rule from FoundationDB-style Buggify is “do bad things, but not too many of them.” Preserve enough successful work for the system to make progress and recover. A chaos phase should eventually stop injecting background lifecycle faults so the workload can settle and validate recovery.

Exit criterion: each Buggify point is observed across seeds, has a named reason to find bugs, and leaves the system in a state the real implementation could encounter.

Stage 8: Layer Runner Chaos One Axis at a Time

Once the oracle catches planted bugs and the smoke seed is stable, expand breadth gradually:

1. Multiple baseline/default seeds.
2. Network Random mode.
3. Storage Random mode if the system persists state.
4. Attrition with at most one unavailable process.
5. Swarm mode for each fault surface.
6. Workload operation swarming.
7. Correlated locality failures and distance latency if deployment shape matters.

use std::time::Duration;

use moonpool_sim::{Attrition, AttritionScope, AttritionVictims, Chaos, ChaosMode};

let report = SimulationBuilder::new()
    .processes(3, || Box::new(Node))
    .workload_factory(|| Box::new(ClientWorkload::new(200)))
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition {
            config: Attrition {
                max_dead: 1,
                prob_graceful: 1.0,
                prob_crash: 2.0,
                prob_wipe: 0.0,
                recovery_delay_ms: None,
                grace_period_ms: None,
                scope: AttritionScope::PerProcess,
                victims: AttritionVictims::Any,
            },
            mode: ChaosMode::Swarm,
        },
        Chaos::BuggifyKnobs,
    ])
    .until_coverage_stable(10, 5_000)
    .run();

The attrition prob_* values are weights, not percentages. Wipe is only valid when the model permits losing that process’s durable state. Failure-domain attrition needs .cluster(...) and a budget large enough to remove the chosen machine, zone, or datacenter. With several process groups, set victims to AttritionVictims::group("acceptor") (or ::tagged(key, value)) so kills land on the role under test rather than on spares; max_dead then counts dead processes in that pool only, and one Chaos::Attrition entry per group gives each role its own regime and budget.

.chaos_duration(...) bounds the window in which Moonpool may inject new faults of any kind. At the cutoff the runner stops attrition and custom fault injectors and calls SimWorld::enter_recovery_mode(), which switches off every configuration-driven network, storage, and block-device fault family and heals the partitions in force.

It stops fault generation, it does not repair anything: corrupted sectors, lost or misdirected writes, connections already closed, and processes already killed all survive the cutoff, and finite effects already started (disk stall/throttle episodes, clogs, delayed packets) expire on their own schedule. The cluster is not healthy at the cutoff — the environment merely stops making it worse.

Leave a recovery segment inside Workload::run after the cutoff for reconvergence and final live queries: that quiet tail, with the processes still alive, is the only window where protocol recovery can happen (settle runs after the processes are aborted). Then use check for retained state and timeline assertions.

Swarm the operation alphabet

Faults can suppress one another, and so can workload actions. Assign stable u8 IDs to operation families and filter them with swarm_op_enabled:

const PUT: u8 = 0;
const GET: u8 = 1;
const DELETE: u8 = 2;

let mut enabled: Vec<u8> = [PUT, GET, DELETE]
    .into_iter()
    .filter(|op| moonpool_sim::swarm_op_enabled(*op))
    .collect();
if enabled.is_empty() {
    enabled.extend([PUT, GET, DELETE]);
}

Opt in with .swarm_operations(). Keep the operation picker RNG discipline stable: consume the same number of simulation RNG draws per logical step where practical. This keeps exploration recipe coordinates meaningful when the enabled alphabet changes.

Exit criterion: each chaos axis has passed alone before the combined campaign is trusted.

Stage 9: Configure Seeds and Stop Conditions for the Job

Use the runner mode that matches the current stage:

UntilCoverageStable is the builder default: 10 quiet seeds and a 1,000-seed cap. Under cargo xtask sim run, LLVM sanitizer coverage supplies the plateau signal. Under ordinary test execution, Moonpool falls back to reached sometimes/reachable assertion slots. report.saturation names the signal.

set_debug_seeds supplies seeds but does not change the iteration limit; pair it with .set_iterations(seeds.len()). Campaign seeds that are not supplied explicitly start from wall-clock entropy. Always retain report.seeds_used and report.seeds_failing so individual timelines remain reproducible.

Set .run_time_budget(...) to catch self-perpetuating simulated-time loops; the default is intentionally generous.

A strict campaign binary should print the report and fail CI on all of these:

report.eprint();

let failed = report.failed_runs > 0
    || !report.assertion_violations.is_empty()
    || !report.coverage_violations.is_empty()
    || report.convergence_timeout;

if failed {
    std::process::exit(1);
}

For a deliberately short smoke or single-seed replay, coverage misses are expected; fail only on run and safety failures there. Never silently accept convergence_timeout in the campaign intended to establish coverage.

Run simulation binaries through xtask:

nix develop --command cargo xtask sim list
nix develop --command cargo xtask sim run my-system

Exit criterion: the report policy distinguishes safety failure, coverage miss, and saturation timeout.

Stage 10: Enable Exploration Last

Exploration adds depth after the deterministic lifecycle, oracle, and semantic signals are trustworthy. It replays every timeline from the beginning, reaches an assertion discovery coordinate, and then changes the counted simulation RNG for the continuation.

Start in deterministic in-process mode:

use moonpool_sim::ExplorationConfig;

let report = SimulationBuilder::new()
    .processes(3, || Box::new(Node))
    .workload_factory(|| Box::new(ClientWorkload::new(200)))
    .invariant(Agreement::new())
    .enable_exploration(ExplorationConfig {
        workers: 0,
        max_runs_per_seed: 2_000,
        branching_factor: 4,
        max_frontier: 512,
        max_recipe_len: 32,
    })
    .until_coverage_stable(10, 1_000)
    .run();

Exploration rejects .workload(instance) because it cannot be reconstructed with fresh state. Use workload factories; fault injectors are always factory-built (fault_factory and the built-in Chaos surfaces). Increase workers only in a standalone single-threaded simulation binary after the fork boundary has been audited.

Assertion discoveries guide the frontier. Sanitizer code coverage is a report and plateau signal, not a recipe anchor. Numeric watermarks, assert_sometimes_all!, and bounded assert_sometimes_each! states provide more sustained guidance than many one-shot booleans.

Recipes change the counted simulation RNG; executor scheduling, select! offsets, and configuration/swarm choices still derive from the root seed. Use many root seeds for scheduling breadth and exploration for semantic depth.

Replay every captured failure with a fresh builder:

let Some(bug) = report
    .exploration
    .as_ref()
    .and_then(|exploration| exploration.bug_recipes.first())
else {
    eprintln!("expected an exploration bug recipe");
    std::process::exit(1);
};

let replay = SimulationBuilder::new()
    .processes(3, || Box::new(Node))
    .workload_factory(|| Box::new(ClientWorkload::new(200)))
    .invariant(Agreement::new())
    .replay_timeline(bug.seed, bug.recipe.clone())
    .run();

If replay differs, audit external I/O, reused state, unseeded collections, direct Tokio calls, changed Buggify line locations, and inconsistent RNG draw counts. See Exploring a Consensus Protocol for consensus-specific signals and operational limits.

Exit criterion: a captured planted failure replays on a fresh cluster before real findings are trusted.

Stage 11: Debug in a Narrow Loop

When a safety assertion catches a bug:

1. Stop broad random changes.
2. Record the ordinary failing seed or exploration recipe.
3. Rebuild exactly the same processes, workload factories, invariant state, chaos surfaces, and operation alphabet.
4. Run one seed or one recipe at ERROR logging first.
5. Add stable structured tracing around the violated state transition.
6. Trace the full data flow and persistence boundary.
7. Fix the implementation, never the assertion.
8. Replay the exact failure.
9. Restore the full coverage-stable campaign.

Do not turn a discovered bug into a narrow seed-only regression test and stop. The value of Moonpool is continuing to search the neighboring state space after the root cause is fixed.

Agent Definition of Done

Before claiming that a simulation integration is complete, verify all of the following:

• Production behavior uses provider traits for task, time, network, storage, and randomness.
• Server state lives in a factory-created Process; durable state uses simulated storage.
• Every timeline receives a fresh workload and reference model.
• The operation alphabet is finite, named, and able to reconcile ambiguous results.
• Local safety uses always-type macros; global safety uses cursor-based tracing invariants.
• Assertion messages are stable, unique, and non-dynamic.
• Sometimes signals describe meaningful situations, with numeric/frontier/ bucket guidance for depth.
• Each Buggify point creates a possible dangerous state and has paired coverage plus safety checks.
• Network, storage, attrition, Buggify knobs, and operation swarm were added gradually.
• The report is checked for run failures, safety violations, intended coverage misses, and convergence timeout.
• Failing seeds and recipes replay from fresh state.
• The repository’s required formatting, Clippy, tests, portability checks, simulation binaries, and book build pass.

For this repository, the final validation baseline is:

nix develop --command cargo fmt
nix develop --command cargo clippy
nix develop --command cargo nextest run
nix develop --command mdbook build book/

Run the relevant cargo xtask sim run <filter> campaign as well. When crate features or portability change, also run the no-default-feature and wasm checks documented in the repository instructions.

Tested Examples to Read Next

Use current tested code as an API reference instead of inventing one giant example:

• crates/moonpool-sim/tests/exploration/tests.rs: process, simulated TCP, tracing invariant, semantic guidance, exploration, and recipe replay.
• crates/moonpool-sim-examples/src/topology.rs: topology, provider timeouts, and correlated attrition.
• crates/moonpool-sim-examples/src/tonic_grpc.rs: production-style transport integration and recovery coverage.
• crates/moonpool-sim-examples/src/dungeon.rs: fixed-draw operation swarming, compound guidance, and quality watermarks.
• crates/moonpool-sim/tests/chaos/swarm.rs: layering Chaos::BuggifyKnobs onto enabled fault surfaces.
• crates/moonpool/examples/metastable_grpc_retry_storm.rs: open-loop load over gRPC, Prometheus series read back through MetricQuery, and an ASCII time-series graph printed to stdout.

Then use the detailed chapters on Process and Workload, assertion concepts, events and invariants, and designing workloads.

External Design Sources

Moonpool’s Buggify placement advice follows the practical patterns documented in BUGGIFY: do minimal valid work, force rare error handling, emphasize concurrency, vary knobs, and leave a recovery phase. The two-stage probabilities above are Moonpool’s current implementation details, not FoundationDB defaults.

The assertion strategy is informed by Antithesis’s official guidance on asserting correctness and sometimes assertions: properties should have stable human-readable identities, and semantic situations are more useful than undifferentiated line coverage. Moonpool’s fixed shared tables, runtime-only cataloging, and replay-from-start frontier are Moonpool-specific and are described explicitly above.

State of Moonpool

• What works
• What is experimental
• How to read this book

Moonpool is a hobby project under active development. It is not production-ready, and the APIs will change. This book documents the framework as it exists today, with honest markers for what works and what remains experimental.

What works

The simulation engine is the most mature piece. Single-threaded deterministic execution, seed-driven reproducibility, and simulated time advancement all function as described in this book and exercised by the simulation binaries shipped with the repository. A failing seed gives you a reproducible local debugging session, every time.

Chaos testing covers network faults (partitions, latency, connection drops, reordering) and storage faults (corruption, torn writes, misdirected I/O) with per-process scoping so each node experiences independent failures. The BUGGIFY-style injection system runs at configurable probability, and the Hurst exponent manipulation produces correlated, cascading failures that mirror real datacenter behavior.

The assertion suite implements the full Antithesis-inspired taxonomy: always, sometimes, reachable, unreachable, numeric, and sometimes-all assertions. These live in shared memory and survive fork boundaries for multiverse exploration.

Provider-backed networking runs the same application protocol over real TCP or simulated byte streams. The hyper integration carries real HTTP, axum, and tonic traffic through that same boundary.

Frontier-based multiverse exploration is operational: deterministic recipe replay, a bounded worker pool, exemplar-guided continuations, and multi-seed exploration with cumulative novelty across seeds.

What is experimental

Parallel exploration (multi-process exploration across CPU cores) is not yet implemented.

How to read this book

Part I stands alone as philosophy. You can read it without ever touching moonpool code, and the ideas apply to any simulation framework.

Parts II through VI are practical and reflect current APIs. Code examples compile against the latest version, but expect them to evolve. When an API is experimental, the text says so.

Part VI on multiverse exploration describes the most novel piece of moonpool. If you are evaluating whether fork-based exploration matters for your use case, start there.

The Case for Simulation

• The gap
• The combinatorial problem
• Why coverage metrics lie
• The structural impossibility

You deploy on a Friday afternoon. The tests are green. Code review was thorough. You even ran the integration suite twice. You go home.

At 2am, your phone lights up. A network partition isolated two nodes for eleven seconds. During recovery, a message arrived out of order. A retry collided with a timeout. The system entered a state that nobody on your team imagined was reachable. Data was lost.

The code was correct for the world your tests described. It was not correct for the world production delivered.

The gap

Development environments are clean. The network is localhost. Disks never fail. Clocks agree. Messages arrive in order, exactly once. We know this is fictional, yet our test environments faithfully reproduce the fiction. We test against a world that does not exist, then express surprise when the real world finds the bugs we missed.

Production is a different animal. A study of 198 failures in Cassandra, HBase, HDFS, MapReduce, and Redis found that 74% were deterministic, most could be reproduced on 3 or fewer nodes, and 77% could have been caught by a unit test asserting against the correct error condition. The bugs were not exotic. They were ordinary mistakes in error handling code that nobody thought to test. Research on network partition failures in cloud services showed that 80% of catastrophic failures in distributed systems are caused by partition-related bugs, and 27% of those result in data loss. These are not exotic edge cases. They are Tuesday.

The gap between development and production is not a minor oversight we can close by writing more careful tests. It is structural.

The combinatorial problem

Consider a modest distributed system: three nodes, a leader election protocol, and replicated state. Now list the things that can go wrong. Any node can crash and restart. The network between any pair of nodes can partition. Messages can be delayed, reordered, or duplicated. Disk writes can be torn or lost. Clocks can drift.

A single test scenario might be: “Node B crashes during a leader election while Node A has an in-flight write and the network between A and C drops for two seconds.” That is one scenario. How many are there?

Even with coarse-grained modeling, a three-node cluster with five failure types and ten time steps produces thousands of distinct failure histories. A five-node cluster with realistic failure granularity produces millions. And that is before considering application-level state: what data was in flight, which transactions were uncommitted, which clients were retrying.

Consider a simple e-commerce API as an example. Six variable dimensions (user types, payment methods, delivery options, promotions, inventory status, currencies) require 648 unique test combinations for basic coverage. Adding one option to each dimension pushes it past 4,000. A real system has hundreds of dimensions. In one real project, a 300-line feature required a 10,000-line test PR to maintain combinatorial coverage.

This is not a tooling problem. No test framework makes writing 10,000 tests sustainable. The combinatorial space of a distributed system grows faster than any team can write tests for it.

Why coverage metrics lie

You might look at your coverage report and feel reassured. 85% line coverage. 70% branch coverage. These numbers measure how much code your tests execute. They say nothing about how much state space your tests explore.

A distributed system can execute the same lines of code in thousands of different orderings with thousands of different timing relationships. Line coverage treats all of those as identical. Branch coverage is slightly better but still blind to interleaving. You can have 100% branch coverage and never once test what happens when a leader election overlaps with a network partition during a compaction.

Will Wilson put it precisely: the very reason tests are needed (humans cannot enumerate all control flow paths) is exactly what makes it impossible for humans to write comprehensive tests. This is not a failure of discipline. It is a logical impossibility. Manual tests verify what developers imagined. They cannot verify what developers did not imagine. And bugs, by definition, live in the places nobody imagined.

The structural impossibility

Here is the argument in its sharpest form.

Distributed systems fail in ways that depend on the ordering of concurrent events. The number of possible orderings grows combinatorially with system size. Humans write tests based on scenarios they can imagine. The scenarios that cause bugs are, almost by definition, the ones nobody imagined. Therefore, comprehensive manual testing of a distributed system is not merely difficult. It is structurally impossible.

This does not mean we should give up on testing. It means we need a fundamentally different approach. Instead of writing individual tests by hand, we need to generate tests automatically. Instead of testing against a clean, predictable environment, we need to test against one that is worse than production. Instead of hoping we covered the important cases, we need infrastructure that systematically explores the space of all possible failures.

That is what simulation gives us. Not better tests. A different kind of testing entirely.

Prevention vs Discovery

• Prevention: guarding what you know
• Discovery: finding what you do not know
• Three principles that enable discovery
• The feedback loop
• The spectrum, not the binary

Most teams think about testing as one thing. Write tests, run tests, check the results. But there are actually two fundamentally different activities hiding under that single word, and conflating them is one of the most consequential mistakes in software engineering.

Prevention: guarding what you know

The first kind of testing is prevention. You ship a feature. A user reports a bug. You fix the bug, then write a test that would have caught it. Next time someone modifies that code, the test fails before the bug can return.

This is regression testing. It is valuable, well-understood, and nearly universal. CI pipelines run thousands of these tests on every commit. The mental model is defensive: we are building a wall around known-good behavior, brick by brick. Each bug report adds a brick.

Prevention testing answers one question: did we break what already worked?

Discovery: finding what you do not know

The second kind of testing is discovery. Rather than guarding known behavior, discovery testing actively searches for unknown failure modes. It asks: what else is broken that we have not found yet?

Prevention tests encode specific scenarios a developer imagined. Discovery testing generates scenarios no developer imagined. Prevention is a wall. Discovery is a search party sent into unmapped territory.

Most teams have a testing portfolio that is 100% prevention and 0% discovery. Every test was written in response to a specific requirement or a specific bug. No test was written to find bugs the team does not yet know about. This is not a criticism of those teams. Until recently, discovery testing at scale was only available to a handful of organizations willing to invest years of infrastructure work. But the tools are catching up.

Three principles that enable discovery

What does it take to shift from prevention to discovery? Three enabling principles, each building on the last.

Deterministic simulation. Discovery testing must generate enormous numbers of scenarios and be able to reproduce any that fail. If a bug depends on a specific ordering of network events, we need to replay that exact ordering. Determinism makes every execution reproducible from a single seed value. A failing seed is a bug report that anyone on the team can replay, inspect, and fix. Without determinism, discovery testing produces irreproducible ghosts.

Controlled fault injection. Discovery testing must exercise failure paths that production encounters but development environments do not. Network partitions, disk corruption, message reordering, clock skew, process crashes during recovery. These are not exotic scenarios. They are the normal operating conditions of any distributed system at scale. Controlled injection means we decide when and how faults occur, driven by a seeded random number generator so they are reproducible and tunable.

Sometimes assertions. This is the subtle one. Prevention tests use assertions that say “this must always be true.” Discovery testing needs a second kind: assertions that say “this should sometimes be true.” A sometimes assertion on a timeout retry path does not check that retries work every time. It checks that our simulation actually exercised the retry path at all. Without sometimes assertions, you can run a million simulations and never know whether they explored the interesting parts of the state space or just repeated the happy path a million times.

The feedback loop

Put the three principles together and something powerful emerges. Simulation generates scenarios. Fault injection pushes those scenarios into failure territory. Always assertions catch violations. Sometimes assertions tell us whether we explored deeply enough.

When a sometimes assertion never fires, it means our simulation is not reaching some region of the state space. That is a signal to adjust: inject different faults, change the workload, add more concurrent operations. When all sometimes assertions are firing and all always assertions hold, we have evidence (not proof, but strong evidence) that our system handles the failure modes we care about.

This is the discovery feedback loop. It runs continuously. It finds bugs that no developer anticipated. And critically, it does not require writing new tests for each new scenario. The infrastructure does the exploration. The developer’s job shifts from writing tests to defining correctness properties and interpreting results.

Lawrie Green from Antithesis captured this duality perfectly: assertions are memos to two audiences. They tell the computer what to check during exploration. And they tell the developer what they believe about their system. When an assertion fails on correct code, the developer’s mental model was wrong. That is itself a critical finding.

The spectrum, not the binary

Prevention and discovery are not competing approaches. Every team needs prevention testing. But adding even a small amount of discovery testing changes the game fundamentally.

You do not need to go from zero to full deterministic simulation overnight. Injecting random delays into your existing integration tests is discovery testing. Property-based tests with randomized inputs are discovery testing. Adding sometimes assertions to your simulation suite is discovery testing.

The question is not whether you should do discovery testing. The question is how far along the spectrum you want to go. Every step reveals bugs that prevention testing, by construction, will never find.

From Mocks to Simulation

• Why mocks fail at scale
• The #[cfg(test)] trap
• The alternative: trait-based simulation
• The fidelity spectrum
• Error injection over expectations
• Moonpool’s provider pattern

Every experienced developer has a mock story. You spend a day writing mocks for a database client, carefully specifying which methods return what, in which order. The tests pass. Then someone refactors the internal call sequence without changing any external behavior, and every mock breaks. The mocks were not testing your system’s correctness. They were testing its implementation details.

This is not a failure of any particular mocking library. It is a structural problem with how mocks work.

Why mocks fail at scale

Mocks operate by replacing a dependency with a fake that returns pre-programmed responses. To program those responses, you need to know exactly which methods your code will call, in which order, with which arguments. This means the test author must carry a mental model of the entire internal call stack between the code under test and the mocked dependency.

For a unit test of a single function, this is manageable. For an integration test of a distributed protocol with concurrent operations, retries, timeouts, and failure handling, it becomes a maintenance nightmare. Every internal refactor risks breaking mocks that were testing behavior, not implementation. Every new failure path requires manually programming new mock responses. The mock setup code grows until it rivals the complexity of the system it is supposed to test.

And mocks jump abstraction layers. Your production code talks to a TCP socket. Your mock replaces the database client. Between those two layers live connection pooling, serialization, retry logic, timeout handling, and error translation. None of that code runs during mock-based tests. You are testing a different system than the one you deploy.

The #[cfg(test)] trap

Rust developers often reach for conditional compilation: #[cfg(test)] to swap in test-specific implementations. This is tempting because it requires no runtime cost and no trait indirection. But it means the binary you test is literally different from the binary you deploy. Different code paths, different struct fields, different behavior.

If a bug lives in the interaction between your retry logic and your connection pool, and your test build replaces the connection pool with an in-memory stub via #[cfg(test)], that bug is invisible to your test suite. You have not tested the system. You have tested a system-shaped thing that happens to share some code.

The alternative: trait-based simulation

There is a different approach. Instead of replacing entire subsystems with hand-programmed fakes, define a trait that describes the interface your code needs. Implement it once for production (real TCP, real disk, real clock). Implement it once for simulation (simulated network, simulated disk, simulated clock). Your application code depends on the trait, not the implementation. The trait implementation runs real logic, not pre-programmed responses, so every execution path that production exercises, simulation exercises too.

#![allow(unused)]
fn main() {
pub trait TimeProvider: Clone + Send + Sync + 'static {
    async fn sleep(&self, duration: Duration) -> Result<(), TimeError>;
    fn now(&self) -> Duration;
}
}

In production, sleep calls tokio::time::sleep. In simulation, sleep registers a timer with the simulated event loop and advances simulated time. The application code is identical in both cases. No #[cfg(test)]. No conditional compilation. The exact same binary logic runs in production and in simulation.

This is the provider pattern. It is the same architectural decision FoundationDB made with their INetwork interface: one trait, two implementations, zero conditional logic in the application.

The fidelity spectrum

Not every simulated implementation needs full fidelity. There is a spectrum.

No-op: the simplest fake. sleep returns immediately, send discards the message. Useful for testing pure logic that happens to call I/O functions.

In-memory: messages go into a queue, disk writes go into a BTreeMap. Fast and deterministic, but does not model timing, failures, or ordering.

Controlled simulation: messages are delayed by randomized amounts, connections drop according to a fault schedule, disk writes can be torn or corrupted. This is where bugs hide, because the system must handle not just the happy path but all the ways the real world deviates from it. Critically, these trait-based fakes scale across the entire codebase. You write each implementation once, and every component that uses the trait gets simulation for free. No per-test mock setup. No maintenance burden that grows with the test suite.

Full simulation: an entire cluster of processes with simulated network topology, coordinated fault injection, and time advancement. FoundationDB runs hundreds of simulated processes in a single thread, compressing hours of cluster behavior into seconds.

The right level depends on what you are testing. A serialization function needs no-op I/O. A retry loop needs controlled failure injection. A consensus protocol needs full cluster simulation.

Error injection over expectations

The deepest difference between mocks and simulation is the direction of control.

Mocks specify outputs: “when this method is called with these arguments, return this value.” The test author must predict every call. If the code takes a different path, the mock panics.

Simulation injects conditions: “connections drop with 5% probability, messages are delayed 1-100ms, disk writes fail 1 in 1000.” The simulation does not care which methods are called or in which order. It cares whether the system recovers correctly regardless of which failures occur.

This is a fundamental shift. Mock-based tests verify that your code follows a specific execution path. Simulation-based tests verify that your code produces correct results across all execution paths the simulation explores. One tests implementation. The other tests behavior.

Moonpool’s provider pattern

This is exactly what moonpool implements. Every interaction with the outside world goes through a provider trait: TimeProvider for clocks and timers, TaskProvider for spawning concurrent work, NetworkProvider for connections and messages, StorageProvider for disk I/O, RandomProvider for randomness.

Your application code calls time.sleep() instead of tokio::time::sleep(). It calls task_provider.spawn_task() instead of tokio::spawn(). In production, these call through to the real runtime. In simulation, they feed into a deterministic event loop where every timer, every message, every disk operation is controlled, reproducible, and subject to fault injection.

No mocks to maintain. No #[cfg(test)] to diverge your test binary from your production binary. The same code, running in a simulated world that is deliberately worse than reality.

A Brief History

• FoundationDB’s radical bet (2009-2015)
• Antithesis generalizes it (2020-present)
• The adoption wave (2020-present)
• The unifying philosophy

Simulation-driven development did not appear from nowhere. It evolved over fifteen years across three distinct eras, each expanding who could use it and how.

FoundationDB’s radical bet (2009-2015)

In 2009, a small team set out to build a distributed transactional database. Before writing a single line of database code, they spent roughly two years building a simulator. No storage engine, no query layer, no client protocol. Just the simulation.

The idea was radical and, to most observers, insane. But the team had a thesis: distributed systems fail in ways that depend on the ordering of concurrent events, and the only way to test those orderings systematically is to control them completely.

They built Flow, a custom extension to C++ that provided actor-model concurrency compiled down to single-threaded callbacks. Every network connection, every disk operation, every timer went through an abstract interface (INetwork) with two implementations: Net2 for production (real TCP via Boost.ASIO) and Sim2 for simulation (in-memory buffers with deterministic delays). A single seeded random number generator controlled everything. Same seed, same execution. Every time.

Two techniques made the simulation radically effective. BUGGIFY injected faults at strategic points throughout the codebase: sending packets out of order, truncating disk writes, skipping timeouts, shrinking buffer sizes. Active at 25% probability during simulation, BUGGIFY broadened the effective contract being tested far beyond what any specification documented. It tested not what the system was supposed to handle, but what it could handle.

Hurst exponent manipulation produced correlated, cascading hardware failures. Real datacenters exhibit failure correlation: a hard drive failing in a rack makes nearby drives more likely to fail. Naive testing models failures as independent events, making cascading failures astronomically rare. FoundationDB’s simulator cranked the correlation up, producing failure patterns that would take years to encounter in production but that simulation could generate in milliseconds.

The scale was staggering. Tens of thousands of simulations every night, each simulating minutes to hours of cluster behavior under extreme fault injection, amounting to roughly one trillion CPU-hours of simulated testing. A physical validation cluster called Sinkhole, real server motherboards wired to programmable power switches and toggled continuously, served as the ultimate reality check. Sinkhole never found a single database bug that simulation had missed. It only found bugs in other software and hardware.

The cost was equally staggering. The approach required a custom language, simulation-first architecture, and two years of pure infrastructure investment before building the actual product. Only an elite team with unusual patience and conviction could pull it off. For over a decade, FoundationDB’s simulation remained a benchmark that others admired but could not replicate.

Antithesis generalizes it (2020-present)

Will Wilson, one of FoundationDB’s original engineers, co-founded Antithesis with a different question: what if you could apply the same techniques to any software, without rewriting it?

The answer was a deterministic hypervisor. Instead of requiring developers to build simulation into their system from day one, Antithesis intercepts nondeterminism at the virtual machine level. Any program, in any language, running on any framework, becomes deterministically reproducible. The hypervisor turns arbitrary software into a pure function from an input byte stream to a sequence of states. Save, restore, fork, replay. No Flow. No custom language. No architectural prerequisites.

On top of the hypervisor, Antithesis built a guided exploration engine. Their SDK provides declarative assertions that tell the platform what to explore without specifying how: Sometimes (ensure this condition fires at least once), Always (ensure this invariant holds), SometimesAll (drive exploration across a frontier of sub-goals). The platform uses these signals, combined with coverage-guided forking and adaptive search, to systematically explore the state space.

The results validated the approach. Customers with mature, well-tested systems found new bugs within 2 to 3 weeks of onboarding. Not shallow bugs. Deep concurrency issues, subtle data corruption, failure recovery defects that had survived years of conventional testing. The exploration engine required minimal domain knowledge to be effective. In one demonstration, Antithesis beat the entire game of Gradius by tracking just 3 bytes of game memory and maximizing time alive. No game-specific strategy. Just depth and coverage.

The adoption wave (2020-present)

The third era is happening now. Simulation-driven development is spreading beyond the teams that invented it.

TigerBeetle, building a financial transactions database, adopted deterministic simulation from day one, adding storage fault patterns (misdirected reads, phantom writes, uninitialized memory) that go beyond network-level testing. Dropbox used simulation to validate their sync engine rewrite. WarpStream applied deterministic simulation testing across their entire SaaS platform.

Clever Cloud uses simulation-driven development to build Materia, a distributed multi-model multi-tenant database. 30 minutes of simulation covers roughly 24 hours of equivalent chaos testing.

What took an elite team two years of custom infrastructure in 2009 is becoming accessible to small teams building on existing frameworks and languages. The investment required is shrinking. The bugs found per engineering-hour are increasing.

The unifying philosophy

Across all three eras, the core philosophy is the same. Make all execution deterministic so bugs are reproducible. Inject faults that are worse than production so surviving systems are overqualified for the real world. Explore systematically so bugs are found by infrastructure, not by imagination.

The tools change. The languages change. The accessibility changes. The philosophy does not.

Why Moonpool Exists

• What I found
• What moonpool synthesizes
• Where moonpool sits
• What makes it different

After ten years of operating distributed systems, one pattern keeps repeating: the bugs that hurt most are the ones nobody imagined.

You build a system. You test the scenarios you can think of. You deploy. Then a network partition overlaps with a retry storm during a rolling upgrade, and the system enters a state no developer on the team anticipated. The code was not wrong. It just was not tested against that particular combination of events.

The combinatorial problem makes this inevitable. A simple e-commerce API with six variable dimensions (user types, payment methods, delivery options, promotions, inventory status, currencies) requires 648 unique test combinations for basic coverage. Adding one option to each dimension pushes it past 4,000. A distributed system with network partitions, node crashes, disk faults, and clock drift has orders of magnitude more dimensions. You cannot test what you do not know. Manual tests ensure regression coverage, not absence of bugs.

That realization is what led to moonpool: the testing toolbox I wish I had when I started building distributed systems.

What I found

The answer already existed, scattered across several projects and decades of work.

FoundationDB had shown that simulating a distributed system’s network inside a single-threaded process, with deterministic fault injection and seed-driven reproducibility, could find more bugs in a night than production found in a year. Their BUGGIFY technique and Hurst exponent manipulation made the simulated world worse than reality. Their Sinkhole cluster never found a database bug that simulation missed.

TigerBeetle extended the simulation philosophy to storage. Where FoundationDB focused on network faults (partitions, latency, connection drops), TigerBeetle modeled disk-level faults drawn from real hardware failure modes: torn writes, misdirected reads, phantom writes, read corruption, sync failures, uninitialized memory. A financial transactions database cannot afford to trust that disks behave according to spec.

Antithesis generalized FoundationDB’s assertion system into a declarative SDK. Instead of hand-coding what to explore, you declare properties: Sometimes (this condition should fire), Always (this invariant must hold), SometimesAll (drive exploration along a frontier). The platform figures out how to reach those states. Their work on NES games demonstrated that coverage-guided forking and adaptive exploration could beat complex state spaces with minimal domain knowledge.

What moonpool synthesizes

Moonpool brings these ideas together in a single Rust framework.

From FoundationDB: the simulation engine. A single-OS-thread runtime with deterministic execution, so every scheduling decision is reproducible from a seed. Trait types are Send + Sync, which means your code can use familiar Arc<RwLock<...>>, DashMap, Arc<AtomicBool>, and Send-bounded task patterns while spawning through the provider. A seeded PRNG controls all network timing, fault injection, and process scheduling. Simulated time jumps forward when all tasks are blocked, compressing hours of cluster behavior into seconds. The same code runs against real networking or the simulated network, swapped at the provider level.

From TigerBeetle: storage fault injection. Simulated disk operations that can corrupt reads, tear writes, fail syncs, and misdirect I/O. Not just network chaos but disk chaos, because real systems fail at both layers.

From Antithesis: the assertion suite and fork-based exploration. Always, sometimes, reachable, unreachable, numeric, and frontier assertions that live in shared memory. Coverage-guided forking that branches the simulation at interesting points, exploring multiple futures from a single state. Adaptive energy budgets that allocate exploration effort where coverage is still improving.

Where moonpool sits

Moonpool is a library-level simulation framework. There is no hypervisor, no custom virtual machine, no special runtime. You add a dependency to your Rust project, implement a few provider traits, and your system becomes simulatable.

This is a deliberate tradeoff. Antithesis’s hypervisor can simulate any software without modification. Moonpool requires that your code use provider traits for I/O, which means designing for simulation from the start or refactoring existing code. In exchange, you get zero-cost abstractions in production, full control over fault injection, and the ability to run simulations as ordinary cargo test invocations. No infrastructure to deploy. No service to call. Everything runs in your CI pipeline.

The provider pattern means your production binary and your simulation binary share the same application logic. No #[cfg(test)]. No conditional compilation. The same code, tested in a world that is deliberately worse than production.

What makes it different

Many simulation frameworks stop at network-level fault injection. Moonpool adds two things.

First, storage simulation at the same fidelity as network simulation. Disk faults drawn from the TigerBeetle and FoundationDB fault models, deterministically controlled by the same seed that controls the network.

Second, fork-based multiverse exploration. When the simulation reaches an interesting state (a new assertion fires, a coverage bit flips, a numeric watermark improves), moonpool forks the process. The parent continues its original trajectory. The child explores from the interesting state with a different seed. This turns a single simulation run into a tree of timelines, each branching at points of maximum discovery potential.

This is not a gimmick. The sequential luck problem (finding a bug that requires getting lucky multiple times in sequence, where the probability is p^n) is the central bottleneck in simulation-based testing. Fork-based exploration attacks it directly: instead of hoping a single timeline stumbles through all the right doors, we branch at each door and explore both sides.

Part VI covers multiverse exploration in depth. For now, the key point is that moonpool is not just a simulation engine. It is a simulation engine with a built-in search strategy for the state spaces that matter most: the ones where bugs require sequences of unlikely events to surface.

Determinism as a Foundation

• The Two Enemies
• Moonpool’s Answer
• What This Gives You

A distributed system dropped writes for 47 seconds. Logs show three nodes disagreed about leadership, but the window where it happened has already passed. The cluster self-healed. You cannot reproduce it locally. You cannot reproduce it in staging. You spend three days staring at logs, form a theory, ship a fix you are 60% sure about, and wait. Until it happens again.

This is what debugging looks like without determinism. The bug is real, but the conditions that triggered it are gone. You cannot replay the execution. You cannot step through it. You cannot even confirm your fix addresses the right cause.

The Two Enemies

Rust programs that use async networking have three sources of non-determinism: thread scheduling, real I/O, and random number generation.

Thread scheduling means the OS decides when threads run. Two threads racing on a shared counter might increment it correctly 999,999 times, then corrupt it once. The interleaving that triggers the bug depends on CPU load, thermal throttling, how many browser tabs you have open. A multi-threaded tokio runtime executes tasks across a thread pool. The order tasks resume after .await is up to the scheduler. Run the same program twice, get two different execution orders.

Real I/O means the outside world injects randomness. A TCP packet arrives 2ms late. A DNS lookup takes 800ms instead of 5ms. A disk write returns ENOSPC because another process filled the partition. Network calls complete in unpredictable order. Timeouts race against responses. Your code is a deterministic function, but its inputs are chaos.

Random number generation introduces a subtler form of non-determinism. Calling rand::rng() pulls entropy from the OS. Two runs with identical inputs can make different random choices, leading to different retry delays, different leader elections, different shard assignments.

Together, these forces make distributed systems bugs the hardest kind to find. The bug only appears under a specific thread interleaving, during a specific network delay pattern, with a specific disk latency. Reproducing it means reproducing all three. Which is effectively impossible.

Moonpool’s Answer

Moonpool brings all three sources under deterministic control.

Single-core execution removes thread scheduling from the equation. One thread. One execution order. No races, no interleavings, no “works on my machine but fails in CI.” We cover this in detail in the next chapter.

Provider abstraction replaces real I/O and real randomness with simulated equivalents. Every system call your code makes (network, disk, time, random numbers) goes through a trait. In production, the trait calls tokio. In simulation, the trait calls moonpool’s deterministic runtime. Same code, different wiring. No #[cfg(test)] branching. The production code path is the tested code path.

With all three sources controlled, the simulation becomes a pure function of its seed. A single u64 value determines everything: which connections fail, when timeouts fire, what order messages arrive, whether disk writes corrupt. Same seed, same execution, same bugs. Every time.

The event side of that guarantee is explicit. A global Scheduler<Event> owns logical time, cancellation, and a monotonically increasing sequence number. Scheduling into the past clamps to the current time. Events with the same timestamp execute in sequence order, so they are FIFO without relying on heap iteration details. Cancelled work is removed without advancing time.

The scheduler does not own the simulated resources. NetworkSimulation owns connections, topology, network faults, operations, and wakers. StorageEngine owns persistent files, open handles, disk episodes, operations, results, and wakers. Each component consumes its targeted event and returns ordered effects for the coordinator to schedule. This boundary matters for replay: random decisions happen in the component that owns the state, global ordering happens in one scheduler, and wakers run only after the world lock is released.

What This Gives You

A failing seed turns a production incident into a local debugging session. Instead of “it happened once between nodes 7 and 12 under load,” you get:

FAILED seed=8839214571 — leadership invariant violated at sim_time=12.4s

You paste that seed into your test, run it, hit a breakpoint. The bug is right there. You fix it. You run 10,000 more seeds. They pass. You ship with confidence.

This is not aspirational. FoundationDB ran 5 to 10 million simulation iterations per night, each with a different seed. Their physical validation cluster (real servers with programmable power switches, toggled continuously) never found a single database bug that simulation missed. Determinism made that possible.

Everything in moonpool builds on this foundation. Chaos testing, assertion coverage, multiverse exploration, all of it requires one guarantee: given the same seed, the simulation produces the same result. The rest of this part explains how we achieve that guarantee.

The Single-Core Constraint

• One Thread, One Order
• How We Get There
• Send-Bounded Traits
• What This Buys, What It Costs
• The Parallelism Trade-Off

Moonpool runs every simulation on a single OS thread. This is not a limitation we reluctantly accept. It is the first design decision we made, and every other decision follows from it.

One Thread, One Order

A multi-threaded tokio runtime uses a work-stealing thread pool. When a task yields at an .await, any thread in the pool might pick it up. Two tasks that resume “at the same time” can execute in either order, and that order changes between runs. This is fine for production throughput. It is fatal for deterministic simulation.

With a single thread, there is exactly one legal execution order for any given set of ready tasks. Task A resumes before Task B, or Task B before Task A, and the choice is controlled by our scheduler, not the OS. The RNG picks the order. The seed determines the RNG. Done.

How We Get There

Moonpool runs every simulation on its own deterministic executor:

let mut executor = moonpool_sim::executor::Executor::new(seed);
executor.block_on(async move {
    // the entire simulation runs here
});

One OS thread drives the entire simulation. No work-stealing pool. No preemption between yield points. When a future resumes, it resumes on the same thread that suspended it. And which ready task runs next is a seeded-random pick from the ready queue: same seed, same order, bit for bit, while a different seed explores a different task interleaving. A FIFO scheduler would run ready tasks in registration order on every seed forever, structurally hiding any race between them. The Deterministic Executor covers the machinery in depth.

This is the whole mechanism. The executor gives us determinism. Nothing fancier is needed.

Send-Bounded Traits

Here is where moonpool diverges from earlier designs. A single-threaded runtime could drop Send bounds entirely and let users sprinkle Rc<RefCell<T>> across .await points. We deliberately do not.

Provider traits use native AFIT with Send + Sync + 'static supertraits and futures that return impl Future<…> + Send:

#![allow(unused)]
fn main() {
pub trait TimeProvider: Clone + Send + Sync + 'static {
    fn sleep(&self, duration: Duration) -> impl Future<Output = Result<(), TimeError>> + Send;
    fn now(&self) -> Duration;
    // ...
}
}

Dyn-stored traits (Process, Workload, FaultInjector) use #[async_trait] without ?Send, also with Send + Sync + 'static supertraits.

Send is paid for interoperability, not for parallelism.

The runtime still runs on one thread. Futures and types are never actually moved between threads at runtime. But the type system commits to Send + Sync at every trait boundary so customer code can use the ecosystem it already knows:

• Arc<RwLock<State>> for shared state
• DashMap for concurrent maps
• Arc<AtomicBool> for flags
• Send-bounded spawning to fan out work: task.spawn_task(...) in the sim, plain tokio::spawn in production

A team adopting moonpool to test an existing service should not have to rewrite their type hierarchy. If their production code uses Arc<RwLock<…>> (it does), moonpool wraps it without contortions. The simulation runtime drives those types on one thread, but the types themselves look exactly like the production ones.

What This Buys, What It Costs

We get determinism from the runtime and ergonomics from the bounds. The combination is rare. Most deterministic simulators force you to pick: either a single-threaded runtime with !Send types that don’t compose with anything, or a multi-threaded runtime that loses determinism.

The cost is a small one. Send + Sync constrains a few exotic patterns (raw Rc, non-Send futures from external crates). In a year of building on this, we have not hit a case where it mattered. The patterns that do matter — locks, channels, spawned tasks — are all Send-friendly already.

The Parallelism Trade-Off

Single-core simulation cannot exploit CPU parallelism within a simulation run. A simulation of 20 nodes runs on one core, not 20.

In practice, this barely matters. Simulation time compression means a single core simulates hundreds of seconds of cluster time in a few seconds of wall-clock time. And we run many simulations in parallel across cores, each with a different seed. A 16-core machine runs 16 independent simulations concurrently, each fully deterministic on its own thread.

Production code is unaffected. The same code that runs single-threaded under our simulation runtime can run on a multi-threaded tokio runtime in production. The provider pattern (covered next) makes this swap transparent.

Seed-Driven Reproducibility

• The RNG Core
• Call Count Tracking
• The Determinism Canary
• Multi-Seed Testing
• Debugging a Failing Seed

One number. A single u64. That is all moonpool needs to fully determine a simulation: which connections fail, when packets arrive, whether BUGGIFY triggers, how long disk writes take, what order events process. Same seed, same execution, same bugs. This is the property that makes everything else work.

The RNG Core

Moonpool uses ChaCha8Rng from the rand_chacha crate, seeded from the u64 value. ChaCha8 is fast, produces high-quality randomness, and is deterministic across platforms. The RNG lives in thread-local storage, which is correct because each simulation runs on a single thread.

#![allow(unused)]
fn main() {
// At the start of every simulation run
set_sim_seed(seed);

// Inside the simulation engine, every decision uses the thread-local RNG:
let latency: u64 = sim_random_range(1..50);  // network delay in ms
let should_fail: bool = sim_random_f64() < 0.25;  // 25% fault probability
let value: f64 = sim_random();  // general-purpose random value
}

These are framework-internal functions used by the simulation engine itself. Your application code should use providers.random().random() and providers.random().random_range() instead, which route through the same underlying RNG but go through the provider abstraction.

There is exactly one stream. sim_random(), sim_random_range(), sim_random_f64(), the RandomProvider your code sees, the executor’s choice of which task to poll next, select! branch offsets, the per-seed swarm subsets, buggify activation and firing, and every network, storage and block-device fault coin all draw from the same thread-local RNG. Moonpool keeps no private stream for its own decisions: the framework draws its randomness the way the code it simulates does. This means the order of calls matters. Adding a new sim_random() call anywhere in the simulation shifts every subsequent random value. This is intentional. It means small code changes produce different simulation trajectories, naturally exploring new parts of the state space.

Call Count Tracking

Every RNG call increments a thread-local counter. You can read it with get_rng_call_count(). This sounds mundane, but it is one of the most useful debugging tools in the framework.

When a seed produces a bug, you can narrow down exactly where the execution diverges from expected behavior by watching the call count. “The bug triggers after RNG call 847” tells you precisely which decision in the simulation started the chain of events leading to failure. Combined with breakpoints and logging, this turns a mysterious distributed failure into a step-through debugging session.

The explorer framework takes this further: it records call counts at fork points, creating a “recipe” of count@seed transitions that can replay an exact exploration path.

The Determinism Canary

Everything above assumes nothing escapes the seed. Something always tries to: a HashMap iterated in its randomized order, a static that survives from one run into the next, a wall-clock timestamp, an OS random number pulled in by a dependency. Each one makes the simulation draw something different, or draw at a different moment, and the seed stops reproducing anything. moonpool ships a canary for exactly this, the mechanism madsim’s Runtime::check_determinism uses:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .check_determinism()
    .workload_factory(|| Box::new(MyWorkload::new()))
    .run();
}

Every seed now runs twice. During the first run, each draw on the simulation stream records a 64-bit fingerprint: a probe of the generator’s state after the draw, taken on a clone so the check consumes no randomness, mixed with the logical clock. During the second run the same hook compares each draw’s fingerprint against the record, and when the run ends the whole record must have been consumed. Because task scheduling, select! offsets, swarm masks and every fault coin are draws on that one stream, any uncontrolled difference in scheduling, ordering, draw count or simulated timing shows up as a fingerprint that does not match, and the seed fails with the always-assertion determinism canary: replay matched the recorded draw sequence, naming the first diverging draw. A replay that matches a prefix and then stops early fails too: the leftover record is the evidence.

Two things to know. It is a canary, not a trace: it tells you that the runs diverged and at which draw, and the call-count tools above take it from there. And it doubles the cost of every seed, so it belongs in a dedicated CI job or a debugging session, not in the default sweep. Like exploration it needs factory workloads; an instance workload would carry its state into the second run and diverge by construction.

Multi-Seed Testing

A single seed tests one execution path. To build confidence, you need many paths. Moonpool’s builder supports two modes:

FixedCount runs a set number of iterations, each with a different random seed:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .set_iterations(100)  // 100 different seeds
    // ... workloads ...
    .run();
}

UntilCoverageStable (the default) keeps drawing seeds until every observed assert_sometimes! / assert_reachable! has fired and code coverage has stopped growing, capped at a safety limit:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .until_coverage_stable(10, 5_000)  // 10 quiet seeds, 5_000 safety cap
    // ... workloads ...
    .run();
}

The power of seed-driven testing compounds over time. Run 1,000 seeds in CI on every commit. Run 100,000 overnight. Each seed explores a different combination of timing, faults, and ordering. Bugs that require three independent unlikely events to coincide will surface within a few thousand seeds because the simulation amplifies failure probability through BUGGIFY.

Debugging a Failing Seed

When CI reports a failure, the output includes the seed:

FAILED seed=17429853261 — connection timeout during leader election

Pin that seed and run it locally with logging enabled:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .set_iterations(1)
    .set_debug_seeds(vec![17429853261])
    // ... same workloads ...
    .run();
}

The simulation replays the exact same execution. Set a breakpoint. Step through. The bug is deterministic now.

The Deterministic Executor

• Why not tokio?
• Randomized AND deterministic
• Where is the reactor?
• The drive loop
• Coming from tokio
• Kill-on-drop
• Exploration process safety
• Verifying it yourself

Moonpool simulations do not run on tokio. They run on a purpose-built, single-threaded executor whose every scheduling decision derives from the iteration seed. This chapter explains why it exists, how it works, and what changes if you are used to tokio.

Why not tokio?

Two reasons, one practical and one fundamental.

The practical one: seeding tokio’s runtime randomness requires the unstable RngSeed API, which forces --cfg tokio_unstable onto every crate in every consumer’s workspace. For stable-pinned projects that is a hard adoption barrier (issue #151).

The fundamental one: tokio’s current-thread scheduler is a FIFO queue, and that is the wrong policy for a bug-finding simulator. When one event makes two tasks runnable, FIFO runs them in registration order, on every seed, forever. Any race between those tasks is structurally unexplorable: no amount of seeds will ever try the other order. A simulation executor should treat “which runnable task goes next” as a decision to explore, the same way it explores message latencies and fault timings.

Randomized AND deterministic

Those two properties sound contradictory, but they are not. Deterministic means “same seed, same execution, bit for bit”, not “one fixed schedule”. The executor keeps its ready tasks in a Vec and picks the next one with a swap_remove at a seeded-random index. madsim uses the same distribution, and FoundationDB’s sim2 randomizes the same way:

seed 42:  task-3, task-0, task-2, task-1, ...
seed 42:  task-3, task-0, task-2, task-1, ...   (always)
seed 43:  task-1, task-3, task-0, task-2, ...

Each seed replays exactly. The population of seeds covers the ordering space. Ordering bugs, the kind that survive code review because both orders look fine locally, live exactly there.

The scheduling index is a draw on the simulation’s one random stream, the same sim_random_range a process reaches through ctx.random(). There is no private scheduling RNG: which task runs next is counted like every other decision, so a fork-explorer count@seed recipe replays the schedule along with the faults and the workload. The only time the executor does not draw is when exactly one task is runnable, which is a forced choice, not a decision.

Where is the reactor?

There is none, and that is the deep reason this executor is small. A production runtime pairs its executor with a reactor (epoll, timers) that turns OS events into waker calls. In the simulation, SimWorld and its global Scheduler<Event> play that role. The scheduler owns logical time, stable same-time ordering, and cancellation. It dispatches targeted events to NetworkSimulation and StorageEngine, which own their resource state, operation results, and waker registries. The returned wake batches are invoked after the world lock is released. The executor only answers one question: of the tasks that are runnable right now, which runs next?

The drive loop

Executor::block_on(main) pins the orchestrator future on the stack as the driver. It never enters the ready queue and is never scheduled randomly:

loop {
    poll driver ── Ready? ─────────────► return
    run_until_stalled()   // poll ready tasks in seeded-random order
                          // until none is runnable
    (driver not woken AND scheduler empty) ► panic: deadlock (with the seed)
}

The orchestrator’s step loops interleave the simulation with the task pool:

sim.step();                              // process one virtual event
crate::executor::until_stalled().await;  // let every woken task run

Because the driver is only re-polled after a complete drain, resuming from until_stalled().await guarantees that every task that was runnable has been polled to Pending or completion. Under tokio this guarantee was approximated by yield_now plus FIFO ordering. Under randomized scheduling “the back of the queue” does not exist, so the executor provides the contract directly.

Inside a task, use executor::yield_now() instead: it reschedules the task at a seeded-random position. until_stalled() is driver-only and asserts it, in every build profile, because a task awaiting “run until nothing is runnable” would be waiting for itself.

One consequence deserves a warning sign: a task must not busy-wait on progress only the driver can make. A loop { yield_now().await } polling for a flag that a simulation event sets keeps the ready queue non-empty forever, so the drain never finishes and sim.step() never runs. The executor kills such a loop with a poll-bound panic naming the task and the seed. Park on a real wake instead: a sleep, a Notify, a channel.

Coming from tokio

The task API mirrors what the orchestrator (and your process/workload code, through TaskProvider) already expected from tokio:

Spawned futures stay Send + 'static, exactly as before: execution is one OS thread, but the bounds let application code use Arc<RwLock<...>>, DashMap, and friends without contortion.

The raw executor keeps panic isolation deliberately narrow. A detached task spawned with executor::spawn records its panic in its own JoinHandle, so a low-level driver can decide how to recover. Tasks spawned through an actor’s ctx.task().spawn_task() carry a different contract: the simulation runner records an unobserved child panic with its process or workload identity and marks that seed as failed. If the caller awaits the handle and handles Err(JoinError::Panicked), the panic is observed and the seed can recover. Dropping or detaching the handle leaves the panic unobserved, even if the task already finished. Dropping a task during process shutdown or executor teardown remains cancellation, not a panic.

The runner also records each unobserved panic as an unobserved_task_panic event with actor, task, and panic fields. It runs invariants once more after task teardown, so those diagnostics remain visible even when setup or check stalls and orchestration returns early.

time.sleep(Duration::ZERO) still schedules a real same-time timer, preserving the scheduler’s FIFO ordering for a burst of immediate work. The runner allows that fan-out, then requests shutdown after a generous fixed count of events that keep rearming at the same logical instant. If the tasks ignore shutdown, the next stagnant burst fails the seed. Time advancing or a task completing resets the count, so ordinary cleanup and long timer-driven runs keep working.

Two behaviors are deliberately better than tokio’s:

• A genuine deadlock (driver not woken, nothing runnable) panics with the seed in the message instead of parking the thread forever.
• A task that stays runnable forever, whether a busy-yield loop or a select over a source it synchronously re-readies (something tokio’s cooperative budget used to interrupt), trips a poll-bound panic naming the task and the seed, in every build profile, instead of hanging the drain loop.

Kill-on-drop

Simulation iterations must be hermetic: a task leaked by seed N must never run during seed N+1. Dropping the per-iteration tokio runtime used to guarantee that, and the executor replicates it with a waker registry (the same pattern async-executor uses). Every live task registers one waker. Drop wakes them all, which schedules every parked task, then pops and drops Runnables until the queue is empty. Dropping a Runnable cancels the task and drops its future, and any tasks that drop wakes land in the same queue and are consumed by the same loop.

Exploration process safety

The explorer only forks between complete timelines, after the deterministic executor and all of its tasks have been dropped. Assertion macros record discovery coordinates; they are not fork points. Forked workers are still an explicit performance mode because unrelated host threads and resources may exist outside the simulation. The default workers: 0 mode runs continuations sequentially in-process and is the conservative choice for deterministic CI.

Verifying it yourself

The executor’s contracts are enforced by crates/moonpool-sim/tests/executor.rs (join/abort/detach/panic/kill-on-drop, same-seed replay, cross-seed diversity) and crates/moonpool-sim/tests/determinism.rs, an end-to-end tripwire: two full SimulationBuilder runs of racing workloads on the same seed must produce byte-identical execution traces. If any component, executor, select!, providers, ever consults an unseeded randomness source, that test fails.

The Provider Pattern

• The Core Idea
• The Providers Bundle
• One Line Changes Everything

Every distributed system does five things: it talks over the network, it reads the clock, it spawns concurrent tasks, it generates random values, and it reads and writes files. That is the entire surface area where non-determinism leaks in. Moonpool’s provider pattern seals all five.

The Core Idea

Define a trait for each category of I/O. Implement the trait twice: once backed by real tokio calls for production, once backed by a deterministic simulation engine for testing. Your application code is generic over the trait. It never knows which implementation it is running against.

                  Application Code
           (generic over Providers trait)
                        |
            +-----------+-----------+
            |                       |
      TokioProviders          SimProviders
     (real TCP, real         (simulated TCP,
      clock, real disk)       logical clock,
                              fault-injected disk)

This is interface swapping, the same technique FoundationDB used with their INetwork interface (production Net2 vs. simulation Sim2). The difference is that Rust’s type system enforces it at compile time. If your code compiles with P: Providers, it works with both implementations. No runtime surprises.

The Providers Bundle

Carrying five separate type parameters through every function signature would be painful:

#![allow(unused)]
fn main() {
// Nobody wants to write this
fn run_server<N, T, K, R, S>(net: N, time: T, task: K, rand: R, storage: S)
where
    N: NetworkProvider, T: TimeProvider, K: TaskProvider,
    R: RandomProvider, S: StorageProvider,
{ /* ... */ }
}

Moonpool solves this with a single bundle trait called Providers:

#![allow(unused)]
fn main() {
pub trait Providers: Clone + Send + Sync + 'static {
    type Network: NetworkProvider;
    type Time: TimeProvider;
    type Task: TaskProvider;
    type Random: RandomProvider;
    type Storage: StorageProvider;

    fn network(&self) -> &Self::Network;
    fn time(&self) -> &Self::Time;
    fn task(&self) -> &Self::Task;
    fn random(&self) -> &Self::Random;
    fn storage(&self) -> &Self::Storage;
}
}

Now your code carries one type parameter:

#![allow(unused)]
fn main() {
fn run_server<P: Providers>(providers: P) {
    let time = providers.time().clone();
    let net = providers.network().clone();
    // Use them naturally
}
}

Two implementations exist: TokioProviders for production, and SimProviders (in moonpool-sim) for simulation. SimProviders::new(sim, seed, ip) takes an IP address so its storage provider is scoped to the correct process. Your application code never imports either one directly. It only sees P: Providers.

One Line Changes Everything

The swap between “testing a real distributed system” and “testing inside a deterministic simulation” happens at the call site, not inside your application logic:

#![allow(unused)]
fn main() {
// Production
let providers = TokioProviders::new();
run_server(providers);

// Simulation (inside a workload, the builder provides SimProviders via SimContext)
let providers = ctx.providers().clone();  // SimProviders
run_server(providers);
}

Same run_server. Same code path. Same binary. The only difference is which Providers implementation gets plugged in. This is the architectural foundation that makes everything else in moonpool possible: chaos testing, assertion coverage, multiverse exploration, all of it rests on the guarantee that your production code runs unmodified inside the simulator.

On the simulation side, each provider is intentionally thin. Time requests go to the global Scheduler<Event>. Network calls go to NetworkSimulation, and storage calls go to StorageEngine. The engines own their state, exact pending results, faults, and wakers, then return scheduling effects to SimWorld. This keeps the application-facing trait familiar without collapsing the simulator back into one global state object.

Quick Start: Swapping Implementations

• A Function Generic Over Providers
• The Forbidden List
• Running in Production
• Running in Simulation

Here is what using providers looks like in practice. We will write a function that uses time and network providers, then show it running in both production and simulation contexts.

A Function Generic Over Providers

#![allow(unused)]
fn main() {
use moonpool_core::{Providers, TimeProvider, NetworkProvider};
use std::time::Duration;

/// Connect to a peer and retry with exponential backoff.
async fn connect_with_retry<P: Providers>(
    providers: &P,
    addr: &str,
    max_retries: u32,
) -> std::io::Result<<P::Network as NetworkProvider>::TcpStream> {
    let mut delay = Duration::from_millis(100);

    for attempt in 0..max_retries {
        match providers.network().connect(addr).await {
            Ok(stream) => return Ok(stream),
            Err(e) if attempt + 1 < max_retries => {
                // Backoff before retrying — uses provider, not tokio directly
                providers.time().sleep(delay).await.ok();
                delay *= 2;
            }
            Err(e) => return Err(e),
        }
    }
    unreachable!()
}
}

Notice what is not in this code: no tokio::time::sleep(). No tokio::net::TcpStream::connect(). The function uses providers.time().sleep() and providers.network().connect(). That is the entire discipline.

The Forbidden List

These direct tokio calls break determinism. Never use them in application code:

Any direct tokio call in your application code is a hole in the simulation. The call will use real I/O, real time, and the simulation has no control over it. The result is non-determinism: different behavior between runs with the same seed.

Running in Production

#![allow(unused)]
fn main() {
use moonpool_core::TokioProviders;

let providers = TokioProviders::new();

// Real TCP connection, real exponential backoff with wall-clock delays
let stream = connect_with_retry(&providers, "10.0.1.1:9000", 5).await?;
}

TokioProviders bundles TokioTimeProvider, TokioNetworkProvider, TokioTaskProvider, TokioRandomProvider, and TokioStorageProvider. Each one delegates to the real tokio equivalent.

Running in Simulation

Inside a simulation workload, the builder gives you SimProviders:

#![allow(unused)]
fn main() {
// The simulation provides SimProviders to your workload
// SimProviders bundles simulated time, network, tasks, random, and storage

let stream = connect_with_retry(&providers, "10.0.1.1:9000", 5).await?;
}

Same function call. But now sleep() advances simulation time instead of wall-clock time. connect() goes through the simulated network where connections can be delayed, dropped, or partitioned. The retry loop exercises the exact same code path, but under controlled, deterministic conditions.

That is the entire provider workflow: write your code generic over P: Providers, use provider methods instead of raw tokio, and the framework handles the rest.

Deep Dive: Why Providers Exist

• Why Not #[cfg(test)]
• Why Not Mocks
• The Pattern
• What You Get

Providers are not the first idea anyone reaches for when testing distributed systems. Most teams start with #[cfg(test)] or mock frameworks. Both approaches have fundamental problems that providers solve.

Why Not #[cfg(test)]

Conditional compilation is tempting. Wrap the real network call in production code, swap in a fake during tests:

#![allow(unused)]
fn main() {
#[cfg(not(test))]
async fn connect(addr: &str) -> io::Result<TcpStream> {
    TcpStream::connect(addr).await
}

#[cfg(test)]
async fn connect(addr: &str) -> io::Result<TcpStream> {
    FakeStream::new()  // returns immediately, no real network
}
}

The problem: you are no longer testing your production code. The test binary compiles a different function. Every #[cfg(test)] block is a fork in your codebase where production and test behavior can silently diverge. As Oxide Computer’s engineering team documented across their five major repositories, #[cfg(test)] blocks “prevent testing real code paths.” The tested code is not the shipped code.

Providers eliminate this entirely. There is one connect() implementation in your application. It calls providers.network().connect(). In production, that dispatches to tokio. In simulation, that dispatches to the simulator. The application code is identical in both cases. Same binary, same compiler output, same code path.

Why Not Mocks

Mock frameworks like mockall record expectations: “this method should be called with these arguments and return this value.” They test that your code calls the right methods in the right order. They do not test what happens when the network does something unexpected.

A mock for TCP might say: “when connect("10.0.1.1:9000") is called, return Ok(stream).” This tells you nothing about what happens when the connection takes 3 seconds, or when it succeeds but the first read returns ConnectionReset, or when the connection succeeds on the third retry but the remote peer has rebooted and lost state.

Providers are full implementations, not recorded expectations. The simulation network provider maintains connection state, buffers packets, injects delays, simulates TCP half-close, drops messages under partition. It is a complete in-memory TCP implementation. When your code connects through the simulation provider, it gets a real (simulated) TCP session with real (simulated) failure modes.

That fidelity includes asynchronous establishment. Simulated bind, connect, and accept do not manufacture an immediately ready socket. They park the calling future, schedule a targeted network operation, and wake it when the configured delay expires. Storage follows the same contract for read, write, sync, and set-length operations. A provider is therefore an adapter into a stateful engine, not a shortcut around the scheduler.

This is the fidelity spectrum that Oxide’s codebases demonstrate: from no-op fakes (methods return Ok(())), through in-memory implementations (real data operations without real I/O), all the way to full protocol simulation. Providers operate at the highest fidelity level because simulation needs to exercise the same edge cases that production encounters.

The Pattern

The recipe is simple. Oxide’s engineering teams converged on it independently across five repositories with near-zero mock framework usage:

1. Define a trait for the external dependency
2. Implement it for production (real tokio calls)
3. Implement it for simulation (deterministic, controllable)
4. Inject via generics (compile-time dispatch, zero runtime overhead)

#![allow(unused)]
fn main() {
// Step 1: The trait (native AFIT, no #[async_trait])
pub trait TimeProvider: Clone + Send + Sync + 'static {
    async fn sleep(&self, duration: Duration) -> Result<(), TimeError>;
    fn now(&self) -> Duration;
}

// Step 2: Production implementation
impl TimeProvider for TokioTimeProvider {
    async fn sleep(&self, duration: Duration) -> Result<(), TimeError> {
        tokio::time::sleep(duration).await;  // real wall-clock delay
        Ok(())
    }
}

// Step 3: Simulation implementation (in moonpool-sim)
// sleep() schedules a timer event, the simulator advances logical time

// Step 4: Inject via generics
async fn heartbeat_loop<T: TimeProvider>(time: T) {
    loop {
        send_heartbeat().await;
        time.sleep(Duration::from_secs(5)).await.ok();
    }
}
}

The compiler guarantees correctness: if heartbeat_loop compiles with T: TimeProvider, it works with both TokioTimeProvider and the simulation time provider. No runtime dispatch. No dynamic casts. No “forgot to wire up the mock” surprises.

What You Get

The provider pattern gives you three things that mocks and #[cfg(test)] cannot:

Same code path. Your production code is your test code. No divergence, no conditional compilation, no “it works in test but fails in production” surprises.

Full behavior, not just call verification. Providers simulate the actual behavior of the subsystem, not just whether your code calls the right methods. A simulated network can partition, delay, reorder, and corrupt. A mock just records calls.

Resource ownership stays local. The network and storage engines own their state, pending results, faults, and wakers. SimWorld coordinates their effects through the global scheduler. This makes cancellation and shutdown explicit and prevents a missing result from being mistaken for a successful operation.

Compile-time safety. If you forget to use a provider and call tokio directly, the simulation still compiles and runs, but the behavior will not be deterministic. Moonpool’s code conventions (enforced in review) catch these: no direct tokio::time::sleep(), no tokio::spawn(), no tokio::net::*. Always go through the provider.

The Five Providers

• TimeProvider
• NetworkProvider
• TaskProvider
• RandomProvider
• StorageProvider
• The Providers Bundle

Moonpool abstracts every interaction between your code and the outside world into five provider traits. Each trait covers one category of I/O. Together, they form a complete boundary around your application, giving the simulator full control over every source of non-determinism.

The sim runs single-threaded on the moonpool deterministic executor, but every provider trait is Send + Sync + 'static. One OS thread runs everything for determinism, yet the types are Send-bounded so customer code stays normal: Arc<RwLock<…>>, DashMap, Arc<AtomicBool>, and Send-bounded task spawning all just work. The async methods use native AFIT (async fn in trait) with explicit -> impl Future<…> + Send desugarings to propagate the Send bound, so no #[async_trait] and no ?Send anywhere in the provider layer.

TimeProvider

Time is the most pervasive dependency in distributed systems. Every timeout, backoff, heartbeat, and lease check goes through TimeProvider.

#![allow(unused)]
fn main() {
pub trait TimeProvider: Clone + Send + Sync + 'static {
    /// Sleep for the specified duration.
    fn sleep(
        &self,
        duration: Duration,
    ) -> impl Future<Output = Result<(), TimeError>> + Send;

    /// Get exact current time.
    fn now(&self) -> Duration;

    /// Get drifted timer time (simulates clock drift between nodes).
    /// Defaults to `now()`; the simulation overrides it.
    fn timer(&self) -> Duration {
        self.now()
    }

    /// Run a future with a timeout.
    fn timeout<F, T>(
        &self,
        duration: Duration,
        future: F,
    ) -> impl Future<Output = Result<T, TimeError>> + Send
    where
        F: Future<Output = T> + Send,
        T: Send;
}
}

The distinction between now() and timer() is borrowed from FoundationDB’s sim2. timer() is a default method that returns now(), which is what production keeps; only the simulation overrides it. In simulation, timer() can drift up to 100ms ahead of now(), testing how your code handles clock skew between processes. Use now() for event scheduling. Use timer() for application-level time checks like lease expiry and heartbeat deadlines.

Production: TokioTimeProvider delegates sleep to tokio::time::sleep, timeout to tokio::time::timeout, and now to std::time::Instant::elapsed.

Simulation: Sleep schedules a Timer in the global Scheduler<Event>. The scheduler owns monotonic logical time and same-time FIFO sequence order. When all tasks are blocked, the simulator performs “time travel” by popping the next scheduled event. This compresses hours of simulated cluster time into seconds of wall-clock time.

NetworkProvider

#![allow(unused)]
fn main() {
pub trait NetworkProvider: Clone + Send + Sync + 'static {
    type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;
    type TcpListener: TcpListenerTrait<TcpStream = Self::TcpStream> + 'static;

    /// Create a TCP listener bound to the given address.
    fn bind(
        &self,
        addr: &str,
    ) -> impl Future<Output = io::Result<Self::TcpListener>> + Send;

    /// Connect to a remote address.
    fn connect(
        &self,
        addr: &str,
    ) -> impl Future<Output = io::Result<Self::TcpStream>> + Send;
}

pub trait TcpListenerTrait: Send + Sync + 'static {
    type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;

    /// Accept a single incoming connection.
    fn accept(
        &self,
    ) -> impl Future<Output = io::Result<(Self::TcpStream, String)>> + Send;

    /// Get the local address this listener is bound to.
    fn local_addr(&self) -> io::Result<String>;
}
}

The associated types TcpStream and TcpListener let each implementation provide its own concrete types. Production gives you tokio::net::TcpStream. Simulation gives you an in-memory stream backed by buffers with controllable latency, reordering, and connection failures.

The API deliberately matches what you would expect from tokio networking. bind, connect, accept behave like their tokio counterparts. The streams implement AsyncRead + AsyncWrite + Send, so they work with any tokio-compatible codec or framing layer and they cross task boundaries cleanly.

Production: TokioNetworkProvider wraps tokio::net.

Simulation: NetworkSimulation owns listeners, connections, topology, faults, pending operation results, and network wakers. Bind, connect, and accept park until their scheduled latency expires. Established streams use in-memory buffers with deterministic delivery delays, TCP half-close simulation, and fault injection such as connection drops, partitions, and corruption. For numeric socket addresses, binding port zero assigns a distinct dynamic port; listener.local_addr() returns that resolved address for clients to connect to. Opaque logical addresses, such as the process IP alone, remain valid in the simulated provider. Dropping a stream with unread received bytes resets the peer, as TCP does when the application discards acknowledged data. A drained stream closes gracefully; AsyncWrite::close shuts down only the write half so the stream can still read a reply. Each simulated listener has a bounded accept backlog (128 unaccepted connections by default). A full backlog parks new connects in FIFO order until an accept returns a stream; a delayed accept’s reservation still occupies a slot. SimulationBuilder::accept_backlog_capacity sets the capacity for each run.

TaskProvider

#![allow(unused)]
fn main() {
pub trait TaskProvider: Clone + Send + Sync + 'static {
    /// Join handle returned by `spawn_task`. `Detach` provides explicit
    /// fire-and-forget: `spawn_task(...).detach()` leaves the task running
    /// without keeping the handle.
    type JoinHandle: Future<Output = Result<(), JoinError>> + Detach + Send + 'static;

    /// Spawn a named task.
    fn spawn_task<F>(&self, name: &str, future: F) -> Self::JoinHandle
    where
        F: Future<Output = ()> + Send + 'static;

    /// Yield control to allow other tasks to run.
    fn yield_now(&self) -> impl Future<Output = ()> + Send;
}
}

Spawned futures are Send + 'static. The runtime still pins everything to one OS thread for determinism, but the bound matches what tokio::spawn expects, so customer code reads exactly like normal tokio code. The name parameter is diagnostic only. In simulation the executor stores it with the task and traces every poll under it; in production TokioTaskProvider merely emits a tracing::trace! event when the task starts and when it completes.

Production: TokioTaskProvider uses plain tokio::spawn. The name feeds those two trace events and nothing else — it is not attached to the tokio task.

Simulation: SimTaskProvider spawns onto the deterministic executor, so scheduling order is a seeded-random, fully reproducible function of the iteration seed.

RandomProvider

#![allow(unused)]
fn main() {
pub trait RandomProvider: Clone + Send + Sync + 'static {
    /// Generate a random value of type T.
    fn random<T>(&self) -> T
    where
        StandardUniform: Distribution<T>;

    /// Generate a random value within a specified range (start..end).
    fn random_range<T>(&self, range: Range<T>) -> T
    where
        T: SampleUniform + PartialOrd;

    /// Generate a random f64 between 0.0 and 1.0.
    fn random_ratio(&self) -> f64 {
        self.random()
    }

    /// Generate a random bool with the given probability of being true.
    fn random_bool(&self, probability: f64) -> bool {
        self.random_ratio() < probability
    }
}
}

Only random and random_range are required; random_ratio and random_bool are default methods derived from them.

RandomProvider is fully synchronous. The other four providers expose async methods via native AFIT, but random number generation never needs to suspend, so its trait has no async fn at all. The supertrait shape (Clone + Send + Sync + 'static) stays consistent with the rest of the provider family.

Production: TokioRandomProvider uses rand::rng() (thread-local, non-deterministic).

Simulation: Uses the seeded ChaCha8Rng from the simulation’s RNG system. Every call draws from the same deterministic stream, maintaining reproducibility.

StorageProvider

#![allow(unused)]
fn main() {
pub trait StorageProvider: Clone + Send + Sync + 'static {
    type File: StorageFile + 'static;

    fn open(
        &self,
        path: &str,
        options: OpenOptions,
    ) -> impl Future<Output = io::Result<Self::File>> + Send;

    fn exists(&self, path: &str) -> impl Future<Output = io::Result<bool>> + Send;
    fn delete(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
    fn rename(
        &self,
        from: &str,
        to: &str,
    ) -> impl Future<Output = io::Result<()>> + Send;

    fn create_dir_all(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;

    fn sync_dir(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
}

pub trait StorageFile: AsyncRead + AsyncWrite + AsyncSeek + Unpin + Send + Sync + 'static {
    fn constraints(&self) -> IoConstraints;
    fn is_direct_io(&self) -> bool;

    fn read_at(
        &self,
        offset: u64,
        buf: &mut [u8],
    ) -> impl Future<Output = io::Result<usize>> + Send;
    fn write_at(
        &self,
        offset: u64,
        buf: &[u8],
    ) -> impl Future<Output = io::Result<usize>> + Send;

    fn sync_all(&self) -> impl Future<Output = io::Result<()>> + Send;
    fn sync_data(&self) -> impl Future<Output = io::Result<()>> + Send;
    fn size(&self) -> impl Future<Output = io::Result<u64>> + Send;
    fn set_len(&self, size: u64) -> impl Future<Output = io::Result<()>> + Send;
}
}

Storage is the newest provider, and the one with the richest fault model. The split is that the provider owns the filesystem namespace — open, exists, delete, rename, and the directory sync that makes those durable — while the file owns already-open bytes. The two never cross: a delete, or a rename over a name, removes the name and nothing else, so a file opened before it keeps reading and writing the same bytes, exactly as on Unix, and the bytes are freed only when the last name and the last handle are both gone. Log rotation and atomic replacement lean on that in production, and the simulator holds to it too. And the namespace is per process: a provider is created for one process IP, so two processes opening data.db open two files on two disks, one node’s delete or rename never reaches another’s, and a sync_dir commits only the syncing node’s directory entries, as it would on two machines.

OpenOptions mirrors std::fs::OpenOptions with read, write, create, truncate, and append, plus the one thing a database needs that std::fs::OpenOptions does not name portably: direct_io(DirectIo). That is deliberately an option on opening an ordinary file rather than a second provider stack — there is no BlockProvider and no DirectIoProvider, and a journal or pager is written directly against StorageFile. The flags obey std’s rules too, on both backends: truncate, create and create_new need write or append, append and truncate contradict each other, and a combination std refuses with InvalidInput (OpenOptions::validate) is refused by the simulator before it touches the file, so a read-only truncate destroys nothing in simulation that it would not destroy in production.

read_at/write_at are positioned: they take &self, never touch the stream cursor, and return the number of bytes moved, so non-overlapping ranges can be read and written concurrently and partial transfers stay visible to the caller. On a file with I/O constraints they are the only way to transfer: the stream half of the trait is refused there, because a shared cursor cannot be kept aligned. BlockFile<F> wraps one open file to add block arithmetic and the loops that turn those partial transfers into whole ones; it holds no path and opens nothing.

Production: TokioStorageProvider wraps tokio::fs.

Simulation: StorageEngine owns an in-memory filesystem with fault injection inspired by TigerBeetle and FoundationDB patterns: read and write corruption, EIO, crash and torn writes, misdirected I/O, sync failures, short transfers, unsynced directory-entry loss, and IOPS/bandwidth timing. There is exactly one simulated file implementation behind every API — a positioned write is visible to a stream read, and a block write is visible to both. Persistent file contents are separate from open-handle state, so two handles have independent cursors, access options, and pending operations. Each delayed operation has an exact ID, explicit pending or completed result, and its own waker. Crash and shutdown complete pending work with errors rather than treating absence as success.

Each SimStorageProvider is scoped to a process IP (SimStorageProvider::new(sim, ip)), so the engine resolves the correct per-process configuration and disk-degradation episode for every operation.

The Providers Bundle

All five come together in the Providers trait:

#![allow(unused)]
fn main() {
pub trait Providers: Clone + Send + Sync + 'static {
    type Network: NetworkProvider;
    type Time: TimeProvider;
    type Task: TaskProvider;
    type Random: RandomProvider;
    type Storage: StorageProvider;

    fn network(&self) -> &Self::Network;
    fn time(&self) -> &Self::Time;
    fn task(&self) -> &Self::Task;
    fn random(&self) -> &Self::Random;
    fn storage(&self) -> &Self::Storage;
}
}

TokioProviders bundles all five production implementations. SimProviders bundles all five simulation implementations and requires an IP address at construction (SimProviders::new(sim, seed, ip)) so that the storage provider is scoped to the correct process. Your application code sees P: Providers and nothing else.

System Under Test vs Test Driver

• Two Perspectives on the Same System
• Process: The Thing You Are Building
• Workload: The Thing That Tests
• Different Lifecycles
• Why Not One Trait?
• What Comes Next

Every simulation in moonpool has two distinct roles. Understanding this separation is the single most important concept before you write your first line of simulation code.

Two Perspectives on the Same System

Think about how you test a web server in production. You have the server itself, handling requests, managing connections, storing data. And you have something else: a client, a load generator, a test harness that sends requests and checks responses.

These two things have fundamentally different jobs. The server is the system. The test driver exercises the system.

Moonpool formalizes this split into two traits: Process and Workload.

Process: The Thing You Are Building

A Process is your server, your node, your distributed system participant. It is the system under test. The code inside a Process is the code you ship to production.

Processes have a difficult life in simulation:

• They crash. The simulation kills them without warning.
• They reboot. After a crash, the simulation creates a fresh instance from scratch.
• They lose state. All in-memory fields vanish on reboot. Only data written to storage survives.
• They get partitioned. Network connections break, packets get delayed, peers become unreachable.

This is by design. The whole point of simulation testing is to throw chaos at your server code and see what breaks.

In FoundationDB’s simulation, the equivalent is the fdbd process. Each simulated FDB node runs the same code as production, but inside a controlled environment where the simulator decides when clocks advance, when packets arrive, and when machines die.

Workload: The Thing That Tests

A Workload is your test driver. It lives outside the system under test. It sends requests, observes responses, and checks that the system behaved correctly.

Workloads have a much easier life:

• They never crash. The simulation does not reboot workloads.
• They survive everything. Processes come and go, but workloads keep running.
• They see the whole picture. Workloads know about all processes, all IPs, the full topology.
• They judge correctness. After the simulation ends, workloads run their check() method to validate final state.

In FoundationDB’s simulation, the equivalent is tester.actor.cpp. The tester knows the cluster topology, drives operations against it, and validates results.

Different Lifecycles

This distinction matters because their lifecycles are completely different.

A Process lifecycle looks like this: created from factory, runs, gets killed, factory creates a fresh one, runs again, gets killed again. Each incarnation starts with empty in-memory state. The factory produces a blank slate every time.

A Workload lifecycle is linear: setup() runs once at the start, run() executes the test logic, check() validates at the end. One continuous life from start to finish.

Here is the key insight: a Process does not know when it will die, and a Workload does not care when Processes die. The Process just tries to do its job correctly. The Workload just keeps sending requests and tracking what happened.

Why Not One Trait?

You might wonder why we need this separation at all. Why not just have “participants” in the simulation?

Because mixing the two concerns creates a mess. If your server code also has to track test state, you cannot reboot it cleanly. If your test driver also runs server logic, it cannot survive process crashes.

The separation also matches real production architecture. You deploy servers. You run integration tests against them. Different code, different lifecycles, different concerns.

What Comes Next

The next two chapters cover each trait in detail. We will look at the Process trait with its factory pattern and reboot semantics, then the Workload trait with its three-phase lifecycle. After that, we will build a complete simulation from scratch.

Process: Your Server

• The Process Trait
• The Factory Pattern
• State and Reboots
• IP Addressing
• Tags for Role Assignment
• Graceful vs Crash Reboots
• A Concrete Example

A Process represents the system under test. It is the code you would ship to production, running inside the simulation where the framework controls time, network, and failure.

The Process Trait

The trait is minimal by design:

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Process: Send + Sync + 'static {
    fn name(&self) -> &str;
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()>;
}
}

Two methods. name() identifies this process type for reporting. run() is where your server logic lives. All Process impls are Send + Sync + 'static, which makes them composable with tokio::spawn and Arc-based shared state like Arc<RwLock<…>> or DashMap. The runtime itself is still single-thread for determinism, but your code reads like ordinary tokio code.

When run() returns Ok(()) or an Err, the process has exited voluntarily. When the simulation kills the process, the future is cancelled and run() never returns at all, and so is every task the process spawned through ctx.task().spawn_task(), however deep: a crash ends the whole process, not just its root future, so a background worker never survives to run beside the rebooted process’s own. When run() panics, the seed fails: the panic is caught and recorded against the process’s IP, and the iteration is reported as failed even if every workload returned Ok, so an assertion firing inside a server (or one of its dependencies) never disappears from the report.

The Factory Pattern

You never construct a Process once. You give the builder a factory that can produce fresh instances:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(MyServer::new()))
}

Why a factory? Because of reboots. When the simulation kills a process and restarts it, the framework calls your factory to get a brand new instance. This guarantees that each boot starts with a clean slate, just like restarting a real server.

The factory is called once per process per boot. Three processes with two reboots each means the factory runs nine times total.

State and Reboots

This is the rule you must internalize: all in-memory state is lost on reboot.

If your Process has a BTreeMap<String, Vec<u8>> field tracking client sessions, that map is gone after a reboot. The new instance from the factory starts empty. Only data written to the simulated storage layer survives.

This matches reality. When a server process crashes and restarts, it does not magically recover its heap. It reads persistent state from disk and rebuilds from there.

IP Addressing

Each process instance gets its own IP address in the 10.0.1.0/24 range:

Process 0 → 10.0.1.1
Process 1 → 10.0.1.2
Process 2 → 10.0.1.3

Workloads get IPs in the 10.0.0.0/24 range. This clean separation makes it easy to identify what is a server and what is a test driver when reading logs.

Your process accesses its IP through ctx.my_ip(). Other process IPs are available through ctx.topology().all_process_ips().

Tags for Role Assignment

Many distributed systems need nodes with different roles: leader and follower, primary and secondary, different data centers. Tags handle this:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(5, || Box::new(MyNode::new()))
    .tags(&[
        ("role", &["leader", "follower"]),
        ("dc", &["east", "west", "eu"]),
    ])
}

Tags distribute round-robin. With 5 processes and 2 role values, the assignment looks like:

Inside your process, read tags from the context:

#![allow(unused)]
fn main() {
async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    let role = ctx.topology().my_tags().get("role");
    match role.as_deref() {
        Some("leader") => run_leader(ctx).await,
        Some("follower") => run_follower(ctx).await,
        _ => Ok(()),
    }
}
}

Graceful vs Crash Reboots

Not all deaths are equal. Moonpool supports three reboot kinds:

Graceful: The simulation signals the shutdown token. Your process has a grace period to finish in-flight work, flush buffers, and close connections cleanly. If it does not exit in time, it gets force-killed anyway.

Crash: Instant death. The process task is cancelled immediately. All connections abort. No cleanup, no buffer drain. Peers see connection reset errors.

CrashAndWipe: Same as Crash but also wipes all persistent storage owned by that process. Simulates total disk failure or a fresh node joining the cluster. The wipe is immediate and scoped to the process’s IP address, so other processes’ storage is unaffected.

To handle graceful shutdown, check the cancellation token in your main loop:

#![allow(unused)]
fn main() {
async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    let listener = ctx.network().bind(ctx.my_ip()).await?;

    loop {
        if ctx.shutdown().is_cancelled() {
            break;
        }
        // Accept connections, handle requests...
    }
    Ok(())
}
}

You do not need to handle crash reboots. There is nothing to handle. The simulation cancels your future and moves on.

A Concrete Example

Here is a simple echo server as a Process:

#![allow(unused)]
fn main() {
struct EchoServer;

#[async_trait]
impl Process for EchoServer {
    fn name(&self) -> &str {
        "echo"
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        loop {
            if ctx.shutdown().is_cancelled() {
                break;
            }

            match ctx.time().timeout(
                Duration::from_millis(100),
                listener.accept()
            ).await {
                Ok(Ok((mut stream, _addr))) => {
                    let mut buf = vec![0u8; 4096];
                    while let Ok(n) = stream.read(&mut buf).await {
                        if n == 0 { break; }
                        let _ = stream.write_all(&buf[..n]).await;
                    }
                }
                _ => continue,
            }
        }
        Ok(())
    }
}
}

Notice the patterns: use ctx.network() for connections, ctx.time() for timeouts, ctx.shutdown() for graceful termination. Never call tokio directly.

Workload: Your Test Driver

• The Workload Trait
• The Three Phases
  • Setup: Prepare the Ground
  • Run: Drive the System
  • Check: Validate the Outcome
• Workloads Survive Everything
• The SimContext
• The Operation Alphabet
• Multiple Workload Instances
• What a Good Workload Looks Like

A Workload exercises the system under test and validates its correctness. While Processes are the code you ship, Workloads are the code that finds your bugs.

The Workload Trait

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Workload: Send + Sync + 'static {
    fn name(&self) -> &str;

    async fn setup(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
        Ok(())
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()>;

    async fn check(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
        Ok(())
    }
}
}

Four methods, two with defaults. The lifecycle follows a strict order: setup(), then run(), then check(). Each phase has different rules and a different purpose.

The Three Phases

Setup: Prepare the Ground

setup() runs sequentially across all workloads. Workload A’s setup completes before workload B’s setup starts. No concurrency, no surprises.

Use setup to establish connections, initialize state, prepare data structures. By the time run() starts, every workload should be ready to go.

#![allow(unused)]
fn main() {
async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    let server_ip = ctx.topology().all_process_ips()
        .first()
        .ok_or(SimulationError::InvalidState("no servers".into()))?;

    self.connection = Some(ctx.network().connect(server_ip).await?);
    Ok(())
}
}

Run: Drive the System

run() is where the action happens. All workloads run concurrently. Multiple clients hammering the same servers, competing for resources, triggering race conditions.

This is where you send requests, observe responses, track expected state, and use assertions to flag anomalies. The run phase continues until all workloads return or the simulation shuts them down.

#![allow(unused)]
fn main() {
async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
    for _ in 0..self.num_operations {
        if ctx.shutdown().is_cancelled() {
            break;
        }

        let op = random_op(ctx.random(), &self.accounts);
        match op {
            Op::Write { key, value } => {
                self.send_write(&key, &value).await?;
                self.model.write(&key, &value);
            }
            Op::Read { key } => {
                let result = self.send_read(&key).await?;
                let expected = self.model.read(&key);
                assert_always!(
                    result == expected,
                    format!("read mismatch for key '{}'", key)
                );
            }
        }
    }
    Ok(())
}
}

Check: Validate the Outcome

check() runs sequentially after all workloads finish and all pending events drain. The system is quiescent. No more messages in flight, no more timeouts pending.

Use check for final state validation. Did the conservation law hold? Are all balances non-negative? Did every committed write survive? The verdict counts: a check() that returns Err (or panics) fails the seed exactly as a failing run() does, even when every run() returned Ok.

#![allow(unused)]
fn main() {
async fn check(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
    let total = self.model.total_balance();
    let expected = self.model.total_deposited - self.model.total_withdrawn;
    assert_always!(
        total == expected,
        format!("conservation law violated: {} != {}", total, expected)
    );
    Ok(())
}
}

Workloads Survive Everything

This is the fundamental difference from Processes. When the simulation kills a server, your workload keeps running. When a connection breaks, your workload can reconnect. When the network partitions, your workload observes the failures and adapts.

Your workload tracks what happened across the entire simulation lifetime, including across process reboots. This is how you verify that a server recovers correctly after a crash: the workload sent a write before the crash, the server rebooted, and the workload reads the value back to check that it survived.

The SimContext

Every lifecycle method receives a SimContext that gives workloads access to everything they need:

• ctx.my_ip() returns this workload’s IP address
• ctx.topology() has process IPs, peer info, and tag registries
• ctx.network() provides simulated TCP connections
• ctx.time() provides simulated clocks and timeouts
• ctx.random() provides deterministic random numbers
• ctx.state() provides cross-workload shared state for invariants
• ctx.shutdown() provides the cancellation token

Use ctx.topology().all_process_ips() to find your servers. Use ctx.peer("server") if you know the name. Use ctx.topology().ips_tagged("role", "leader") if you need role-specific targeting.

The Operation Alphabet

Strong workloads define an “operation alphabet”: the set of actions they can perform. Deposits, withdrawals, reads, writes, delays. Each operation has a weight controlling how often it fires.

#![allow(unused)]
fn main() {
pub fn random_op(random: &SimRandomProvider, accounts: &[String]) -> Op {
    let roll = random.random_range(0..100);
    match roll {
        0..30  => Op::Deposit { ... },
        30..50 => Op::Withdraw { ... },
        50..70 => Op::Read { ... },
        70..90 => Op::Transfer { ... },
        _      => Op::SmallDelay,
    }
}
}

The weights matter. Too many reads and you never test write conflicts. Too many writes and you never test read-after-crash consistency. The alphabet should cover normal operations, adversarial inputs, and small delays that let background work complete.

Multiple Workload Instances

Sometimes one client is not enough. Use workloads() to create multiple instances:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .workloads(WorkloadCount::Fixed(3), |i| {
        Box::new(ClientWorkload::new(i))
    })
}

Each instance gets its own client_id (accessible via ctx.client_id()) and its own IP. Multiple clients hitting the same server concurrently is where the interesting bugs hide.

For variable topology testing, use WorkloadCount::Random(1..6) to spawn a different number of clients each iteration. The count is determined by the simulation RNG, so it stays deterministic per seed.

What a Good Workload Looks Like

The best workloads follow a pattern:

1. Define an operation alphabet with weighted random selection
2. Track a reference model (expected state computed locally)
3. Assert on every response using assert_always! for invariants
4. Use assert_sometimes! for coverage of interesting paths
5. Validate final state in check() using the reference model
6. Publish state via ctx.state() for cross-workload invariant checking

The next part walks through building a complete simulation from scratch, demonstrating all of these patterns.

Your First Simulation

• What We Are Building
• Prerequisites
• The Plan
• Why This Order

We have covered the theory. Deterministic execution, providers, the split between Process and Workload. Now we build something real.

What We Are Building

Over the next four chapters, we will create a complete simulation test for a key-value server. By the end, you will have:

• A Process that accepts TCP connections and responds to get/set requests
• A Workload that drives random operations and tracks expected state
• Assertions that verify correctness during and after the simulation
• A SimulationBuilder configuration that runs hundreds of iterations with different seeds

The key-value server is deliberately simple. The interesting part is not the server logic but how the simulation wraps around it, finding bugs you would never catch with unit tests.

Prerequisites

You need Nix for the development environment. All cargo commands run inside nix develop:

nix develop --command cargo build

The simulation binary will live alongside moonpool’s existing simulation binaries, managed by xtask:

cargo xtask sim list    # see what exists
cargo xtask sim run kv  # run our simulation (once we build it)

The Plan

Chapter: Defining a Process covers implementing the Process trait. We will set up a TCP listener, parse incoming requests, and handle graceful shutdown. The process is a simple in-memory key-value store that loses all state on reboot.

Chapter: Writing a Workload covers implementing the Workload trait. We will define an operation alphabet (get, set, delete), build a reference model that tracks expected state, and use assertions to catch bugs.

Chapter: Configuring the SimulationBuilder covers the builder’s fluent API. We will configure the number of processes, set iteration control, add invariants, and enable chaos phases with attrition.

Chapter: Running and Observing covers execution and output. We will run the simulation, read the report, understand what success and failure look like, and debug a failing seed.

Why This Order

We build bottom-up. The Process comes first because it is the system under test, the thing everything else depends on. The Workload comes second because it exercises the Process. The builder ties them together. Running is last because you need all the pieces in place before you can execute.

Each chapter produces code that builds on the previous one. By the end of the fourth chapter, you will have a working simulation you can extend with your own chaos experiments.

Defining a Process

• The Process Struct
• Implementing the Trait
• Handling Requests
• Registering with the Builder
• Process Groups
• What the Shutdown Token Gives You
• Key Takeaways

Our key-value server is a Process. It listens for connections, handles get/set requests, and respects shutdown signals. Everything it does goes through providers, never raw tokio calls.

The Process Struct

Start with the struct. A Process is recreated from a factory on every boot, so the struct starts empty:

#![allow(unused)]
fn main() {
use async_trait::async_trait;
use moonpool_sim::{Process, SimContext, SimulationResult};

struct KvServer;
}

No fields. Each time the simulation boots this process, the factory returns a fresh KvServer. Any data it accumulates lives only until the next crash.

Implementing the Trait

The trait has two methods: name() for identification and run() for the main logic.

#![allow(unused)]
fn main() {
#[async_trait]
impl Process for KvServer {
    fn name(&self) -> &str {
        "kv"
    }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        let mut store: BTreeMap<String, Vec<u8>> = BTreeMap::new();

        loop {
            if ctx.shutdown().is_cancelled() {
                break;
            }

            let accept_result = ctx
                .time()
                .timeout(Duration::from_millis(100), listener.accept())
                .await;

            match accept_result {
                Ok(Ok((stream, _addr))) => {
                    handle_connection(stream, &mut store).await;
                }
                Ok(Err(e)) => {
                    tracing::warn!("accept error: {}", e);
                }
                Err(_) => {
                    // Timeout, loop back and check shutdown
                    continue;
                }
            }
        }
        Ok(())
    }
}
}

Walk through this line by line.

Binding the listener: ctx.network().bind(ctx.my_ip()) creates a TCP listener on the process’s assigned IP. In the simulated world, this registers the IP for incoming connections. No real ports are opened.

The store: A plain BTreeMap that lives on the stack. When this process crashes, the map vanishes. When the factory creates a new instance, it starts empty. This is the “all in-memory state is lost on reboot” principle in action; the ordered map also keeps iteration deterministic.

The main loop: We loop forever, checking the shutdown token each iteration. ctx.shutdown().is_cancelled() returns true during graceful reboots, giving us a chance to break cleanly. For crash reboots, the framework cancels the entire future, so we never reach this check.

Timeout on accept: We use ctx.time().timeout() instead of tokio::time::timeout(). The simulated timer means the framework controls when the timeout fires, keeping everything deterministic.

Handling Requests

The connection handler parses a simple wire protocol. Production systems can keep their existing framing or hand the stream to hyper. A raw protocol shows the provider boundary directly:

#![allow(unused)]
fn main() {
async fn handle_connection(
    mut stream: SimTcpStream,
    store: &mut BTreeMap<String, Vec<u8>>,
) {
    let mut buf = vec![0u8; 4096];
    loop {
        let n = match stream.read(&mut buf).await {
            Ok(0) => break,       // Connection closed
            Ok(n) => n,
            Err(_) => break,      // Connection error
        };

        // Parse and handle the request
        let request = &buf[..n];
        let response = match request[0] {
            b'G' => {  // GET
                let key = String::from_utf8_lossy(&request[1..]);
                store.get(key.as_ref())
                    .cloned()
                    .unwrap_or_default()
            }
            b'S' => {  // SET
                // Format: S<key_len:u8><key><value>
                let key_len = request[1] as usize;
                let key = String::from_utf8_lossy(
                    &request[2..2 + key_len]
                ).to_string();
                let value = request[2 + key_len..].to_vec();
                store.insert(key, value.clone());
                value
            }
            _ => vec![],
        };

        let _ = stream.write_all(&response).await;
    }
}
}

Registering with the Builder

The Process is registered through .processes() on the builder. The first argument is how many instances to run, the second is the factory:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
}

This creates 3 KvServer instances at IPs 10.0.1.1, 10.0.1.2, and 10.0.1.3. Each one runs independently. Each one can be killed and restarted independently.

For variable cluster sizes, pass a range:

#![allow(unused)]
fn main() {
.processes(3..=7, || Box::new(KvServer))
}

Now each iteration randomly picks between 3 and 7 servers, deterministically based on the seed.

Process Groups

Most systems have more than one kind of server. A consensus tier and a pool of spares. Acceptors and the matchmakers that reconfigure them. Each .processes() call registers one group, named after its process type, with its own factory and its own per-seed count:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3..=5, || Box::new(Acceptor))
    .processes(0..=3, || Box::new(Matchmaker))
}

Every group owns its own IP range. The first group registered lives on 10.0.1.x, the second on 10.0.2.x, and so on, so a group’s members are contiguous and a single-group simulation keeps the addresses it always had. Inside a process, ctx.topology().my_group() names the group it belongs to and ctx.topology().ips_in_group("acceptor") lists a group’s members. ctx.client_id() and ctx.client_count() are the process’s index within and the size of its own group, so an acceptor can number itself 0..n without knowing how many matchmakers this seed drew.

The counts are drawn in registration order from the seed, which keeps two things true at once: a seed replays the same shape, and each role’s cluster size varies independently of the others’. A 0..=3 group really can draw zero on a plain seed, and it still appears in ctx.topology().groups() so the workload can tell “no matchmakers this seed” from “this system has no matchmakers”.

What the Shutdown Token Gives You

Checking ctx.shutdown() is optional but valuable. During graceful reboots, the simulation cancels the token and gives a grace period. Your process can:

• Finish in-flight requests
• Flush write buffers
• Close connections cleanly so peers see EOF instead of reset errors

If you do not check the token, graceful reboots still work. The framework just force-cancels your future after the grace period expires. But checking gives your process a chance to exit cleanly, which tests a different code path than a hard crash.

Key Takeaways

The pattern for every Process is the same:

1. Bind a listener using ctx.network()
2. Accept connections in a loop
3. Check ctx.shutdown() for graceful termination
4. Use ctx.time() for timeouts, never raw tokio
5. Keep your state in-memory, expect to lose it

The factory produces a blank instance. The simulation manages the lifecycle. Your job is to write the server logic and let the framework handle chaos.

Writing a Workload

• The Workload Struct
• Setup: Finding the Servers
• Run: The Operation Loop
• Check: Final Validation
• Publishing State for Invariants
• Patterns That Find Bugs

The workload drives our key-value server and checks that it behaves correctly. It sends requests, tracks expected state in a reference model, and uses assertions to catch bugs as they happen.

The Workload Struct

When registered via .workload(), a single workload instance is reused across all iterations, letting it accumulate state. (Workloads registered via .workloads() with a factory are recreated each iteration.) The struct holds everything the workload needs to track:

#![allow(unused)]
fn main() {
use async_trait::async_trait;
use moonpool_sim::{
    SimContext, SimulationResult, Workload,
    assert_always, assert_sometimes,
};

struct KvWorkload {
    /// Number of operations per run
    num_ops: usize,
    /// Reference model: what we expect the server to contain
    model: BTreeMap<String, Vec<u8>>,
    /// Keys we use for operations
    keys: Vec<String>,
}
}

The model is the most important field. It mirrors what the server should contain. Every time we write to the server, we write the same value to the model. Every time we read from the server, we compare the result against the model.

Setup: Finding the Servers

The setup() method runs before any workload’s run() starts. Use it to locate processes and prepare connections:

#![allow(unused)]
fn main() {
#[async_trait]
impl Workload for KvWorkload {
    fn name(&self) -> &str {
        "kv-client"
    }

    async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        // Verify we have servers to talk to
        let process_ips = ctx.topology().all_process_ips();
        if process_ips.is_empty() {
            return Err(moonpool_sim::SimulationError::InvalidState(
                "no server processes available".into(),
            ));
        }
        self.model.clear();
        Ok(())
    }
}

The key discovery mechanism is ctx.topology(). It knows about every participant in the simulation: which IPs are processes, which are workloads, what tags each process has.

Common patterns for finding servers:

#![allow(unused)]
fn main() {
// All server IPs
let all_servers = ctx.topology().all_process_ips();

// A specific peer by name
let server_ip = ctx.peer("server").expect("server exists");

// Servers with a particular role tag
let leaders = ctx.topology().ips_tagged("role", "leader");
}

Run: The Operation Loop

The run() method is where bugs get found. We generate random operations, execute them against the server, and verify each response:

#![allow(unused)]
fn main() {
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let server_ips = ctx.topology().all_process_ips().to_vec();

        for i in 0..self.num_ops {
            if ctx.shutdown().is_cancelled() {
                break;
            }

            // Pick a random server
            let server_idx = ctx.random().random_range(0..server_ips.len());
            let server_ip = &server_ips[server_idx];

            // Generate a random operation
            let roll = ctx.random().random_range(0..100);
            match roll {
                0..40 => {
                    // SET operation
                    let key = &self.keys[
                        ctx.random().random_range(0..self.keys.len())
                    ];
                    let value = format!("v{}", i).into_bytes();

                    match self.send_set(ctx, server_ip, key, &value).await {
                        Ok(()) => {
                            self.model.insert(key.clone(), value);
                            assert_sometimes!(true, "set_succeeded");
                        }
                        Err(e) => {
                            tracing::warn!("set failed: {}", e);
                            assert_sometimes!(true, "set_failed_network");
                        }
                    }
                }
                40..80 => {
                    // GET operation
                    let key = &self.keys[
                        ctx.random().random_range(0..self.keys.len())
                    ];

                    match self.send_get(ctx, server_ip, key).await {
                        Ok(value) => {
                            let expected = self.model.get(key)
                                .cloned()
                                .unwrap_or_default();
                            assert_always!(
                                value == expected,
                                format!(
                                    "read mismatch for '{}': got {} bytes, expected {}",
                                    key, value.len(), expected.len()
                                )
                            );
                        }
                        Err(e) => {
                            tracing::warn!("get failed: {}", e);
                        }
                    }
                }
                _ => {
                    // Small delay to let simulation events process
                    let _ = ctx.time().sleep(Duration::from_millis(10)).await;
                }
            }
        }
        Ok(())
    }
}

Notice the two assertion types working together:

assert_always! guards invariants that must never be violated. If a GET returns data that does not match our model, something is broken. An always-assertion failure is a definite bug.

assert_sometimes! marks paths that should fire at least once across all iterations. If "set_succeeded" never triggers across hundreds of seeds, something is wrong with our test setup. If "set_failed_network" never triggers, we might not have enough chaos.

Check: Final Validation

After all workloads finish and pending events drain, check() runs for final state validation:

#![allow(unused)]
fn main() {
    async fn check(&mut self, _ctx: &SimContext) -> SimulationResult<()> {
        // Verify model consistency
        let total_keys = self.model.len();
        assert_always!(
            total_keys <= self.keys.len(),
            format!(
                "model has more keys than expected: {} > {}",
                total_keys, self.keys.len()
            )
        );

        Ok(())
    }
}
}

The check phase is your last chance to validate. The system is quiet. No messages in flight, no operations pending. What the model says should match what the server actually contains.

Publishing State for Invariants

Workloads can publish state that invariant functions read after every simulation step. This enables cross-workload validation:

#![allow(unused)]
fn main() {
// In run(), after each operation:
ctx.state().publish("kv_model", self.model.clone());
}

An invariant function (registered on the builder) can then read this state:

#![allow(unused)]
fn main() {
fn check_model_size(q: &dyn TraceQuery, _sim_time_ms: u64) {
    // Snapshot the latest size events; assert_always if any exceeds bounds.
    let entries = q.snapshot("kv_model_size");
    if let Some(latest) = entries.last() {
        assert_always!(
            latest.u64("size").is_some_and(|s| s <= 100),
            "model grew beyond expected bounds"
        );
    }
}
}

This is the publish-and-check pattern: the workload publishes its reference model, and invariants validate cross-workload properties after every step. The events and invariants chapter covers this in depth.

Patterns That Find Bugs

The strongest workloads combine several techniques:

Reference model: Track expected state locally. Compare against actual server responses. Any divergence is a bug.

Weighted operation alphabet: Mix writes, reads, and delays. Control the distribution. Too predictable means you only test happy paths.

Both assertion types: assert_always! for correctness properties that must hold on every single call. assert_sometimes! for coverage goals that should fire at least once across the full run.

Handle failures gracefully: Network errors during chaos are expected. Log them, maybe track them in the model, but do not treat them as test failures. The bug is when the server returns the wrong answer, not when it returns an error.

Configuring the SimulationBuilder

• The Minimal Builder
• Adding Processes
• Iteration Control
• Seed Control
• Tags for Role Distribution
• Invariants
• Chaos and Attrition
• Enabling Chaos
• Putting It All Together

The SimulationBuilder is the glue. It takes your Process, your Workload, your invariants, and your chaos configuration, and wires them into a runnable simulation.

The Minimal Builder

The simplest possible simulation has one workload and runs once:

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .workload(KvWorkload::new(100, keys))
    .run()
    .await;
}

This creates a single workload at IP 10.0.0.1, runs it with a random seed, and produces a SimulationReport. No processes, no chaos, no multiple iterations. Useful for smoke testing, but not for finding bugs.

Adding Processes

To test a client-server system, add processes alongside the workload:

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(100, keys))
    .run()
    .await;
}

The builder creates 3 server processes at 10.0.1.1 through 10.0.1.3 and one workload at 10.0.0.1. The workload finds server IPs through ctx.topology().all_process_ips().

Call .processes() once per role of the system under test. Each call is an independent group with its own factory and per-seed count, laid out on its own 10.0.{group}.x range, and ctx.topology().ips_in_group("matchmaker") answers a group by name. Defining a Process covers the layout, and Attrition shows how each group gets its own reboot regime.

Iteration Control

One iteration is not enough. Different seeds produce different scheduling orders, different random choices, different failure patterns. You need hundreds or thousands of iterations to find bugs hiding in rare interleavings.

Fixed count runs a specific number of iterations:

#![allow(unused)]
fn main() {
.set_iterations(100)
}

Until coverage stable is the default. It stops once every observed assert_sometimes! / assert_reachable! has fired and code coverage has not grown for a configurable number of consecutive seeds. Useful when you want chaos to run as long as it keeps finding new behaviour and stop automatically once it does not:

#![allow(unused)]
fn main() {
.until_coverage_stable(10, 5_000)  // 10 quiet seeds, 5_000 safety cap
}

Each iteration gets a different seed, producing a different execution. The seeds are deterministic and derived from the iteration manager, so the same configuration always explores the same seeds.

Seed Control

When a simulation fails on a specific seed, you need to reproduce it. Use set_debug_seeds() to run exactly those seeds:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(100, keys))
    .set_debug_seeds(vec![42, 7891])
    .run()
    .await;
}

This runs exactly 2 iterations with seeds 42 and 7891. Combined with RUST_LOG=error, this is the primary debugging workflow: find the failing seed in the report, reproduce it in isolation, add logging, find the bug.

Tags for Role Distribution

When your distributed system has roles, tags assign them to processes:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(5, || Box::new(ConsensusNode))
    .tags(&[
        ("role", &["leader", "follower"]),
        ("dc", &["east", "west", "eu"]),
    ])
}

Tags distribute round-robin. Process 0 gets role=leader, dc=east. Process 1 gets role=follower, dc=west. Process 2 gets role=leader, dc=eu. And so on, wrapping around.

Inside a Process, read tags via ctx.topology().my_tags().get("role"). Inside a Workload, query the tag registry: ctx.topology().ips_tagged("role", "leader") returns the IPs of all leader processes.

Invariants

Invariants run after every simulation event. They check cross-workload properties that must hold at all times, not just at the end.

Trait-based invariant:

#![allow(unused)]
fn main() {
struct AgreementInvariant;

impl Invariant for AgreementInvariant {
    fn name(&self) -> &str { "agreement" }

    fn check(&self, state: &StateHandle, _sim_time_ms: u64) {
        if let Some(model) = state.get::<ConsensusModel>("consensus_model") {
            for (slot, values) in &model.committed_values {
                let unique: BTreeSet<_> = values.iter().collect();
                assert_always!(unique.len() <= 1, "agreement violated");
            }
        }
    }
}

// Register on builder:
.invariant(AgreementInvariant)
}

Closure-based invariant for simpler cases:

#![allow(unused)]
fn main() {
.invariant_fn("key_count_bounded", |state, _time| {
    if let Some(model) = state.get::<KvModel>("kv_model") {
        assert_always!(model.len() <= 1000, "too many keys");
    }
})
}

Invariants read from the StateHandle, which workloads write to via ctx.state().publish(). This is how the test driver communicates its reference model to the invariant checker.

Chaos and Attrition

Real distributed systems do not just run cleanly. Servers crash, networks partition, and then things have to recover. The builder models this with chaos_duration:

#![allow(unused)]
fn main() {
use moonpool_sim::{Attrition, Chaos, ChaosMode};

SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(200, keys))
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([Chaos::Attrition {
        config: Attrition {
            max_dead: 1,
            prob_graceful: 0.3,
            prob_crash: 0.5,
            prob_wipe: 0.2,
            recovery_delay_ms: Some(1000..5000),
            grace_period_ms: Some(2000..4000),
        },
        mode: ChaosMode::Random,
    }])
    .set_iterations(100)
    .run()
    .await;
}

The simulation lifecycle:

1. Chaos phase (30 simulated seconds): Workloads run concurrently with fault injectors. Attrition randomly kills and restarts processes, respecting max_dead to avoid killing everything at once. Configuration-driven network and storage faults fire in this window too.

2. Chaos cutoff: chaos_duration expires and the simulation enters recovery mode. Fault injectors are told to stop, every configuration-driven fault family (network, storage, block device) is switched off, and the partitions the simulator is holding are healed.

   The cutoff bounds fault generation only. It does not repair anything: corrupted sectors stay corrupted, lost writes stay lost, connections the application already saw close stay closed, and a disk stall or clog already running is waited out rather than erased. The cluster is not healthy at the cutoff — the environment has merely stopped making it worse.

3. Recovery tail: The processes are still running, so this is the window where the system under test re-elects, re-replicates, and re-syncs. Give your workloads enough simulated time after the cutoff to converge, and assert on convergence from them.

4. Workload completion: Workloads should be finite (do N operations, or sleep for a sim-time duration, then return).

5. Settle: Processes are aborted, then the orchestrator drains remaining events. If the system does not settle within the timeout (sim time), the test fails with diagnostics, surfacing cleanup bugs like leaked tasks or unclosed connections. Because the processes are already gone, settle is not a recovery window — no protocol work can happen there.

6. Check: The check() methods run inside the event loop, so network RPCs work normally.

max_dead: 1 means at most one process is down at any time. The probability weights control the mix of graceful shutdowns (shutdown token fired, grace period) versus instant crashes (no warning, connections abort).

Enabling Chaos

For additional chaos, enable one or more fault surfaces and choose how each is sampled per seed:

#![allow(unused)]
fn main() {
.enable_chaos([Chaos::Network(ChaosMode::Random)])
}

ChaosMode::Random varies latency, packet delay distributions, and other network parameters per iteration, based on the seed. A surface absent from the list stays off, using its default configuration (consistent, low-latency, no faults).

For stronger coverage, use ChaosMode::Swarm. It enables a random subset of that surface’s fault families per seed, fully disabling the rest, rather than turning every fault slightly on at once. This defeats passive suppression, where active faults crowd each other out and the extreme single-family configs that surface bugs almost never occur. See Network Faults for the full reasoning.

#![allow(unused)]
fn main() {
.enable_chaos([
    Chaos::Network(ChaosMode::Swarm),
    Chaos::Storage(ChaosMode::Swarm),
])
}

The same call drives storage faults and the attrition reboot regime, each with its own ChaosMode. The workload operation-alphabet swarm is a separate, test-driver concern: opt in with .swarm_operations().

Putting It All Together

A production-grade simulation configuration looks like this:

#![allow(unused)]
fn main() {
let report = SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .tags(&[("role", &["primary", "replica"])])
    .workload(KvWorkload::new(500, keys.clone()))
    .invariant(ConservationLaw)
    .chaos_duration(Duration::from_secs(30))
    .enable_chaos([
        Chaos::Attrition {
            config: Attrition {
                max_dead: 1,
                prob_graceful: 0.3,
                prob_crash: 0.5,
                prob_wipe: 0.2,
                recovery_delay_ms: None,
                grace_period_ms: None,
            },
            mode: ChaosMode::Swarm,
        },
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
    ])
    .set_iterations(100)
    .run()
    .await;
}

The builder takes care of the rest: creating the simulated world, assigning IPs, seeding the RNG, running the orchestration loop, collecting metrics, and producing the report.

Running and Observing

• Running with xtask
• Reading the SimulationReport
• What Success Means
• When Things Fail
• Debugging a Failing Seed
• Stop Conditions
• cargo nextest vs cargo xtask sim
• Exit Codes
• The Feedback Loop

You have a Process, a Workload, and a builder configuration. Now we run the simulation and make sense of the output.

Running with xtask

The primary way to run simulation binaries is through xtask:

cargo xtask sim list         # List all simulation binaries
cargo xtask sim run kv       # Run binaries matching "kv"
cargo xtask sim run-all      # Run everything

The run subcommand matches against binary names. For example, cargo xtask sim run tonic-grpc selects the sim-tonic-grpc example.

Each simulation binary is a standalone Rust binary that constructs a SimulationBuilder, calls .run(), and prints the report. A typical main function:

fn main() {
    let _ = tracing_subscriber::fmt()
        .with_max_level(tracing::Level::WARN)
        .try_init();

    let report = SimulationBuilder::new()
        .processes(3, || Box::new(KvServer))
        .workload(KvWorkload::new(200, keys))
        .set_iterations(100)
        .enable_chaos([Chaos::Network(ChaosMode::Random)])
        .run();

    report.eprint();

    if !report.seeds_failing.is_empty() || !report.assertion_violations.is_empty() {
        std::process::exit(1);
    }
}

Notice what is missing: there is no runtime to build. run() drives each iteration on a fresh moonpool deterministic executor, Executor::new(seed) plus executor.block_on(...) under the hood. Determinism still demands a single OS thread, and every moonpool trait and future is still Send + 'static, so customer code can hold Arc<RwLock<…>> and DashMap naturally while the executor polls everything on one thread. Which ready task runs next is a seeded-random pick, so each seed also explores a different task interleaving. And because nothing here needs a tokio runtime, simulation binaries build on plain stable Rust with no special rustflags.

Reading the SimulationReport

The report prints to stderr with .eprint(). Here is what a healthy report looks like:

━━━ Simulation Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  100 iterations   100 passed   0 failed   ✓ 100.0%

  Wall Time    12ms avg       1.20s total
  Sim Time     45.23s avg
  Events       8,432 avg

━━━ Assertions (4) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓  always       "conservation law"  8,200 pass  0 fail
  ✓  always       "read matches model" 12,847 pass  0 fail
  ✓  sometimes    "set_failed_network" 412 / 12,847  (3.2%)
  ✓  sometimes    "set_succeeded"     6,102 / 12,847  (47.5%)

The same text, coloured when stderr is a terminal, is what .eprint() prints; format!("{report}") gives the plain form.

The critical lines:

• ✓ 100.0% means no iteration panicked or returned an error
• 0 fail on always-assertions means no invariant violations
• ✓ on sometimes-assertions means every coverage goal was hit at least once (a never-fired one shows as ○)

What Success Means

A simulation succeeds when two conditions hold simultaneously:

1. No always-assertion violations: Every assert_always! passed on every evaluation across all iterations
2. All sometimes-assertions fired: Every assert_sometimes! evaluated to true at least once across all iterations

Both matter. A simulation that never violates invariants but also never exercises error paths is not testing enough. A simulation that hits every code path but tolerates wrong answers is not checking enough.

When Things Fail

A failing report shows faulty seeds and violations:

━━━ Simulation Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  100 iterations   98 passed   2 failed   ✗ 98.0%

  Wall Time    12ms avg       1.20s total
  Sim Time     45.23s avg
  Events       8,432 avg

  Faulty seeds: [7891, 42033]

━━━ Assertions (4) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✗  always       "read matches model" 12,800 pass  47 fail
  ✓  always       "conservation law"  8,200 pass  0 fail
  ✓  sometimes    "set_failed_network" 412 / 12,847  (3.2%)
  ✓  sometimes    "set_succeeded"     6,102 / 12,847  (47.5%)

━━━ Violations ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✗  Always "read matches model": 47 failures out of 12,847 evaluations

The report tells you:

• Which seeds failed: 7891 and 42033
• Which assertion broke: “read matches model” had 47 failures
• How often: 47 out of 12,847 evaluations, so it is a rare condition

Debugging a Failing Seed

Take the failing seed and isolate it:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(KvServer))
    .workload(KvWorkload::new(200, keys))
    .set_debug_seeds(vec![7891])
    .run()
    .await;
}

Now increase logging. Set the environment variable:

RUST_LOG=debug cargo xtask sim run kv

Because the simulation is deterministic, seed 7891 reproduces the exact same scheduling, the exact same random choices, the exact same failure. You can add tracing::debug! statements, rerun with the same seed, and see exactly what happened.

The debugging workflow:

1. Find the seed in the report’s faulty seeds list
2. Isolate it with set_debug_seeds(vec![seed])
3. Add logging in the Process and Workload code
4. Rerun and trace the execution
5. Fix the root cause in the Process code
6. Verify by running the full iteration suite again

Stop Conditions

How long should a chaos run last? Any fixed seed count is wrong: too small misses bugs, too large burns time. The right answer is to stop on a signal, not a count — when the run has nothing left to discover. Moonpool exposes two answers via IterationControl.

UntilCoverageStable { plateau_seeds, max_iterations } is the default, and the one you want most of the time. It stops when every observed assert_sometimes!, numeric sometimes assertion, or assert_reachable! has fired at least once and code coverage has not grown for plateau_seeds consecutive seeds. The max_iterations field is a safety cap. Under cargo xtask sim run the binaries are sancov-instrumented, so the progress signal is real code coverage, the count of distinct edges the seeds have exercised. Under plain cargo nextest run there is no instrumentation, so it falls back to assertion coverage, the set of sometimes, numeric-sometimes, and reachable messages that have fired. The report names which signal it used, so the fallback is never silent. This works with or without exploration; no fork happens unless you call .enable_exploration().

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .workload(KvWorkload::new(200, keys))
    .until_coverage_stable(10, 5_000)  // 10 quiet seeds, 5_000 safety cap
    .run();
}

FixedCount(n) is the workhorse for reproducible replay. You commit to running exactly n seeds, the duration is bounded, and the report is identical across machines. Reach for this when debugging a specific seed or when budgets must be predictable.

UntilCoverageStable sets report.convergence_timeout = true when the safety cap is hit before the run saturates, so CI can fail loudly instead of silently treating “we ran out of seeds” as success.

Each workload phase also has a generous virtual-time budget. It bounds a setup, run, or final check that keeps rearming timers without completing. The runner first requests shutdown, then rejects a phase that still cannot finish, so a stuck precondition cannot leave CI waiting forever.

If a setup, run, or check task panics or is cancelled before it returns its owned workload, Moonpool stops the campaign. Reusing the surviving workloads on a later seed would leave the workload set incomplete and can assign an instance another workload’s identity. An ordinary error returned from a workload preserves its instance, so later seeds continue normally.

cargo nextest vs cargo xtask sim

Moonpool has two ways to run tests:

cargo nextest run runs unit tests and integration tests. These are fast, focused tests for specific modules. Use nextest during development for quick feedback.

cargo xtask sim run runs simulation binaries. These are comprehensive, multi-iteration chaos tests that take longer but find deeper bugs. Use xtask for validation before merging.

Both should pass before work is complete. The typical workflow: write code, run nextest for fast iteration, then run xtask sim for thorough validation.

Exit Codes

Simulation binaries should exit with code 0 on success and code 1 on failure. The standard pattern:

#![allow(unused)]
fn main() {
if !report.seeds_failing.is_empty()
    || !report.assertion_violations.is_empty()
{
    std::process::exit(1);
}
}

Check both seeds_failing (iterations that panicked or errored) and assertion_violations (always-type assertions that failed). Coverage violations (coverage_violations) indicate sometimes-assertions that never fired, which may or may not be worth failing the build over depending on your testing philosophy.

The Feedback Loop

Simulation testing is iterative. You write a workload, run it, find a bug, fix the bug, add assertions to prevent regression, run again. Each round makes the system more robust.

The report’s assertion table is your scoreboard. When every row shows ✓ with high hit counts, you know your test is both thorough and correct. When a sometimes-assertion shows ○, you know there is a code path your chaos is not reaching. When an always-assertion shows ✗, you know there is a real bug to fix.

This is the rhythm of simulation-driven development: build, test, observe, improve.

Chaos Testing vs Simulation

• Two Schools of Breaking Things
• Chaos Engineering: Learning from Production
• Simulation: Breaking Things Before They Exist
• Simulation Subsumes Chaos
• Complementary, Not Competing
• Moonpool’s Approach

Two Schools of Breaking Things

There are two dominant approaches to testing distributed systems under failure. Chaos engineering takes a running production system and injects faults into it: kill a VM, drop a network link, fill a disk. Simulation testing builds a fake world and runs the system inside it, injecting faults by rewriting the laws of physics. Both break things on purpose. But they break things in fundamentally different ways, and understanding those differences shapes how we use them.

Chaos Engineering: Learning from Production

Chaos engineering, popularized by Netflix’s Chaos Monkey, treats production (or a production-like staging environment) as the laboratory. You pick an experiment (“what happens if we kill this database node?”), form a hypothesis (“traffic reroutes within 5 seconds”), run the experiment against real infrastructure, and observe what happens.

This approach has real strengths. It tests the actual system with real configurations, real dependencies, and real network stacks. It catches problems that no amount of pre-production testing would find: misconfigured load balancers, stale DNS caches, monitoring gaps. When chaos engineering finds something, you know it matters because it happened in the real world.

But chaos engineering has structural limitations:

• Non-deterministic. You cannot reproduce the exact sequence of events that caused a failure. The bug happened once, under conditions you cannot fully reconstruct.
• Reactive. You find problems after they exist in production, not before.
• Slow. Each experiment takes real time against real infrastructure. Running one scenario might take minutes. Running a million is not practical.
• Blast radius. Real users can be affected. Even with careful scoping, an experiment that goes wrong can cause actual outages.

Chaos engineering finds symptoms. The database went down and something bad happened. But why? Was it a race condition? A missing retry? A lock held too long during recovery? Getting from symptom to root cause requires forensic debugging after the fact, often without a way to reproduce the exact failure.

Simulation: Breaking Things Before They Exist

Simulation testing takes the opposite approach. Instead of injecting faults into a real system, you build a simulated world where faults are a first-class feature. The network drops packets because you told it to. Disks corrupt writes because the configuration says they should. Clocks drift because the simulator makes them drift.

Everything runs in a single process on one OS thread, driven by a seeded pseudorandom number generator. The same seed produces the same execution, every time. A failing test is not a flaky signal. It is a reproducible bug with an exact replay.

The advantages follow directly:

• Deterministic. Every failure is reproducible. A failing seed is a permanent regression test.
• Proactive. You find bugs before code reaches production, before it reaches staging, often before it leaves a developer’s machine.
• Fast. No real I/O, no real time. Moonpool simulates hundreds of seconds of cluster behavior in single-digit real seconds. FoundationDB ran 5 to 10 million simulation runs per night.
• Exhaustive. Different seeds explore different fault combinations. Run enough seeds and you cover failure scenarios no human would think to write tests for.

Simulation Subsumes Chaos

Here is the key insight: simulation subsumes chaos engineering. Everything chaos engineering does, simulation does too, but with reproducibility, speed, and exhaustiveness added on top.

Chaos engineering injects a random partition? Simulation injects a random partition, and you can replay the exact moment it happened with the exact timing. Chaos engineering kills a process? Simulation kills a process, restarts it, and verifies the recovery path, all in milliseconds of real time.

Will Wilson made this point concrete with FoundationDB’s Sinkhole: a rack of real servers wired to programmable power switches, toggled continuously to validate the real system. Sinkhole never found a single database bug that simulation had missed. It only found bugs in other software and in hardware. The simulation was a stricter, more adversarial environment than reality.

Complementary, Not Competing

This does not mean chaos engineering is useless. The two approaches are complementary:

Simulation is for development. It runs on every commit, explores millions of fault scenarios, and catches bugs before they ship. It tests the logic of your system against the worst possible world.

Chaos engineering is for production validation. It verifies that your real deployment, with its real configuration and real dependencies, behaves as expected. It catches the things simulation cannot model: actual kernel behavior, real cloud provider quirks, misconfigured infrastructure.

Think of simulation as the wind tunnel and chaos engineering as the flight test. You would never skip the wind tunnel because you plan to fly the plane. And you would never skip the flight test because the wind tunnel looked good.

Moonpool’s Approach

In moonpool, chaos injection is a tool within the simulation, not an alternative to it. Network faults, storage corruption, process reboots, code path perturbation: all of these are chaos techniques, but they run inside a deterministic simulator where every fault is reproducible and every failure is debuggable.

The next chapters cover four dimensions of chaos that moonpool provides: buggify for code-level fault injection, attrition for process lifecycle chaos, network faults, and storage faults. Each one amplifies failure probabilities so that rare bugs become common, while keeping everything controlled by a single seed.

Chaos in Moonpool

• The Philosophy: Make Rare Bugs Common
• Four Dimensions of Chaos
  • Code Paths: Buggify
  • Infrastructure: Attrition
  • Network Faults
  • Storage Faults
• Determinism Is Non-Negotiable
• The Next Four Chapters

The Philosophy: Make Rare Bugs Common

Real distributed systems fail in ways that are individually rare but collectively inevitable. A network partition happens once a month. A disk corrupts a write once a year. A process crashes during recovery once in a thousand deployments. Individually, each event is unlikely. But multiply enough low-probability events across enough nodes and enough time, and the question is not if but when.

The core philosophy of chaos in moonpool is simple: amplify failure probabilities so that rare bugs become common. If a network partition happens once a month in production, make it happen every few seconds in simulation. If a disk write corrupts once a year, corrupt one every hundred writes. If a process crashes at the worst possible moment once in a thousand runs, make it crash at the worst possible moment every run.

This is not reckless. It is strategic. By making failures frequent, we force the code to handle them every time, not just when a developer remembers to write a test for them. And because everything runs inside a deterministic simulation, every failure is reproducible. A bug found at high fault probability is the same bug that would have appeared at low probability in production, just found a thousand times faster.

Four Dimensions of Chaos

Moonpool provides chaos injection along four distinct dimensions, each targeting a different layer of the system:

Code Paths: Buggify

Inspired by FoundationDB’s BUGGIFY macro, buggify!() scatters fault injection points throughout your application code. Each point randomly activates once per simulation run, then fires probabilistically on each call. This forces your code down error paths, timeout branches, and recovery logic that would otherwise require precise failure timing to exercise.

Infrastructure: Attrition

Processes crash. Processes restart. Sometimes they restart cleanly, sometimes they lose all their data. Attrition automatically cycles your server processes through graceful shutdowns, crashes, and data-wiping restarts during the chaos phase, while respecting a max_dead constraint that keeps enough processes alive for the system to remain operational.

Network Faults

TCP connections fail in subtle ways. Moonpool simulates connection drops, latency spikes, byte corruption, clock drift, partial writes, and both symmetric and one-way network partitions. The simulation operates at the TCP connection level, not the packet level, because connection-level faults are what distributed systems actually need to handle.

Storage Faults

Disks lie. Following TigerBeetle’s fault model, moonpool injects read corruption, write corruption, torn writes, misdirected I/O, phantom writes, and sync failures. These are the faults that data-integrity code must survive, and they are nearly impossible to test without simulation.

Determinism Is Non-Negotiable

Every chaos mechanism in moonpool is controlled by the simulation seed. The same seed produces the same faults in the same order at the same times. This means:

• A failing seed is a permanent, reproducible bug report
• You can debug a failure by replaying it with logging turned up
• Fixing a bug and re-running the seed verifies the fix
• Different seeds explore different fault combinations automatically

This is what separates simulation chaos from production chaos. In production, a network partition happened and something went wrong, but you cannot reproduce the exact sequence of events. In simulation, you hand someone a seed number and they see exactly what you saw.

The Next Four Chapters

Each dimension of chaos has its own chapter with configuration details, code examples, and the design reasoning behind the choices moonpool makes:

• Buggify: Fault Injection covers the buggify!() macro and code-level fault injection patterns
• Attrition: Process Reboots covers automatic process lifecycle chaos
• Network Faults covers TCP-level network fault simulation
• Storage Faults covers TigerBeetle-inspired storage fault patterns

Buggify: Fault Injection

• The Idea
• Two-Phase Activation
• Five Injection Patterns
  • 1. Error Injection on Success Paths
  • 2. Artificial Delays
  • 3. Parameter Randomization
  • 4. Alternative Code Paths
  • 5. Process Restarts
• Spiking Config Knobs
• Probability Calibration
• Anti-Patterns
• Production Safety
• The Standalone moonpool-buggify Crate

The Idea

Most distributed system bugs do not live in the happy path. They hide in error handlers, timeout branches, retry logic, and recovery code. These paths are exercised only when something goes wrong, which means they are the least tested code in the system and the most critical when failures happen.

FoundationDB solved this with a technique called BUGGIFY: scatter conditional fault injection points throughout the codebase, activated only during simulation. Each point perturbs code behavior in a small way: return an error instead of success, add an artificial delay, randomize a buffer size. Run enough seeds and these perturbations force the code through every error path, every retry loop, every recovery sequence.

Moonpool implements this as the buggify!() macro.

Two-Phase Activation

A naive approach would fire every buggify point on every call. That produces chaos but not useful chaos. If every operation fails, the system never makes progress and you never test the interesting interactions between partial failures.

Moonpool uses FoundationDB’s two-phase activation model:

Phase 1: Activation. The first time a buggify location is encountered during a simulation run, it is randomly activated or deactivated. This decision is fixed for the entire run. A location that is deactivated will never fire, no matter how many times it is reached.

Phase 2: Firing. Each time an activated location is reached, it fires with a fixed probability (25% by default). This means an active buggify point fires roughly one in four times, creating a mix of successful and failed operations.

#![allow(unused)]
fn main() {
// Each call site is a unique location (identified by file:line).
// First encounter: 50% chance of activation.
// Subsequent encounters at an active site: 25% chance of firing.
if buggify!() {
    return Err(Error::timeout("simulated timeout"));
}
}

Neither rate is a builder setting: the runner activates each location at a fixed 50% per seed, and buggify! fires an active location at a fixed 25% per call. What can vary is the firing probability, per call site, with buggify_with_prob!:

#![allow(unused)]
fn main() {
// Fire at 50% probability instead of the default 25%
if buggify_with_prob!(0.5) {
    buffer_size = 1; // Force single-byte reads
}
}

This two-phase design means each seed tests a different combination of active fault injection points. Seed 42 might activate a request timeout but deactivate a storage error. Seed 43 might do the reverse. Run enough seeds and you cover the combinatorial space of fault interactions.

Five Injection Patterns

FoundationDB’s codebase uses BUGGIFY in five recurring patterns, all of which translate directly to moonpool:

1. Error Injection on Success Paths

The most common pattern. After a successful operation, sometimes return an error anyway:

#![allow(unused)]
fn main() {
let result = connection.send(message).await;
if result.is_ok() && buggify!() {
    // Force the caller through its error handling path
    return Err(Error::io("buggified send failure"));
}
}

2. Artificial Delays

Inject delays to expose race conditions and timing-dependent bugs:

#![allow(unused)]
fn main() {
if buggify!() {
    // Slow down this operation to widen race windows
    time.sleep(Duration::from_millis(100)).await?;
}
}

3. Parameter Randomization

Vary sizes, timeouts, and limits to test edge cases:

#![allow(unused)]
fn main() {
let batch_size = if buggify!() {
    // Test with tiny batches to exercise boundary conditions
    random.random_range(1..3)
} else {
    DEFAULT_BATCH_SIZE
};
}

4. Alternative Code Paths

Force the system down paths it rarely takes:

#![allow(unused)]
fn main() {
let should_compact = needs_compaction() || buggify_with_prob!(0.1);
if should_compact {
    // Exercise compaction logic more frequently
    compact_storage().await?;
}
}

5. Process Restarts

Trigger restarts at specific points to test crash recovery:

#![allow(unused)]
fn main() {
if buggify_with_prob!(0.01) {
    // Simulate a crash right after writing but before syncing
    return Err(Error::crash("buggified crash after write"));
}
}

Spiking Config Knobs

The patterns above scatter buggify!() through your own code. But the simulation has its own knobs: network latencies, disk IOPS, partition durations, fault probabilities. FoundationDB randomizes these the same way it randomizes everything else, with a one-liner at config construction:

if (randomize && BUGGIFY) KNOB = deterministicRandom()->random(lo, hi);

Moonpool gives us buggify_knob!(default, lo..hi). It returns default on most seeds, but when its call site activates and fires it returns a random value inside the range:

#![allow(unused)]
fn main() {
// Most seeds keep the sampled IOPS. A few push the disk to a crawl.
self.iops = buggify_knob!(self.iops, 100..5_000);
}

This is a different coverage axis from swarm testing. Swarm decides which fault families are on. Buggify-knobs decides how hard an enabled knob gets pushed. A seed might run with only storage faults active, and within that, a disk throttled to 100 IOPS instead of the usual 25,000. The two compose, multiplying the configurations a night of seeds explores.

We opt in through the same enable_chaos list we use for everything else:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::BuggifyKnobs,
    ])
    .workload(MyWorkload)
    .run();
}

Chaos::BuggifyKnobs is a modifier, not a surface of its own. It only perturbs knobs on surfaces we already enabled, so it never silently switches on a fault family we left off. Like every other buggify decision, each spike is deterministic per seed, so a failing seed replays exactly.

Probability Calibration

Not all buggify points should fire at the same rate. FoundationDB uses a three-tier calibration:

The default 25% firing probability works well for most injection points. Use buggify_with_prob!() when you need more control. High-probability points are useful for paths you want exercised frequently. Low-probability points model events that are individually rare but must be handled correctly.

Anti-Patterns

Do not use buggify for business logic. Buggify is for simulating infrastructure failures, not for feature flags or A/B testing. If the buggified branch changes application semantics rather than injecting a fault, it belongs in your application logic, not in a buggify block.

Do not use non-deterministic random. Buggify uses sim_random() internally, which is controlled by the simulation seed. Never mix in rand::random() or other non-deterministic entropy. That breaks reproducibility.

Do not use excessive probabilities without reason. A buggify_with_prob!(0.9) that fails 90% of attempts means the system almost never succeeds. That tests error handling but misses the interesting interactions between partial success and partial failure.

Production Safety

Buggify is gated behind simulation state. When the simulation is not running, buggify!() always returns false. There is no runtime cost in production: the check is a thread-local boolean read. You can leave buggify calls in your production code without worrying about them firing outside simulation.

This is the same guarantee FoundationDB provides: BUGGIFY is gated behind g_network->isSimulated(), ensuring zero production impact regardless of how aggressively chaos is injected during testing.

The Standalone moonpool-buggify Crate

The buggify!() and buggify_with_prob!() macros live in the zero-dependency moonpool-buggify crate. Sans-I/O and production code that wants buggify points can depend on it directly, without pulling the simulation runtime into its dependency graph:

[dependencies]
moonpool-buggify = "0.8"

The crate owns only the disabled-by-default state and the macros. When a simulation run starts, moonpool-sim installs its deterministic seeded RNG into that shared state, so macros imported through either crate share activation decisions during simulation — and stay inert everywhere else. moonpool-sim re-exports both macros, so existing moonpool_sim::buggify! call sites are unchanged.

buggify_knob! remains in moonpool-sim: knob randomization is simulation-specific configuration spiking, not application-level fault injection.

Attrition: Process Reboots

• Why Processes Need to Die
• Three Kinds of Reboot
  • Graceful
  • Crash
  • CrashAndWipe
• The Attrition Configuration
• Using Attrition
• The max_dead Constraint
• Role-Aware Victims: One Regime per Group
• Failure Domains: Correlated Reboots
• Custom Fault Injection
• Scripted Lifecycle Faults
  • Fault Factories and Exploration

Why Processes Need to Die

A distributed system that only works when all nodes are healthy is not a distributed system. It is a single point of failure with extra network hops. Real clusters lose nodes constantly: rolling deployments restart processes, kernel panics crash them, power failures wipe their storage. Your system must handle all of these, and it must handle them while continuing to serve requests.

Attrition is moonpool’s built-in mechanism for automatically killing and restarting server processes during simulation. It runs during the chaos phase, picks random processes, kills them in various ways, waits a recovery delay, and restarts them. The goal is to continuously verify that your system can tolerate node failures without manual intervention.

Three Kinds of Reboot

Moonpool provides three reboot types, each modeling a different real-world failure:

Graceful

The controlled shutdown. A cancellation token fires, giving the process a grace period to drain buffers, close connections cleanly, and flush pending writes. If the process does not exit within the grace period, it gets force-killed. After shutdown, connections deliver remaining buffered data (FIN semantics), then the process restarts with fresh state.

This models rolling deployments, planned maintenance, and well-behaved process managers. The process has a chance to clean up, but that chance is time-bounded.

Crash

The sudden death. The process task is cancelled at the crash instant – before its connections abort, so nothing the crash disturbs can wake it again. All connections abort with no buffer drain. Peers see connection reset errors. Any in-memory state is lost. The process runs no further work until it restarts, after a recovery delay.

This models kernel panics, OOM kills, and hardware failures. There is no warning and no cleanup. Code that assumes a graceful shutdown will always happen gets a rude surprise.

CrashAndWipe

The worst case. Same as Crash, but all persistent storage for the process is also deleted. The process restarts as if it were a brand new node joining the cluster for the first time.

This models total disk failures, accidental data deletion, or replacing a failed machine with a fresh one. The wipe is scoped to the crashed process’s IP address, so other processes’ storage is unaffected. Systems that rely on durable state for recovery must handle the case where that state is gone.

The Attrition Configuration

Attrition is configured through the Attrition struct:

#![allow(unused)]
fn main() {
Attrition {
    max_dead: 1,
    prob_graceful: 0.3,
    prob_crash: 0.5,
    prob_wipe: 0.2,
    recovery_delay_ms: Some(1000..10000),
    grace_period_ms: Some(2000..5000),
    scope: AttritionScope::PerProcess,
    victims: AttritionVictims::Any,
}
}

max_dead is the most important field. It caps the number of simultaneously dead processes. If you have a 3-node cluster with max_dead: 1, attrition will never kill a second node before the first has restarted. This ensures the system always has enough live nodes to remain operational (assuming your replication factor matches).

prob_graceful, prob_crash, prob_wipe are weights, not probabilities. They do not need to sum to 1.0. The attrition injector normalizes them internally and picks a reboot kind by weighted random selection. Setting prob_wipe: 0.0 disables wipe reboots entirely.

recovery_delay_ms controls how long a dead process stays dead before restarting. The actual delay is drawn randomly from this range, so different seeds test different recovery timings. The default is 1 to 10 seconds of simulated time.

grace_period_ms controls how long a graceful shutdown has to complete. Again, drawn randomly from the range. The default is 2 to 5 seconds.

scope decides which failure domain each reboot targets. The default, AttritionScope::PerProcess, kills one random process at a time. PerMachine, PerZone, and PerDatacenter kill all collocated processes together, which is the subject of a later section.

victims decides which processes are eligible at all. The default, AttritionVictims::Any, draws over every server process. A group or tag filter keeps attrition on one role, which is the subject of Role-Aware Victims.

Using Attrition

Attrition is configured on the simulation builder and requires a chaos duration:

#![allow(unused)]
fn main() {
use moonpool_sim::{Attrition, Chaos, ChaosMode};

SimulationBuilder::new()
    .processes(3, || Box::new(MyProcess::new()))
    .enable_chaos([Chaos::Attrition {
        config: Attrition {
            max_dead: 1,
            prob_graceful: 0.3,
            prob_crash: 0.5,
            prob_wipe: 0.2,
            recovery_delay_ms: None,  // use defaults
            grace_period_ms: None,    // use defaults
            scope: AttritionScope::PerProcess,
            victims: AttritionVictims::Any,
        },
        mode: ChaosMode::Random,
    }])
    .chaos_duration(Duration::from_secs(60))
    .workload(MyWorkload::new())
    .run()
    .await;
}

The .chaos_duration() call is required because attrition runs only during the chaos phase. After the chaos duration elapses the simulation enters recovery mode: fault injectors stop, every configuration-driven network and storage fault family is switched off, and partitions are healed, while the system continues until all workloads complete. That quiet tail is where a rebooted process rejoins and the cluster reconverges — the damage from the chaos phase is still there to recover from, since recovery mode stops new faults rather than repairing old ones. A settle phase then drains remaining events before checks run, surfacing cleanup bugs rather than hiding them behind an arbitrary timer.

ChaosMode::Random uses your configured weights, recovery window, and scope as written every seed. Switch to ChaosMode::Swarm to swarm the reboot regime itself. Each seed can select the never-reboot case (which surfaces slow-leak and timer bugs that constant restarting hides), a single reboot kind such as always-crash or graceful-only, and a recovery-delay range scaled to between 50% and 200% of the configured range. With .cluster(), the seed also selects among PerProcess and correlated failure scopes whose groups fit the sampled max_dead budget. A flat .processes() campaign keeps its configured scope because it has no failure-domain topology to swarm.

These regime choices are six draws on the simulation stream, taken once the world is built, in registration order. The injector still makes its usual single runtime draw per reboot for the actual recovery delay and victim, so a regime costs a fixed footprint at a fixed position and replays with the seed. Same reasoning as swarming network faults.

The max_dead Constraint

max_dead deserves special attention because it is the bridge between chaos and correctness. Without it, attrition could kill all your nodes simultaneously, which is technically chaos but not useful chaos. No distributed system survives simultaneous failure of all replicas.

Set max_dead to match your system’s fault tolerance. A system with replication factor 3 can tolerate 1 failure, so max_dead: 1. A system that needs 3 of 5 nodes alive should use max_dead: 2. This ensures attrition tests failures your system should survive, not failures that are inherently unrecoverable.

The budget counts dead processes among the injector’s victims. With AttritionVictims::Any that is the whole cluster. With a group or tag filter it is that pool alone, so “at most one dead acceptor” is exactly what it says, whatever is happening to the matchmakers.

Role-Aware Victims: One Regime per Group

A cluster rarely has one kind of node. Once you register process groups, uniform attrition has a problem: it draws its victim over every process, so on a seed with three acceptors and five spares most kills land on the spares, and the consensus tier you meant to stress barely notices. The fix is to tell attrition who it is allowed to kill:

#![allow(unused)]
fn main() {
use moonpool_sim::AttritionVictims;

Attrition {
    max_dead: 1,
    victims: AttritionVictims::group("acceptor"),   // never a spare
    ..base
}
}

AttritionVictims::group("acceptor") restricts the draw to one .processes() / .cluster() group, named after its process type. AttritionVictims::tagged("role", "voter") restricts it to processes carrying a tag. The filter costs no randomness of its own: the victim is drawn uniformly over the eligible processes, so a filter that excludes nothing leaves every existing seed’s draw schedule untouched, and a filter that leaves nothing eligible skips the round without drawing. It applies to every scope too. A machine-scoped reboot only picks machines that host an eligible process and only kills the eligible processes on it.

Because the budget is per pool, you can give each group its own regime. Every Chaos::Attrition entry adds one independent injector:

#![allow(unused)]
fn main() {
.enable_chaos([
    Chaos::Attrition {
        // Acceptors: at most one down at a time, regime swarmed per seed.
        config: Attrition { max_dead: 1, victims: AttritionVictims::group("acceptor"), ..base.clone() },
        mode: ChaosMode::Swarm,
    },
    Chaos::Attrition {
        // Matchmakers: hammered independently, up to three down at once.
        config: Attrition { max_dead: 3, victims: AttritionVictims::group("matchmaker"), ..base },
        mode: ChaosMode::Random,
    },
])
}

The two injectors run side by side during the chaos phase and never block each other: a dead matchmaker does not spend the acceptors’ budget. Swarm regimes are drawn in registration order from the simulation stream, so a campaign with several injectors replays exactly like one with a single injector. In the report each injector carries its filter in its name (attrition[group=acceptor]), so a failing seed says which regime was active.

Failure Domains: Correlated Reboots

max_dead: 1 protects you from killing two random nodes at once. But real outages are rarely random and rarely one node at a time. A rack loses power and takes ten machines with it. A datacenter link cuts and a whole region goes dark. A host reboots and every process pinned to it dies together. These are correlated failures, and they are exactly the failures that break quorum logic, because the nodes that die were never independent to begin with.

Flat IPs cannot express this. 10.0.1.{1..N} is a list of peers with no notion of which ones share fate. So moonpool borrows FoundationDB’s model: a cluster is a hierarchy of Datacenter → Zone → Machine → Process, and processes on the same machine fail together because they share a machine.

You describe that hierarchy with .cluster() instead of .processes():

#![allow(unused)]
fn main() {
use moonpool_sim::{LocalityConfig, SimulationBuilder};

SimulationBuilder::new()
    .cluster(
        // 3 datacenters × 3 zones × 3 machines × 2 processes = 54 processes.
        // Ranges like 1..=3 are sampled per seed, so every seed runs a
        // different cluster shape, the way FoundationDB generates topology.
        LocalityConfig::new(3, 3, 3, 2),
        || Box::new(MyProcess::new()),
    )
    .workload(MyWorkload::new());
}

The topology, not a flat count, decides how many processes exist. Each one is assigned a globally unique datacenter, zone, and machine id (dc1, dc1-z1, dc1-z1-m1), and processes read their own placement plus query the cluster through their topology:

#![allow(unused)]
fn main() {
let me = ctx.topology().my_locality().expect("clustered process");
let machine_mates = ctx.topology().peers_on_my_machine();
let same_dc = ctx.topology().ips_in_domain(DomainLevel::Datacenter, me.datacenter());
}

Now scope earns its keep. With AttritionScope::PerMachine, attrition picks a machine instead of a process and reboots every process on it in the same instant. The two processes that share dc1-z1-m1 die together and recover together, exactly as they would when their host kernel panics. PerZone does the same one level up, and PerDatacenter one level above that, for the region-wide blackout that only a replication scheme spanning datacenters is supposed to survive.

#![allow(unused)]
fn main() {
.enable_chaos([Chaos::Attrition {
    config: Attrition {
        max_dead: 2,  // one machine's worth of processes
        prob_graceful: 0.3,
        prob_crash: 0.5,
        prob_wipe: 0.2,
        recovery_delay_ms: None,
        grace_period_ms: None,
        scope: AttritionScope::PerMachine,
        victims: AttritionVictims::Any,
    },
    mode: ChaosMode::Random,
}])
}

max_dead still counts dead processes, and a machine reboot is atomic against that budget: attrition only kills a machine when the whole group fits within the remaining budget. With two processes per machine, max_dead: 2 lets exactly one machine be down at a time and never leaves a machine half-dead. Set it to a multiple of your machine size that matches how many machines your replication can lose.

The same topology also shapes the network, not just the reboots. PartitionStrategy::IsolateZone and IsolateDatacenter cut a whole domain off the rest of the cluster, and LinkLatencyConfig gives cross-datacenter links their real cost. Both are covered in Network Faults.

For targeted correlated faults, FaultContext exposes the group reboots directly. ctx.reboot_machine("dc1-z1-m1", RebootKind::Crash) kills a named machine and returns the IPs it took down, and ctx.reboot_domain(DomainLevel::Datacenter, "dc1", kind) blacks out a whole datacenter. Combined with ctx.ips_in_domain(...), the same domain ids drive network partitions across a zone or datacenter boundary, so you can model a region cut as cleanly as a host reboot.

Custom Fault Injection

Attrition covers the common case of random process reboots. For more targeted fault injection, implement the FaultInjector trait:

#![allow(unused)]
fn main() {
#[async_trait]
impl FaultInjector for RollingRestart {
    fn name(&self) -> &str { "rolling_restart" }

    async fn inject(&mut self, ctx: &FaultContext) -> SimulationResult<()> {
        // Restart each process in order, waiting for recovery between each
        for ip in ctx.process_ips().to_vec() {
            ctx.reboot(&ip, RebootKind::Graceful)?;
            ctx.time().sleep(Duration::from_secs(15)).await
                .map_err(|e| SimulationError::InvalidState(e.to_string()))?;

            if ctx.chaos_shutdown().is_cancelled() {
                break;
            }
        }
        Ok(())
    }
}
}

The FaultContext provides access to process reboots, network partitions, and tag-based targeting. You can combine built-in attrition with custom fault injectors by registering both on the builder.

Scripted Lifecycle Faults

reboot() combines the kill with a timer-based restart, which is right for random attrition but cannot express a scripted sequence like “crash node X, let the workload make progress, then bring X back”. For that, FaultContext splits the primitives:

• ctx.crash(ip) force-kills the process without scheduling a restart. The node stays down — no application work runs — until an explicit restart.
• ctx.restart(ip) boots a fresh instance from the process factory, ending the hold-down.
• ctx.state() exposes the same per-iteration StateHandle that workloads and processes see, so the injector can wait on a deterministic workload milestone (an operation counter, a published flag) instead of a wall of sleeps.

#![allow(unused)]
fn main() {
async fn inject(&mut self, ctx: &FaultContext) -> SimulationResult<()> {
    let target = ctx.process_ips()[0].clone();
    ctx.crash(&target)?;                       // crash now, hold down

    // Wait until the workload has executed 100 operations.
    while ctx.state().get::<u64>("ops").unwrap_or(0) < 100 {
        ctx.time().sleep(Duration::from_millis(20)).await
            .map_err(|e| SimulationError::InvalidState(e.to_string()))?;
        if ctx.chaos_shutdown().is_cancelled() { return Ok(()); }
    }

    ctx.restart(&target)?;                     // explicit recovery
    Ok(())
}
}

Everything the script reads is per-iteration simulated state, so the crash target, milestone, and restart point replay exactly from the seed.

Fault Factories and Exploration

A custom injector is always registered through a factory, never as an instance: a mutated instance could not be rewound for every explored timeline, so the runner rebuilds the injector from the factory for every seed and every continuation.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(MyProcess))
    .workload_factory(|| Box::new(MyWorkload))
    .fault_factory(|| Box::new(ScriptedCrashInjector::new()))
    .chaos_duration(Duration::from_secs(30))
}

Like workload_factory, the factory builds a fresh injector for every root seed and every explored continuation timeline, so scripted fault sequences replay exactly from the root seed plus recipe — with in-process exploration (workers: 0) and forked workers alike.

Network Faults

• TCP, Not Packets
• The Fault Catalog
  • Latency Injection
  • Connection Drops
  • Black Holes
  • Clogging
  • Partial Writes
  • Partial Reads
  • Bit Flips
  • Clock Drift
  • Buggified Sleep Delay
  • Network Partitions
  • Connect Failures
  • Accept backlog
• Graceful vs Abort Disconnect
• Flow control
• The Swizzling Insight
• Configuration in Practice
• Swarm Testing: Less Is More

TCP, Not Packets

Moonpool simulates network faults at the TCP connection level, not the individual packet level. This is a deliberate design choice, inherited from FoundationDB. In practice, distributed systems rarely deal with individual packets. They deal with connections: connections that drop, connections that stall, connections that report success on one side and failure on the other. These are the faults that matter for application correctness.

Packet-level simulation (what TigerBeetle does) is useful for testing network stacks themselves. But for application-level distributed systems, connection-level faults exercise the code paths that actually fail in production: reconnection logic, request retries, leader election on disconnect, and state reconciliation after a partition.

The Fault Catalog

Moonpool’s ChaosConfiguration controls a wide range of network faults. Each fault is independently configurable and randomized per seed when using NetworkConfiguration::random_for_seed().

Latency Injection

Network timing uses configurable latency distributions. Bind, connect, and accept are genuinely asynchronous: each allocates an operation, schedules its completion, registers the caller’s waker, and remains pending until that exact operation becomes ready. Dropping one of these delayed futures cancels its scheduled work. Reads wait for data delivery when the receive buffer is empty. Writes accept available send-buffer capacity and schedule later byte delivery, applying write and link latency on the delivery path.

This distinction is observable. A timeout can beat bind, connect, or accept, and dropping the losing branch does not leave a stale completion behind. The simulator still picks every delay deterministically from the configured distribution, so the same seed reproduces the same race.

For tail latency testing, moonpool supports bimodal latency distribution, following FoundationDB’s halfLatency() pattern. In bimodal mode, 99.9% of operations use normal latency, but 0.1% experience latencies multiplied by 5x to 20x. This is how real networks behave: most requests are fast, but a small fraction hit GC pauses, cross-datacenter hops, or congestion.

Re-sampling latency per operation has a blind spot: no single connection ever stays slow. Real clusters break differently. One machine sits behind a degraded switch port and every message to it lags, run after run, while the rest of the fleet is healthy. That stably-slow peer is exactly what stalls a quorum or reorders a consensus round, and uniform per-operation jitter never produces it. So moonpool borrows FoundationDB’s SimClogging trick: set max_pair_latency to a range and each ordered IP pair draws one fixed latency at first contact, then carries it for the whole run. Off by default (ZERO..ZERO), it adds nothing. Turn it on and some pairs are permanently sluggish while others are quick, which is how you find the bug where the slow replica is the one holding the lease.

There is a second blind spot, and it is not a fault at all. A message between two processes on the same machine and a message between Paris and Montreal are both “one network operation” to the simulator, drawn from the same range. Real deployments do not work that way: loopback is tens of microseconds, a rack hop is hundreds, a cross-region hop is tens of milliseconds. If a replication protocol only ever sees uniform links, its cross-datacenter behavior is untested. NetworkConfiguration::link_latency fixes that. Give it a LinkLatencyConfig and each locality distance gets its own distribution:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .cluster(LocalityConfig::new(2, 2, 2, 1), || Box::new(MyNode::new()))
    // Same machine, same zone, same datacenter, cross datacenter.
    .link_latency(LinkLatencyConfig::default())
}

The engine classifies every IP pair through the .cluster() topology, samples the matching distribution once at first contact, and keeps that value for the whole run, in the same per-pair budget as max_pair_latency (when both are on, the two samples are summed). A pair where either side has no locality, a workload client for instance, gets nothing extra, so plain .processes() runs are unaffected. FoundationDB does not model this at all, it runs a single global latency distribution, but a library that wants to find cross-region replication bugs needs the distance to be visible in the timing.

Connection Drops

Random close injects spontaneous connection failures during I/O operations, at a configurable probability (default 0.001%). When triggered, 30% of closes are explicit (the caller gets an error) and 70% are silent (the connection just stops working). This ratio, taken from FoundationDB, tests both error-handling paths and timeout-based failure detection.

Fault decisions are sampled only when an operation can make progress. Re-polling a read with no buffered data or a write under backpressure returns Poll::Pending without consuming simulation randomness, so runtime-specific spurious polls cannot shift the seed’s later fault and latency choices.

A cooldown period prevents cascading closes from overwhelming the system. The goal is to test recovery, not to make the system completely inoperable.

Black Holes

A silent close still ends: the peer eventually reads EOF and learns something. A black hole never ends. When black_hole_probability fires, one direction of a connection (this side’s sends, the peer’s, or both, the same three-way draw random close makes) starts delivering nothing. The bytes are acknowledged into the sender’s window and vanish when they would land, the peer’s reads stay Pending with no data and no EOF, and a graceful close from the holed side never arrives either. Both ends see a connection that looks perfectly alive. Because the window is end-to-end (see Flow control) the swallowed bytes stay charged to the writer: a small request goes in and times out at the application, exactly the failure the fault exists for, while a bulk stream fills its window and its writer parks forever, as a TCP sender whose peer stopped acknowledging would. That is what a peer whose kernel keeps acknowledging into a frozen application looks like, or a middlebox that dropped its connection state, and it is the fault that finds a request without a timeout: nothing errors, nothing closes, and only the caller’s own deadline can notice. An application-level timeout, an HTTP/2 keep-alive ping, a heartbeat: whatever detects it in production is what has to detect it here.

#![allow(unused)]
fn main() {
let mut config = NetworkConfiguration::fast_local();
config.chaos.black_hole_probability = 0.0001; // per I/O, like random close
config.chaos.black_hole_cooldown = Duration::from_secs(5);
}

The coin is rolled on the same operations as random close, under the same progress rule, with its own cooldown, and it is recorded once as a black_hole fault event. Three properties keep it honest. A black hole is permanent for the connection: it never recovers, and the only way out is the one production has, a new connection. An abort still crosses: a process kill resets its connections and the peer finds out, as a kernel RST would once the host is back, so a hung peer is never mistaken for a dead one at the wrong time. And recovery mode stops new black holes but keeps the ones in force, like a closed connection; the quiet tail is where the reconnect has to happen. SimWorld::black_hole_connection(id, hole_send, hole_recv) is the scripted form for fault injectors, and the family is off by default, masked as NetworkFault::BlackHole.

Clogging

Write clogging stalls data delivery on a connection for a random duration (100-300ms by default). This simulates network congestion, TCP backpressure, and flow control contention. Code that assumes writes complete promptly will fail under clogging.

Partial Writes

Writes are truncated to a random length (0 to 1000 bytes by default), following FoundationDB’s approach. This tests TCP fragmentation handling and message framing logic. If your wire protocol assumes that a single write delivers a complete message, partial writes will break that assumption immediately.

Partial Reads

Reads are the mirror image: a read returns a random prefix of what is buffered (1 to 1000 bytes by default), and the rest stays buffered for the next read. FoundationDB’s Sim2Conn does the same 50/50 split on the receiver. The one asymmetry with writes is that a partial read always delivers at least one byte when data is available, because a zero-byte read is how the stream signals end-of-file. This exercises the reassembly side of your wire protocol: code that calls read once and assumes it got a whole message will drop bytes the moment a short read splits a header or a length prefix.

Bit Flips

Packet data is corrupted with random bit flips at low probability (0.01% by default). The number of flipped bits follows a power-law distribution between 1 and 32. This tests checksum validation and corruption detection. Without bit-flip injection, corruption bugs only surface in production when cosmic rays or faulty NICs flip bits for you.

Clock Drift

Simulated clocks can drift by up to 100ms (configurable) between nodes. This tests anything that depends on time agreement: lease expiration, distributed consensus, TTL handling, and cache invalidation. Clock drift is subtle because the code often works correctly with small drift and fails catastrophically when drift exceeds a threshold.

Buggified Sleep Delay

FoundationDB occasionally stretches a process timer with a power-law delay to exercise timeouts and timing-dependent races. Moonpool applies the same fault to TimeProvider::sleep. In a SimulationBuilder campaign it is active only during the configured chaos_duration: setup is clean, and sleeps scheduled at or after the chaos deadline receive exactly their requested duration. This makes the post-chaos quiet tail a real recovery period, so a claim such as “the cluster converges within five seconds after faults stop” is measuring the cluster rather than a fault that silently continued running.

The gate is checked before either buggified-delay RNG draw. A configuration that disables the fault, a NetworkFaultMask without BuggifiedDelay, and sleeps outside the chaos window therefore leave the counted simulation RNG untouched. Direct low-level SimWorld construction has no campaign phases and retains the historical whole-run behavior, unless you call SimWorld::enter_recovery_mode() yourself.

The same quiet-tail guarantee holds for every other network fault family, not just this one: at the chaos_duration cutoff the runner calls SimWorld::enter_recovery_mode(), which zeroes the whole ChaosConfiguration fault surface and heals any partition in force. What it does not do is repair damage: bytes already corrupted stay corrupted, a connection the application already saw close stays closed, and a link that has already sampled its permanent extra latency stays slow. See the configuration appendix for the full boundary contract.

Network Partitions

Moonpool supports seven partition strategies:

Partitions have configurable probability and duration. They can be programmatic (via FaultContext::partition) or automatic (via partition_probability in the chaos config).

At the lower-level SimWorld API, partition_pair(from, to, duration) adds one directed pair entry. Call it for both directions to create a bidirectional pair cut. A single restore_partition(from, to) deliberately removes both directed entries, preventing a half-healed pair during recovery. Send-wide and receive-wide partitions remain separate and expire through their targeted restore events.

A partition never punches a hole in an established connection. A queued send whose direction is cut stalls: the bytes stay at the front of the send buffer, the writer keeps seeing backpressure, and the stream resumes in order the moment the partition heals, whether that is at its deadline or through an early restore_partition / FaultContext::heal_partition. The peer therefore either reads the original bytes in order, or sees the connection fail. It never reads a later chunk in place of one that was silently dropped, which for a framed protocol (h2 on the same connection) would corrupt the frame that follows. This mirrors FoundationDB’s SimClogging, where a clogged pair adds delay and only an explicit disconnect fails the connection.

The same holds for bytes that are already on the wire. A chunk that left the send buffer has a delivery time sampled from the write latency, and that time was sampled before the partition existed; the partition still decides its fate. Every byte moves through four places, in order:

application write
    │  poll_write takes window
local send queue          stalls under a cut
    │  ProcessSendBuffer samples latency
in flight                 frozen under a cut, re-timed by the heal
    │  Delivery event
peer receive buffer       the peer's now; nothing takes it back
    │  poll_read returns window
application read

When a cut lands, every chunk (and any FIN) in flight on the cut direction is frozen where it is. Its Delivery event still fires at the old time and does nothing. When every partition blocking the direction has healed, each frozen item is re-timed by exactly the time the direction spent cut, so a cut of D delays every byte it caught by D, the stream keeps its order (a chunk sent after the heal lands behind the last frozen one), and no randomness is drawn: the latencies already sampled are reused. A response written at t=0 with 100ms of latency does not slip through a partition that starts at t=50ms; it lands at 100ms + (heal − 50ms). Directed, send-wide, and receive-wide cuts all freeze the same way, an explicit heal and the recovery-mode heal both thaw the same way, and a FIN is an item like any other: the peer sees the last bytes and then EOF, never EOF first.

The first three strategies are IP-blind: they pick nodes out of a hat. IsolateZone and IsolateDatacenter read the .cluster() topology instead, so the cut lands exactly where a real one would, on the boundary that shares a switch or a region. Without a topology they fall back to Random selection, which keeps them safe to draw for any seed.

The asymmetric pair is worth dwelling on. A node whose sends are blocked still hears every heartbeat from the cluster, so it happily believes it is healthy while everyone else marks it dead. Systems that infer liveness from “I can see you” rather than “you can see me” split brains here. Both arms record their own fault events (SendPartitionCreated, RecvPartitionCreated) on the timeline, so an invariant can correlate application behavior with the exact one-way cut that caused it.

Connect Failures

Connection establishment can fail in two modes, following FoundationDB’s SIM_CONNECT_ERROR_MODE:

• AlwaysFail: Every buggified connect attempt returns ConnectionRefused
• Probabilistic: 50% fail with ConnectionRefused, 50% hang forever (never complete)

The hanging mode is particularly nasty. Code that does not implement connect timeouts will block forever, which is exactly the kind of bug simulation should find.

Accept backlog

Each listener holds at most 128 unaccepted connections by default (NetworkConfiguration::accept_backlog_capacity). The builder’s .accept_backlog_capacity(n) sets that capacity for every seed; n must be positive. A full backlog parks connect() calls in FIFO order. Each resumes only after an accept() has returned a stream, including its configured accept latency. An endpoint reserved by an accept still occupies its slot, and canceling that accept returns the endpoint to the backlog without freeing the slot. Canceling a parked connect discards its unpublished connection pair. Aborting a parked connect’s pair wakes it to an error before it can publish. Resetting an unaccepted endpoint frees its slot immediately and fails any accept that reserved it. Dropping a listener aborts its backlog and fails its parked connects and accepts, even if a new listener later binds the same address.

Graceful vs Abort Disconnect

When a connection closes, moonpool models two distinct TCP behaviors:

Graceful close implements TCP half-close semantics. The closing side marks its send direction as closed and puts a FIN on the wire behind every byte still queued or in flight, so it lands after all of them and is held by a partition with them. The remote side continues reading buffered data normally and sees EOF only after the FIN arrives. A drained stream drop closes gracefully; dropping a stream with unread received bytes aborts instead, because the application discarded data the peer had sent.

Write shutdown is what AsyncWrite::close does, on both backends: the production provider’s compat wrapper maps it onto tokio’s poll_shutdown, that is shutdown(SHUT_WR), and the simulated stream does the same. The FIN goes out behind the queued bytes and further writes fail with BrokenPipe, but the read half stays open and nothing is discarded, so the reply to an EOF-delimited request still lands and is still readable. Dropping the stream is what closes the connection, and a drop after a shutdown adds the receive half without sending a second FIN.

Abort close immediately terminates both directions. No FIN, no buffer drain: the send queue, the flight, and the unread bytes are gone. The remote side gets a connection reset error on its next read or write, and a writer parked on a full window is woken to that error rather than left waiting on credits nothing can return. This models a crashed process or a force-killed connection.

Flow control

Each direction of a connection has one byte window (NetworkConfiguration::tcp_send_window_bytes, SimulationBuilder::tcp_send_window_bytes, 64 KiB by default). The window is taken when poll_write accepts bytes and returned only when the peer’s application reads them, byte for byte:

outstanding = queued + in flight + unread at the peer (+ swallowed by a black hole)
outstanding ≤ window
available   = window − outstanding

Moving bytes from the send queue onto the wire, or from the wire into the peer’s receive buffer, releases nothing; they are the same outstanding bytes in a different place. A write accepts min(len, available) (a short write, like write(2)) and parks with Poll::Pending when available is zero; the peer’s next read wakes it with exactly the bytes it consumed, so a partial read of 137 bytes lets the writer send 137 more. A reader that stops reading therefore backs its writer up end to end, even after every byte has reached the receive buffer, which is the TCP behavior behind “the server kept streaming into a client that had stopped consuming” and the pipeline that deadlocks on its own replies. A partition returns no credit (delay is not consumption), a black hole never returns it, a peer that closes with bytes unread returns them, and an abort wakes the parked writer to the error.

Backpressure is entropy-neutral: a Poll::Pending on a full window, the wake that follows a read, and every move between the three places consume no simulation randomness, so a run that blocks and resumes replays draw for draw. Congestion control, retransmission, and segment boundaries are out of scope: the model is a reliable ordered byte stream with a bounded number of bytes outstanding, which is what exposes the distributed-system behaviors a simulation is after.

The distinction matters because many protocols depend on reading remaining data after the peer signals shutdown. HTTP/1.1 relies on this for chunked transfer encoding. gRPC uses it for trailing metadata. If your simulation only models abort closes, you will miss bugs in graceful shutdown handling.

The Swizzling Insight

One finding from FoundationDB’s simulation work deserves special mention: restoring network connections in reverse order of disconnection finds more bugs than restoring in forward order. This is called swizzling. As Will Wilson described it: “for reasons that we totally don’t understand, this is better at finding bugs than normal clogging.”

Why does this work? Forward restoration tests the easy case: the first connection dropped is the first restored, so recovery happens in the order the system expects. Reverse restoration forces the system to handle partial recovery where the most recently dropped connection comes back first. This creates asymmetric states that exercise recovery logic in ways no developer would think to test manually.

This is the kind of insight that only falls out of running thousands of simulations. No one sat down and reasoned that reverse-order restoration would find more bugs. The simulator tried both and the data spoke for itself.

Configuration in Practice

For maximum chaos testing, use NetworkConfiguration::random_for_seed(). This randomizes all parameters based on the simulation seed, so different seeds test different network conditions:

#![allow(unused)]
fn main() {
let network_config = NetworkConfiguration::random_for_seed();
}

For fast unit tests where network chaos would just slow things down, use NetworkConfiguration::fast_local():

#![allow(unused)]
fn main() {
let network_config = NetworkConfiguration::fast_local();
// Minimal latencies, all chaos disabled
}

For targeted testing of specific fault types, start with defaults and override:

#![allow(unused)]
fn main() {
let mut config = NetworkConfiguration::default();
config.chaos.partition_probability = 0.05;
config.chaos.partition_strategy = PartitionStrategy::IsolateSingle;
// Everything else at defaults
}

When a builder should keep its per-seed Random or Swarm profile but exclude one environmental fault family, use a NetworkFaultMask. For example, this campaign retains clogs, partitions, short reads and writes, closes, latency, and the other sampled faults while disabling wire-data corruption:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .enable_chaos([Chaos::Network(ChaosMode::Swarm)])
    .network_fault_mask(
        NetworkFaultMask::all().without(NetworkFault::BitFlip),
    )
    .enable_exploration(exploration_config)
}

The builder samples the complete profile first, applies buggify knob changes, then applies the mask immediately before creating SimWorld. Masking performs no RNG draws. The sampled values and Swarm subset therefore stay identical to an unmasked run, the in-run RNG call count does not move, and an exploration recipe replays deterministically. With the default all-family mask, this step is a no-op.

Swarm Testing: Less Is More

There is a subtle trap hiding inside random_for_seed(). It sets every fault family to a random non-zero probability. Clogging is a little bit on, partitions are a little bit on, bit flips are a little bit on, all at once, on every seed. That sounds thorough. It is actually the opposite.

The problem is called passive suppression, and it comes straight from the “Swarm Testing” paper by Groce and colleagues (ISSTA 2012), popularized by Will Wilson’s Antithesis talks. When every feature is always slightly active, the features crowd each other out. To find a clogging bug you often need clogging cranked hard, for a sustained stretch, with nothing else interfering. If partitions keep tearing down the connection you were trying to clog, you never drive clogging deep enough to hit its bug. The undirected default explores a narrow band in the middle of the configuration space and never visits the extremes where bugs actually live. In the paper, undirected swarm testing found 42% more distinct compiler crashes than a heavily hand-tuned default.

The fix is counterintuitive: for each run, turn off a random subset of fault families entirely. Each seed enables roughly half the families and fully disables the rest. One seed is clogging-only. Another is partition-plus-bitflip. Another is the all-off no-fault baseline. Across many seeds you cover the single-family extremes that the all-on config can never reach.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    // each seed enables a random subset of network fault families
    .enable_chaos([Chaos::Network(ChaosMode::Swarm)])
    // ...
}

ChaosMode::Swarm builds on random_for_seed(), then masks each of the eight network fault families to off with 50% probability: clog, partition, bit-flip, random-close, connect-failure, clock-drift, buggified-delay, and permanent pair latency. The all-off subset is allowed on purpose, because a pure no-fault run is a useful, valid config. The same enable_chaos call swarms storage faults (Chaos::Storage) and the attrition reboot regime (Chaos::Attrition) the same way.

The interesting engineering detail is where the randomness comes from: the same place as everything else. The subset is drawn from the simulation’s one stream, seeded for the iteration, before the world is built and before any process runs, as a fixed number of draws per family (one coin each, consumed whether or not the family was on). Same seed, same subset, every time: a replay, fork-explorer recipes included, rebuilds the subset from the seed before the counted run starts. There is no separate configuration RNG: moonpool draws its own decisions from the stream it hands to the code under test.

You already met a tiny version of this idea earlier in the chapter. Connect failures have always had a one-in-three chance of being fully disabled per seed. Swarm testing simply generalizes that one disabled branch to every fault family at once.

Storage Faults

• Disks Lie
• The Fault Taxonomy
  • Read Corruption
  • Write Corruption
  • I/O Errors
  • Crash Faults (Torn Writes)
  • Short Transfers
  • Lost Directory Entries
  • Misdirected Writes
  • Misdirected Reads
  • Phantom Writes
  • Sync Failures
• Performance Simulation
• Dynamic Disk Degradation Episodes
• Disk Failure: I/O That Never Completes
• Exact Asynchronous Operations
• StorageEngine Ownership
• Independent Open Handles
• Per-Process Storage Configuration
• Crash and Wipe Operations
• Configuration in Practice
• Positioned I/O, Direct I/O, and Alignment
• File Durability Is Not Directory Durability
• The Barrier-Bounded Crash Model
  • The Lost-Synced-Write Oracle
  • Aiming Faults
• Blocks Are a View, Not a Device

Disks Lie

Every database developer eventually learns this lesson. write() returns success, but the data never reaches the platter. fsync() completes, but the drive’s firmware lied about flushing its cache. A cosmic ray flips a bit in DRAM between computing a checksum and writing to disk. A firmware bug directs a write to the wrong sector.

These are not hypothetical failures. TigerBeetle’s documentation catalogs them with references to real incidents: LSE studies showing 8.5% of SATA drives developing silent corruption, firmware bugs causing misdirected writes across drives in a RAID array, and enterprise SSDs that acknowledge fsync without actually flushing.

Moonpool’s storage fault injection is modeled on TigerBeetle’s fault taxonomy. The goal is to test that your data integrity code actually works, not by hoping these faults happen in production, but by making them happen deterministically in simulation.

The Fault Taxonomy

Moonpool’s StorageConfiguration controls the fault families below. All are off by default, and a family whose probability is zero draws no randomness at all, so a run with faults disabled stays byte-for-byte deterministic.

Read Corruption

A read operation returns wrong data. The file contains correct bytes, but the value returned to the application has been corrupted. This models ECC failures, DRAM bit flips, and controller firmware bugs.

What it tests: Checksum validation on reads. If your system trusts data without verifying checksums, read corruption will silently propagate bad data through the system.

Write Corruption

A write operation stores wrong data. The application writes correct bytes, but what lands on disk is different. This models controller bugs, bad sectors, and write buffer corruption.

What it tests: Read-after-write verification and end-to-end checksums. Systems that compute checksums before writing and verify after reading will detect write corruption. Systems that do not will store garbage.

I/O Errors

A read or a write fails outright: the device reports that it could not serve the request. This is an operating condition, and it is a different thing from a read that succeeds and returns corrupt bytes — read_eio_probability and write_eio_probability are separate families from the two above for exactly that reason.

What it tests: Error paths. Code that treats an error as “corrupt data” — or worse, ignores the result — is wrong on real disks, and both halves have to be driven.

Crash Faults (Torn Writes)

The system crashes mid-write. Some sectors are written, others are not. This models power failures, kernel panics, and OOM kills during I/O.

What it tests: Write-ahead logging, atomic write protocols, and crash recovery. Any system that performs multi-step writes without a journal or atomic commit is vulnerable to torn writes. See The Barrier-Bounded Crash Model below for the shapes a crash can leave behind.

Short Transfers

A read or write moves fewer bytes than asked for and returns the count. This is ordinary POSIX behaviour that callers routinely forget.

What it tests: Whether a caller loops. Code that treats a read_at return value as “all of it” silently drops data, which short_transfer_probability makes happen on demand.

Lost Directory Entries

A create, delete, or rename that was never followed by a directory sync does not survive a crash — however thoroughly the file’s contents were synced.

What it tests: Whether an engine syncs the directory holding its files, not just the files. See File Durability Is Not Directory Durability below.

Misdirected Writes

A write lands at the wrong location. The application writes to offset A, but the data ends up at offset B. This models firmware bugs and controller errors that TigerBeetle specifically documents as real-world failures.

What it tests: Per-record addressing verification. Systems that embed the expected offset in each record’s header can detect misdirected writes. Systems that trust the filesystem to put data where it was told will read the wrong records.

Misdirected Reads

A read returns data from the wrong location. The application reads offset A, but gets the contents of offset B. Same root causes as misdirected writes, from the read side.

What it tests: Same as misdirected writes. Checksums that include the expected position catch this.

Phantom Writes

A write appears to succeed but does not persist. The write() call returns Ok(n) and even fsync() completes, but the data is gone after a restart. This models drive firmware that lies about durability.

What it tests: Durability verification after recovery. Systems that write, sync, crash, and restart must verify that their data survived. Phantom writes ensure this verification logic works.

Sync Failures

sync_all() returns an error. This models disk errors during flush, full disks, and I/O errors that only manifest at sync time.

What it tests: Error handling in durability-critical code paths. Many systems call fsync() but do not check the return value. In simulation, a sync failure is a loud signal that your error handling has a gap.

Performance Simulation

Beyond faults, moonpool simulates realistic storage performance characteristics:

These parameters ensure that storage-heavy code paths experience realistic timing, which is important for testing timeout logic and concurrent I/O patterns.

Dynamic Disk Degradation Episodes

Steady-state timing is a lie of a different kind. Real disks do not degrade at a fixed rate. They degrade episodically: a garbage-collection pause freezes I/O for 100-500ms, a thermal event throttles throughput for seconds, firmware stalls under load. These episodes are what trigger the interesting failures: timeout cascades, backpressure collapse, and recovery bottlenecks that a constant 150 MB/s never produces. FoundationDB models this with its DiskFailureInjector, and moonpool borrows the idea.

Two episode kinds sit on top of the steady-state formula, both off by default and scoped per process (per owning IP):

Before each read, write, or sync, the owning process’s episode state machine runs: an expired episode clears, an active one stays, and an idle disk rolls the dice to enter a new episode. The episode is keyed by the process IP, not the file, because real degradation is a property of the physical disk: a garbage-collection pause or a firmware stall hits every file the machine owns at once. So one episode freezes all of a process’s open files together, and a second machine in the same simulation keeps running at full speed. That correlated freeze, where a whole machine’s I/O completes at the same moment the window lifts, is exactly the backpressure spike that overflows queues in real systems.

#![allow(unused)]
fn main() {
// A disk that stalls on every operation for 50ms
let stalling = StorageConfiguration {
    disk_stall_probability: 1.0,
    disk_stall_duration: Duration::from_millis(50),
    ..StorageConfiguration::fast_local()
};
}

The key property is that a disabled disk never touches the random number stream. When both probabilities are zero, the state machine returns before drawing any randomness, so steady-state runs stay byte-for-byte deterministic. Chaos runs enable low-rate episodes through random_for_seed(), swarm masking, and buggify knob spikes, the same machinery every other storage fault family uses.

Disk Failure: I/O That Never Completes

A stall is a disk that answers late. A failed disk is a disk that never answers. Every read, write, sync, or set_len issued to it after the failure is accepted and stays Pending for the rest of the run. Nothing returns an error, nothing times out on the disk’s behalf, and the world’s event queue is empty while the caller waits. FoundationDB models this with failedDisk, where waitUntilDiskReady() returns Never(); moonpool does the same through disk_failure_probability, a per-operation coin that is off by default.

This is the storage fault that finds a missing timeout. Code that awaits an fsync inline, with no deadline and no way for the rest of the process to notice, hangs. The runner’s stall detector then sees a world with no events and no progress, requests a graceful shutdown, and reports a deadlock. That verdict is the point: a disk that stops answering is something a real machine does, and a process that cannot survive it has a bug.

#![allow(unused)]
fn main() {
// A disk that fails on its first operation
let failing = StorageConfiguration {
    disk_failure_probability: 1.0,
    ..StorageConfiguration::fast_local()
};
}

The failure is keyed by the owning process, like an episode: one failure hangs every file the machine owns. Two rules keep it a fault a system can be expected to survive rather than a way to make a run unwinnable:

• At most one disk is failed at a time. The coin is not drawn while another disk is failed, so a quorum system never loses more than one member to a hung disk. This is the family’s floor.
• A crash or wipe of the owning process replaces the disk. The operations it parked fail with OperationInterrupted along with the rest of the process’s in-flight I/O, the failure is cleared, and the budget is free for the next disk to draw the coin. Until then the failure has no expiry: recovery mode stops new failures but keeps one already in force, exactly as it keeps a killed process.

A scripted fault injector can fail a disk directly with SimWorld::fail_disk_for_process(ip). That path consumes no randomness and ignores the one-at-a-time budget, because the injector owns its own budget. A parked operation samples no latency and enters no episode, so a hung I/O never moves the seed’s random stream either.

Exact Asynchronous Operations

Read, write, sync, and set-length calls schedule work and return Poll::Pending. Each submission receives a unique OperationId. Its StorageEvent carries that exact ID, the submitting handle, and the expected operation kind. StorageEngine keeps an explicit pending entry and later an explicit Result for the same ID. A missing entry is an invalid operation, not implicit success.

Network establishment is asynchronous too. Bind, connect, and accept also need the scheduler to advance. Established stream writes can accept bytes into their send buffer immediately, and reads can complete immediately when bytes are already buffered. Do not use the old rule that all network operations are ready while all storage operations are pending.

SimulationBuilder::run() already interleaves the deterministic executor and the scheduler for normal process and workload tests. If you write a low-level provider test, drive its future and SimWorld together on Moonpool’s executor:

async fn drive<F: Future>(sim: &mut SimWorld, future: F) -> F::Output {
    futures::pin_mut!(future);
    futures::future::poll_fn(|cx| match future.as_mut().poll(cx) {
        Poll::Ready(output) => Poll::Ready(output),
        Poll::Pending if sim.has_pending_events() => {
            sim.step();
            cx.waker().wake_by_ref();
            Poll::Pending
        }
        Poll::Pending => Poll::Pending,
    })
    .await
}

let mut executor = moonpool_sim::executor::Executor::new(seed);
executor.block_on(async move {
    let provider = sim.storage_provider(ip);
    drive(&mut sim, async move {
        let mut file = provider.open("test.txt", OpenOptions::create_write()).await?;
        file.write_all(b"hello").await?;
        file.sync_all().await
    })
    .await
})?;

The helper polls the provider future, steps one scheduled event when necessary, and lets the event’s waker make the future runnable again. If the future is pending with no scheduled path to progress, the executor reports a deadlock with the seed.

StorageEngine Ownership

StorageEngine owns the whole simulated disk surface: persistent file data, path lookup, open handles, default and per-process configurations, disk episodes, pending operations, completed results, fault decisions, and storage wakers. SimWorld only schedules the engine’s requested events and cancellations, records returned faults, and invokes the returned wake batch after releasing the world lock.

That split keeps operation ordering explicit. Completion events never search for the oldest operation with the same file and kind. Concurrent operations on one file can finish in scheduler order and wake only their own callers. Dropping a future cancels its schedule and removes its pending result state.

Independent Open Handles

Persistent contents belong to a file record. Cursor position, access options, closed state, and pending-operation IDs belong to an open handle. Opening the same path twice therefore creates two handles over one file:

• Seeking or reading through one handle does not move the other handle’s cursor
• Read-only and write-only permissions are enforced per handle
• Dropping or closing one handle cancels its pending work without deleting the persistent file or closing sibling handles
• Append handles resolve the current end of the shared file when scheduling a write

This matches the behavior applications expect from real file descriptors and makes concurrent-handle races meaningful rather than accidentally sharing one cursor.

Per-Process Storage Configuration

Storage fault injection is scoped per process. Each process is identified by its IP address, and you can assign different StorageConfiguration to different processes. This models real-world heterogeneous hardware: one node with a flaky SSD, another with a healthy disk.

The engine maintains a global configuration as the default plus optional per-process overrides. For each file operation it resolves the profile by the file owner’s IP, then updates that process’s disk-degradation episode before calculating latency and faults.

Set per-process configuration through SimWorld:

#![allow(unused)]
fn main() {
// Give process 10.0.1.2 a degraded disk
let degraded = StorageConfiguration {
    read_corruption_probability: 0.01,  // 1% read corruption
    write_corruption_probability: 0.005,
    ..StorageConfiguration::default()
};
let degraded_ip = "10.0.1.2".parse().expect("valid process IP");
sim.set_process_storage_config(degraded_ip, degraded);
}

Every persistent file is tagged with its owning process IP. Fault injection decisions such as corruption, latency, and sync failure use that owner rather than a single global profile.

Crash and Wipe Operations

Two SimWorld methods handle storage lifecycle during process failures:

simulate_crash_for_process(ip, close_files) applies crash behavior to the process’s persistent files, including torn-write fault injection. Every pending read, write, sync, or set-length operation for those files completes with an interrupted error and wakes its exact waiter. When close_files is true, the affected handles are also marked closed.

wipe_storage_for_process(ip) deletes all persistent storage owned by the given process and invalidates its handles. This models total disk failure or replacing a machine. The CrashAndWipe reboot kind calls both: crash first, then wipe. The wipe happens immediately.

Global simulation shutdown follows the same explicit-result rule. It cancels every pending storage schedule, records a shutdown error for each operation, marks handles closed, and returns all waiters for wakeup. No task is left parked behind a synthetic timer, and no crash-cleared operation can report Ok(()).

Configuration in Practice

For chaos testing, use StorageConfiguration::random_for_seed(). This randomizes both performance parameters and fault probabilities based on the simulation seed:

#![allow(unused)]
fn main() {
let storage_config = StorageConfiguration::random_for_seed();
// Fault probabilities: 0.001% to 0.1% (low but present)
// IOPS: 10K to 100K
// Bandwidth: 50-500 MB/s
}

For fast unit tests, use StorageConfiguration::fast_local():

#![allow(unused)]
fn main() {
let storage_config = StorageConfiguration::fast_local();
// 1M IOPS, 1 GB/s, 1us latencies, zero faults
}

The fault probabilities in random_for_seed() are intentionally low (0.001% to 0.1%). Storage faults at higher rates would prevent the system from making progress. The goal is a steady trickle of faults that occasionally exercises corruption detection and recovery, not a deluge that makes every I/O fail.

Positioned I/O, Direct I/O, and Alignment

A journal or a pager does not want a seek cursor. It wants to read page 7 and write page 12, possibly at the same time, from several tasks, without any of them disturbing the others’ idea of “where I am”.

StorageFile::read_at(offset, buf) and write_at(offset, buf) are that primitive. They take &self, address the offset literally, and never touch the stream cursor, so non-overlapping ranges of one file can be read and written concurrently. They are read and write, not read_exact and write_all: both return the number of bytes moved, and a read returns 0 at end of file. short_transfer_probability makes the simulator move a non-empty prefix instead of the whole buffer, deterministically, so a caller that assumes a full transfer can be driven red.

Everything a database needs from an open is on OpenOptions, not on a second provider:

#![allow(unused)]
fn main() {
let file = provider
    .open(
        "db/pages",
        OpenOptions::read_write().direct_io(DirectIo::Required),
    )
    .await?;
}

• DirectIo::Disabled is ordinary buffered I/O.
• DirectIo::Optional asks for uncached I/O and accepts a documented fallback where the filesystem refuses it; file.is_direct_io() says which one you got.
• DirectIo::Required fails the open rather than downgrading silently, and opens a file that already exists — it never creates one.

That last rule is a deliberate boundary, and the native open shows why. A refused O_CREAT | O_DIRECT still leaves the file behind: the file is created during path resolution and O_DIRECT is rejected afterwards. Cleaning that up is a namespace protocol — stage an inode elsewhere, publish the name — not a file open, and it is not the provider’s to run on your behalf. Whoever owns the file’s format owns that protocol, because only they know whether the name has to be durable, what a half-created file means, and how recovery finds one. So a journal bootstraps its own file, in the order its recovery expects:

#![allow(unused)]
fn main() {
// 1. create it with an ordinary open, 2. make the file and its name durable
provider.create_dir_all("db").await?;
provider.sync_dir(".").await?; // durable name for the db directory
let created = provider.open("db/wal", OpenOptions::create_new_write()).await?;
created.sync_all().await?;
drop(created);
provider.sync_dir("db").await?;

// 3. now require the capability of the file that exists, 4. format and recover
let wal = provider
    .open("db/wal", OpenOptions::read_write().direct_io(DirectIo::Required))
    .await?;
}

A Required open of an existing file does honour truncate, and applies it only after direct I/O is secured, so a refused capability check cannot have modified the file.

Direct I/O is not durability. An uncached write is still only visible until a sync_all() / sync_data() makes it durable, exactly like a buffered one. What it does buy is that a read comes from the device rather than from a page the kernel may have marked clean after a failed flush (Rebello, ATC’20).

file.constraints() reports what the open demands, as three independent numbers — offset alignment, length alignment, and memory alignment — because a device may constrain them differently. None of them is your block or page size, and none is a crash-atomicity unit. Every transfer is checked against them, so a misaligned direct-I/O call fails with InvalidInput in simulation exactly as it earns EINVAL from a kernel. AlignedBuf is the one aligned-buffer facility in moonpool; layers above it reuse it rather than growing their own.

The numbers are discovered, not assumed: production asks the kernel (statx(STATX_DIOALIGN)) for the device’s real requirement, falls back to the page size as a documented bound where the kernel will not say, and refuses to offer direct I/O at all where it can do neither — advertising an alignment that turns out to be too weak would hand callers a buffer the device rejects. The simulation reports the disk geometry it was configured with.

The stream API and alignment are mutually exclusive. A file with I/O constraints refuses AsyncRead / AsyncWrite with ErrorKind::Unsupported, in both backends, because a shared cursor cannot be kept aligned: a stream transfer starts wherever the cursor happens to be, and a short one leaves it somewhere no subsequent request may start from. Seeking still works — it moves a cursor without transferring anything — and read_at / write_at are what such a file offers instead.

File Durability Is Not Directory Durability

create_dir_all("db")   // makes the directory visible
sync_dir(".")          // makes the db name durable
open("db/wal", create)   // creates a directory entry
write(..)                // fills the file
sync_all(file)           // the bytes are durable

After that sequence a crash may leave no db/wal at all. The bytes were made durable; the name that reaches them was not. Engines that get this right (SQLite, PostgreSQL, LMDB) follow every create, delete, and rename with a directory sync, and StorageProvider::sync_dir(path) is that call. It lives on the provider, beside rename and delete, because it concerns the filesystem namespace rather than the contents of an open file.

The simulator keeps two namespaces: the visible one, which a create, delete, or rename changes immediately, and the durable one, which only sync_dir promotes into. create_dir_all creates visible directories, but each new directory’s name becomes durable only when its parent is synced. Syncing db can save db/wal, but cannot save the new db name without syncing .. No file or child directory survives if its parent name disappears on a crash. On a crash each divergence between the visible and durable names resolves independently under unsynced_dir_entry_loss_probability: a created name may not be there, a deleted one may be back. The family is off by default and draws no randomness while off, so a crash keeps the namespace it had unless a test asks otherwise.

The virtual relative root . and absolute root / are distinct, with no host filesystem behind either. Paths normalize aliases such as ./db/wal, but every directory component is checked before .. is folded: an open of missing/../wal still fails because missing is not a directory.

When an unsynced rename rolls back, the recovered name is also the file’s fault coordinate for that crash and later I/O. Independent name outcomes can rarely leave both the old and new names pointing to one image; the simulator uses the first surviving name in lexical order for path-based fault masks and reports.

The Barrier-Bounded Crash Model

A sync is the only barrier a file has, and everything written since the last one is up for grabs.

Every simulated file keeps two images: the durable bytes as of its last successful sync, and the visible bytes reads observe. A write dirties sectors in the visible image; a sync commits them. On a crash each dirty sector resolves independently:

An occasional fully clean crash (clean_crash_probability, 10% — FDB’s number) and an occasional correlated rollback of a contiguous run (erase-block damage, Zheng FAST’13) round out the shapes, and an unsynced length change resolves the same way. This is what makes “the later record landed while the earlier one is partial” — the case journal recovery exists to survive — actually reachable.

Two properties are worth stating plainly, because code hides behind their opposites:

• A lost sector reads the fill pattern, which is zeros on some files and garbage on others (garbage_fill_probability, drawn per file). Zeros are the dangerous real case (SATA RZAT, NVMe DLFEAT, unwritten extents), so recovery code that infers “never written” from “reads as zero” has to be driven red on both.
• Damage is deterministic. A corrupted sector returns the same wrong bytes on every read; a retry never heals it.

The Lost-Synced-Write Oracle

At every sync, each committed sector is stamped with a CRC of the content the caller was told is durable (FoundationDB’s AsyncFileWriteChecker pattern). After a crash, a stamped sector that no longer matches is a simulator bug and fails the run loudly — unless the opt-in barrier-violation family is armed (barrier_violation_probability > 0), in which case a sync occasionally lies about a sector and the oracle flips to must-detect mode, reporting the loss as an expected LostSyncedWrite. That family is how a consumer proves cluster-level recovery heals a single lying disk (the fsyncgate class).

Aiming Faults

Random fault families are gated by a caller-provided eligibility mask, (path, sector) -> bool, so a replication-aware harness can enforce “never damage all copies of one record” without moonpool knowing what a replica is (TigerBeetle’s ClusterFaultAtlas pattern). Rolls happen before the mask is consulted, so installing one never shifts the random stream.

Directed tests reach for the targeted API on SimWorld instead: corrupt_file(path, sectors), fail_file_with_eio(path, sectors, target), clear_file_eio, and — to test the oracle itself — corrupt_durable_out_of_band. What each crash did is available from take_storage_crash_reports(), and every fault injected from take_storage_fault_records().

A failed sync_dir reports an I/O error to its caller and emits a storage_sync_fault event on the simulation fault timeline. An invalid or missing directory fails before the sync fault coin is considered.

Fault coordinates are file plus flat sector offset. There is no region and no sub-file namespace: what the bytes at an offset mean belongs to the format written on top.

Blocks Are a View, Not a Device

Storage engines address data in fixed-size blocks. BlockFile<F> is that arithmetic, and only that arithmetic:

#![allow(unused)]
fn main() {
let file = provider.open(path, options).await?;
let blocks = BlockFile::new(file, 4096)?;

blocks.grow_to_blocks(1024).await?;
let mut page = blocks.buffer(1);     // aligned for this file
blocks.write_blocks(7, page.as_slice()).await?;
blocks.sync().await?;
}

read_blocks fills its buffer completely or fails UnexpectedEof, and write_blocks writes every byte or fails — looping over the partial transfers the file below is allowed to return is the job this layer exists to do.

What it deliberately is not:

• It wraps one already-open file. It takes no path, stores no path, and cannot open, create, rename, or delete anything. There is no BlockFile::open and no BlockProvider: an operation that needs a path is an operation for StorageProvider.
• It is not a directory, several files, a region table, a manifest, a namespace, or a virtual disk. A database’s logical zones — journal, superblock, pages — are byte offsets in one file. The file API does not grow a superblock-file because the format has a superblock.
• The block size is the caller’s unit. It must be a multiple of the file’s I/O alignment, and it is neither that alignment nor a crash-atomicity unit: a two-block write tears across a crash like any other.
• It is not durability. write_blocks returns when bytes are visible; sync is what makes them durable.

Because it is generic over StorageFile, there is one block layer over two backends — the simulated file and the real one — rather than a production block device and a separate simulated one.

Assertions: Finding Bugs

• Why Assertions Never Crash
• Dual Purpose
• The Reporting Pipeline
• What Comes Next

Imagine your simulation catches a process violating an invariant. You have two choices: crash immediately, or record the violation and keep going. In traditional testing, we crash. The logic seems obvious: something went wrong, stop everything, report the failure.

But that logic is wrong for simulation testing.

Why Assertions Never Crash

Here is the core insight: the first thing that goes wrong is rarely the worst thing that goes wrong. When a process violates a consistency invariant, that violation is a bug. But if we abort the simulation right there, we will never see what happens next. Maybe that corrupted state propagates to two other processes. Maybe the replication protocol silently accepts the bad data and commits it to all replicas. Maybe the system continues operating “normally” for a thousand more steps before the corruption surfaces in a way that would actually harm a user.

An early abort masks the cascade. And the cascade is where the real damage lives.

Moonpool follows the principle that Antithesis pioneered: assertions record violations and continue. When assert_always! detects a violation, it logs an error, increments a counter, and lets the simulation keep running. The simulation report at the end shows everything that went wrong, in what order, and how often.

This is not about being lenient. It is about being thorough. A single simulation run that records three violations across two different subsystems tells you far more than three separate runs that each crash on the first violation they find.

Dual Purpose

Assertions in moonpool serve two purposes that reinforce each other.

First, they verify correctness. An assert_always! is a property that must hold every time it is checked. If it fails even once across thousands of iterations, there is a bug. The system tracks pass and fail counts, giving you a precise success rate rather than a binary pass/fail.

Second, they guide exploration. When you enable multiverse mode (covered in Part VI), certain assertions become active signals to the explorer. An assert_sometimes! that fires true for the first time tells the explorer “this is interesting, continue from here.” The explorer records a replay recipe to that RNG coordinate, then runs new timelines that reproduce the prefix and diverge afterward. This is what turns a random walk through state space into a directed search.

The same assertion does both jobs. You write it once, thinking about correctness. The exploration framework uses it automatically to find more bugs.

The Reporting Pipeline

After each simulation iteration, the runner checks has_always_violations(). If any always-type assertion failed during the iteration, the runner marks that seed as a failure. But the simulation is not interrupted mid-run. The entire iteration completes, accumulating all violations.

When the simulation finishes, assertion_results() returns an AssertionStats for every tracked assertion:

#![allow(unused)]
fn main() {
pub struct AssertionStats {
    /// Total number of times this assertion was evaluated
    pub total_checks: usize,
    /// Number of times the assertion condition was true
    pub successes: usize,
}
}

You can ask for success_rate() to get a percentage. An always-assertion with a 99.7% success rate means it failed in 0.3% of checks. That is a bug, and the numbers tell you how hard it is to trigger.

At the end of all iterations, validate_assertion_contracts() performs final validation. It returns two categories of violations: always violations (definite bugs where invariants were broken, including assertion slot-table overflow) and coverage violations (sometimes-assertions that were never satisfied, meaning the simulation did not exercise certain paths). The distinction matters because always violations indicate bugs regardless of iteration count, while coverage violations are only meaningful after enough iterations for statistical confidence. The shared table tracks up to 512 assertion sites; if it fills, SimulationReport::dropped_assertion_allocations reports the number of untracked evaluations and the run fails loudly instead of appearing green.

What Comes Next

The next four chapters walk through the assertion types from simple to complex. We start with the conceptual taxonomy (invariants, discovery, guidance), then cover boolean assertions, numeric watermarks, and compound assertions. Each type has a different relationship with the simulation and the explorer, and understanding that relationship is the key to writing assertions that actually find bugs.

Invariants vs Discovery vs Guidance

• Invariants: What Must Always Hold
• Discovery: What Must Happen Eventually
• Guidance: What to Optimize Toward
• Why the Taxonomy Matters

Not all assertions ask the same kind of question. Some say “this must always be true.” Others say “this should happen at least once.” And a few say “try to make this better.” These are fundamentally different kinds of statements, and the simulation framework treats them differently.

Moonpool organizes assertions into three categories based on their purpose. Getting the category right matters because it determines how the assertion interacts with the runner, how it interacts with the explorer, and what a violation actually means.

Invariants: What Must Always Hold

Invariant assertions state properties that must be true every single time they are checked. If they fail even once, there is a bug.

#![allow(unused)]
fn main() {
assert_always!(committed_count <= total_count, "commits cannot exceed total");
}

This is the assertion you reach for most often. In Antithesis’s own Pangolin database, about 90% of assertions are always-type. That ratio matches what we see in practice: the vast majority of what we want to say about a system is “this property holds.”

Invariant assertions do not interact with the explorer. They never count as discoveries. Their job is purely to catch violations. When one fails, the runner records the violation and continues. The value is in the recording, not in the stopping.

There is an important subtlety: assert_always! also fails if it is never reached. An assertion that is never evaluated gives false confidence. If you have an assertion guarding a recovery path but your simulation never triggers recovery, the assertion tells you nothing. Moonpool flags unreached always-assertions as violations so you know your coverage has gaps.

For optional code paths where non-reachability is acceptable, use assert_always_or_unreachable! instead. It validates the property when reached but passes silently if the code path is never exercised.

Discovery: What Must Happen Eventually

Discovery assertions state properties that must occur at least once across all simulation iterations. They do not need to hold every time, but they must fire true at some point.

#![allow(unused)]
fn main() {
assert_sometimes!(leader_elected, "a leader should eventually be elected");
assert_reachable!("recovery path exercised");
}

Where invariants validate, discovery assertions prove coverage. They answer questions like: Did our simulation actually trigger a failover? Did a leader election happen? Did the retry path execute? Without discovery assertions, a simulation could run thousands of iterations exercising only the happy path and report zero failures. Everything looks green, but nothing interesting was tested.

Discovery assertions become exploration amplifiers in multiverse mode. When assert_sometimes! fires true for the first time, the explorer records the seed and RNG position that reached it. Follow-up timelines replay that prefix and diverge just afterward. Reaching the state was hard, and there are likely more interesting states reachable from it.

Consider a bug that requires a failover (probability 1/1000) followed by a specific timing condition during recovery (probability 1/1000). Without exploration amplification, finding this bug requires roughly 1,000,000 random iterations. With a sometimes-assertion on the failover, the explorer takes a shortcut: find the failover once in ~1000 iterations, retain its replay prefix, then try roughly 1000 continuations from that prefix. The effective probability drops from multiplicative to additive.

This is why discovery assertions have “superpowers” in moonpool. They are not just coverage markers. They are active participants in the search for bugs.

Guidance: What to Optimize Toward

Guidance assertions steer the explorer toward interesting regions of the state space. They express goals that the system should try to achieve, and the explorer actively works to satisfy them.

#![allow(unused)]
fn main() {
assert_sometimes_greater_than!(throughput, 1000, "should sometimes achieve high throughput");

assert_sometimes_all!("full_cluster_ready", [
    ("leader_elected", has_leader),
    ("all_replicas_synced", replicas_in_sync),
    ("clients_connected", clients_ready),
]);
}

Numeric guidance assertions track watermarks. The system remembers the best value it has observed, and an improvement is a discovery the explorer anchors continuations to. This creates a ratchet effect: the explorer steadily pushes toward boundary conditions where bugs tend to hide.

Compound guidance assertions track frontiers. assert_sometimes_all! counts how many sub-goals are simultaneously true, and an increase is a discovery. The explorer is driven to satisfy more sub-goals at once, which naturally leads it toward complex system states that are hard to reach by random exploration alone.

Why the Taxonomy Matters

The three categories are not just organizational labels. They determine what the framework does with the assertion:

When writing an assertion, ask yourself: Am I checking a property that must never be violated? Am I verifying that something interesting happened? Or am I telling the explorer where to look? The answer determines which macro to use, and using the right one makes the entire framework more effective.

Always and Sometimes

• assert_always!
• assert_always_or_unreachable!
• assert_sometimes!
• assert_reachable!
• assert_unreachable!
• Choosing the Right One

The boolean assertion macros are the foundation of everything else in moonpool’s assertion system. There are five of them. Each takes a message string that identifies the assertion across iterations and across forked timelines.

assert_always!

The workhorse. This states a property that must hold every time it is evaluated.

#![allow(unused)]
fn main() {
assert_always!(
    replica_count >= min_replicas,
    "replica count must meet minimum"
);
}

If the condition is false, moonpool records the violation at ERROR level with the current seed and increments the failure counter. It does not panic. The simulation continues, and subsequent assertions in the same iteration can still fire and record their own results.

There is one behavior that surprises people at first: assert_always! also fails if it is never reached. The reasoning is that an untested invariant provides false confidence. If you write an assertion guarding your recovery path but the simulation never exercises recovery, the post-run validation will flag it. This forces you to either fix your simulation to reach that path or use a different assertion type.

assert_always_or_unreachable!

Same semantics as assert_always!, except it passes silently when the code path is never executed.

#![allow(unused)]
fn main() {
assert_always_or_unreachable!(
    balance >= 0,
    "balance must be non-negative after withdrawal"
);
}

Use this for properties that guard optional or conditional code paths. If a particular fault injection scenario never triggers a withdrawal, this assertion will not flag a coverage gap. But if the path is reached and the balance goes negative, that is a violation.

The distinction between assert_always! and assert_always_or_unreachable! prevents a common trap: littering your code with always-assertions, then getting flooded with “never reached” violations because your simulation configuration does not exercise every path. Reserve assert_always! for the paths you must test. Use assert_always_or_unreachable! for the rest.

assert_sometimes!

This states a property that should be true at least once across all iterations. It does not need to hold every time.

#![allow(unused)]
fn main() {
assert_sometimes!(
    connections_dropped > 0,
    "chaos should sometimes drop connections"
);
}

If the condition is never true after all iterations complete, the post-run validation reports a coverage violation. But unlike always-violations, coverage violations are statistical. They become meaningful only after enough iterations.

The real power of assert_sometimes! shows up in multiverse mode. When the condition fires true for the first time, the explorer records a replay anchor. New timelines reconstruct that prefix and diverge after the interesting state. This transforms sometimes-assertions from passive coverage checks into active exploration amplifiers.

Think of it this way: assert_sometimes! is how you tell the explorer “this state is worth investigating.” The explorer does the rest.

Here is a practical pattern. You want to verify that your system handles leader re-election after a crash:

#![allow(unused)]
fn main() {
// In your workload
assert_sometimes!(
    leader_changed_after_crash,
    "leader should change after crash"
);
}

Without multiverse exploration, this assertion just checks that your simulation exercises the re-election path. With exploration enabled, it creates a checkpoint at the moment re-election succeeds. The explorer then branches from that checkpoint, increasing the chance of finding bugs in the post-election state.

assert_reachable!

A simplified form of sometimes-assertion for code paths that should be hit at least once. No condition needed.

#![allow(unused)]
fn main() {
fn handle_timeout(&mut self) {
    assert_reachable!("timeout handler executed");
    // ... handle the timeout
}
}

This is equivalent to assert_sometimes!(true, "...") but reads more clearly when you just want to confirm a path is exercised. Like assert_sometimes!, its first reach is a discovery in multiverse mode.

assert_unreachable!

The inverse: marks a code path that should never execute.

#![allow(unused)]
fn main() {
fn handle_message(&mut self, msg: Message) -> Result<(), Error> {
    match msg.kind {
        MessageKind::Request => { /* ... */ }
        MessageKind::Response => { /* ... */ }
        MessageKind::Unknown => {
            assert_unreachable!("received unknown message kind");
            return Err(Error::UnknownMessage);
        }
    }
    Ok(())
}
}

If this code path is reached, moonpool records a violation (equivalent to an always-violation). Unlike assert_always!, there is no condition to check. Being reached at all is the violation.

Note that the code after the assertion still executes. The assertion records the problem but does not prevent the error path from running. This lets the simulation discover what happens when “impossible” conditions occur.

Choosing the Right One

The decision tree is straightforward:

• Must hold every time, must be tested: assert_always!
• Must hold every time, might not be reached: assert_always_or_unreachable!
• Must happen at least once: assert_sometimes!
• Path must be exercised: assert_reachable!
• Path must never execute: assert_unreachable!

In practice, the distribution follows a pattern similar to what Antithesis found with their Pangolin database: roughly 90% always-type, with sometimes/reachable filling in the coverage gaps. That ratio makes sense because most of what we want to say about a system is “this must hold,” not “this should eventually happen.”

Start with always-assertions on your core invariants. Add sometimes-assertions on the interesting error paths and recovery scenarios. The explorer will use those sometimes-assertions to find the bugs hiding behind the invariants.

Numeric Assertions

• Always: Numeric Bounds
• Sometimes: Numeric Goals
• Watermark Mechanics
• Use Cases

Boolean assertions answer yes-or-no questions. But many of the properties we care about in distributed systems are numeric: latency must stay below a threshold, throughput should reach a target, queue depth should not exceed a bound. Numeric assertions let us express these properties directly and give the explorer something extra to work with: a value to optimize.

Always: Numeric Bounds

The always-numeric macros state bounds that must hold on every check. They work like assert_always! but compare two numeric values:

#![allow(unused)]
fn main() {
assert_always_less_than!(queue_depth, max_queue_size, "queue must not overflow");
assert_always_greater_than_or_equal_to!(replica_count, 1, "must have at least one replica");
}

Four comparison operators are available: assert_always_greater_than!, assert_always_greater_than_or_equal_to!, assert_always_less_than!, and assert_always_less_than_or_equal_to!. Each takes a value, a threshold, and a message.

When the comparison fails, the behavior is the same as boolean always-assertions: the violation is recorded, an error is logged, and the simulation continues.

But something extra happens behind the scenes. The framework tracks a watermark for the value: the most extreme value observed across all checks. For assert_always_less_than!(x, 100, ...), the framework remembers the highest x it has seen. Even if x never reaches 100, the watermark tells you how close the system came to the boundary.

Why does this matter? Because assert_always_less_than!(x, 100) implicitly tells the explorer to maximize x. The explorer gravitates toward states where x is highest, naturally pushing toward the boundary condition. If there is a bug that only manifests when x reaches 99, the explorer will find it faster than random exploration would.

This boundary-seeking behavior is automatic. You write a bound, and the framework tries to find states that approach or violate it.

Sometimes: Numeric Goals

The sometimes-numeric macros state goals that should eventually be achieved. The comparison must hold at least once across all iterations:

#![allow(unused)]
fn main() {
assert_sometimes_greater_than!(throughput, 500, "should achieve >500 ops/sec");
assert_sometimes_less_than!(p99_latency, 100, "p99 should sometimes drop below 100ms");
}

Like their boolean counterparts, these are coverage assertions. If the goal is never achieved after all iterations, the validation reports a coverage violation.

The difference from boolean sometimes is in how they interact with the explorer. Sometimes-numeric assertions signal a discovery when the comparison distance improves. Guidance tracks left - right, while the report still shows the best value of left. Including both operands matters when the threshold changes during a run: a larger left value is not progress toward left > right if the right value grew even faster.

This creates a ratchet. Suppose you write:

#![allow(unused)]
fn main() {
assert_sometimes_greater_than!(committed_transactions, 1000, "high commit throughput");
}

The first time committed_transactions reaches 50, the explorer branches. Then it finds a timeline where it reaches 200 and branches again. Then 450. Then 800. Each improvement creates a new branch point, and the explorer explores from progressively better states. The watermark only moves in one direction: toward the goal.

Watermark Mechanics

Every numeric assertion, whether always or sometimes, maintains a reporting watermark in shared memory. This watermark is the best value of the left operand observed so far:

• For assertions that track the highest value (maximize=true): assert_always_less_than!, assert_always_less_than_or_equal_to!, assert_sometimes_greater_than!, assert_sometimes_greater_than_or_equal_to!. These seek the boundary from below (always) or ratchet upward (sometimes).
• For assertions that track the lowest value (maximize=false): assert_always_greater_than!, assert_always_greater_than_or_equal_to!, assert_sometimes_less_than!, assert_sometimes_less_than_or_equal_to!. These seek the boundary from above (always) or ratchet downward (sometimes).

The watermark lives in the shared assertion region, so in multiverse mode every timeline sees the same current best. When one timeline improves it, the improvement is immediately visible to all subsequent timelines. This means the explorer collectively pushes toward the boundary rather than each timeline searching independently.

For sometimes-numeric assertions, a second watermark tracks the best comparison distance, left - right, seen by the explorer. Greater-than goals maximize this distance; less-than goals minimize it. A new discovery only fires when that distance improves past the shared mark, so a moving threshold cannot falsely steer exploration in the wrong direction.

Use Cases

Resource bounds:

#![allow(unused)]
fn main() {
assert_always_less_than!(
    memory_usage_mb, max_memory_mb,
    "memory must stay within budget"
);
}

The explorer pushes toward high memory states, testing the system under pressure.

Convergence targets:

#![allow(unused)]
fn main() {
assert_sometimes_less_than!(
    convergence_time_ms, 500,
    "cluster should converge within 500ms"
);
}

The explorer ratchets toward fast convergence, branching each time it finds a faster path.

Throughput validation:

#![allow(unused)]
fn main() {
assert_always_greater_than_or_equal_to!(
    processed_count, expected_count,
    "must process all submitted requests"
);
}

Catches dropped requests. The explorer seeks states where the gap between processed and expected is smallest (i.e., where the system is closest to dropping something).

The key insight with numeric assertions is that expressing a bound also expresses a direction for exploration. You get correctness checking and guided search from the same line of code.

Compound Assertions

• assert_sometimes_all!: Simultaneous Sub-Goals
• assert_sometimes_each!: Per-Value Coverage
• Quality Watermarks
• State Explosion and Practical Limits
• Choosing Between the Two

Boolean assertions check one condition. Numeric assertions track one value. But the most interesting system states involve multiple things being true at once: a leader is elected and all replicas are synced and clients are connected. Or we want every distinct configuration to be tested, not just the first one we stumble on. Compound assertions handle both of these cases.

assert_sometimes_all!: Simultaneous Sub-Goals

Some properties only matter when multiple conditions hold at the same time. A cluster is not truly healthy unless the leader is elected, replicas are caught up, and no partitions are active. Testing each condition individually does not tell you whether the system can achieve them all at once.

assert_sometimes_all! takes a message and a list of named boolean sub-goals:

#![allow(unused)]
fn main() {
assert_sometimes_all!("cluster_fully_operational", [
    ("leader_elected", has_leader),
    ("replicas_synced", all_replicas_in_sync),
    ("no_partitions", partition_count == 0),
    ("clients_connected", active_clients > 0),
]);
}

The assertion tracks a frontier: the maximum number of sub-goals that have been simultaneously true. It starts at zero. The first time any sub-goal is true, the frontier advances to 1. When two are true at once, it advances to 2. When all four are true simultaneously, the frontier reaches 4 and the assertion is fully satisfied.

Each frontier advance is a discovery the explorer schedules continuations from. Distinct truth combinations are useful too: leader_elected alone and replicas_synced alone both have frontier 1, but they may lead to very different futures. Moonpool therefore records new partial combinations as structured state discoveries even when their true-count does not beat the frontier. A bounded 64-bit bitmap per assertion keeps this extra guidance cheap; hash collisions can discard an optional hint but cannot affect assertion correctness or the reported frontier.

This creates a progression without collapsing every same-cardinality state: the explorer can retain several ways to satisfy one sub-goal, then continue toward two, three, and finally all of them. Reports show the frontier as reached/target and keep the assertion in MISS until the full target is reached.

This is powerful for multi-step objectives. Consider a distributed transaction system. The full operation requires: prepare all participants, get all votes, commit, and acknowledge to the client. As a sometimes-all assertion:

#![allow(unused)]
fn main() {
assert_sometimes_all!("distributed_commit_complete", [
    ("all_prepared", all_participants_prepared),
    ("all_voted_yes", all_votes_received),
    ("committed", transaction_committed),
    ("client_acked", client_received_ack),
]);
}

The explorer works through the stages. First it finds a state where all participants are prepared. It forks from there and pushes for votes. Then it finds committed states. Each stage is a stepping stone to the next.

Without this assertion, the explorer would need to stumble on the complete end-to-end scenario by chance. With it, the explorer gets intermediate checkpoints that guide it through the sequence.

assert_sometimes_each!: Per-Value Coverage

Sometimes you want to ensure that every distinct value of something is explored, not just any one value. If your system has 10 different error codes, you want the simulation to exercise all 10. If a consensus protocol has 5 possible states, you want coverage of each.

assert_sometimes_each! creates a separate assertion for each unique combination of identity keys:

#![allow(unused)]
fn main() {
// Ensure every node ID is tested
assert_sometimes_each!("node_tested", [("node_id", node_id)]);

// Ensure every (state, role) combination is explored
assert_sometimes_each!("state_coverage", [
    ("state", state as i64),
    ("role", role as i64),
]);
}

Each unique combination of key values gets its own bucket. Discovering a new bucket registers a replayable exemplar the explorer can continue from. This equalizes exploration across all discovered values, preventing the simulation from over-concentrating on values it finds easily while neglecting harder-to-reach ones.

The Antithesis team demonstrated this dramatically with The Legend of Zelda. They used SOMETIMES_EACH with screen coordinates to ensure the explorer visited all 128 overworld screens and 230 dungeon rooms. Without per-value bucketing, the explorer would revisit the starting area thousands of times while leaving distant rooms unexplored. With it, every room gets roughly equal attention.

For distributed systems, the same pattern applies. If you have a state machine with states {Follower, Candidate, Leader, Observer}, a sometimes-each assertion ensures the simulation exercises each state rather than getting stuck in the most common one:

#![allow(unused)]
fn main() {
assert_sometimes_each!("raft_state_exercised", [
    ("node", node_id),
    ("state", raft_state as i64),
]);
}

Quality Watermarks

Sometimes discovering a value is not enough. You also want to explore each value under good conditions. assert_sometimes_each! supports an optional second list of quality keys that track watermarks per bucket:

#![allow(unused)]
fn main() {
assert_sometimes_each!(
    "dungeon_floor_explored",
    [("floor", floor_number)],           // identity: which floor
    [("health", current_health)],         // quality: explore each floor with good health
);
}

Each bucket remembers the best quality values observed. When a bucket is revisited with better quality, the explorer re-forks from that improved state. This prevents the “doomed state” problem described in Antithesis’s Castlevania analysis: the explorer reaches an area but in such bad shape that no useful exploration can happen from there. Quality watermarks ensure the representative state for each bucket is the best one found so far.

State Explosion and Practical Limits

Compound assertions can create a lot of buckets. If you use two identity keys with 100 distinct values each, that is 10,000 potential buckets. Each new bucket is a discovery the explorer may anchor continuations to, and the bucket table itself is bounded (256 slots), so individual assertion effectiveness degrades when buckets proliferate.

Keep identity keys coarse enough to be useful. If you are tracking screen positions in a 256x256 grid, bucket them into 16x16 regions rather than tracking exact pixels. The goal is coverage of meaningful distinct states, not exhaustive enumeration of every possible value.

A good rule of thumb: if you would not manually write a separate test for each distinct value, it probably should not be its own bucket.

Choosing Between the Two

The choice is straightforward:

• Multiple conditions that must hold simultaneously: assert_sometimes_all!
• One condition that must hold for each distinct value: assert_sometimes_each!

Use assert_sometimes_all! when the challenge is getting several things true at once. Use assert_sometimes_each! when the challenge is covering a space of distinct configurations or states.

Both are guidance assertions. Both trigger forks. Both make the explorer smarter about where to search. And both turn a single line of code into something that would otherwise require dozens of hand-crafted test scenarios.

Events and Invariants

• Emitting Events
• The Invariant Trait
• Querying the Timeline
• Reading from a Workload
• The Fault Timeline
• Simulation Time in Log Output
• Snapshots vs Timelines
• When to Use Invariants vs Assertions
• Performance

Assertions live inside workloads. They validate local properties: this key should exist, this response matched what we sent. But some correctness properties span the entire system. The conservation law in a consensus protocol is a good example. All committed values must agree across replicas, and no committed value can be lost. No single workload owns that property. We need something that watches the whole world.

Other properties span time rather than space. Monotonicity says a term number never decreased. Causal ordering says the lock was acquired before the write. Conservation over time says total money stayed constant through a sequence of transfers. A snapshot of the latest state cannot tell us any of these. If a balance went 100, then 50, then 100 again, a snapshot-based check sees 100 and declares everything fine. The dip to 50 is invisible.

Both kinds of properties need the same two ingredients: an append-only timeline of events, and a watcher that runs as the simulation advances.

How would you find a dual leader in production? You would query your traces: every leader_elected event, grouped by term, flagged when two nodes claim the same one. Moonpool’s invariant system runs exactly that query, against exactly those traces, inside the simulation. The instrumentation you write for production observability is the test oracle.

Emitting Events

Correctness facts are plain tracing events. No markers, no derives, no moonpool-specific API:

#![allow(unused)]
fn main() {
// Inside a process or workload — exactly what you'd write for production
// observability anyway.
tracing::info!(target: "raft", term, leader = %my_ip, "leader_elected");
}

Three rules:

1. The message is the event name. Use a constant name like "leader_elected", not an interpolated sentence. Events are grouped and queried by name, the same way you would query Loki or your OTel backend.
2. Use % for strings. %value records the Display form without quotes. ?value on a String keeps Debug quotes, which makes field matching annoying.
3. INFO or above. debug! and trace! events are not captured.

That’s the whole convention. Sim time is stamped automatically from the simulation clock, so do not include a time_ms field.

Where does the source come from? The orchestrator wraps every process and workload task in a tracing span carrying its ip (info_span!("process", ip = %ip)). When an event fires inside that task, the capture layer walks the span scope and attributes the event to the nearest enclosing actor. A task spawned through ctx.task().spawn_task() runs inside the span that was current when it was spawned, exactly as tokio::spawn does, so a background request handler’s events carry the same ip as the actor that spawned it, however deep the spawning goes. In production the same role is played by host or pod attributes on your trace resource. Events emitted outside any actor span (runtime internals, the orchestrator itself) are not captured.

Each captured event is a TraceEvent:

#![allow(unused)]
fn main() {
pub struct TraceEvent {
    pub seq: u64,        // global monotonic sequence number, reset per seed
    pub time_ms: u64,    // simulation time at capture
    pub source: String,  // ip of the enclosing actor span, or "sim" for faults
    pub target: String,  // tracing target, e.g. "raft"
    pub level: tracing::Level,
    pub name: String,    // the message, e.g. "leader_elected"
    pub fields: BTreeMap<String, FieldValue>,
}
}

The seq field gives total ordering across all event names. If leader_elected gets seq 0 and client_ack gets seq 1, the election was captured first, even at the same time_ms. Typed values come out per field: e.u64("term"), e.str("leader"), e.bool("ok"), e.f64("ratio") all return Options.

One emission, two audiences. In simulation, SimulationLayer captures the event into the timeline and invariants cross-validate it. In production, where no SimulationLayer is installed, the same event flows to whatever subscriber is configured: fmt, OpenTelemetry, structured JSON. Nothing in the emission is moonpool-specific.

Below the floor is free in simulation. SimulationLayer::install (what SimulationBuilder::run calls) stacks the layer on a LevelFilter at the run’s trace floor, INFO by default, so a DEBUG or TRACE span or event is disabled at the subscriber and never reaches the registry. Instrument hot paths freely with #[tracing::instrument(level = "trace")]: in a sweep each call costs one level compare, and the timeline captures exactly the INFO-and-above events it always did.

Lowering the floor to debug one seed. SimulationBuilder::trace_level(LevelFilter::DEBUG) (or TRACE) turns those spans on and captures their events into the timeline for the whole run; SimulationLayer::new().with_level(..) does the same for a hand-built layer. The floor never touches scheduling or randomness, so the seed replays identically at any level — only the run is slower and the capture larger, which is why it is a per-investigation setting rather than a sweep default.

The Invariant Trait

An invariant is a check the orchestrator runs after every simulation step. If the invariant records a failure, the simulation reports the failing seed.

#![allow(unused)]
fn main() {
pub trait Invariant: 'static + Send {
    fn name(&self) -> &str;
    fn observe(&self, q: &dyn TraceQuery, sim_time_ms: u64);
    fn reset(&mut self) {}
}
}

The invariant gets a TraceQuery view of all captured events plus the current simulation time. The contract: if the property holds, return normally. If it does not, report with assert_always! so the failure is recorded with the seed. The reset hook is called between seeds to clear cursors and tracking sets. Because steps batch events, one observe call may see several new events.

Register invariants on the builder:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .workload(ConsensusWorkload::new(3))
    .invariant(AgreementInvariant::default())
    .invariant(ValidityInvariant::default())
    .run();
}

For quick one-off checks there is a closure shorthand:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .invariant_fn("commit_volume", |q, _t| {
        assert_always!(q.len("commit") <= 10_000, "runaway commit volume");
    });
}

Querying the Timeline

TraceQuery has three methods. since(name, &cursor) returns only the events newer than the cursor and advances it — cheap, and the right default. snapshot(name) re-reads everything from the start of the seed. len(name) counts.

A real consensus example. Multiple nodes must agree on committed values. The agreement invariant checks that no two nodes have committed different values for the same slot. This catches subtle bugs: a leader change that replays a proposal, a vote that arrives after a new ballot, a crash during the accept phase.

Each node emits tracing::info!(slot, value, "commit") whenever it commits. The invariant scans the timeline and verifies no two commits for the same slot disagree.

#![allow(unused)]
fn main() {
use std::cell::{Cell, RefCell};
use std::collections::BTreeMap;
use moonpool_sim::{Invariant, TraceQuery, assert_always};

#[derive(Default)]
pub struct AgreementInvariant {
    cursor: Cell<usize>,
    committed: RefCell<BTreeMap<u64, u64>>,
}

impl Invariant for AgreementInvariant {
    fn name(&self) -> &str { "agreement" }

    fn observe(&self, q: &dyn TraceQuery, _t: u64) {
        // Cursor-based: only new events since the last call.
        let mut committed = self.committed.borrow_mut();
        for e in q.since("commit", &self.cursor) {
            let slot = e.u64("slot").expect("commit carries a slot");
            let value = e.u64("value").expect("commit carries a value");
            if let Some(prev) = committed.get(&slot) {
                assert_always!(
                    *prev == value,
                    format!("agreement violated at slot {slot}: {prev} vs {value}")
                );
            } else {
                committed.insert(slot, value);
            }
        }
    }

    fn reset(&mut self) {
        self.cursor.set(0);
        self.committed.borrow_mut().clear();
    }
}
}

If a leader change causes two nodes to commit different values for the same slot, the invariant fires on the step that captured the second commit. Reset between seeds is the invariant’s responsibility: clear the cursor in reset() and the simulation builder calls it for you between iterations.

The canonical runnable example is crates/moonpool-sim/tests/leader_election.rs: a workload emits leader_elected events, and a SplitBrainInvariant detects two leaders claiming the same term.

Reading from a Workload

Inside Workload::check() or anywhere you have a SimContext, the observability handle implements TraceQuery too:

#![allow(unused)]
fn main() {
let q = ctx.observability();
let transfers = q.snapshot("transfer");
let total: u64 = transfers.iter().filter_map(|e| e.u64("amount")).sum();
assert_always!(total <= max_funds, "spent more than reserves");
}

snapshot returns an empty Vec if nothing was emitted under that name. No Option to unwrap.

The Fault Timeline

Every fault the simulator injects, from network partitions to storage corruption to process kills, lands in the same timeline under the well-known name "sim_fault", with source = "sim" and a kind field identifying the variant. No workload code needed. These are not production traces — there is no simulator in production — so the engine records them internally and the runner merges them into the timeline after each step (engine-level tests can drain them directly with SimWorld::take_faults()).

#![allow(unused)]
fn main() {
use std::cell::Cell;
use moonpool_sim::{Invariant, SIM_FAULT_EVENT_NAME, TraceQuery, assert_always};

pub struct KillRateInvariant {
    cursor: Cell<usize>,
    kill_count: Cell<usize>,
}

impl Invariant for KillRateInvariant {
    fn name(&self) -> &str { "kill_rate" }

    fn observe(&self, q: &dyn TraceQuery, _t: u64) {
        for e in q.since(SIM_FAULT_EVENT_NAME, &self.cursor) {
            if e.str("kind") == Some("process_force_kill") {
                self.kill_count.set(self.kill_count.get() + 1);
            }
        }
        assert_always!(
            self.kill_count.get() <= 10,
            format!("too many kills: {}", self.kill_count.get())
        );
    }

    fn reset(&mut self) {
        self.cursor.set(0);
        self.kill_count.set(0);
    }
}
}

The kind values cover three categories. Network and storage faults originate inside the component that owns the affected state. The component returns the fault beside its ordered scheduler effects, and SimWorld stamps it with the current monotonic scheduler time before the runner merges it into the trace timeline.

• Process lifecycle: process_graceful_shutdown (with grace_period_ms), process_force_kill (with cause: grace_period_expired, crash, or crash_and_wipe), process_restart (all with ip)
• Network: partition_created, partition_healed (with from/to), random_close, bit_flip (with connection_id), send_partition_created, recv_partition_created (with ip)
• Storage: storage_read_fault, storage_write_fault (with write_kind), storage_sync_fault, storage_crash, storage_wipe (all with ip)

The real power is correlation. When an application-level invariant fires, cross-reference the fault events to understand what the infrastructure was doing at that moment. A conservation law violation at t=5000 that coincides with a process_force_kill at t=4980 tells a very different story than one with no faults nearby. Both kinds of events sit in one timeline with one clock, which is exactly how you would correlate an alert window against infrastructure events in a production log aggregator.

Simulation Time in Log Output

By default, tracing_subscriber::fmt::layer() prefixes every log line with wall-clock time. That is exactly wrong for simulation. A seed runs millions of simulated milliseconds inside a few real-time milliseconds, so every line gets the same wall-clock timestamp and reading the log feels like reading a stack trace with the line numbers ripped off.

SimTime is a FormatTime impl that prints the current sim time in milliseconds reported by a Clock. Clock is a narrow Send + Sync trait, implemented for SimulationLayerHandle out of the box, so the typical setup is one line:

#![allow(unused)]
fn main() {
use moonpool_sim::{SimTime, SimulationLayer};
use tracing_subscriber::layer::SubscriberExt;

let sim_layer = SimulationLayer::new();
let handle = sim_layer.handle();

let subscriber = tracing_subscriber::registry()
    .with(sim_layer)
    .with(
        tracing_subscriber::fmt::layer()
            .with_timer(SimTime::new(handle.clone())),
    );

let _guard = tracing::subscriber::set_default(subscriber);
}

Output:

sim+    1.234s  INFO raft: term=3 leader=10.0.1.2 leader_elected
sim+    1.235s  INFO myservice::handler: processing request id=42

The orchestrator advances the clock by calling handle.set_sim_time_ms(...) after each sim.step(). All tracing::*! calls between steps, captured or not, see the right time. The handle is per-layer, so two parallel sims keep their clocks isolated.

In production, no SimulationLayer is registered and the formatter falls back to its default zero. Use plain wall-clock fmt::layer() there, and only pull SimTime in for sim and test runs.

Snapshots vs Timelines

Use both. They solve different problems.

StateHandle::publish/get stores the latest snapshot. Good for properties about the current state: the sum of all balances equals total deposits minus total withdrawals. Workloads call ctx.state().publish("model", model.clone()) and read the snapshot at the end of the simulation in Workload::check.

The trace timeline stores append-only history. Good for properties about how the state changed over time: no message was received before it was sent, the leader term never decreased, every debit has a matching credit.

Invariants can derive snapshots from event histories by keeping an internal counter and updating it as events arrive. The fault timeline adds infrastructure context for free, and every event you emit is one that production observability also sees.

When to Use Invariants vs Assertions

Assertions (assert_always!, assert_sometimes!) belong inside workloads. They validate local properties from the workload’s perspective: this response has the right balance, this error path was exercised.

Invariants validate global, cross-workload properties from an omniscient perspective. They watch the trace timeline and check that the system’s history is consistent. Use them for:

• Conservation laws (messages, resources, committed values)
• No-phantom properties (never receive something that was not sent)
• Consistency across processes (leader election: at most one leader per term)
• Monotonicity properties (ballot numbers only increase)
• Causal ordering (a write to x must come before a read that returns it)
• Time-shaped anomalies (retry rates that stay elevated after the trigger fault healed — the metastable failure signature)

A useful rule of thumb: if the property involves state from more than one process or workload, it is an invariant. If it is about one workload’s local view, it is an assertion. Inside an invariant, prefer the assertion macros over a raw panic! so failures are recorded with the seed.

Performance

Invariants run after every simulation step. A typical simulation processes thousands of steps per iteration, and you might run hundreds of iterations. Keep invariants fast.

Concretely: use cursor-based since rather than full snapshot, iterate only the new events, compare a few counters, check a simple predicate. Avoid expensive operations like sorting large datasets or formatting strings on the happy path. The format! in the failure message is fine because it only runs when the invariant fails.

If you find yourself wanting a slow invariant, like replaying a log to verify consistency, run it only in the workload’s check() method at the end of the simulation rather than on every step.

Application Metrics

Your service is probably already instrumented. There is a prometheus::Registry somewhere, a requests_total counter, a latency histogram, a gauge for the queue depth. That instrumentation exists because it is how you understand the system in production.

Moonpool can report on it. Register a metrics source per simulated node and the counters, gauges and histograms your code already keeps show up in the simulation report — aggregated across seeds, attributed per node, with an exact time series of every mutation.

use std::sync::Arc;
use moonpool_prometheus::PrometheusSource;

let report = SimulationBuilder::new()
    .metrics_factory(|_ip| Arc::new(PrometheusSource::default()))
    .processes(3, || Box::new(MyNode::new()))
    .workload(MyWorkload::default())
    .set_iterations(10)
    .run();

report.eprint();

━━━ Metrics (15 series) ━━━━━━━━━━━━━━━━━━━━━━━━━━
  kv_keys_stored{instance="10.0.1.1"}                  15.81 avg        1 min       16 max
  kv_request_latency_seconds{instance="10.0.1.1"}     130.83 total  6,667 obs     0.02 avg
  kv_requests_in_flight{instance="10.0.1.1"}            2.22 avg         0 min        3 max
  kv_requests_rejected_total{instance="10.0.1.1"}      8,620 total     862 per seed
  kv_requests_served_total{instance="10.0.1.1"}        6,667 total  666.70 per seed

Reading your metrics from process code

The source is reachable from any process or workload through its context:

let metrics = ctx.metrics::<PrometheusSource>()
    .ok_or_else(|| SimulationError::InvalidState("no metrics factory".into()))?;

let served = metrics.counter("requests_served_total", "Requests served")?;
let in_flight = metrics.gauge("requests_in_flight", "Requests running")?;
let latency = metrics.histogram_with_buckets(
    "request_latency_seconds",
    "Time to serve one request",
    vec![0.001, 0.005, 0.010, 0.050, 0.100],
)?;

served.inc();

These are get-or-register, which matters because a source outlives process reboots (see below). A rebooted process looks its metrics up instead of failing with AlreadyReg.

To wrap a registry your application already built, use PrometheusSource::new(my_registry).

Why per node, and why per iteration

Every simulated node shares one OS process. A single shared registry would merge three nodes’ counters into one number, so the factory is keyed by IP and each node gets its own registry. Each sample is stamped with an instance label naming the node — Prometheus’ own label for a scrape target.

The factory also runs afresh every iteration. Most registries have no reset, so a reused instance would make seed 50 report the sum of fifty runs. This is the same reason a custom fault injector is registered through fault_factory rather than as an instance.

A reboot is the exception: a source is keyed by IP, so a crashed and restarted process keeps its counters, exactly as a real node’s /metrics does across a restart on the same host.

Time series, not samples

Metrics created through the source hand back instrumented handles. Every inc(), set() and observe() records a point, stamped with the simulated clock. There is no scrape interval and no sampling: the series is exact, and it replays bit-for-bit for a given seed.

let metrics = report.individual_metrics[0].as_ref().expect("seed passed");

for (series, points) in &metrics.app_series {
    for point in points {
        println!("{series} {} at {}ms", point.value, point.time_ms);
    }
}

That is what makes a latency spike line up with the partition that caused it. The aggregated view on report.app_metrics uses this series for min, max and mean, which is why a gauge reports the range it actually moved over rather than whatever value it happened to rest at when the run ended.

Series are capped (10,000 points each by default; see PrometheusSource::set_series_capacity) so a hot loop cannot grow a test’s memory without bound. SimulationMetrics::dropped_metric_points is non-zero when a series was truncated.

Asking questions: metric queries

A raw series is data, not an answer. After five hundred seeds the question is usually “what throughput did this hold, and which seed was the worst?” — so the runner declares what it wants to know, and the report answers it.

use std::time::Duration;
use moonpool_sim::{Fill, Mean, MetricQuery, Percentile};

let report = SimulationBuilder::new()
    .metrics_factory(|_ip| Arc::new(PrometheusSource::default()))
    .metric(
        MetricQuery::select("requests_total")
            .label("operation", "write")
            .rate()
            .bucketize(Duration::from_secs(60), Mean)
            // A minute with no requests is zero per second, not missing data.
            .fill(Fill::Value(0.0))
            .reduce(Mean)
            .named("write_throughput"),
    )
    .metric(
        MetricQuery::select("request_latency_seconds")
            .bucketize(Duration::from_secs(60), Percentile(0.99))
            .named("write_p99"),
    )
    .processes(3, || Box::new(MyNode::new()))
    .workload(MyWorkload::default())
    .set_iterations(500)
    .run();

━━━ Metric Queries (2) ━━━━━━━━━━━━━━━━━━━━━━━━━━
  write_throughput
    requests_total{operation="write"} → rate → 60s buckets (mean) → reduce (mean)  ·  scalar per run  ·  runs: 500
    min            5,421   seed=9182
    mean           5,973
    max            6,102   seed=224
    across runs:  p50 5,991  p95 6,071  p99 6,094

min and max name the seed that produced them, so the worst run goes straight into set_debug_seeds(vec![9182]).

Queries are declared before run() and their results travel with the report, in SimulationReport::metric_queries. Nothing reconstructs them afterwards from raw snapshots: the runner already knew what mattered.

The operations

The aggregators are Min, Mean, Max and Percentile(p) — nothing else.

Counter semantics are explicit. .rate() is the only place a series is read as a counter; bucketize(_, Mean) on a raw counter honestly reports the mean cumulative value, not a throughput. A drop in a counter’s value is read as a reset, Prometheus style. Because a fresh source is built per iteration, every counter is zero at simulated time zero, so the very first increment is rated too.

Empty buckets

bucketize omits a bucket nothing landed in, because a gap and a zero are different facts: a workload that stopped reporting is not a workload reporting zero. Two ops say which one this metric means, split the way Warp 10 splits them — .fill(policy) repeats a value the series already has, while .interpolate() computes one it never had:

Reach for a fill when the quantity holds its value between observations (or is simply zero when nothing happened), and .interpolate() when it moves continuously — a queue depth, a replication lag.

bucket                     0 |    1 |    2 |    3 |    4
no fill                   10 |    - |    - |   40 |    -
.fill(Fill::Previous)     10 |   10 |   10 |   40 |   40
.fill(Fill::Next)         10 |   40 |   40 |   40 |    -
.fill(Fill::Value(0.0))   10 |    0 |    0 |   40 |    0
.interpolate()            10 |   20 |   30 |   40 |    -

They compose, in call order: .interpolate().fill(Fill::Value(0.0)) draws the line through the interior gaps and zeroes the ends interpolation cannot reach.

Either way the grid runs from bucket zero to the bucket holding the run’s end, so a workload that went quiet halfway through gets buckets for the silence instead of the series just stopping.

That matters for more than tidiness. Without a fill, a seed’s bucket set depends on when that seed happened to be busy, so the across-run summary splits one window into several — each with a fraction of the runs in it, and min/max no longer comparing like with like:

# unfilled: two seeds busy in different minutes
[0s,60s)     runs: 1
[120s,180s)  runs: 1

# Fill::Value(0.0): one grid, every window sees every seed
[0s,60s)     runs: 2   min 0  max 5
[60s,120s)   runs: 2   min 0  max 0
[120s,180s)  runs: 2   min 0  max 5

Neither op collapses nor restores a distribution, so both leave the stage — and with it whether Percentile still applies — unchanged. Both read the recorded buckets only, never the ones they just synthesized, so a carried value is the last real observation and a run of interpolated gaps is one straight line rather than a curve that drifts. A series that recorded nothing at all stays empty: a policy alone must not conjure a metric the run never touched.

Series identity is name plus label set, canonically ordered, so two label sets that differ only in insertion order are the same series. Matching is by subset: a query naming operation="write" also matches a series carrying instance="10.0.1.1".

Moonpool’s own metadata stays out of your labels. Every result row carries seed (the replayable identity of one iteration) and run_id (the identity of the whole run() invocation) as moonpool-side fields, never injected into the application’s Prometheus labels.

Percentiles that mean something

mean(p99(node_a), p99(node_b)) is not a p99, and p99(p99(a), p99(b)) is not one either. Once observations are collapsed into a percentile, the percentile of that is a number without a meaning.

The builder enforces this at compile time. Each stage tracks what its values are, and Percentile only applies where the individual observations survive:

// Compiles: the raw histogram observations are still there.
MetricQuery::select("request_latency_seconds")
    .bucketize(Duration::from_secs(60), Percentile(0.99))
    .named("p99");

// Does NOT compile: p99 of per-bucket p99s.
MetricQuery::select("request_latency_seconds")
    .bucketize(Duration::from_secs(60), Percentile(0.99))
    .reduce(Percentile(0.99))
    .named("nonsense");

Min, Mean and Max apply anywhere — “the mean of each node’s p99” is a useful number as long as nobody calls it a p99, which is why the report prints each query’s provenance (observations, scalar or percentile-derived).

The percentiles in the summary block are a different dimension: they are taken across runs, where every seed contributes one equally-weighted value. p95 there names the 95th-percentile run, which is why the report labels it across runs.

Worked example: watching a metastable failure

Totals tell you what a run did; the recorded series tells you when it went wrong, which is the only way to see a failure that outlives its cause. crates/moonpool/examples/metastable_grpc_retry_storm.rs is the demonstration: a real gRPC service with a bounded worker pool, an open-loop client with an aggressive timeout and retries, and exactly one temporary slowdown in the middle of the run.

cargo run --release --example metastable_grpc_retry_storm \
    --features hyper,prometheus -- --seed 4

It reads back the series the run recorded, runs the query pipelines above over them — .rate().bucketize(1s, Mean).fill(Fill::Value(0.0)) for counters, .bucketize(1s, Mean).fill(Fill::Previous) for gauges — and prints the buckets as an ASCII graph on stdout, one column per simulated second, with the trigger window marked. No Grafana, no browser, no export step.

What the graph shows is a metastable failure. The slowdown lasts 650 ms. The service never recovers from it: retries provoked by that excursion keep the server saturated for the remaining 29 seconds, at exactly the offered load it carried comfortably before. The trigger is gone; the bad state is not.

--search 0..64 runs a range of seeds and prints one summary line each. The trigger sits on the tipping point, so some seeds absorb it and return to the healthy operating point while others fall into the storm and stay there — same code, same configuration, different seed. That is the clearest evidence available that the bad state is a second stable operating point rather than a broken setting, and it is exactly the kind of question the recorded series is there to answer.

Timing on the simulated clock

Use start_timer from this crate, not prometheus’ own:

let timer = latency.start_timer(ctx.time());
let response = client.request(req).await?;   // observed even on an early return
timer.stop_and_record();

prometheus::Histogram::start_timer() reads the wall clock. Inside a simulation that records how fast your laptop happened to be, not the latency the simulation modeled, and it gives a different answer on every replay of the same seed. SimHistogram::start_timer reads the provider clock instead.

Pick bucket bounds that bracket what your simulation produces. Prometheus’ defaults are tuned for real HTTP latency; simulated latencies often live on a different scale, and a distribution that lands entirely in the last bucket tells you nothing.

What metrics are not

Metric values are reported, never used to steer the simulation. Not everything in a registry is deterministic — a metric fed from the wall clock, or from a library’s own internal timing, is not — so the runner treats a scrape as an observation. If you want a property enforced, assert on it: metrics tell you the shape of a run, assertions tell you whether it was correct.

Under fork-based exploration, only root timelines are scraped. Explored timelines report back through the shared assertion table, which carries no metric values.

Beyond Prometheus

PrometheusSource is one implementation of MetricsSource, a registry-agnostic trait in moonpool-core:

pub trait MetricsSource: Send + Sync + 'static {
    fn collect(&self) -> Vec<MetricSample>;
    fn set_clock(&self, clock: Arc<dyn MetricClock>) {}
    fn series(&self) -> BTreeMap<String, Vec<MetricPoint>> { BTreeMap::new() }
    fn dropped_points(&self) -> u64 { 0 }
}

collect alone is enough for a scrape-only adapter over any registry. The other three are what an adapter implements to deliver an exact time series (dropped_points reports how many points a capped series discarded). An OpenTelemetry adapter slots in the same way, with no change to the simulation runner.

Designing Workloads That Find Bugs

• Strategy vs Tactics
• The Operation Alphabet
• Swarm the Alphabet
• Invariant Patterns
• Assertions as Memos
• Generate Sufficient Workload

A workload that exercises only the happy path is a workload that finds nothing. The previous chapters covered how to write a workload, wire up assertions, and register invariants. This chapter is about what to put inside them. What operations to generate, what to measure, and how to think about the relationship between input design and bug discovery.

Strategy vs Tactics

Will Wilson offers a useful framework for thinking about simulation workloads. There are two independent dimensions: strategy and tactics.

Strategy is what you measure and optimize. Code coverage, grid cells visited, time the system stays alive, request throughput. Strategy determines how the framework judges whether a run was productive.

Tactics is how you generate inputs. Uniform random? Weighted toward writes? Biased toward boundary values? Tactics determines the raw material the simulator works with.

These dimensions are independent. A workload with brilliant tactics (generating exactly the right fault sequences) can get by with crude strategy (just measuring crashes). A workload with sophisticated strategy (coverage-guided exploration) can get by with simple tactics (pure random inputs). You do not need both to be perfect. You need at least one to be good.

But there is a trap in “pure random” that deserves attention.

The white noise paradox: random inputs are maximally random at the micro level but minimally random at the macro level. Consider random per-packet network faults. Each packet has a 5% chance of being dropped. Sounds adversarial. But the probability of a sustained 10-second partition is 0.05 raised to the power of however many packets cross that link in 10 seconds. Essentially zero. Random drops average out to a slightly degraded but fundamentally healthy network. No sustained partition ever forms.

The same applies to random user inputs. Random button presses in a video game average out to no sustained action. Random API calls average out to no sustained workflow. The system never enters the deep states where bugs hide.

This is why moonpool uses correlated fault injection rather than random per-event drops. “The probability that there’s a network partition happening at time t=1 is highly dependent on whether there’s a network partition happening at time t=0.” A new partition starting is rare. A partition continuing is nearly certain. Correlated distributions produce the sustained fault patterns that expose real bugs.

The Operation Alphabet

From practical experience building workloads, we have found that the operations a workload generates fall into three categories. We call these the Operation Alphabet.

Normal operations are production-like traffic. Reads, writes, queries, transactions. The distribution should match what real users do. If your system handles 80% reads and 20% writes, your workload should approximate that ratio.

Adversarial inputs are what users actually send, whether you planned for it or not. Empty strings. Boundary values (0, -1, u64::MAX). Unicode edge cases. Maximum-length fields. Keys that collide in your hash function. A workload that skips adversarial inputs is testing a polite fiction.

Nemesis operations deliberately break things. Kill a process mid-transaction. Trigger a compaction during peak write load. Send conflicting writes to the same key from multiple clients simultaneously. These push the system into states that normal operation reaches only after months of uptime.

Only normal operations? You test the happy path. Add adversarial inputs and you test the validation path. Add nemesis operations and you test the recovery path. Bugs overwhelmingly live in recovery paths. Your alphabet needs all three letters.

#![allow(unused)]
fn main() {
let roll = ctx.random().random_range(0..100);
match roll {
    0..60 => {
        // Normal: production-like read/write mix
        self.do_normal_operation(ctx).await?;
    }
    60..80 => {
        // Adversarial: boundary values, empty inputs, collisions
        self.do_adversarial_operation(ctx).await?;
    }
    80..100 => {
        // Nemesis: conflict storms, concurrent mutations, stress
        self.do_nemesis_operation(ctx).await?;
    }
}
}

The percentages are tunable. Start with a heavy normal-operation bias and adjust based on what your assert_sometimes! statements tell you about coverage.

Swarm the Alphabet

A complete alphabet has a hidden failure mode. Picking operations uniformly over the full set on every step means every operation is always slightly active, and they crowd each other out. The classic example is a bounded queue with a 50/50 push/pop mix. That is a one-dimensional random walk, and by the Hoeffding bound it takes tens of millions of operations to drift far enough to ever fill. The capacity-overflow bug is real, reachable, and never hit, because pop keeps undoing the progress push makes. This is passive suppression, the same effect we saw with network faults, now at the level of the workload’s own operations.

The fix is the same too. Instead of running every seed with the whole alphabet, enable a random subset per seed and turn the rest fully off. Disable pop for some runs and the queue fills on the first handful of steps. We call this swarming the alphabet, after the Swarm Testing paper.

moonpool_sim::swarm_op_enabled(op_id) tells you whether operation op_id is in this seed’s subset. Opt in with SimulationBuilder::swarm_operations(). Without it, every operation reports enabled, so default runs use the full alphabet and nothing changes.

#![allow(unused)]
fn main() {
// Cache the subset once per run. Each operation is independently ~50% on,
// a bit of a mask the runner drew once at the start of the iteration, so
// asking consumes nothing and is the same answer every time.
// Empty-mask fallback: if a seed disables everything, use the full alphabet
// so the workload always has something to do.
let enabled: Vec<u8> = (0..NUM_OPS).filter(|&i| swarm_op_enabled(i)).collect();
let enabled = if enabled.is_empty() { (0..NUM_OPS).collect() } else { enabled };

// Per step: take a single draw and remap it into the enabled subset.
let raw = moonpool_sim::sim_random::<u64>();
let op = enabled[(raw % enabled.len() as u64) as usize];
}

The remap matters. We draw once and fold the result into the subset rather than resampling in a loop until we land on an enabled operation. A resample loop would consume a variable number of SIM_RNG values per step, and the fork explorer keys its replay on the in-run RNG call count. Keep the per-step draw count fixed and replay stays exact. When the full alphabet is enabled, enabled[raw % NUM_OPS] == raw % NUM_OPS, so a non-swarm run is byte-identical to one with no swarm code at all.

Like the fault-family version, the mask is drawn from the simulation stream: 256 bits, one per u8 operation id, four draws taken by the runner immediately after seeding and before the fault-family subsets. swarm_op_enabled only reads a bit, so querying consumes no randomness and the answer is independent of how often or in what order you ask. The two swarm axes compose: a seed picks both a fault subset and an operation subset.

The payoff is a demonstrator assertion that only the swarm can reach. Emit it on the path where it holds, so it never records a check on the runs where the subset keeps it impossible:

#![allow(unused)]
fn main() {
// Reachable only when every movement operation was masked off this seed.
// Under the always-on full alphabet, this can never happen.
if !enabled.iter().any(|&i| i < NUM_MOVE_OPS) {
    moonpool_sim::assert_sometimes!(true, "movement suppressed by swarm");
}
}

The dungeon example in crates/moonpool-sim-examples/src/dungeon.rs is the reference implementation, and crates/moonpool-sim/tests/swarm_op_alphabet.rs shows the assertion firing under .swarm_operations() and staying unreachable without it.

Invariant Patterns

Generating interesting operations is half the problem. The other half is knowing whether the system behaved correctly under those operations. Four patterns have proven reliable across many simulation workloads.

Reference models maintain an in-memory expected state and compare after operations. We saw this in the workload chapter: a BTreeMap mirrors what the server should contain, updated on every write, compared on every read. Use BTreeMap, not HashMap. Deterministic iteration order makes failures reproducible.

Conservation laws check quantities that must remain constant. Total record count across shards. Messages sent minus messages received. Committed values in a consensus protocol must agree across replicas. The events and invariants chapter covers this pattern in depth.

Structural integrity validates data structures after chaos. Traverse a linked list and verify no cycles. Walk a B-tree and check balance at every level. Count children and verify parent pointers. These catch corruption that reference models miss because they operate at a different abstraction level.

Operation logging resolves commitment uncertainty. When a write fails with a network error, did it commit or not? Log the intent alongside the mutation. After recovery, read back the state and reconcile against the log. Essential for workloads that operate through unreliable networks, which is all simulation workloads.

Assertions as Memos

Lawrie Green offers a perspective on assertions that changes how you think about placing them. Assertions serve two audiences simultaneously.

They inform the computer about expected behavior. In moonpool, assert_sometimes! tells the explorer “this state is interesting, branch from here.” assert_always! tells the runner “if this fails, stop and report.” Assertions are active participants in the search.

They document the developer’s mental model. When you write assert_always!(balance >= 0, "balance should never go negative"), you are recording a belief about how the system works. The assertion is a memo to your future self and to every developer who reads the code after you.

When an assertion fails on correct code, the mental model was wrong. That is itself a critical finding. Maybe the behavior is fine and the assertion needs updating. Maybe it reveals a design assumption that will cause trouble later. Either way, the mismatch between belief and reality is worth knowing about.

This dual purpose means assertions belong everywhere a developer has an opinion about what should happen. On the error path. On the recovery path. On the “this should never happen” path that definitely will happen in simulation.

Generate Sufficient Workload

A single client sending sequential requests will never trigger the race conditions where distributed bugs hide. The system needs enough concurrent activity for interesting interactions to emerge.

Define all actions the system can take. Not just the obvious ones. Multi-step workflows: “begin transaction, write three keys, read one back, commit.” Interacting operations: “two clients write the same key simultaneously.” Failure-spanning operations: “write during a partition, read after recovery.”

Then run many instances concurrently:

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(3, || Box::new(KvServer::new()))
    .workloads(5, || Box::new(KvWorkload::new(200)))
    .run()
    .await
}

Five workloads running 200 operations each, against three servers, with chaos enabled. The combinatorial interactions between concurrent operations, across servers experiencing faults, produce the complex interleavings where bugs live.

There is a deeper principle here. Individual test cases scale multiplicatively with system complexity. A 300-line feature can require 10,000 lines of manual test code for combinatorial coverage. Test generators scale linearly. Adding one variant to your operation enum covers all new combinations with that operation. You write one line. The simulator explores thousands of new scenarios.

#![allow(unused)]
fn main() {
enum Operation {
    Get { key: String },
    Set { key: String, value: Vec<u8> },
    Delete { key: String },
    Scan { start: String, end: String },
    // Adding this one variant automatically generates
    // combinations with every other operation, every
    // fault type, and every timing interleaving.
    CompareAndSwap { key: String, expected: Vec<u8>, new: Vec<u8> },
}
}

This is the real payoff of simulation testing. Not that each individual test is better, but that the cost of coverage changes from exponential to linear. One well-designed workload, with a complete operation alphabet, strong invariants, and enough concurrency, finds more bugs than a thousand hand-written test cases.

Debugging a Failing Seed

• The Workflow
• What Makes This Different
• A Note on Non-Determinism

A seed failed. The simulation report printed a number, maybe seed=17429853261, next to a red line. What now?

In production distributed systems, debugging a concurrency bug means staring at interleaved logs from multiple nodes, trying to reconstruct a sequence of events that you cannot replay. You form a theory, add logging, deploy, wait for the bug to happen again, and hope your new logs captured enough. It is slow, painful, and often inconclusive.

Deterministic simulation changes this completely. A failing seed is not a clue. It is a recording. Same seed, same execution, same bug, every time. Debugging becomes mechanical rather than archaeological.

The Workflow

The process has five steps:

1. Reproduce the failure with the exact seed and FixedCount(1)
2. Isolate by reading the event trace to find the triggering event
3. Understand the causal chain that led to the violation
4. Fix the root cause in your code
5. Verify by re-running the original seed and then the full chaos suite

Each of these steps is straightforward because determinism gives us something rare in distributed systems: repeatability. We do not hunt ghosts. We replay recordings.

What Makes This Different

Traditional debugging tools assume non-determinism. You set a breakpoint and hope the thread schedule cooperates. You add print statements and hope the race condition still manifests. You run the test ten times and it passes nine.

With simulation, the RNG seed controls everything: which connections fail, when timers fire, what order events process, how long delays take. Pin the seed, and the entire execution is frozen in time. You can add logging, set breakpoints, restructure your investigation, and the bug will be there waiting, exactly where you left it.

The next three sections cover the practical details: how to pin a seed and reproduce, how to read the event trace, and what common mistakes look like so you can recognize them quickly.

A Note on Non-Determinism

If you reproduce with a failing seed and the bug disappears, you have a different problem: something in your system is non-deterministic. This is actually valuable information. Common sources include direct tokio calls that bypass providers, HashMap iteration order leaking into behavior, or system randomness sneaking in through a dependency. The reproducing chapter covers how to track these down.

Reproducing with FixedCount

• Pinning a Seed
• Turning Up the Logging
• When the Bug Disappears
• From Reproduction to Investigation

The simulation report told you that seed 17429853261 failed. The first step is always the same: reproduce it.

Pinning a Seed

Take your simulation binary and add two things: the failing seed, and a single iteration.

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .workload(MyWorkload::new())
    .set_iterations(1)
    .set_debug_seeds(vec![17429853261])
    .run()
    .await
}

set_debug_seeds fixes the RNG seed. set_iterations(1) tells the runner to execute exactly one iteration with that seed instead of sweeping through random seeds. Together, they replay the exact execution that failed.

Run it. You should see the same failure, the same panic message, the same assertion violation. Every time.

Turning Up the Logging

By default, simulation runs are quiet. When debugging a specific seed, turn up logging via the RUST_LOG environment variable. Start with ERROR to see assertion violations and failure messages:

RUST_LOG=error cargo xtask sim run my_simulation

For deeper investigation, RUST_LOG=trace shows the full event trace: every event processed, every timer fired, every connection state change. This is the firehose, but with a pinned seed it is a reproducible firehose.

When the Bug Disappears

You pinned the seed, ran it, and the bug is gone. This means one thing: something in your code is non-deterministic.

The simulation engine controls time, networking, and randomness through providers. But if your code bypasses those providers, it introduces real-world non-determinism into the simulated world. Common culprits:

• Direct tokio calls: tokio::time::sleep() instead of time.sleep(), tokio::spawn() instead of task_provider.spawn_task(). These escape the simulation’s control.
• System randomness: Using rand::thread_rng() or std::collections::HashMap iteration order instead of the simulation’s RandomProvider.
• Wall-clock time: Calling std::time::Instant::now() or SystemTime::now() instead of using the simulated clock.
• External I/O: Reading files, making real network calls, or anything that touches the actual operating system.

If you suspect non-determinism, run the same seed twice with RUST_LOG=trace and diff the output. The first divergence point tells you exactly where determinism broke.

From Reproduction to Investigation

Once you have a reliable reproduction, the next step is understanding what happened. The event trace (covered in the next section) shows you the causal chain: which events fired, in what order, and what state they produced. With a pinned seed, you can also set breakpoints in your IDE and step through the exact execution path that triggers the bug.

The key insight: you are not searching for a bug anymore. You are reading a recording of a bug. That is a fundamentally easier problem.

Reading the Event Trace

• The Scheduler
• Key Event Types
• Tracing the Causal Chain
• Using RNG Call Count
• Infrastructure vs Workload Events
• Practical Tips

You have a pinned seed reproducing the failure. Now you need to understand what happened. The event trace is your primary tool.

The Scheduler

Moonpool’s simulation engine is built around Scheduler<Event>, a priority scheduler ordered by logical time and a stable ScheduleId. It owns the one monotonic simulation clock, same-time FIFO sequence allocation, and eager cancellation. Scheduling into the past clamps to the current time. Cancelled entries disappear without advancing time.

The scheduler coordinates delayed work without owning resource state. It dispatches NetworkEvent values to NetworkSimulation and StorageEvent values to StorageEngine. Those components update their own state, resolve the exact pending operation, and return ordered effects, faults, and wake batches. The coordinator applies those effects and wakes tasks after releasing the world lock.

When you enable trace-level logging (RUST_LOG=trace), you see every event as it fires:

  Processing event at t=1.234s seq=47: Network(Delivery { connection_id: 3, seq: 12 })
  Processing event at t=1.234s seq=48: Timer { task_id: 12 }
  Processing event at t=2.500s seq=49: Storage(operation_id=9, WriteComplete)

Each line tells you what happened, when, and in what order.

Key Event Types

The simulation has a small set of event types, and learning to recognize them makes traces much easier to read.

Timer events wake sleeping tasks. When your workload calls time.sleep(Duration::from_secs(1)), that schedules a Timer event one second in the future. These are the heartbeat of your simulation.

Network events target the network engine. OperationReady completes one delayed bind, connect, or accept latency. ProcessSendBuffer moves the next chunk of a connection’s send queue onto the wire. Delivery says an in-flight item (a data chunk, or the FIN of a graceful close) has reached its delivery time; the item itself lives in the sender’s in-flight queue, and the event lands it only if no partition is holding that direction, so a Delivery that fires under a cut does nothing and the heal re-times the item.

Network maintenance events change connection-wide state. PartitionRestore, SendPartitionClear, and RecvPartitionClear remove expired cuts. ClogClear and ReadClogClear release parked stream operations only when the event still matches the active deadline, so an old expiry cannot clear a newer clog.

Storage events carry an exact OperationId, handle ID, and operation kind. Reads, writes, syncs, and set-length operations complete only their matching pending entry. The engine stores an explicit success or error result and wakes only that operation’s waiter.

Process lifecycle events manage reboots. ProcessGracefulShutdown signals a process to clean up. ProcessForceKill aborts it after the grace period. ProcessRestart brings it back.

Shutdown wakes all tasks for orderly termination at the end of a simulation.

Tracing the Causal Chain

When an assertion fires, the question is: what caused this? The event trace gives you the answer, but you read it backwards.

Start at the failure. Look at the last few events before the assertion. Usually one of them is the trigger: a Delivery that landed a stale message, a Timer that expired causing a timeout, or an OperationReady that let a connection race complete. Then ask which component requested that schedule. Follow the chain back through the trace.

For example, suppose your conservation law invariant fires after event #312. Look at event #312: it is a Delivery on connection 7. What was connection 7? The trace shows it was established at event #201 between the workload and a KV server process. What did the delivery contain? A withdraw response. But the model expected a deposit. Now you have a lead.

Using RNG Call Count

Every random decision in the simulation consumes one or more calls to the deterministic RNG. The total call count at any point in the execution is a precise fingerprint of “where we are.”

When comparing a working seed against a failing seed, the RNG call count tells you exactly where their executions diverge. If both seeds process events identically through RNG call 847, but diverge at call 848, the code executing at that point made a different random choice that led down the failing path.

This technique is especially useful for regression testing: if you fix a bug and the RNG call pattern changes, you know your fix altered the execution path (which is expected). If it does not change, your fix might not be reaching the right code.

Infrastructure vs Workload Events

Not every event in the trace matters to your investigation. The simulation marks some events as infrastructure: PartitionRestore, SendPartitionClear, RecvPartitionClear, and ProcessRestart. These maintain simulation state but do not represent application work.

The simulation uses this distinction internally to decide when to terminate. After all workloads finish, if only infrastructure events remain in the scheduler, the simulation can safely end. When reading traces, you can often skip them and focus on OperationReady, Delivery, Timer, and Storage events that directly affect application progress.

Practical Tips

Start narrow. Use RUST_LOG=error first to see just the failure. Then widen to RUST_LOG=debug or RUST_LOG=trace only if you need more context.

Search for the event sequence number. The invariant failure happens after a specific sim.step() call. The event processed in that step has a sequence number. Search for it in the trace.

Count backwards. If the failure is at event #312, the cause is often in the 5-10 events before it, not 200 events earlier.

Compare two seeds. Run a passing seed and a failing seed side by side with trace output. Diff the two traces. The first divergence point is where the bug’s path begins.

Common Pitfalls

• Delayed Operations Need the Scheduler
• Confusing Task Yield With Driver Drain
• Using unwrap()
• Direct Tokio Calls
• Wrong Runtime Flavor
• Using ?Send on Dyn Traits
• Holding a Lock Across .await
• Smuggling !Send Types Into Spawned Futures
• Moving Resource State Back Into world.rs
• Unordered Collection Non-Determinism
• Forgetting to Emit Events for Invariants

A reference list of mistakes we have seen (and made) when building simulations with Moonpool.

Delayed Operations Need the Scheduler

Storage read, write, sync, and set-length operations wait for targeted completion events. Network bind, connect, and accept are delayed operations too. If a low-level test polls one of these futures without stepping SimWorld, the future cannot complete.

Fix: Prefer SimulationBuilder::run(), which already interleaves executor drains and scheduler steps. For a low-level provider test, use Executor::block_on(...) and a poll_fn driver that steps SimWorld while the future is pending. The complete helper is in Storage Faults.

Confusing Task Yield With Driver Drain

Spawned tasks do not run until the current task yields or parks. If application code spawns a task and immediately checks shared state, that task may not have run yet. Use task_provider.yield_now().await when the application genuinely needs to yield.

The simulation driver has a different job. After sim.step() wakes tasks, it uses executor::until_stalled().await to drain every runnable task before the next event. until_stalled() is driver-only. Awaiting it from a spawned task is an error because that task would be waiting for a drain that includes itself.

Using unwrap()

Moonpool follows a strict no-unwrap() policy. In simulation, a panic from unwrap() is not a clean error report. It is an uncontrolled crash that may mask the real failure and confuse the assertion system.

Fix: Use Result<T, E> with ? everywhere. Map errors with context when needed. For RwLock poison, use .expect("RwLock poisoned: prior task panicked") since poisoning means a prior panic already happened.

Direct Tokio Calls

Calling tokio::time::sleep(), tokio::time::timeout(), or tokio::spawn() bypasses the simulation’s control of time and task scheduling. Your code will use real wall-clock time instead of simulated time, and the simulation cannot inject faults.

Fix: Use provider traits: time.sleep(), time.timeout(), task_provider.spawn_task().

Wrong Runtime Flavor

Moonpool runs on a single OS thread for determinism. Wrapping a simulation in #[tokio::main] or building a tokio runtime around it hands scheduling to tokio: at best redundant, at worst (multi-thread) real parallelism that destroys reproducibility, even though traits are Send-bounded.

Fix: Drive the simulation with the moonpool executor. SimulationBuilder::run() does this for you, and standalone code uses Executor::new(seed) plus executor.block_on(...) (see The Deterministic Executor). Do not use LocalSet, spawn_local, or build_local(): they are gone from the model.

Using ?Send on Dyn Traits

Moonpool’s dyn-stored traits (Process, Workload, FaultInjector) carry Send + Sync + 'static supertraits so customer code can share state through Arc<RwLock<…>> and DashMap. Writing #[async_trait(?Send)] on your impl removes the Send bound the trait promises and the compiler rejects it.

Fix: Use plain #[async_trait] (no ?Send) on dyn-stored impls. Provider traits use native AFIT with -> impl Future<…> + Send and need no attribute at all.

Holding a Lock Across .await

Because spawned futures must be Send, any local you hold across an .await point must also be Send. A std::sync::MutexGuard is !Send, so holding one across an await fails to compile with a confusing “future is not Send” error pointing at the spawn site, not the lock.

Fix: Scope the guard tightly. Lock, read or mutate, drop the guard, then .await. Prefer parking_lot::Mutex or tokio::sync::Mutex when you genuinely need to hold state across awaits — and even then, drop the guard before any long-running await.

#![allow(unused)]
fn main() {
// Bad: guard lives across the await
let guard = state.lock().expect("RwLock poisoned: prior task panicked");
network.send(&guard.payload).await?;

// Good: drop the guard first
let payload = {
    let guard = state.lock().expect("RwLock poisoned: prior task panicked");
    guard.payload.clone()
};
network.send(&payload).await?;
}

Smuggling !Send Types Into Spawned Futures

Rc, RefCell, and raw pointers are !Send. Capturing them in a future that you hand to tokio::spawn or task_provider.spawn_task() poisons the whole future and the compiler refuses it.

Fix: Use Arc<RwLock<…>> or Arc<AtomicX> for shared mutable state. If you have a legitimate single-task data structure, keep it on the stack of a non-spawned future, never close over it in a spawn.

Moving Resource State Back Into world.rs

SimWorld coordinates a global Scheduler<Event> and lifecycle. Network state, faults, operations, and wakers belong to NetworkSimulation. The corresponding storage data belongs to StorageEngine. Reaching through SimInner to mutate a resource and then scheduling directly couples ownership again and usually creates difficult borrows.

Fix: Add a targeted component transition. Let it return ordered schedule and cancel effects, fault records, and a WakeBatch. The world applies those effects through the scheduler and invokes the wakers after releasing its lock. Never wake while holding the world lock because a waker may immediately poll and re-enter the same component.

Unordered Collection Non-Determinism

std::collections::HashMap does not guarantee iteration order, and the order can vary between runs. If your workload iterates a HashMap and the iteration order affects behavior (choosing which account to process, which message to send), you have introduced non-determinism.

Fix: Use BTreeMap / BTreeSet, or collect dependency-owned unordered data into a Vec and sort before iterating. Moonpool’s own workspace denies HashMap and HashSet through Clippy so this class of replay leak fails CI.

Forgetting to Emit Events for Invariants

Invariants read plain tracing events captured into the timeline. If your workload changes state but forgets to emit a corresponding event, invariants see an incomplete history and either miss bugs or report false violations.

Fix: Emit a trace event for every state change you want invariants to observe. The same event also flows to production observability subscribers, so this serves a dual purpose.

#![allow(unused)]
fn main() {
self.model.record_commit(slot, value);
tracing::info!(slot, value, "commit");
}

Discovering Properties

• The Attention Focus Pattern
• Eight Focuses for Moonpool Code
• Using the Focuses
• What Discovery Produces
• A Quick Example

You know the assertion macros. You know the buggify patterns. But when you sit down with a fresh Process and Workload, the hardest question is not how to assert — it is what to assert. Where should assert_always! go? Which code paths need assert_sometimes!? Where would buggify!() expose the most interesting failures?

A single unstructured pass through your code will find the obvious properties. The non-obvious ones — the properties that catch real bugs — require looking at the same code from multiple independent angles.

The Attention Focus Pattern

The idea comes from Antithesis’s property discovery methodology. Instead of reading through code once and noting assertions as they occur to you, you examine the same code eight times, each time through a different lens. Each lens is called an attention focus.

Why does this work? Because different failure modes hide in different mental models. A developer thinking about crash recovery notices different things than one thinking about concurrency. A focus on protocol contracts surfaces properties that a focus on resource boundaries would miss entirely. The structured repetition is the point.

Eight Focuses for Moonpool Code

State Integrity examines in-memory invariants and storage persistence. What state must survive reboots? What write ordering assumptions exist? If a process is killed between writing field A and field B, does recovery handle the inconsistency? Look for monotonicity properties (ballot numbers, sequence IDs), derived state that could diverge from its source, and reference model expectations that the process might violate.

Concurrency looks at races between workloads hitting the same process. Moonpool runs on a single OS thread, but our traits are Send-bounded so customer code naturally uses Arc<RwLock<…>>, Arc<Mutex<…>>, or DashMap. Async interleaving across .await points creates real concurrency hazards even on one thread. Check-then-act patterns where another task could mutate state between the check and the act. Multiple workloads accessing the same key. Arc<RwLock<…>> state read, then mutated, with a yield point in between.

Crash Recovery asks what happens when a Process is killed and restarted from its factory. The factory returns a blank instance — all in-memory state is gone. Partially-written storage operations (write without sync_all()) may leave corrupt data. Recovery code that assumes clean state will miss torn writes. Operations interrupted between a storage write and a sync are the classic source of subtle bugs.

Network Faults examines behavior under connection drops, partitions, and reordering. RPC calls without timeouts. Retry logic that is not idempotent. Stale cached state after a partition heals (old leader references, expired peer info). Fire-and-forget sends where delivery actually matters.

Timing & Scheduling checks sensitivity to event ordering. Hardcoded timeouts that interact with other timeouts. Logic that assumes timers fire before network events arrive. Election timeouts that could overlap with heartbeat timeouts. Races between timer expiry and data delivery.

Resource Boundaries hunts for unbounded growth. Vec or VecDeque that grows without limit under sustained load. Missing backpressure on incoming requests. Operations that assume connections or file handles are always available. These bugs only surface under chaos, which is exactly when they matter most.

Protocol Contracts looks for guarantees that are claimed but not enforced. Doc comments that say “returns error if not found” but the code returns a default. Ordering assumptions between RPC calls (create before update) that nothing validates. Response types that mask partial failures as success.

Lifecycle Transitions examines startup, shutdown, and reboot sequences. Requests arriving before initialization completes. In-flight work silently dropped during graceful shutdown. State published to StateRegistry before it is valid. Shutdown ordering dependencies where closing connections before flushing storage loses data.

Using the Focuses

There are two ways to work through the focuses.

Ensemble mode runs all focuses in parallel. For each focus, a separate analysis pass examines the code through that single lens and produces candidate assertions and buggify points. The results are then synthesized: duplicates found by multiple focuses are high-confidence properties, while unique finds from a single focus are high-value catches that a single pass would have missed.

Sequential mode works through the focuses as a checklist. The key discipline is making an explicit pass for each focus. Do not skip a focus because an earlier pass “already covered” that area. The value is in the independent perspective — the same line of code looks different through the lens of crash recovery than through the lens of concurrency.

What Discovery Produces

Each discovered property maps to a concrete assertion or buggify placement:

• Location: exact file and line range
• Macro: which assertion type (assert_always!, assert_sometimes!, buggify!(), etc.)
• Message: unique, descriptive assertion message
• Rationale: what bug this catches and why it matters
• Provenance: which focus or focuses surfaced it

Properties found by multiple focuses independently are strong candidates. They represent invariants that sit at the intersection of multiple failure modes. Properties found by only one focus are equally important — they represent the blind spots that unstructured review misses.

A Quick Example

Consider a key-value Process that accepts writes over RPC and persists them to storage. A single review pass might add assert_always! for “response matches stored value” and call it done.

Running through the focuses reveals more:

• State Integrity: the process maintains a write counter. Is it monotonically increasing? assert_always!(new_count > old_count, "write counter regression")
• Crash Recovery: writes go to an in-memory map, then flush to storage on a timer. A crash between write and flush loses data. buggify!() on the flush path to test this. assert_sometimes! to verify the recovery path is actually exercised.
• Concurrency: two workloads writing the same key. The last-write-wins semantics should hold. assert_always! on read-after-write consistency from each workload’s perspective.
• Network Faults: the client retries on timeout, but the write is not idempotent. A retry after an ambiguous failure could double-apply. assert_always! on the write count matching expected count.

Four focuses, four distinct properties, one of which (the retry idempotency bug) is the kind of subtle issue that crashes production systems. A single pass would likely have caught only the first.

The /discover-properties skill automates this process. It examines your Process and Workload code through all eight focuses and produces a structured list of assertion and buggify placements, ready to implement.

Using moonpool-sim Standalone

• The Technical Foundation
• Proof: Real HTTP Over Simulated TCP
• Spawning Inside a Simulation
• What This Means For You

moonpool-sim is a standalone simulation engine. Provider traits, chaos injection, assertions, and fork-based exploration work with your existing application protocol. That may be raw TCP, axum behind hyper, tonic gRPC, or a service whose database boundary is replaced with an in-memory fake.

Why does this matter? Most teams are not designing a new networking stack. They are running services behind a load balancer, talking to databases, and shipping features. Moonpool controls the runtime boundaries while their application code stays recognizable.

The Technical Foundation

The key fact that makes this possible lives in the NetworkProvider trait:

#![allow(unused)]
fn main() {
pub trait NetworkProvider: Clone + Send + Sync + 'static {
    type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;
    // ...
}
}

SimTcpStream implements futures::io::AsyncRead + AsyncWrite + Unpin (the runtime-agnostic IO traits) and is Send + Sync + 'static. One thin adapter turns that into whatever shape a library wants, which makes it a drop-in replacement for tokio::net::TcpStream anywhere the ecosystem uses trait-based I/O. And the tokio ecosystem uses trait-based I/O everywhere that matters: hyper, tonic, tower, axum (via hyper), sqlx’s wire protocol, redis-rs.

This isn’t an accident. We designed the provider traits to match tokio’s interfaces exactly because we wanted existing libraries to work unchanged.

Proof: Real HTTP Over Simulated TCP

The hyper integration test in crates/moonpool-sim/tests/hyper_http.rs demonstrates this concretely. Unmodified hyper HTTP/1.1 running over simulated TCP with chaos injection:

#![allow(unused)]
fn main() {
struct HyperServer;

#[async_trait]
impl Process for HyperServer {
    fn name(&self) -> &str { "server" }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        // moonpool's select!, not tokio's: `biased;` because shutdown is a
        // guard on the accept work, not a peer racing it.
        let (stream, _addr) = moonpool_sim::select! {
            biased;
            result = listener.accept() => result?,
            _ = ctx.shutdown().cancelled() => return Ok(()),
        };

        // SimTcpStream implements the futures-io traits; HyperIo presents
        // them as the IO traits hyper defines for itself.
        let io = HyperIo::new(stream);

        hyper::server::conn::http1::Builder::new()
            .serve_connection(io, service_fn(handle_request))
            .await?;

        Ok(())
    }
}
}

Hyper 1.x defines its own rt::Read and rt::Write traits, and moonpool_hyper::HyperIo implements them directly over the futures-io shape every provider stream already has. From hyper’s perspective nothing has changed: HTTP parser, chunked encoding, keep-alive logic, content-length validation, all exercised for real over simulated networking.

The client side follows the same pattern. Connect via ctx.network().connect(), wrap in HyperIo, hand to hyper::client::conn::http1::handshake. Real HTTP/1.1 request-response cycles over a network that drops packets, injects latency, and kills connections.

HTTP/2 asks more of the runtime than this, because it spawns work and reads a clock. The hyper Stack covers what that takes, and what moonpool-hyper hands you for gRPC.

Spawning Inside a Simulation

The sim is single-threaded by construction. Every iteration runs on the moonpool deterministic executor, so every spawned task runs on the one OS thread that drives simulation events. Determinism depends on it.

What changed compared to earlier moonpool versions: the types crossing trait boundaries are now Send + 'static. Provider traits, SimTcpStream, Process, Workload, the SimContext you get handed, all of them. Spawning goes through the executor, either directly with moonpool_sim::executor::spawn (named tasks, the name shows up when debugging a seed) or through the provider seam, and customer state can use Arc<RwLock<…>>, DashMap, Arc<AtomicBool> without ceremony.

#![allow(unused)]
fn main() {
// Works: SimTcpStream is Send, the future is Send, the executor accepts it.
let handle = moonpool_sim::executor::spawn("serve-one", async move {
    serve_one(stream).await
});
}

If you want to keep the provider seam (so the same code runs against a real tokio runtime later), reach for TaskProvider::spawn_task from ctx.task() instead. Same semantics, an injectable interface, and inside the sim it lands on the same executor.

Do not use tokio::task::spawn_local or build a LocalSet. There is no tokio runtime inside a simulation, so there is no LocalSet for spawn_local to attach to and the spawn would never poll. Whenever you reach for a !Send workaround, you have probably wandered off the supported path.

The one case that still bites people: some third-party connection futures (hyper’s Connection is the canonical example) are themselves !Send for reasons unrelated to moonpool. They hold internal state that isn’t Send. You cannot spawn those on any Send-bounded executor, moonpool’s included. The fix is to drive them inline alongside your other work:

#![allow(unused)]
fn main() {
let (mut sender, conn) = hyper::client::conn::http1::handshake(io).await?;

// hyper's connection future is !Send by its own design.
// Drive it inline with select! instead of spawning.
let driver = async move {
    let _ = conn.await;
};

moonpool_sim::select! {
    biased;
    result = send_requests(&mut sender) => result?,
    _ = driver => {}
    _ = ctx.shutdown().cancelled() => {}
}
}

That is a hyper API decision, not a moonpool constraint. Axum handlers, tonic services, tower middleware, all Send. Spawn them freely.

What This Means For You

If your application uses tokio::net::TcpStream through trait-based I/O (which hyper, axum, tonic, and most of the ecosystem do), you can simulate it. The process is:

1. Define a Process that binds a listener and serves connections
2. Define a Workload that connects and sends requests
3. Wire them together with SimulationBuilder
4. Run thousands of iterations with chaos injection

No actor system required. No RPC framework. No new programming model. Just your existing HTTP handlers running over a network that tries to break them.

Where to Draw the Line

• The Partial Failure Problem
• Fakes Give You Control
• The Fidelity Spectrum
• Per-Dependency Guidance
• Rules of Engagement
• The 80% Argument

The instinct when someone says “fake your dependencies” is to feel like you’re cutting corners. Real databases catch real bugs. Test containers give you the real thing. A BTreeMap pretending to be Postgres is a compromise.

That instinct is wrong. Fakes aren’t a compromise. They’re more powerful than real dependencies.

The Partial Failure Problem

Test containers give you binary failure: the whole service is up, or the whole service is down. Docker kills the container, your test sees a connection refused, you verify your retry logic works.

Production failures are never binary. Kafka loses partition 3 while partitions 1, 2, and 4 stay healthy. A Postgres replica lags 800ms behind the primary while the primary is fine. One Redis shard OOMs while five others serve traffic normally. Your S3 bucket returns 503 on 2% of PUTs while GETs succeed at full speed.

These partial failures are where bugs hide. The request that reads from the lagging replica and writes to the healthy primary. The consumer that rebalances partitions and loses its offset for exactly one topic. The cache lookup that fails for one key prefix while the rest of the keyspace works.

Test containers cannot produce these failures. A container is a black box. You can start it or stop it. You cannot reach inside and make partition 3 return errors while partition 4 succeeds.

Fakes Give You Control

A trait-based fake controls the failure surface at arbitrary granularity. Your MessageBroker trait fake can return Ok for partition 1 and Err for partition 3 in the same call. Your Database fake can inject 200ms latency on reads from replica 2 while replica 1 responds instantly. Your Cache fake can evict entries for keys matching a pattern while retaining everything else.

Combined with moonpool’s buggify!() macro, these fakes become probabilistic fault injectors. Every operation has a chance of failure, controlled by a deterministic seed. When a test fails, you replay the exact same seed and get the exact same sequence of partial failures.

#![allow(unused)]
fn main() {
impl Store for InMemoryStore {
    fn create(&self, name: &str) -> Result<Item, StoreError> {
        // 25% chance of write failure when buggify is enabled.
        // A real Postgres can only be fully up or fully down.
        // This fake can fail individual writes.
        if buggify!() {
            return Err(StoreError::WriteFailed("buggified".into()));
        }
        // ... normal implementation
    }
}
}

The Fidelity Spectrum

Not every dependency needs full simulation. Think of fakes on a spectrum:

No-op: Returns Ok(()) for everything. Useful when you don’t care about a dependency’s behavior, just that calls don’t crash. Logging, metrics, tracing facades.

In-memory: BTreeMap-based storage, VecDeque-based queues. Correct behavior without persistence. Good for most unit and integration tests.

Fault-injectable: In-memory with buggify!() on operations. Correct behavior most of the time, controllable failures when chaos is enabled. This is where most simulation fakes live.

Full simulation via moonpool: The dependency runs as a Process in the simulation. Network traffic is simulated with latency, drops, and corruption. Reserved for components where the network interaction IS the interesting behavior (your own services, consensus protocols).

Per-Dependency Guidance

Network (HTTP, gRPC, TCP): Simulate via moonpool. This is the sweet spot. Real HTTP parsing, real serialization, simulated transport. Your handlers run unchanged.

Database: Trait fake with BTreeMap. Model the operations your code actually uses (CRUD, transactions, queries by index). Inject failures per-operation. Don’t try to simulate SQL parsing.

Message brokers: Trait fake with per-partition control. A VecDeque<Message> per partition, with injectable failures per-partition. Model consumer group rebalancing if your code depends on it.

External HTTP APIs: Canned responses with injectable failures. Your Stripe client fake returns a known charge object, but buggify!() returns a 429 or network timeout 10% of the time.

Cache: Trait fake with partial cluster modeling if you need it, simple BTreeMap if you don’t. Inject evictions and connection failures.

Rules of Engagement

BTreeMap, not HashMap. Deterministic iteration order matters for reproducibility. A HashMap iterating in different order across runs makes your simulation non-deterministic.

Send + Sync for axum State. Axum requires State to be Send + Sync. Reach for Arc<RwLock<BTreeMap<...>>>, not Rc<RefCell<...>>. Everything in moonpool is Send-bounded now, so Arc<RwLock<…>> is the natural shape and plugs straight into axum without contortions. Execution still happens on a single OS thread (the moonpool deterministic executor), so in practice the lock is never contended, but it’s the Send bounds that make the type fit, not the thread count.

The Oxide convention. Oxide’s omicron and crucible projects keep fakes in a fakes/ module alongside the real implementation. The trait and the fake ship together. When you change the trait, you update the fake in the same PR. This keeps fakes from drifting.

The 80% Argument

A fake covering 80% of a dependency’s behavior with determinism and fault injection is strictly better than a test container covering 100% with no control over failures.

Test containers are non-deterministic. They’re slow (seconds to start). They require Docker (not available in all CI environments). They break across versions (Postgres 15 vs 16 container images). They can’t produce partial failures. A failing test gives you a log and a prayer.

A fake compiles with your project. It runs in microseconds. It reproduces from a seed. It produces failures that containers physically cannot. The 80% you model covers the behavior your code actually depends on.

The remaining 20% you don’t model? That belongs in separate integration tests, run less often, against real infrastructure. Nightly CI with actual Postgres and Kafka containers. Those tests verify your fakes are faithful. The simulation tests, run on every commit, verify your code handles failure correctly.

Both kinds of tests improve each other. When an integration test reveals a failure mode your fake doesn’t model, you add it to the fake. When a simulation test finds a bug under partial failure, you verify the fix against real infrastructure. The two approaches compound.

Wiring a Web Service

• Step 1: The Store Trait
• Step 2: The InMemoryStore
• Step 3: The Axum Router
• Step 4: The Process
• Step 5: The Workload
• Step 6: Wire It Together

Theory is cheap. Here’s a complete worked example: an axum web service running inside moonpool-sim with chaos injection, fault-injectable storage, and assertion-based validation. The full source lives in crates/moonpool-sim-examples/src/axum_web.rs.

Step 1: The Store Trait

Every dependency boundary starts with a trait. This one models item persistence:

#![allow(unused)]
fn main() {
pub trait Store: Send + Sync + 'static {
    fn create(&self, name: &str) -> Result<Item, StoreError>;
    fn get(&self, id: u64) -> Result<Option<Item>, StoreError>;
}
}

Send + Sync + 'static because axum requires State to be Send + Sync. In production, this trait is backed by Postgres or SQLite. In simulation, it’s backed by a BTreeMap.

Notice these are synchronous methods. The real database calls would be async, but for a fake that never does I/O, synchronous is simpler and equally correct. If your production trait has async methods, that works too.

Step 2: The InMemoryStore

BTreeMap for deterministic ordering. AtomicU64 for ID generation. RwLock because axum needs Send + Sync.

#![allow(unused)]
fn main() {
pub struct InMemoryStore {
    items: RwLock<BTreeMap<u64, Item>>,
    next_id: AtomicU64,
}

impl Store for InMemoryStore {
    fn create(&self, name: &str) -> Result<Item, StoreError> {
        // Fault injection: randomly fail writes.
        // Models disk full, replication lag, constraint violations.
        if buggify!() {
            return Err(StoreError::WriteFailed("buggified".into()));
        }

        let id = self.next_id.fetch_add(1, Ordering::Relaxed);
        let item = Item { id, name: name.to_string() };
        self.items.write()
            .map_err(|e| StoreError::WriteFailed(format!("{e}")))?
            .insert(id, item.clone());
        Ok(item)
    }

    fn get(&self, id: u64) -> Result<Option<Item>, StoreError> {
        // Lower probability: reads fail less often than writes in practice.
        if buggify_with_prob!(0.05) {
            return Err(StoreError::ReadFailed("buggified".into()));
        }

        Ok(self.items.read()
            .map_err(|e| StoreError::ReadFailed(format!("{e}")))?
            .get(&id).cloned())
    }
}
}

The buggify!() calls are the whole point. A Postgres container is either up or down. This fake can fail a write while the next read succeeds. It can fail creates at 25% while gets fail at 5%, modeling asymmetric failure that actually happens in production.

Step 3: The Axum Router

Standard axum. Nothing moonpool-specific here:

#![allow(unused)]
fn main() {
pub fn build_router(store: Arc<dyn Store>) -> axum::Router {
    axum::Router::new()
        .route("/health", get(health))
        .route("/items", post(create_item))
        .route("/items/{id}", get(get_item))
        .with_state(store)
}
}

The handlers use State(store): State<Arc<dyn Store>> and return standard axum responses. create_item returns 201 on success, 500 when the store fails. get_item returns 200, 404, or 500. If you already have an axum app, your existing router works here.

Step 4: The Process

This is where moonpool enters the picture. A Process is the system under test, running on a simulated server node:

#![allow(unused)]
fn main() {
#[async_trait]
impl Process for WebProcess {
    fn name(&self) -> &str { "web" }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        use futures::{FutureExt, stream::{FuturesUnordered, StreamExt}};

        let store = InMemoryStore::new();
        let app = build_router(store);
        let listener = ctx.network().bind(ctx.my_ip()).await?;

        // hyper's serve_connection future is !Send (the HTTP1 state machine
        // holds internal Rc<…>), so no Send-bounded spawn will accept it.
        // Drive multiple in-flight connections inline via FuturesUnordered.
        let mut connections = FuturesUnordered::new();

        loop {
            // Deliberately NOT `biased;`: accept and connection progress are
            // peer data sources, so the seed picks which branch wins.
            moonpool_sim::select! {
                accept = listener.accept() => {
                    let (stream, _addr) = accept?;

                    // SimTcpStream implements the futures-io traits, which is all
                    // HyperIo needs to present it as hyper's own IO. Vectored
                    // writes on: the sim delivers each IoSlice separately, with
                    // its own chance of chaos.
                    let io = HyperIo::new(stream).with_vectored_writes(true);
                    // TowerToHyperService bridges axum's tower::Service to hyper's Service
                    let service = TowerToHyperService::new(app.clone());

                    connections.push(async move {
                        if let Err(e) = hyper::server::conn::http1::Builder::new()
                            .serve_connection(io, service)
                            .await
                        {
                            tracing::debug!("hyper error (expected under chaos): {e}");
                        }
                    });
                }
                Some(()) = connections.next(), if !connections.is_empty() => {}
                _ = ctx.shutdown().cancelled() => return Ok(()),
            }
        }
    }
}
}

Two things to note. First, we use hyper::server::conn::http1::serve_connection, not axum::serve(). axum::serve takes tokio::net::TcpListener directly, so it can’t accept our simulated listener. serve_connection takes anything implementing hyper’s own IO traits. SimTcpStream implements futures::io::AsyncRead + AsyncWrite (the runtime-agnostic traits) and is itself Send + Sync, so we wrap it in moonpool_hyper::HyperIo, which is exactly that adapter. TowerToHyperService comes from the same crate, so the axum example needs neither tokio-util nor hyper-util.

Second, FuturesUnordered instead of spawning. Customer code in the sim is Send + Sync, and SimTcpStream is Send. The blocker is hyper itself: http1::serve_connection returns a future that is !Send because the HTTP1 state machine holds internal Rc<…> cells. That future cannot cross any Send-bounded spawn, but it polls perfectly well from the accept loop. FuturesUnordered lets one task drive any number of in-flight connections concurrently. The accept loop pushes new connections in, the connections.next() arm drains completed ones, and shutdown cancels both. The select! is moonpool’s and stays in its default form on purpose: accepting a new connection and progressing an in-flight one are peers that can be ready in the same simulation step, so the seed decides which branch wins and different seeds explore both orders.

Step 5: The Workload

The workload is the test driver. It connects to the process, sends requests, and validates responses:

#![allow(unused)]
fn main() {
#[async_trait]
impl Workload for WebWorkload {
    fn name(&self) -> &str { "client" }

    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> {
        let server_ip = ctx.peer("web").ok_or_else(|| {
            SimulationError::InvalidState("web process not found".into())
        })?;

        for round in 0..5 {
            match self.send_round(ctx, &server_ip, round).await {
                Ok(()) => {}
                Err(e) => {
                    // Under chaos, requests can fail. That's expected.
                    assert_sometimes!(true, "request_round_failed");
                    tracing::debug!("round {round} failed: {e}");
                }
            }
        }
        Ok(())
    }
}
}

Inside send_round, the workload opens a hyper client connection. The client side has the same !Send problem: hyper::client::conn::http1::handshake hands back a conn driver future that is !Send. Rather than spawn it, we pin it on the stack and let the request/response future race against it:

#![allow(unused)]
fn main() {
let io = HyperIo::new(stream).with_vectored_writes(true);
let (mut sender, conn) = hyper::client::conn::http1::handshake(io).await?;

let driver = async move {
    if let Err(e) = conn.await {
        tracing::warn!("client conn driver error: {e}");
    }
};
let driver = std::pin::pin!(driver);

// sender.send_request(...) runs below. The surrounding moonpool
// select! in send_round polls `driver` concurrently.
}

Assertions split into two kinds:

• assert_always! for invariants: health returns 200, read-after-write returns the same data, nonexistent items return 404 or 500.
• assert_sometimes! for coverage: items sometimes created successfully, store reads sometimes fail, request rounds sometimes fail under chaos.

The assert_sometimes! calls are how moonpool knows it’s actually exercising error paths. If store_write_failed never triggers across thousands of iterations, something is wrong with the chaos configuration.

Step 6: Wire It Together

#![allow(unused)]
fn main() {
SimulationBuilder::new()
    .processes(1, || Box::new(WebProcess))
    .workload(WebWorkload)
    .set_iterations(10)
    .run();
}

One web server process, one workload driving requests, ten iterations with different seeds. Each iteration creates a fresh simulation: new network, new processes, new store state, new buggify activation decisions.

The default network configuration injects latency and connection faults. Combined with buggify!() in the store, your handlers face both network-level chaos (connection drops, latency spikes) and application-level chaos (write failures, stale reads) deterministically and reproducibly.

When a seed fails, you replay it with set_debug_seeds(vec![failing_seed]) and set_iterations(1) to reproduce the exact sequence of events that triggered the bug.

The hyper Stack: gRPC and HTTP

• Serving: One Call Per Connection
• Connecting: The Channel Owns the Connection
• The Sealed Bounds
• Why Any of This Is Deterministic

The axum chapter got a web service into simulation by handing hyper a simulated stream. That works because HTTP/1 asks almost nothing of the runtime: parse bytes, write bytes, done. HTTP/2 is a different animal. It multiplexes streams over one connection, it pings the peer to notice death, and it runs per-request work concurrently. To do any of that, hyper needs to spawn tasks and read a clock.

hyper is runtime-agnostic about both. It defines rt::Executor and rt::Timer traits and asks you to supply them. In production everyone supplies hyper-util’s tokio versions, which is why the ecosystem reads as tokio-only. Inside a simulation those are exactly the two things we cannot allow: a tokio spawn never runs on the deterministic executor, and a tokio sleep runs on wall-clock time that no seed controls.

moonpool-hyper supplies the same hooks over the provider traits.

The first four are thin adapters. The last two are the interesting ones, because they are where the plumbing that every hyper user hand-writes gets written once.

Serving: One Call Per Connection

Here is the gRPC server from tonic_grpc.rs, the whole thing:

#![allow(unused)]
fn main() {
let server = H2Server::new(ctx.providers()).with_config(H2ServerConfig {
    keep_alive: Some(keep_alive()),
    vectored_writes: true,
});

loop {
    moonpool_sim::select! {
        accept = listener.accept() => {
            let (stream, addr) = accept?;

            // Wired to the shutdown token: a graceful Attrition reboot
            // signals it, and hyper then finishes the RPCs already in
            // flight before the connection ends.
            let shutdown = ctx.shutdown().clone();
            let connection = server.serve_connection_with_shutdown(
                stream,
                echo.clone(),
                shutdown.clone().cancelled_owned(),
            );

            ctx.task()
                .spawn_task("grpc-server-conn", async move { connection.await })
                .detach();
        }
        () = ctx.shutdown().cancelled() => return Ok(()),
    }
}
}

echo is the generated EchoServer, an ordinary tower service. serve_connection_with_shutdown wraps it, builds the IO, and hands back a future that is Send + 'static, so it spawns as a normal sim task. h2 connection futures are Send, unlike hyper’s HTTP/1 state machine, which is why this is a spawn rather than the inline FuturesUnordered dance the axum example needs.

Notice what stayed with the caller: the accept loop. That is deliberate. The race between accepting a new connection and making progress on existing ones is a real ordering question, and under the seeded select! different seeds explore both answers. Hiding it inside a helper would hide a property worth testing.

The shutdown future is the graceful drain. When Attrition reboots the process politely, the token fires, hyper stops accepting new streams on that connection, and the RPCs already in flight get to finish. The example asserts that this actually happens, with assert_sometimes!(true, "grpc_server_drained_on_shutdown") guarded on the token being cancelled when the connection ends.

Connecting: The Channel Owns the Connection

On the client, tonic::transport::Channel is the thing we cannot use. Its job, though, is one we still need: connect lazily, reconnect after a death, and multiplex everything over one connection. ReconnectingChannel is that job, over the providers:

#![allow(unused)]
fn main() {
let channel: GrpcChannel = ReconnectingChannel::new(
    ctx.providers(),
    server_ip.clone(),
    ChannelConfig {
        connection_timeout: CONNECT_TIMEOUT,
        keep_alive: Some(keep_alive()),
        vectored_writes: true,
        ..ChannelConfig::default()
    },
);

let echo_client = EchoClient::with_origin(channel.clone(), origin.clone());
}

One channel for the whole workload, cloned into as many generated clients as we like. The example creates it once and reuses it across every round, which is the point: when Attrition kills the server mid-round, the next round does not build a fresh connection by hand, it discovers that the channel rebuilt itself. That is reconnection under test rather than reconnection assumed.

When the owner removes a peer or ends a process incarnation, shut down that shared lifecycle explicitly:

#![allow(unused)]
fn main() {
channel.close();
assert!(channel.is_closed());
}

Shutdown is terminal, idempotent, and shared by every clone. It interrupts an active connect or reconnect backoff immediately, drops the live h2 driver (and therefore its keepalive work), and makes parked readiness checks, new requests, and already-issued request futures return ChannelError::Closed. It does not replay logical RPCs; deciding whether a request is safe to retry remains the application’s job.

Failures arrive as Code::Unknown, because tonic’s h2-aware status mapping lives behind features the runtime-free core does not enable. Under chaos that is an expected outcome, not a bug, and the example gives it its own sometimes-assertion so transport death stays distinguishable from an application-level UNAVAILABLE.

At the tower boundary, a transient connect or handshake failure is reserved by poll_ready, which returns Ready(Ok(())); the following call returns the reserved error. Tower treats Ready(Err(_)) as terminal for that service instance, so the channel uses readiness errors only for terminal shutdown or an exhausted max_connection_failures limit. This is also the convention tonic’s own reconnect service follows.

The Sealed Bounds

One wrinkle worth knowing before you write your own helper. hyper’s h2 builders require Http2ClientConnExec and Http2ServerConnExec, traits that are sealed: you cannot implement them, you can only satisfy them by supplying an executor that already implements rt::Executor for hyper’s private future types. Naming them in a generic signature works, but nothing proves the bound is satisfiable until a concrete type instantiates it. moonpool-hyper’s tests do exactly that, naming ReconnectingChannel<TokioProviders, Full<Bytes>> and H2Server<TokioProviders> so the compiler has to discharge both bounds. If you build your own wrapper and it will not compile, that is usually this.

Why Any of This Is Deterministic

Three deliberate choices keep h2 behavior seed-reproducible.

Keepalive runs on provider time. When a timer is configured, hyper reads the clock exclusively through Timer::now(), and HyperTimer answers from the time provider. So a ping interval of three seconds means three seconds of simulated time, and a connection that dies because a clogged link swallowed the PONG dies at the same simulated instant on every replay.

Backoff has no jitter. Production jitter exists to desynchronize a fleet of clients. Here it would only make reconnect timing depend on something other than the seed, so ChannelConfig doubles from initial_reconnect_delay and saturates at max_reconnect_delay, full stop.

Response headers do not read the wall clock. Hyper normally adds a Date header from SystemTime. That timestamp becomes part of the connection’s HPACK state, so two otherwise identical gRPC replays can emit different frame sizes and perturb later simulated network choices. H2Server disables the automatic header. A production caller using the builder escape hatch can re-enable it, or the service can provide a date from an application-controlled clock.

Reset streams do not expire on the wall clock. h2 keeps a locally reset stream around for a while so late frames for it can be ignored, and it judges that expiry on std::time::Instant::now() — the one clock hyper’s Timer does not reach — after one real second by default. A simulation that takes longer than that in wall time then expires those streams at a simulated instant that depends on how fast the machine is, and the client connection’s “last stream closed” self-wake lands on a different poll from run to run (paros’s determinism canary caught it as a draw that diverged at the same simulated nanosecond with the two fingerprints swapped between processes). ReconnectingChannel sets reset_stream_duration to Duration::MAX, so the count cap (max_concurrent_reset_streams) is the only bound on them, which is deterministic. hyper exposes no such knob on the server builder, so a server that resets streams keeps a residual dependence on wall time there.

A connection has to earn its reset. The failure count that drives the backoff clears only once a connection has survived initial_reconnect_delay, not the moment the handshake completes. Without that rule, a peer that accepts and immediately dies would reconnect forever at zero backoff and never reach max_connection_failures. With it, a flapping peer faces escalating delays, which is both the correct production behavior and a bounded one to simulate.

What You’re Testing (and What You’re Not)

• What You Are Testing
• What You Are Not Testing
• The Same Tradeoff Everyone Makes
• Incremental Adoption

Running an axum service inside moonpool-sim exercises a specific slice of your application’s behavior. Understanding what falls inside and outside that slice prevents both false confidence and unnecessary skepticism.

What You Are Testing

Real handler logic. Your axum handlers run unchanged. JSON serialization and deserialization happen for real (serde, not mocked). Routing matches against actual paths. Middleware executes in order. Extractors parse real HTTP requests. If your handler has a bug in its JSON response structure, simulation catches it.

HTTP behavior under chaos. Requests arrive over simulated TCP with injected latency, connection drops, and data corruption. Hyper’s HTTP/1.1 parser processes real wire bytes through a network that actively tries to break things. Half-closed connections, incomplete messages, connection resets mid-response: all exercised.

Concurrent request handling. Multiple connections are served simultaneously by pushing each hyper::serve_connection future onto a FuturesUnordered that the accept loop drives inline. We can’t spawn those futures onto our Send-bounded runtime because hyper’s HTTP/1.1 state machine is !Send, so we poll them all in one task instead. Race conditions between concurrent handlers accessing shared state (your Store fake) still surface under different scheduling orders across seeds.

Error paths. Connection failures, timeouts, process reboots mid-request, store failures via buggify!(). The workload validates that your service handles these gracefully: returns appropriate status codes, doesn’t panic, doesn’t corrupt state.

Recovery after crash. With attrition enabled, moonpool kills and restarts processes. Your Process::run method executes fresh after each reboot. If your service leaks state across restarts or fails to rebind its listener, simulation finds it.

What You Are Not Testing

Production startup code. axum::serve() binds a real tokio::net::TcpListener and manages the accept loop internally. In simulation, we use hyper::server::conn::http1::serve_connection with a manual accept loop. If your production startup has a bug (wrong bind address, missing middleware registration), simulation won’t catch it.

Real database execution. The Store fake is a BTreeMap, not Postgres. SQL query plans, transaction isolation levels, connection pool behavior, schema migrations: none of these are exercised. A query that works on BTreeMap but generates incorrect SQL won’t be caught.

TLS. Simulated TCP streams are plaintext. TLS handshake failures, certificate validation, protocol negotiation: not exercised. If your service has a TLS configuration bug, you need integration tests against real TLS.

Real TCP congestion. moonpool-sim models connection-level faults (drops, latency, corruption) and end-to-end flow control (a fixed byte window per direction that a slow reader fills, see Flow control), but not congestion algorithms, window autotuning, retransmission timers, or real segment boundaries. A bug that only manifests under a kernel’s actual congestion behavior won’t be found here; one that manifests because a writer never expected to block will.

OS resource limits. File descriptor exhaustion, memory pressure, CPU scheduling. The simulation runs in a single process with no OS-level resource constraints. A service that leaks file descriptors will appear healthy in simulation.

The Same Tradeoff Everyone Makes

This is the exact tradeoff every simulation system makes. AWS has run ShardStore for over 15 years with the same architecture: real application logic, simulated network, faked storage. FoundationDB simulates network and disk but not the OS kernel. TigerBeetle simulates I/O but not the filesystem. Antithesis runs real binaries but controls the kernel, not the hardware.

The common thread: simulate the interaction boundaries where interesting failures occur (network, storage I/O), use real implementations for computation (parsers, serializers, business logic), and accept that some classes of bugs require different testing approaches.

Incremental Adoption

You don’t have to simulate your entire application on day one. Start with one module. The trait boundary you create for simulation (Store, Cache, MessageBroker) also improves your production architecture. Dependency injection makes code testable with or without simulation. Trait-based boundaries make components swappable.

Each module you bring under simulation compounds coverage. Your first module finds bugs in its own error handling. Your second module, interacting with the first under chaos, finds bugs in the interaction. By the third module, the simulation is finding bugs you didn’t know to look for.

The entry cost is one trait and one fake per dependency. The ongoing cost is maintaining fakes when traits change (the Oxide convention of shipping fake alongside trait helps). The return is deterministic, reproducible, chaos-tested coverage of your actual application logic, running in milliseconds on every commit.

Using Providers in Production

• The production backend
• A lean dependency tree
• One gotcha: TokioTimeProvider::now()
• Where it runs

Every chapter so far has been about testing. But the whole point of writing against the provider traits is that the code you tested is the code you ship. There is no separate “production version” to keep in sync, no #[cfg(test)] fork in behavior. The function you ran ten million times under simulation is the exact function that now handles real traffic.

So how do we actually deploy it?

The production backend

Simulation swaps in SimProviders. Production swaps in TokioProviders — a zero-cost bundle of the five real implementations: wall-clock time, tokio task spawning, real TCP, OS randomness, and real filesystem I/O. Code that is generic over P: Providers accepts either one.

use moonpool::prelude::*;

// The same signature you used in tests.
async fn fetch_with_retry<P: Providers>(providers: &P, max_attempts: u32)
    -> Result<u32, String>
{
    // time().sleep(), random().random_bool(), task().spawn_task() ...
}

#[tokio::main]
async fn main() {
    let providers = TokioProviders::new();
    let result = fetch_with_retry(&providers, 5).await;
    println!("{result:?}");
}

The complete, runnable version lives in crates/moonpool/examples/retrying_worker.rs. Its main runs on Tokio; its #[test] drives the same fetch_with_retry through SimulationBuilder across 50 seeds. One function, two worlds, verified by cargo test --example retrying_worker.

Networking follows the same swap. Code written against NetworkProvider sees SimTcpStream in a simulation and real Tokio TCP streams in production. If the application speaks HTTP or gRPC, enable moonpool-hyper and give its adapters the same provider bundle.

A lean dependency tree

The simulation runtime and the fork-based explorer have no business in a production binary. The explorer in particular pulls in libc, fork, and mmap — machinery that exists only to branch timelines during testing. A production build should never compile it.

Moonpool keeps it out with features. The default pulls the whole framework (that is what the crate’s audience uses day to day). Production opts out:

[dependencies]
moonpool = { version = "0.8", default-features = false, features = ["tokio"] }

That stanza gives you the provider contract and the production backend, and nothing else. cargo tree on such a build shows no moonpool-sim, no moonpool-explorer, no moonpool-assertions. The only system libraries present are the ones tokio itself needs for real sockets and files.

Keep sim on as a dev-dependency feature and off in your release profile, and your tests still run the full simulator while your shipped binary stays lean.

One gotcha: TokioTimeProvider::now()

In simulation, time().now() returns the canonical simulation clock. In production, TokioTimeProvider::now() returns elapsed time since the provider was created, not wall-clock time. It is a monotonic stopwatch, perfect for measuring durations and scheduling relative deadlines, and deliberately not a calendar. If you need a wall-clock timestamp for logging or persistence, reach for std::time::SystemTime directly at that call site. Everything that drives sleeps, timeouts, and backoff should keep going through the provider so it stays testable.

Where it runs

The provider contract and the production backend compile broadly. The sim runtime is portable too, with one boundary: the fork-based explorer needs a POSIX process model.

The simulation engine compiles to wasm32-unknown-unknown because it derives everything from a seeded RNG, a logical clock, and a cooperative scheduler — no operating system required. The explorer cannot follow it there: it is built on fork(), waitpid(), and MAP_SHARED memory, which both Linux and macOS provide (the explorer stays single-threaded and never touches Apple frameworks, so darwin’s usual fork-without-exec hazards don’t apply — CI runs the fork-based exploration suite on macOS). On wasm the full simulator runs via the in-process assertion table; you lose multiverse forking, not correctness checking. If a build refuses to include the explorer, the fallback is one line: moonpool-sim = { default-features = false }.

Migrating Existing Code to Providers

• Thread one bundle from the top
• The call mapping
• Gotcha 1: the I/O traits move to futures::io
• Gotcha 2: now() is a stopwatch, not a calendar
• Gotcha 3: replace every last one
• Knowing the swap was safe

You have a service that already works. It sleeps, it spawns, it opens sockets and files, and somewhere it calls rand. None of that is testable under simulation yet, because every one of those calls reaches straight into the operating system and the wall clock. The job is to route each of them through a provider instead, so that one day the same function can run under SimProviders and replay deterministically.

The reassuring part is that this is a refactor, not a rewrite. The code keeps running on real Tokio the whole time, because TokioProviders wraps the exact primitives you are replacing. You are not changing behavior, you are changing who you ask for time, tasks, randomness, sockets, and files. Simulation comes later. Today the only thing that has to be true is that TokioProviders behaves like the raw calls, and it does.

Thread one bundle from the top

Before touching any call site, decide how the provider reaches your code. The pattern is a single type parameter, P: Providers, threaded from main down through the call graph. Construct the bundle once at the top and pass it by reference.

use moonpool::prelude::*;

async fn run<P: Providers>(providers: &P) -> Result<(), MyError> {
    // everything non-deterministic goes through `providers`
}

#[tokio::main]
async fn main() -> Result<(), MyError> {
    let providers = TokioProviders::new();
    run(&providers).await
}

The discipline that makes this work is resisting the shortcut. Do not call tokio::time::sleep three layers deep because the provider is not in scope. Plumb the provider through. A function that still reaches for a global is a function simulation cannot control.

The call mapping

Most of the migration is mechanical. Each non-deterministic primitive has a direct provider equivalent.

Two small type changes ride along. time().sleep returns Result<(), TimeError>, so add a ?. time().timeout returns Result<T, TimeError> rather than tokio’s Result<T, Elapsed>, so match on the moonpool error.

Gotcha 1: the I/O traits move to futures::io

This is the one that surprises people. The Tokio network and storage providers wrap their tokio::net::TcpStream and tokio::fs::File in tokio_util::compat::Compat, which exposes the runtime-agnostic futures::io::{AsyncRead, AsyncWrite, AsyncSeek} traits instead of tokio::io::*. The stream is still a real socket and the file is still a real file. Only the extension trait you import changes.

#![allow(unused)]
fn main() {
// Before: tokio's extension traits.
use tokio::io::{AsyncReadExt, AsyncWriteExt};

// After: the same method names, from futures.
use futures::io::{AsyncReadExt, AsyncWriteExt};

stream.write_all(b"hello").await?;
let n = stream.read(&mut buf).await?;
}

write_all, read, read_exact, and read_to_end all live on the futures traits, and seeking comes from futures::io::AsyncSeekExt. The Tokio-only conveniences read_buf and AsyncBufRead have no direct equivalent, so a read loop written around them needs a small restructure into plain read calls.

Gotcha 2: now() is a stopwatch, not a calendar

providers.time().now() returns elapsed time since the provider was created, not wall-clock time. It is perfect for measuring durations and relative deadlines and deliberately useless as a timestamp. If a log line or a persisted record needs a real calendar time, reach for std::time::SystemTime at that exact call site and leave everything that drives sleeps, timeouts, and backoff going through the provider. The Using Providers in Production chapter covers this in more detail.

Gotcha 3: replace every last one

A single surviving tokio::time::sleep or thread_rng() is enough to make a future simulation non-reproducible, and it will not announce itself. Make the sweep exhaustive. Grep the crate for tokio::, rand::, std::fs, and std::time::Instant, and confirm each remaining hit is genuinely outside the deterministic path. The compiler helps once the function is generic over P: Providers, because the providers are the only async machinery in scope, but synchronous calls like Instant::now() slip through type checking and need eyes.

Knowing the swap was safe

Because TokioProviders is the production backend, “did I change behavior?” has a concrete answer. Moonpool ships a conformance suite under crates/moonpool-sim/tests/conformance/ that runs one generic contract per provider against TokioProviders on a real runtime. It asserts the properties your migrated code depends on: a sleep advances the clock by at least its duration, a timeout over a stalled future elapses, a byte-for-byte echo survives a round trip, a write followed by sync_all reads back identical, and missing or already-existing files surface as NotFound and AlreadyExists. Each contract is written generically so the very same body will later run against SimProviders, which is how the production and simulation backends are kept honest.

In practice the loop is short. Make a function generic over P: Providers, swap its non-deterministic calls using the table above, fix the futures::io imports, and run your existing Tokio tests. If they still pass, you have lost no behavior and gained a function that simulation can drive.

Simulation in the Browser

• Why a simulator compiles to wasm at all
• What had to change
• Does block_on park?
• Building a wasm-able crate
• A worked example
• What stays behind

The same simulation that grinds through thousands of seeds in CI also runs in a browser tab, with no server and no install. Type a seed, watch your system take traffic under a chaotic network, and share a link that reproduces the exact run on anyone’s machine. That makes a great demo. It is also a proof. If the engine runs inside a sandbox with no operating system, then nothing in the simulation secretly depends on one.

Why a simulator compiles to wasm at all

In simulation the network and the disk are not devices. SimNetwork is an in-memory state machine that schedules “deliver this packet at logical time T” events on a queue. SimStorage is the same idea for reads and writes. Time is a counter the engine advances. Randomness is a seeded ChaCha8 stream. The scheduler is the moonpool deterministic executor, plain library code. Add those up and there is no syscall anywhere on the path.

The production providers are the opposite. TokioProviders is real TCP, real files, and OS randomness, every one of them a trip into the kernel. That single difference is why the simulator crosses over to wasm32-unknown-unknown and the production backend does not.

One piece used to be borrowed rather than owned. SimProviders once spawned tasks through the real TokioTaskProvider, which meant shipping tokio’s rt feature to the browser. Not anymore: SimTaskProvider spawns onto the moonpool deterministic executor, so the wasm build no longer needs core’s tokio-task feature at all. What remains of tokio in the graph is a thin, syscall-free slice: sync for shutdown tokens and macros for the select! expansion. The net and fs halves never cross over.

What had to change

Getting there was four narrow fixes, not a rewrite.

• The fork-based explorer (libc, fork, mmap) moved behind a default-on exploration feature you switch off for wasm.
• Three wall-clock call sites the harness uses for reporting got a shim, because Instant::now() and SystemTime::now() compile on wasm and then panic the moment you call them.
• The tokio dependency narrowed to sync and macros. The deterministic executor owns the scheduling, so not even rt crosses over.
• rand dropped its default features so it never reaches for getrandom and the OS entropy that a browser has no answer for. The sim never needed it: its RNG is seeded ChaCha8.

Does block_on park?

Compiling is necessary, not sufficient. A conventional runtime that runs out of ready work tries to park the thread, and parking panics on wasm32-unknown-unknown. So the real question was whether block_on ever parks while driving a simulation.

It does not. The moonpool executor never parks at all: the simulation is ready-driven, every wakeup fires inline as the orchestrator steps the event queue, and there is no external reactor sitting on a socket or a timer to wake the thread later. The executor always has the next step in front of it until the run ends, and if nothing is runnable it treats that as a genuine deadlock and panics with the seed instead of parking.

Building a wasm-able crate

Putting a simulation in a browser is mostly a Cargo question, not a code question. Your workload, your processes, and your invariants stay exactly as they are. The one rule is to keep the heavy, non-portable dependencies out of the build:

[dependencies]
moonpool-sim = { version = "0.8", default-features = false }
futures = "0.3"

default-features = false drops the explorer and the production tokio providers and leaves a simulator that targets wasm. Anything native, the same crate’s tests, a multi-seed chaos binary, a CI runner, depends on the same code with the heavier features switched back on. Cargo resolves features per build, so cargo build -p your-wasm-crate never drags exploration into the bundle. Write the simulation once, run it in both places.

A worked example

The repository ships one: crates/moonpool-wasm-demo. It runs a single seed of two nodes trading integrity-checked ping/pong frames over raw TCP, driven by the simulated network, and animates the result. The client and server are ordinary Process and Workload code with no browser awareness. .enable_chaos([Chaos::Network(ChaosMode::Random)]) injects seeded latency and connection drops, so some round trips come back slow and some never come back at all. A generic recorder reads the same trace timeline your invariants read and turns it into the picture.

It is running right here, compiled to wasm and served as a page asset. Press Run, type a seed, or pick a preset. 256 is chaotic, 7 is a storm, 42 is calm. The same seed always replays the same history.

You can also run it outside the book. The build is three commands and no bundler:

cargo build --release --target wasm32-unknown-unknown -p moonpool-wasm-demo --lib
wasm-bindgen --target web --out-dir web/pkg \
  target/wasm32-unknown-unknown/release/moonpool_wasm_demo.wasm
cd web && python3 -m http.server   # open the page, press Run

Run a seed natively with cargo run -p moonpool-wasm-demo 42, then run the same seed in the browser. You get the byte-identical history. Same seed, same run, on your laptop and in a stranger’s tab. That is seed-driven reproducibility with nowhere left to hide.

What stays behind

Two things do not cross over, by design. The production providers cannot run in a browser, because a tab has no raw TCP and no filesystem. And the explorer cannot follow, because there is no fork in wasm. You lose multiverse forking, not correctness: the in-process assertion table still tracks every assert_always! and assert_sometimes!. The full platform matrix lives in Using Providers in Production.

To keep the browser path from quietly rotting, a portability job in CI compiles the simulator to wasm32-unknown-unknown on every change, runs the no-explorer test suite, and fails the build if the lean production tree ever pulls the simulator back in. The browser is not a one-time stunt. It is a gate.

Simulating the Network

• TCP, Not Packets
• What We Simulate
• Who Owns What
• A Raw TCP Process
• What We Do Not Simulate
• Same Code, Two Worlds

When we built the provider traits in Part II, we put TCP behind NetworkProvider. Production gets real sockets from TokioNetworkProvider. Simulation gets provider-backed listeners and SimTcpStream. Application code sees the same futures::io::AsyncRead and AsyncWrite interface in both places.

That seam is deliberately small. Moonpool simulates the network behavior a distributed application can observe without pretending to be an IP stack.

TCP, Not Packets

Many network simulators model individual packets: MTU fragmentation, congestion windows, selective acknowledgments, and routing at the segment level. That fidelity is useful for protocol research. It is usually wasted on an application whose contract begins at a TCP byte stream.

Our systems speak TCP. They open connections, send messages, read responses, and handle disconnections. The bugs that kill production clusters happen at connection granularity, not packet granularity:

• A node loses its connection mid-write and the remote sees a partial message
• A one-way partition lets one side hear heartbeats while its replies disappear
• A process reboots while clients still hold streams to the old process
• A partition heals after several request deadlines have expired

These failure modes need connection establishment, byte delivery, partial I/O, latency, and close semantics. They do not need a simulated router.

FoundationDB reached the same boundary in sim2.actor.cpp: simulated connections can be delayed, corrupted, or severed, while MTU discovery and congestion control stay with the real kernel. Moonpool follows that boundary at the provider level. What runs above the stream is your code. It can be a small binary protocol, hyper, axum, tonic, or something entirely private to your system.

What We Simulate

The simulation network controls:

• Connect, bind, and accept. Each call is a real delayed operation. It remains pending until its targeted scheduler event fires. A simulated listener has a backlog and a connection handshake that can succeed, fail, or hang under chaos.
• Byte delivery. Writes enter the connection’s send buffer and the network engine requests ordered delivery events from the global scheduler. Partial writes and partial reads exercise loops that incorrectly assume one poll transfers a complete application message.
• Latency and clogging. Per-link latency, temporary clogs, and partitions change when or whether bytes arrive.
• Corruption. Bit flips make checksum and validation paths reachable. Raw protocols must provide their own integrity checks when corruption matters.
• Connection lifecycle. Graceful close delivers buffered bytes before EOF. Abrupt close can surface as an explicit I/O error or a silent peer failure.

Every decision comes from the seeded simulation RNG. A connection that hangs on seed 73 hangs the same way every time seed 73 is replayed.

Who Owns What

The networking provider is a thin adapter into NetworkSimulation. That engine owns listeners, connection pairs, pending backlogs, topology, partition state, network faults, delayed-operation results, and every network waker. It does not own logical time or a private queue.

SimWorld coordinates a single Scheduler<Event> for timers, network events, storage events, and process lifecycle. A network transition returns ordered actions such as scheduling a Delivery, cancelling an operation, or recording a fault. The coordinator applies those actions, releases its lock, then wakes tasks. Keeping wake calls outside the lock lets a re-entrant waker poll network state without deadlocking.

Endpoints are owned. A bind holds its address until the listener is dropped or its process is killed, so a second bind of a live address fails with AddrInUse, and a connect is refused with ConnectionRefused when nothing is listening at the address by the time the connection arrives. The address-in-use and connection-refused paths of startup, shutdown and misconfiguration therefore happen in simulation exactly as they do against the kernel. Port zero is ephemeral, as everywhere: such binds never collide.

Delayed bind and connect futures are cancellation-safe. Accept reserves one backlogged connection while its latency elapses, and dropping the future returns that reservation to the front of the backlog. Shutdown fails delayed operations and drains network waiters instead of leaving them parked.

A Raw TCP Process

The server below is ordinary process code. It binds its assigned address, reads one fixed-size request, and echoes it. read_exact matters because the network is allowed to split delivery into smaller reads.

use futures::io::{AsyncReadExt, AsyncWriteExt};
use moonpool_sim::{NetworkProvider, TcpListenerTrait};

let listener = ctx.network().bind(ctx.my_ip()).await?;
let (mut stream, peer) = listener.accept().await?;

let mut frame = [0_u8; 16];
stream.read_exact(&mut frame).await?;
tracing::info!(%peer, "request_received");
stream.write_all(&frame).await?;

The client uses the same provider boundary:

let mut stream = ctx.network().connect(&server_ip).await?;
stream.write_all(&request_frame).await?;
stream.read_exact(&mut response_frame).await?;

No simulation-only socket API appears in either side. In generic application code, accept a P: Providers or N: NetworkProvider and the same functions can run over TokioProviders in production. Inside a Process or Workload, SimContext supplies the simulated implementation.

Your protocol still owns framing, checksums, retries, and deadlines. This is a feature, not missing infrastructure. The simulator should exercise the code you ship, including its behavior when a frame is truncated or a retry is ambiguous. For ecosystem protocols, moonpool-hyper provides the adapters that let hyper and tonic consume the same provider streams.

What We Do Not Simulate

Moonpool deliberately skips:

• Individual packet routing between simulated hosts
• MTU and fragmentation at the IP layer
• Congestion windows and flow control (TCP slow start, etc.)
• DNS resolution
• TLS handshakes inside SimNetwork

Those concerns still need integration and production testing. The simulation focuses on application-visible timing, failure, and byte-stream behavior where seeded replay gives the most leverage.

Same Code, Two Worlds

The key architectural property is that the same application code runs in both environments. Production uses real TCP sockets. Simulation routes each connection and delivery through SimWorld.

Application Code
     │
     ▼
┌─────────────┐
│  Providers   │ ◄── trait bundle
├─────────────┤
│ NetworkProv. │ ◄── connect(), bind(), accept()
└──────┬──────┘
       │
  ┌────┴─────┐
  │          │
  ▼          ▼
Real TCP   SimWorld
(tokio)    (Scheduler<Event>)
                │
                ▼
         NetworkSimulation

There is no #[cfg(test)] branch and no socket mock in the application. The protocol parser, request handling, retry policy, and timeout logic are the real code, running against a hostile deterministic network. Then production swaps the provider and hands those same functions real sockets.

Multiverse Exploration

• The Core Idea
• Vocabulary
• What This Section Covers

Throughout this book, we have built a simulation framework that runs deterministic tests, injects chaos, and validates correctness with assertions. We can run thousands of seeds, each exploring a different corner of the state space. For many bugs, that is enough.

But some bugs are not found by running more seeds. They require a sequence of unlikely events, and no single seed happens to produce that exact sequence. We teased this in Part I when we described the vision of simulation-driven development. Now we deliver the capstone feature: multiverse exploration.

The Core Idea

Because the simulation is deterministic, an execution is not a fleeting thing — it is a reproducible coordinate in the state space. When a timeline reaches an interesting state for the first time (a retry fires, a leader election completes, the dungeon’s next floor is entered), moonpool remembers how to get back there: the seed plus the exact RNG position of the discovery. Later, it replays that prefix and continues with fresh randomness — a short randomized rollout from a proven starting point.

The insight is simple but powerful: if a timeline managed to reach an interesting state, running forward from that state with different random choices is far more likely to find the next interesting event than starting from scratch. Exploration turns deterministic executions into accumulated knowledge: interesting states are remembered, resumed, and extended; barren branches simply die.

Crucially, the tree of explored timelines is a logical structure, not a physical one. The logical multiverse may contain thousands of timelines; the physical footprint is one controller process plus a small, fixed pool of short-lived workers. Reaching a state ten discoveries deep costs a recipe with ten segments — not a tree of live processes.

Vocabulary

Seed is a u64 that completely determines a simulation’s randomness. Same seed means same coin flips means same execution, every time. This is the foundation from Part II.

Timeline is one complete simulation run. A root seed plus a recipe uniquely identifies a timeline.

Recipe is the replay path to a timeline: a sequence of (rng_call_count, seed) breakpoints. “Run with the root seed; after 151 RNG calls, reseed to 8837201; after 80 more, reseed to 1293847”:

151@8837201 -> 80@1293847 -> 42@9918273

Follow those instructions and you arrive at the exact same timeline, deterministically — task scheduling included, because the executor draws its schedule from the same counted stream the recipe replays.

Discovery is a globally-new interesting state: the first pass of an assert_sometimes!, a new assert_sometimes_each! bucket, a watermark or frontier improvement. Discoveries are latched atomically in shared memory, so each fires exactly once across all timelines.

Journal is the list of discoveries one run made, each stamped with the RNG call count where it happened — the coordinate a recipe breakpoint can anchor to.

Frontier is the queue of jobs (recipes) waiting to be explored.

Exemplar is a retained recipe that reaches one semantic state. The controller keeps a few per state, because two executions can hit the same state with very different future potential.

Worker is a forked process that executes exactly one timeline and exits. Workers never fork and never make exploration decisions.

What This Section Covers

1. The Exploration Problem explains why random simulation is not enough for hard bugs, and frames exploration as a resource allocation problem using NES game analogies from Antithesis.

2. The Frontier Controller describes the exploration loop: recipes, discovery journals, the one-expansion rule, and how barren branches die without any budget accounting.

3. Bounded Workers covers the physical execution model: fork() as a bounded job-execution optimization rather than a tree structure, single-owner novelty, crash isolation, and macOS portability.

4. Exemplars and Continuations shows how the controller remembers semantic states, retains multiple exemplars per state, and schedules continuations from the under-explored frontier — the machinery that carries the dungeon workload to floor 8.

5. Multi-Seed Exploration shows how cumulative novelty makes additional root seeds cheap: seeds that discover nothing new stop after one run.

The moonpool-explorer crate that implements all of this is a leaf dependency with exactly one external dependency: libc. It has zero knowledge of processes, networks, or storage. It communicates with the simulation through the assertion accounting hooks and one function pointer that reads the RNG call count. That minimal coupling is deliberate: the exploration engine is a general-purpose controller that works with any deterministic simulation, not just moonpool’s.

Let us start with the problem it solves.

The Exploration Problem

• The Sequential Luck Problem
• State Space Shapes
  • Breadth-Dominated: The Zelda Problem
  • Depth-Dominated: The Gradius Insight
  • Barriers: The Castlevania Trap
  • Continuous Optimization: The Metroid Economy
• Exploration as Resource Allocation

Running thousands of random seeds is a powerful technique. It finds race conditions, timing bugs, and failure-handling errors that hand-written tests never would. But there is a class of bugs it struggles with, and understanding why reveals the fundamental challenge of state-space exploration.

The Sequential Luck Problem

Consider a distributed lock service with a subtle bug: the lock can be granted to two nodes simultaneously, but only when a specific failover event happens during a specific rollback window. The failover has a 1/1000 chance per simulation step. The rollback has a 1/1000 chance per step. For the bug to manifest, both must happen in the same run.

With independent random seeds, the probability of both events occurring is roughly 1/1,000,000. We need about a million simulation runs to have a decent chance of seeing it. Each run takes a few seconds. That is weeks of compute for one bug.

This is the Sequential Luck Problem: a bug requiring N unlikely events in sequence has probability that decreases multiplicatively. Two events at 1/1000 each need ~10^6 trials. Three events need ~10^9. The search space grows exponentially with the depth of the required sequence.

Now consider what happens with replay-and-continue. We run seeds until one of them triggers the failover (about 1000 runs). At that moment, we retain the deterministic replay prefix and schedule new timelines that reproduce the failover, then continue with different randomness. Each continuation has a 1/1000 chance of hitting the rollback window. We need about 1000 continuations, plus the ~1000 root seeds to reach the failover. Total: roughly 2000 timelines instead of 1,000,000.

The improvement is not incremental. It changes the complexity from multiplicative (p1 * p2) to additive (p1 + p2). For three-event bugs, the difference is even more dramatic: ~3000 timelines instead of ~10^9.

State Space Shapes

Not all exploration problems are alike. Antithesis, through their work running NES games with autonomous testing, discovered that state spaces have shapes that require fundamentally different approaches.

Breadth-Dominated: The Zelda Problem

The Legend of Zelda (1986) is an open-world game with 128 overworld screens, 230+ dungeon rooms, 25 prerequisite items, and multiple weapons. The state space is wide: there are many things to discover, and they do not need to happen in a strict sequence. The challenge is not getting past a single barrier but covering a large surface area.

Antithesis solved Zelda with SOMETIMES_EACH, which equalizes exploration across all discovered states. Visit screen #47? Good. Now visit screen #48 with the same frequency. The exploration engine spreads its budget across the breadth of the space, ensuring no region is neglected.

In moonpool terms, this maps to assert_sometimes_each! assertions that track coverage across identity values. A distributed database might use this to ensure all partition ranges see similar test traffic, or all node roles get exercised equally.

Depth-Dominated: The Gradius Insight

Gradius (1985) is a side-scrolling shooter that auto-scrolls. The player cannot go back. Progress is measured almost entirely by survival time. Antithesis tried a complex strategy tracking powerups, weapons, ship position, and score. Then they deleted all of it and ran with a minimal strategy: maximize time since power-on, tracking only 3 bytes of game memory (single-player mode, pause state, death animation).

The minimal strategy beat the entire game.

This revealed that for depth-dominated problems, the platform infrastructure (save/restore, checkpoint-and-branch) matters more than domain-specific strategy. The agent’s gameplay was alien: it burned powerups on speed, clipped through walls, hid from bosses until they timed out, and hammered the pause button. But it worked because the platform kept branching from the deepest point reached.

For simulation testing, this means that sometimes the right approach is a minimal liveness assertion (“the system is still making progress”) combined with aggressive checkpoint-and-branch, rather than elaborate state tracking.

Barriers: The Castlevania Trap

Castlevania’s Stage 6 has stompers: pillars that descend and crush the player. The exploration engine tracked Simon’s position on a 32-pixel grid and tried to equalize coverage across grid cells. It cruised through the first five stages. Then it got stuck.

The problem was doomed states. The best-known exemplar for the critical grid cells was Simon standing under a descending stomper. That state is irrecoverable: no sequence of future inputs can save him. The explorer kept restarting from this doomed state and making zero progress.

Two fixes unlocked progress. First, refining the grid from 32-pixel to 16-pixel cells created safe zones between the stompers where non-doomed exemplars could be established. Second, adding the stomper position to the state tuple distinguished “under a high stomper” from “under a low stomper.”

The lesson for simulation testing: when exploration gets stuck, the fix is either better input distribution (how chaos events are generated) or better output interpretation (what signals guide exploration). Heatmaps, assertion coverage reports, and sometimes-assertion hit rates are the diagnostic tools that tell you which one to adjust.

Continuous Optimization: The Metroid Economy

Metroid (1986) presented the hardest challenge. The exploration engine could reach everywhere accessible without missiles. But red doors require 5 missiles to open, and the engine kept spending missiles on enemies (because it helped short-term exploration) rather than hoarding them for progression.

The naive solutions failed. Adding missile count to the state tuple caused state explosion (position times missile count is too many combinations). Requiring 5+ missiles was too restrictive (some areas need you to spend them).

The solution was continuous optimization: decouple the optimization objective from the exploration objective. Explore all positions, but prefer states with more missiles and health. When a better-resourced path to a state is found, propagate the improvement through all downstream states.

This generalizes beyond games. In distributed systems testing, the analogues are memory usage (prefer states with lower memory to catch leaks), queue depth (prefer states with deeper queues to find overflow bugs), and latency (prefer states with higher latency to find timeout bugs).

Exploration as Resource Allocation

The NES examples reveal a deeper truth: exploration is fundamentally a resource allocation problem, not just a randomness problem.

We have a finite compute budget. We can spend it on breadth (more root seeds, wider coverage) or depth (more continuations from interesting states, deeper sequences). We can spread it evenly across all known states or concentrate it on the most promising ones.

Every one of these decisions affects what bugs we find. A breadth-heavy approach misses deep sequential bugs. A depth-heavy approach misses bugs in unexplored regions. Even allocation wastes budget on doomed states. Concentrated allocation might miss productive ones entirely.

The next three chapters describe how moonpool makes these allocation decisions with a deliberately simple policy: productive exploration produces new work, unproductive exploration dies naturally. We start with the frontier controller (the exploration loop itself), then the bounded worker pool (the physical execution model), and finally the exemplar store and continuation scheduling that keep pushing the frontier deeper.

The goal is to find bugs that random testing cannot reach, using the same compute budget — while keeping the number of live processes small and fixed.

The Frontier Controller

• Jobs Are Recipes
• The Discovery Journal
• One Expansion Per Run
• Who Decides Novelty
• Bug Recipes

The heart of the explorer is a small loop that runs in one process and owns every exploration decision. Nothing else in the system decides anything: assertion macros describe interesting behavior, workers execute timelines, and the controller — the Explorer in moonpool-explorer — turns discoveries into follow-up work.

                explorer / controller
                        |
                     frontier          (FIFO queue of jobs)
                        |
            +-----------+-----------+
            |           |           |
          worker      worker      worker
            |           |           |
            +-----------+-----------+
                        |
               discovery journals
                        |
                novelty decision
                        |
          productive? → expand once
          barren?     → the branch dies

Jobs Are Recipes

The unit of exploration work is deliberately minimal:

pub struct ExploreJob {
    pub recipe: Recipe,   // Vec<(rng_call_count, seed)>
}

A job says: replay this prefix, then keep going with the fresh randomness of the final segment’s seed. Executing a job means running one complete simulation from the beginning with the recipe’s RNG breakpoints installed. Because the simulation is deterministic and every decision — executor scheduling, select! offsets, swarm masks, faults, the workload’s own draws — is a counted draw on the one simulation stream, replay is exact — and cheap enough that no live-checkpoint machinery is needed.

The Discovery Journal

During a run, every globally-new discovery is recorded into a per-run journal: what kind of discovery it was, which semantic state it belongs to, and the RNG call count at that moment. The call count is the crucial part — it is the coordinate where a child recipe can anchor:

root run (seed 42):
  journal: [floor 2 entered @ call 812, key found @ call 1490]

expansion, anchored at the highest-priority discovery:
  progress > structured state novelty > one-shot coverage
  latest call count breaks ties inside a class

Numeric watermark, compound frontier, and bucket-quality improvements are progress. New assert_sometimes_each! buckets and partial assert_sometimes_all! combinations are structured state novelty. A first boolean sometimes/reachable pass is one-shot coverage. This ordering prevents a late, easy coverage hit from pulling immediate children away from an earlier step toward a difficult goal. Every discovery still gets its own retained exemplar for later continuation scheduling.

The journal is bounded to 256 entries. Repeated events for one semantic state are coalesced to its best anchor. If the journal fills, stronger discoveries replace the weakest retained coverage events, so early noise cannot permanently hide late progress after the shared novelty latch has fired.

One Expansion Per Run

A run that made at least one semantic assertion discovery is productive and is expanded exactly once: branching_factor children are enqueued, anchored at its highest-priority discovery. This holds no matter how many discoveries the run made. If one timeline discovers five new assertion states, it is still one productive execution, not five branching events. Sanitizer code coverage is reported separately and does not create frontier jobs.

A run with an empty journal discovered nothing the multiverse had not already seen. It is not punished, scored, or refilled — its branch simply produces no children and dies. This one rule replaces the entire energy system of the previous explorer (global budgets, per-mark allowances, reallocation pools, productive/barren classification): the only global limits are a per-seed run budget and a frontier size cap.

ExplorationConfig {
    workers: 0,             // deterministic in-process mode (the default)
    max_runs_per_seed: 8000, // total timelines per root seed
    branching_factor: 4,    // children per productive run
    max_frontier: 1024,     // queued-job cap
    max_recipe_len: 64,     // depth cap in replay segments
}

Set workers above zero only in a standalone simulation binary when bounded parallel fork() workers are worth the less reproducible search order.

Who Decides Novelty

Discovery detection lives in the shared assertion region (moonpool-assertions): first passes, new buckets, comparison-distance improvements, compound-frontier advances, and partial truth combinations are guarded by atomic shared state. Whichever timeline records the transition journals it; every other timeline, in any process, sees “already known”. There is no check-then-merge race, and the controller is the single owner of everything built on top of these signals (frontier, exemplars, statistics, bug recipes).

Bug Recipes

A failing exploration run — a workload error, an assert_always! violation, or a worker crash — is recorded with its job’s recipe verbatim. Replaying it reproduces the failing timeline bit-for-bit:

SimulationBuilder::new()
    .replay_timeline(bug.seed, bug.recipe.clone())
    .workload(MyWorkload)
    .run();

The next chapter looks at how jobs are physically executed while keeping the process count fixed.

Bounded Workers

• fork() Is an Optimization, Not a Data Structure
• What Crosses the Process Boundary
• Portability

The previous explorer was its process tree: a discovery forked the running simulation mid-poll, children forked again at their own discoveries, and the logical exploration tree materialized as a recursive tree of live OS processes. It worked — and it made deep exploration impractical, because depth multiplied the number of simultaneously live processes.

The redesign inverts the relationship:

The size of the logical exploration space is independent of the number of simultaneously active OS processes.

fork() Is an Optimization, Not a Data Structure

The controller executes each job through a fixed pool of workers:

controller ──fork──▶ worker (slot 0) ── run ONE timeline ──▶ _exit
           ──fork──▶ worker (slot 1) ── run ONE timeline ──▶ _exit
           ◀─waitpid── read journal from the slot, decide, expand

A worker is forked at a quiescent point between runs — never mid-simulation — inherits the whole prepared runner state via copy-on-write, executes exactly one replay-plus-continuation, serializes its discovery journal (and sanitizer-coverage counters) into its MAP_SHARED result slot, and _exits with 0 (clean), 42 (simulation failure), or anything else (crash). Workers never fork, never touch the frontier, and never return into controller code. Nothing catches a panic inside a worker: the worker simply dies, and the controller classifies its waitpid status — 0 is clean, 42 is a reported simulation failure, any other exit code or a signal is a crash — so a panicking run is retained as a crash reproducer like any other, and recursion is structurally impossible.

Live processes are therefore bounded by 1 + workers at all times. Reaching a state fifty discoveries deep costs a fifty-segment recipe, not fifty live processes.

With workers: 0 the controller runs every job in-process: sequential, fork-free, and fully deterministic — the same exploration schedule every time. This is the mode to use when debugging the explorer itself (and the only mode on non-unix targets). With workers > 0 each timeline is still individually reproducible from its recipe, but the search order depends on which worker finishes first.

What Crosses the Process Boundary

Exactly three things are shared between controller and workers:

1. The assertion region (MAP_SHARED): pass/fail counts accumulate across processes, and the discovery CAS latches make novelty a race-free, exactly-once decision.
2. Result slots: one per worker, carrying the run’s discovery journal out of the worker before it exits. A crashed worker leaves an empty journal; its recipe is still retained as a crash reproducer.
3. Sancov buffers: LLVM inline-8bit-counters are copied into a per-worker slot and merged into the controller-owned history map after waitpid — a single owner, no concurrent merges.

Everything else — frontier, exemplars, statistics, bug recipes — lives in ordinary controller memory with exactly one owner.

Portability

The explorer uses only portable POSIX primitives: fork, waitpid (with EINTR retry), and mmap(MAP_SHARED | MAP_ANONYMOUS). There is no clone3, no pidfd, no /proc, no cgroups — the same code runs on macOS and Linux, and CI exercises both. Fork safety holds because the whole simulation stack is single-threaded by design: the deterministic executor runs on one OS thread, and the controller forks only between runs when no locks are held.

Exemplars and Continuations

• Semantic States
• A Few Exemplars Per State
• Continuations From the Under-Explored Frontier
• The Dungeon Walkthrough

Expansion alone is momentum: a productive run spawns a few children, and if a child is productive the wave carries forward. But waves die. Four children from the “key found on floor 5” moment might all get eaten by monsters, and with plain expansion that hard-won state would be lost. This chapter covers the machinery that remembers progress and keeps re-investing in it.

Semantic States

Every discovery carries a state id: the assertion message hash for ordinary slot assertions, the site-plus-truth-combination hash for partial assert_sometimes_all! states, or the bucket hash (message plus identity keys) for assert_sometimes_each!. This is the Castlevania lesson from the problem chapter, applied through the macros you already write: the workload’s assertions project the huge concrete state space into a tractable set of semantic buckets.

assert_sometimes_each!("descended",
    [("to_floor", floor)],       // identity → one state per floor
    [("health", hp_bucket)]);    // quality  → watermark per state

The controller tracks up to 512 distinct states per seed. For each it records whether any discovery there was progress (a watermark, frontier, or quality improvement — monotonic signals like assert_sometimes_greater_than!(level, 0, "dungeon level reached")) rather than plain coverage, and how many continuation batches it has received.

A Few Exemplars Per State

For each state the controller retains up to three exemplars — recipes that reach it, anchored at the discovery’s RNG count. Three, not one, because of the doomed-state trap: “floor 4 at full health” and “floor 4 about to die” hit the same bucket with dramatically different futures. If the only retained exemplar is doomed, every continuation from that state is wasted.

Once a state’s exemplar list is full, new exemplars evict the oldest. Re-discoveries of a known state only arrive through quality and watermark improvements, so recency correlates with better starting points — the dying exemplar rotates out when a healthier path to the same floor is found. This is the Metroid lesson (prefer better-resourced paths to the same place) implemented as a three-slot ring buffer instead of an optimization framework.

Continuations From the Under-Explored Frontier

When the frontier drains and run budget remains, the controller schedules a fresh batch of continuations from the most promising known state:

pick the state minimizing:  visits / (depth + 1)
    ties → progress states first, then deeper recipes
rotate through its exemplars
enqueue branching_factor children with fresh derived seeds

The depth weighting is the entire “scoring system”: a state whose exemplars sit N replay segments deep gets roughly N+1 times the continuation budget of a shallow one, so effort concentrates on the frontier instead of spreading uniformly — while the visit counter still guarantees every state is revisited occasionally. No energy, no reinforcement learning, no corpus manager.

The Dungeon Walkthrough

The dungeon workload (in moonpool-sim-examples) is the acceptance benchmark for this design: eight floors, each hiding a key behind a 3% probability gate, monsters that scale with depth, and a treasure on floor 8 whose brute-force probability is about 0.03^7 ≈ 2×10⁻¹¹. Reaching it requires accumulating progress.

root run:  wanders floor 1, discovers "on key tile floor 1" @ call 812
   └─ continuations replay to the key tile and re-roll the 3% gate
        └─ "key found L1" → "stairs with key" → "descended to_floor 2"
             └─ each new floor adds states, exemplars, deeper anchors
                  └─ ... floor 8, treasure: bug recipe with ~50 segments

With workers: 4 and max_runs_per_seed: 8000, the explorer finds the treasure in a few seconds of wall time with at most five live processes. The captured recipe replays the entire path deterministically. The workload itself contributes structure the explorer relies on but never sees: correlated action sequences (70% chance of repeating the previous move — rollouts, not isolated decisions), a stable two-draws-per-step RNG discipline that keeps replay coordinates meaningful, and semantic assertions that name the states worth remembering.

Multi-Seed Exploration

• Novelty Is Cumulative
• Barren Seeds Die After One Run
• The Complete Picture

A single root seed reaches a particular region of the state space. No amount of continuation can explore regions that the root timeline’s early execution path never touches: if the first 100 RNG calls establish a specific dungeon layout or network topology, every recipe extension shares that foundation. Variations on a theme, not different themes.

The builder’s ordinary multi-seed loop provides the themes. Exploration makes each additional theme cheap.

Novelty Is Cumulative

The discovery latches in the shared assertion region — the CAS guards that make each discovery fire exactly once — are deliberately not reset between seeds, and neither is the sancov history map. Novelty is a property of the whole run, not of one seed:

• If seed 1 already discovered “key found L1”, seed 2’s timelines that find the same key journal nothing.
• If seed 1’s exploration drove the “dungeon level reached” watermark to 5, seed 2 only journals a discovery by beating 5.
• Assertion pass/fail counts also accumulate, so the final report and contract validation reflect every seed.

The consequence is a naturally progressive bar: each seed only earns exploration budget for behavior no earlier timeline exhibited.

Barren Seeds Die After One Run

max_runs_per_seed is a ceiling, not a quota. Each seed’s exploration starts from its root run’s journal; a root run that discovers nothing globally new produces no expansion, no exemplars, and therefore no continuations — the seed finishes after a single timeline:

━━━ Seeds ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  #1  seed=16797655684637781087  1,500 timelines   ← did the deep work
  #2  seed=6458319710110286448       1 timelines   ← nothing new here
  #3  seed=10665710964006250804      1 timelines   ← nothing new here

This replaces the previous design’s warm-start heuristics (warm_min_timelines, coverage-yield ramp-downs): there is no machinery deciding how much to invest in an already-explored seed, because an unproductive seed simply generates no work. Under UntilCoverageStable, cheap barren seeds mean the run samples many more root themes per unit of wall-clock time while the plateau detector watches the same cumulative coverage signals as always.

The Complete Picture

The problem: bugs that require sequences of unlikely events are exponentially hard to find with random seeds.

The frontier controller: deterministic replay turns executions into recipes; discoveries (CAS-latched, exactly-once) turn productive runs into a small number of follow-up jobs; barren branches die. No energy accounting.

Bounded workers: fork() executes jobs cheaply via copy-on-write, but the process tree is never the exploration tree — one controller plus a fixed worker pool, on Linux and macOS alike.

Exemplars and continuations: a few retained recipes per semantic state, scheduled from the under-explored, depth-weighted frontier, keep converting run budget into deeper progress.

Multi-seed exploration: cumulative novelty makes extra seeds nearly free, so breadth across themes composes with depth within them.

The developer’s job is to write good assertions — they are simultaneously the correctness spec and the exploration map. The controller handles the rest, and every timeline it ever visits can be replayed from a one-line recipe.

Exploring a Consensus Protocol

• Start Conservatively
• Put Safety in Invariants
• Add Semantic Replay Anchors
• Combine Exploration with Broad Seeds
• Replay Every Failure
• Current Operational Limits

The explorer is most useful for Paxos, Raft, and replicated logs when it is given two different kinds of signal: strict safety invariants that catch bugs, and semantic progress assertions that tell the frontier which prefixes are worth continuing. Code coverage alone reports that a branch ran; it does not provide the RNG-coordinate anchor used to build a recipe.

Start Conservatively

Use fork-free exploration while validating a new integration:

use moonpool_sim::{ExplorationConfig, SimulationBuilder, WorkloadCount};

let report = SimulationBuilder::new()
    .processes(3, || Box::new(PaxosNode::new()))
    .workloads(WorkloadCount::Fixed(1), |_| {
        Box::new(PaxosWorkload::new())
    })
    .invariant(AgreementInvariant::default())
    .enable_exploration(ExplorationConfig {
        workers: 0,
        max_runs_per_seed: 2_000,
        branching_factor: 4,
        max_frontier: 512,
        max_recipe_len: 32,
    })
    .until_coverage_stable(10, 1_000)
    .run();

workers: 0 makes controller order deterministic and avoids fork() while the test-driver lifecycle is being audited. Increase the worker count only in a standalone simulation binary on Linux or macOS, after confirming the host process has no unrelated threads or external resources that a child could inherit.

Use .workload_factory(...) or .workloads(...) for the client driver. Exploration rejects a single .workload(instance) and custom fault-injector instances because those opaque values cannot be reconstructed for every continuation. Built-in Chaos surfaces are supported. Keep authoritative test state inside fresh processes and factory-created workloads; do not depend on external clocks, randomness, threads, files, or sockets outside Moonpool providers.

Put Safety in Invariants

Emit stable, structured facts from the Paxos implementation:

tracing::info!(
    target: "paxos",
    slot,
    ballot,
    value = %value_hash,
    "value_chosen"
);

Then check cross-node and cross-time properties with cursor-based invariants. For Paxos, the minimum useful set is:

• Agreement: two chosen events for the same slot never name different values.
• Validity: every chosen value was proposed by a client.
• Acceptor persistence: after a promise or acceptance survives sync_all, a reboot never reveals an older ballot or a conflicting accepted value.
• Prefix consistency: for Multi-Paxos, committed logs on any two nodes agree throughout their common prefix.
• Client semantics: retries do not create two successful outcomes for one non-idempotent command.

The agreement example in Events and Invariants shows the full TraceQuery::since pattern. Run these checks after every simulation step; do not wait until the workload finishes, because a later state can hide a transient safety violation.

Add Semantic Replay Anchors

Use discovery assertions for qualitatively different protocol states:

assert_sometimes!(leader_changed, "paxos leader changed");
assert_sometimes!(promise_rejected, "stale ballot was rejected");
assert_sometimes!(recovered_acceptor, "acceptor recovered persisted state");

assert_sometimes_all!("failover made progress", [
    ("old leader unavailable", old_leader_unavailable),
    ("new quorum formed", new_quorum_formed),
    ("client completed", client_completed),
]);

Use watermarks for monotonic depth. They let later runs improve a frontier instead of consuming one static discovery:

assert_sometimes_greater_than!(decided_slots, 32, "many slots decided");
assert_sometimes_greater_than!(ballot_changes, 4, "repeated ballot changes");
assert_sometimes_greater_than!(retry_depth, 3, "deep client retry chain");

Use assert_sometimes_each! for a small, bounded state vocabulary:

assert_sometimes_each!(
    "paxos transition",
    [("phase", phase_code), ("fault regime", fault_regime_code)],
    [("decided slots", decided_slots)]
);

Keep identity keys coarse. The shared table holds 512 assertion sites and 256 sometimes_each buckets. Bucketing every node, ballot, slot, and value tuple will exhaust it quickly and teaches the controller identifiers rather than protocol structure. Good identity dimensions are phase, role, quorum state, recovery mode, and fault regime; good quality values are decided-slot count, ballot depth, retry depth, and inverse replica lag.

Combine Exploration with Broad Seeds

Exploration is semantic-guided randomized testing, not exhaustive model checking or a proof of Paxos safety. Recipes change the counted simulation RNG after an anchor, while executor and select! ordering remain derived from the root seed. Discovery latches are cumulative across seeds, so static boolean milestones mostly guide the first seed that reaches them.

Keep broad multi-seed testing enabled. Dynamic buckets and watermarks let later seeds contribute new semantic progress, while UntilCoverageStable samples new root schedules until assertion and code coverage plateau.

Replay Every Failure

An exploration bug marks its root seed as failed and appears in report.exploration.bug_recipes. Reproduce it with a fresh builder and fresh driver state:

let bug = report
    .exploration
    .as_ref()
    .and_then(|exploration| exploration.bug_recipes.first())
    .expect("exploration found a bug recipe");

let replay = SimulationBuilder::new()
    .processes(3, || Box::new(PaxosNode::new()))
    .workloads(WorkloadCount::Fixed(1), |_| {
        Box::new(PaxosWorkload::new())
    })
    .invariant(AgreementInvariant::default())
    .replay_timeline(bug.seed, bug.recipe.clone())
    .run();

assert_eq!(replay.failed_runs, 1);

If replay does not reproduce, audit the integration for direct Tokio calls, unseeded collections or randomness, reused workload state, and external I/O. A recipe is exact only when the entire test harness is deterministic and starts each timeline from the same logical state.

Current Operational Limits

Before treating explorer results as a CI gate for a consensus system, account for these limits:

• Forked workers have no independent wall-clock deadline; blocking code in a child can stall the controller. Moonpool’s simulated-time budget only helps when the deterministic executor continues making events.
• Parallel workers may discover shared assertion novelty in finish order, so search order is not reproducible even though each captured recipe is.
• A worker crash after claiming a discovery can lose that discovery’s journal anchor for the rest of the run.
• Sanitizer coverage is a reporting and plateau signal, not a frontier anchor.

For a correctness-critical Paxos suite today, use exploration to supplement the ordinary multi-seed simulation, keep workers: 0 for deterministic CI, and replay every captured failure before diagnosing it.

Assertion Reference

• Boolean Assertions
  • assert_always!(condition, message)
  • assert_always_or_unreachable!(condition, message)
  • assert_sometimes!(condition, message)
  • assert_reachable!(message)
  • assert_unreachable!(message)
• Numeric Assertions
  • Always numeric example
  • Sometimes numeric example
• Compound Assertions
  • assert_sometimes_all!(message, [(name, bool), ...])
  • assert_sometimes_each!(message, [(key, value), ...]) / assert_sometimes_each!(message, [(key, value), ...], [(quality_key, quality_value), ...])
• Validation: validate_assertion_contracts()
  • Always violations (definite bugs)
  • Coverage violations (statistical)
• Related Functions

Moonpool provides 15 assertion macros for testing distributed system properties. All macros follow the Antithesis principle: assertions never crash your program. Violations are recorded in shared memory and reported after the simulation completes, allowing the system to continue running and discover cascading failures.

Every macro is tracked in shared memory via moonpool-explorer, enabling cross-process visibility across forked exploration timelines.

Boolean Assertions

These macros test boolean conditions. Always-type assertions record violations but do not panic. Sometimes-type assertions guide exploration by triggering forks on discovery.

assert_always!(condition, message)

Records a violation if condition is false. The simulation continues; violations are collected and reported at the end. Validated by validate_assertion_contracts() which flags the assertion if fail_count > 0, or if must_hit and the assertion was never reached.

#![allow(unused)]
fn main() {
assert_always!(
    granted_count <= 1,
    "lock never granted to two nodes simultaneously"
);
}

assert_always_or_unreachable!(condition, message)

Like assert_always!, but does not require the code path to be reached. If the code is never executed, the assertion passes silently. Useful for guarding optional error-handling paths.

#![allow(unused)]
fn main() {
assert_always_or_unreachable!(
    retry_count < max_retries,
    "retry count within bounds when retry path taken"
);
}

assert_sometimes!(condition, message)

Records pass/fail statistics. The first time the condition is true is a discovery: in exploration mode the explorer anchors alternate timelines at that point. Validated by checking that pass_count > 0 after enough iterations.

#![allow(unused)]
fn main() {
assert_sometimes!(
    saw_leader_election,
    "leader election triggered at least once"
);
}

assert_reachable!(message)

Marks a code path as “should be reached.” Always passes true as the condition. First reach is a discovery. Validated by checking that pass_count > 0.

#![allow(unused)]
fn main() {
if connection_failed {
    assert_reachable!("retry path exercised");
    retry().await;
}
}

assert_unreachable!(message)

Marks a code path that should never execute. If reached, records a violation (but does not panic). Validated by checking that pass_count == 0.

#![allow(unused)]
fn main() {
match state {
    State::Valid => { /* ok */ }
    State::Invalid => {
        assert_unreachable!("invalid state should never occur");
    }
}
}

Numeric Assertions

These macros compare a value against a threshold. Always-type numeric assertions record violations on failure. Sometimes-type numeric assertions track watermarks (best observed value); a watermark improvement is a discovery.

All values are cast to i64 internally.

Always numeric example

#![allow(unused)]
fn main() {
assert_always_greater_than!(
    queue.len(),
    0,
    "queue never empty during processing"
);

assert_always_less_than_or_equal_to!(
    latency_ms,
    timeout_ms,
    "response latency within timeout"
);
}

Sometimes numeric example

#![allow(unused)]
fn main() {
assert_sometimes_greater_than!(
    concurrent_connections,
    5,
    "achieved high concurrency"
);

assert_sometimes_less_than!(
    retry_delay_ms,
    100,
    "fast retry path exercised"
);
}

Watermark tracking: For sometimes numeric assertions, the explorer tracks the best value observed so far. When a new evaluation improves the watermark (higher for gt/ge, lower for lt/le), it creates a replay anchor for timelines that might push the metric even further.

Compound Assertions

These macros track multi-dimensional properties across multiple conditions or identity keys.

assert_sometimes_all!(message, [(name, bool), ...])

Tracks a frontier: the maximum number of conditions that have been simultaneously true. When the frontier advances (more conditions true at once than ever before), a replay anchor is created.

#![allow(unused)]
fn main() {
assert_sometimes_all!("all_nodes_healthy", [
    ("node_a", node_a_healthy),
    ("node_b", node_b_healthy),
    ("node_c", node_c_healthy),
]);
}

If previously at most 2 of the 3 conditions were true at once, and now all 3 are true, the frontier advances from 2 to 3 and a replay anchor is created.

assert_sometimes_each!(message, [(key, value), ...]) / assert_sometimes_each!(message, [(key, value), ...], [(quality_key, quality_value), ...])

Each unique combination of identity keys gets its own bucket. A new bucket is a discovery. If quality keys are provided, the explorer also tracks quality watermarks per bucket, and a quality improvement is a fresh discovery (registering a healthier exemplar for the same state).

#![allow(unused)]
fn main() {
// Identity keys only -- guide on first discovery of each (lock, depth) combo
assert_sometimes_each!("gate", [("lock", lock_id), ("depth", depth)]);

// With quality watermarks -- also guide when health improves for a known bucket
assert_sometimes_each!("descended", [("to_floor", floor)], [("health", hp)]);
}

Validation: validate_assertion_contracts()

After the simulation completes, validate_assertion_contracts() reads all assertion slots from shared memory and checks each one against its kind-specific contract. It returns two vectors:

Always violations (definite bugs)

These indicate real bugs and are safe to check regardless of iteration count.