Index

• Quick Start Routes
• Foreword
• Part I: Why Simulation Testing
• Part II: Foundations
• Part III: Building Simulations
• Part IV: Simulating Existing Applications
• Part V: Raw Network Simulation
• Part VI: Building on Top
• Appendix

This page is a map of the book. Each entry links to one chapter and says what it covers.

Quick Start Routes

Foreword

• State of Moonpool: the crates, what is stable, what is new, what is missing.

Part I: Why Simulation Testing

• The Case for Simulation: the gap between localhost and production, and why coverage numbers lie.
• Prevention vs Discovery: regression tests guard what we know; simulation finds what we do not know.
• From Mocks to Simulation: why mocks and #[cfg(test)] fail at scale, and how traits replace them.
• A Brief History: FoundationDB, Antithesis, TigerBeetle and what moonpool takes from each.
• Why Moonpool Exists: one short paragraph for each moonpool feature, with links.

Part II: Foundations

• Determinism as a Foundation: threads, I/O and randomness, and how moonpool removes each one.
• The Single-Core Constraint: one thread gives one legal order; Send-bounded traits; no tokio runtime in a simulation.
• Seed-Driven Reproducibility: one u64 seed controls the run; RNG call counts; the opt-in determinism canary.
• The Deterministic Executor: seeded task scheduling, task owners, the CPU gate, kill-on-drop.
• The Provider Pattern: the Providers bundle; TokioProviders and SimProviders.
• Quick Start: Swapping Implementations: one generic function, run on Tokio and in the simulator; the forbidden calls.
• Deep Dive: Why Providers Exist: why not mocks, “never more permissive”, static dispatch.
• The Six Providers: Time, Task, Network, Random, Storage and System, plus Resolver and select!.
• Same Code in Production and Simulation: no Sim* wrappers; hint!, buggify! and cpu_charge are no-ops in production; lean release builds.
• System Under Test vs Test Driver: why Process and Workload are two traits.
• Process: Your Server: the factory, reboots, the context, crash_self.
• Workload: Your Test Driver: setup, run, check; the operation alphabet; slowness and system samples.

Part III: Building Simulations

• Your First Simulation: the plan and the full target program.
• Defining a Process: accept loop, select! on shutdown, ranged counts.
• Writing a Workload: topology lookups, a reference model, why run() must finish.
• Configuring the SimulationBuilder: every builder method in one table, with links.
• Running and Observing: nextest vs xtask, the report, stop conditions, stall guards.
• Process Groups, Tags and Topology: several roles, IP ranges, tags, .cluster() failure domains.
• Chaos Testing vs Simulation: two schools of breaking things, and where faults enter.
• Chaos in Moonpool: every Chaos surface, ChaosMode::Random and Swarm, the chaos window.
• Buggify: Fault Injection: buggify!, buggify_with_prob!, buggify_named!, buggify_knob!, two-phase activation.
• hint!: Naming Fault Sites: the code names a moment, the simulator decides; vetoes; the journal example.
• Swarm Testing: Masks and Budgets: fault masks, budgets, swarm_operations, one workload with many behaviours.
• Attrition: Process Reboots: reboot kinds, max_dead, victims per group, failure domains.
• Correlated Outages and Self-Crashes: Chaos::Outage, OutageVeto, SimContext::crash_self.
• Network Faults: the nine families, partitions, black holes, network masks.
• Storage Faults: storage layering, the barrier-bounded crash model, the lost-synced-write oracle.
• Aiming Storage Faults: LayoutRegion, fault focus, targeted injection, replicated fault patterns.
• CPU Time and Gray Failures: the CPU gate, cpu_charge, SystemProvider, Chaos::GrayFailure, slow machines.
• Custom Fault Injectors: FaultInjector, fault_factory, the FaultContext methods.
• Assertions: Finding Bugs: the fifteen macros, where counts live, how assertions decide when to stop.
• Invariants vs Discovery vs Guidance: three jobs, and what each kind does in the runner and the explorer.
• Always and Sometimes: the five boolean macros.
• Numeric Assertions: the eight numeric macros and watermarks.
• Compound Assertions: assert_sometimes_all! and assert_sometimes_each!.
• Events and Invariants: plain tracing events, the Invariant trait, the fault timeline.
• Application Metrics: per-node Prometheus metrics, metric queries, a metastable failure.
• Designing Workloads That Find Bugs: operation alphabets, swarm, invariant patterns.
• Debugging a Failing Seed: the workflow.
• Reproducing with FixedCount: pin the seed, turn up the logs.
• Reading the Event Trace: event kinds, faults in the timeline, RNG call counts.
• Common Pitfalls: seventeen mistakes and their fixes.
• Discovering Properties: eight attention focuses to find what to assert.

Part IV: Simulating Existing Applications

• Using moonpool-sim Standalone: real HTTP over simulated TCP.
• Where to Draw the Line: fakes vs containers, per dependency.
• Wiring a Web Service: an axum service step by step.
• The hyper Stack: gRPC and HTTP: H2Server, ReconnectingChannel, tonic.
• What You’re Testing (and What You’re Not): the limits of the model.
• Using Providers in Production: TokioProviders, the lean dependency tree, inert fault sites.
• Migrating Existing Code to Providers: the call mapping and the gotchas.
• Simulation in the Browser: the wasm build and the live demo.

Part V: Raw Network Simulation

• Simulating the Network: TCP, not packets; who owns what.
• Typed RPC with moonpool-rpc: driver, endpoints, delivery modes, reply streams, failure monitor.
• What moonpool-rpc Promises: the promises and the seven campaigns that check them.
• Admission, Security and Derive: admission queues, buggify points, TLS, JWT, the derive macro.

Part VI: Building on Top

• Multiverse Exploration: fork at discovery.
• The Exploration Problem: the sequential luck problem.
• The Frontier Controller: recipes, the discovery journal, bug recipes.
• Bounded Workers: a fixed pool of forked workers.
• Exemplars and Continuations: semantic states and the under-explored frontier.
• Multi-Seed Exploration: exploration across many seeds.
• Exploring a Consensus Protocol: a worked procedure.
• A Crash-Aware Journal: moonpool-journal, its commit hints, and its tests under Chaos::Storage.

Appendix

• Assertion Reference: every macro in one table.
• Crate Map: every crate, its features and dependencies.
• Configuration Reference: every config struct with defaults.
• Fault Reference: every fault family with defaults.
• Glossary: the terms.
• Sim Compatibility Checklist: ten checks before code runs in a simulation.
• Calibrating Against a Real Machine: moonpool-calibrate measures a disk and a network.
• Running Simulation Binaries with xtask: cargo xtask sim.