Index

• Quick Start Routes
• Part I: Why Simulation Testing
• Part II: Foundations
• Part III: Building Simulations
• Part IV: Simulating Existing Applications
• Part V: Raw Network Simulation
• Part VI: Building on Top
• Appendix

A sitemap of every chapter in the Moonpool book. Each entry links to a chapter with a summary of what it covers.

Quick Start Routes

• “I am a coding agent adding a full simulation harness” — Guide for AI Agents
• “What is Moonpool?” — The Case for Simulation, then Why Moonpool Exists
• “How do I write my first simulation?” — Your First Simulation and its sub-chapters
• “How do providers work?” — The Provider Pattern
• “How do I add chaos/faults?” — Chaos in Moonpool
• “How do I use assertions?” — Assertions: Finding Bugs
• “How do I simulate raw TCP?” — Simulating the Network
• “How do I test an existing app (e.g. axum)?” — Using moonpool-sim Standalone
• “How do I ship this to production?” — Using Providers in Production
• “How do I migrate my existing app onto providers?” — Migrating Existing Code to Providers
• “Can the simulation run in a browser?” — Simulation in the Browser
• “How does multiverse exploration work?” — Multiverse Exploration
• “What assertions are available?” — Assertion Reference
• “What configuration options exist?” — Configuration Reference

Part I: Why Simulation Testing

• The Case for Simulation — Why distributed systems need simulation; the gap between localhost and production; failure statistics
• Prevention vs Discovery — Two testing philosophies: regression (prevention) vs generative (discovery)
• From Mocks to Simulation — Why mocks break at scale; the #[cfg(test)] trap; maintenance cost
• A Brief History — FoundationDB simulator origins, TigerBeetle storage faults, Antithesis assertions
• Why Moonpool Exists — Synthesizing ideas from FDB, TigerBeetle, and Antithesis into one framework

Part II: Foundations

• Determinism as a Foundation — Three non-determinism sources: threads, I/O, randomness; why reproducibility matters
• The Single-Core Constraint — Single-threaded execution guarantees one legal ordering; the moonpool executor
• Seed-Driven Reproducibility — One u64 seed controls entire simulation; ChaCha8Rng; cross-platform determinism
• The Deterministic Executor — Moonpool’s own single-threaded executor; seeded-random task scheduling; no reactor needed
• The Provider Pattern — Five traits (Time, Network, Task, Random, Storage) abstract all I/O; swap real vs simulated
• Quick Start: Swapping Implementations — Practical example: generic function running against TokioProviders or SimProviders
• Deep Dive: Why Providers Exist — Problems with #[cfg(test)] and mocks; providers eliminate both
• The Five Providers — TimeProvider, NetworkProvider, TaskProvider, RandomProvider, StorageProvider details
• System Under Test vs Test Driver — Process (server code) vs Workload (test driver); two distinct roles
• Process: Your Server — Process trait: name(), run(); recreated fresh on every boot from factory
• Workload: Your Test Driver — Workload trait: setup(), run(), check(); survives reboots; drives and validates

Part III: Building Simulations

• Your First Simulation — End-to-end walkthrough: KV server process, workload, assertions, builder
• Defining a Process — KvServer implementing Process trait; handling TCP; respecting shutdown
• Writing a Workload — KvWorkload tracking state; sending requests; validating responses
• Configuring the SimulationBuilder — Builder pattern: .workload(), .processes(), chaos config, iterations
• Running and Observing — cargo xtask sim run; reading reports; simulation binary structure
• Chaos Testing vs Simulation — Chaos engineering (production, reactive) vs simulation (deterministic, proactive)
• Chaos in Moonpool — Four fault dimensions: buggify, attrition, network faults, storage faults
• Buggify: Fault Injection — Two-phase activation; testing error paths; FoundationDB-inspired
• Attrition: Process Reboots — Graceful, crash, wipe reboot types; randomized kills; recovery delay
• Network Faults — Delayed connection establishment, latency, partitions, drops, reordering, and clogging
• Storage Faults — Exact delayed operations, independent handles, corruption, misdirected I/O, crash errors, and per-process disk behavior
• Assertions: Finding Bugs — Record and continue (Antithesis principle); cascade discovery
• Invariants vs Discovery vs Guidance — Three assertion categories: invariants, sometimes, numeric
• Always and Sometimes — assert_always! (must hold) vs assert_sometimes! (exploration guidance)
• Numeric Assertions — assert_always_less_than!; watermark tracking; explorer optimizes bounds
• Compound Assertions — assert_sometimes_all! for simultaneous sub-goals; frontier tracking
• Events and Invariants — plain tracing events as the timeline, the Invariant trait runs after every step, the auto-recorded fault timeline, SimTime formatter for sim-time log prefixes
• Designing Workloads That Find Bugs — Targeted adversarial design vs white noise; strategy matters
• Debugging a Failing Seed — Five-step workflow: reproduce, isolate, understand, fix, verify
• Reproducing with FixedCount — Pin seed with set_debug_seeds() + set_iterations(1); exact replay
• Reading the Event Trace — Scheduler ordering, targeted component events, and causal chain reconstruction
• Common Pitfalls — Delayed-operation driving, executor drains, provider-only calls, and component boundaries
• Discovering Properties — Systematic property discovery using attention focuses; finding where to place assertions and buggify

Part IV: Simulating Existing Applications

• Using moonpool-sim Standalone — Standalone simulation engine for existing code (axum, Postgres, etc.)
• Where to Draw the Line — Fakes vs test containers; binary failure limitations
• Wiring a Web Service — Worked example: axum service in simulation with Store trait fake, chaos, assertions
• The hyper Stack: gRPC and HTTP: moonpool-hyper’s runtime adapters, the reconnecting h2 channel, the serve helper, and why h2 keepalive and backoff stay deterministic
• What You’re Testing (and What You’re Not) — Tests handler logic and HTTP under chaos; doesn’t test TLS, proxies, startup code
• Using Providers in Production — Deploying the same code on TokioProviders; the lean dependency stanza; feature + platform matrices; the now() gotcha
• Migrating Existing Code to Providers — Routing existing tokio/rand/fs calls through providers; the call mapping; the futures::io Compat gotcha; verifying with the conformance suite
• Simulation in the Browser — Why the sim compiles to wasm; whether block_on parks; building a wasm-able crate; the moonpool-wasm-demo example; the portability CI gate

Part V: Raw Network Simulation

• Simulating the Network — Provider-backed TCP streams, connection-level faults, partial I/O, and the boundary between simulation and the kernel

Part VI: Building on Top

• Multiverse Exploration — Deterministic replay turns executions into accumulated exploration knowledge
• The Exploration Problem — Sequential Luck Problem: N unlikely events need exponential trials without branching
• The Frontier Controller — Recipes, discovery journals, one expansion per productive run
• Bounded Workers — fork() as snapshot optimization; 1 + workers live processes, any depth
• Exemplars and Continuations — Bounded exemplars per semantic state; depth-weighted continuation scheduling
• Multi-Seed Exploration — Cumulative novelty across seeds; barren seeds stop after one run

Appendix

• Assertion Reference — Complete table of 15 assertion macros with behavior and parameters
• Crate Map — Workspace crate diagram and dependency hierarchy
• Configuration Reference — SimulationBuilder methods, ChaosConfiguration, AttritionConfiguration, exploration
• Fault Reference — Every fault by category with config fields and defaults
• Glossary — Alphabetical definitions: adaptive forking, always assertion, attrition, buggify, coverage bitmap, etc.
• Sim Compatibility Checklist — Reference checklist for bringing existing Rust code into a moonpool simulation: forbidden APIs vs the provider seam
• Calibrating Against a Real Machine — Measuring real storage and network latency with raw std I/O, then generating LatencyDistribution constants