Guide for AI Agents

• The Rules
• Crates and Features
• Providers
• Process, Workload, Context
• SimulationBuilder
• SimulationReport
• Assertions
• Invariants and Traces
• Buggify and Hints
• Chaos Configuration
• FaultContext (custom injectors)
• Exploration
• Low-Level SimWorld
• Staged Adoption
• Code to Read First

This page is a dense map of moonpool for a coding agent that adds deterministic simulation to a system. Every name on it exists in the current source. When this page and the code disagree, the code wins: read the source and update this page with the change. The narrative chapters explain each mechanism; this page shows how to combine them.

The Rules

1. Production code is generic over P: Providers. It calls time.sleep, task.spawn_task, network.connect, storage.open, random.random_range, system.sample. Never tokio::spawn, tokio::time, tokio::select!, std::fs, std::net, thread_rng, Instant::now.
2. The same code runs in production over TokioProviders and in simulation over SimProviders. No Sim* wrapper in application code.
3. Process is the server, rebuilt by its factory on every reboot. Workload is the test driver and survives reboots. Never merge them.
4. Facts go out as tracing events. Cross-process safety goes in an Invariant. Local checks go in assert_always!. Rare paths get assert_sometimes!. Dangerous moments get buggify! or hint!.
5. Every random decision is a counted draw on one seeded stream. BTreeMap, not HashMap. Futures are Send + 'static. No unwrap().
6. Replay before diagnosis: set_debug_seeds(vec![seed]).set_iterations(1) or replay_timeline(seed, recipe).
7. Never delete an assertion that catches a bug. Fix the bug.
8. Validate with nix develop --command cargo fmt, cargo clippy, cargo nextest run, and mdbook build book/ when a public API changes.

Crates and Features

Production: moonpool = { default-features = false, features = ["tokio"] }. Simulation harness: moonpool-sim in dev-dependencies or a separate binary crate. wasm: moonpool-sim with default-features = false.

Providers

OpenOptions::direct_io(DirectIo::{Disabled, Optional, Required}), BlockFile<F> for block arithmetic over one open file (owned *_blocks_aligned forms for zero copy), AlignedBuf (zeroed, for_constraints, truncate, is_aligned_to), LayoutRegion { path, bytes, kind, stripe, shard } to name byte ranges. moonpool_sim::select! replaces tokio::select! with the same grammar. moonpool_sim::executor::spawn(name, fut) is the sim-only raw spawn.

Process, Workload, Context

#[async_trait]
impl Process for Node {
    fn name(&self) -> &str { "node" }
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> { .. }
}

#[async_trait]
impl Workload for Driver {
    fn name(&self) -> &str { "driver" }
    async fn setup(&mut self, ctx: &SimContext) -> SimulationResult<()> { Ok(()) }  // default
    async fn run(&mut self, ctx: &SimContext) -> SimulationResult<()> { .. }
    async fn check(&mut self, ctx: &SimContext) -> SimulationResult<()> { Ok(()) }  // default, after chaos
}

SimContext methods:

SimulationBuilder

let report = SimulationBuilder::new()
    .processes(3, || Box::new(Node::default()))              // group 0 on 10.0.1.{1..3}
    .tags(&[("role", &["leader", "follower"])])?             // round-robin over the last group
    .workloads(WorkloadCount::Fixed(2), |i| Box::new(Driver::new(i)))
    .enable_chaos([
        Chaos::Network(ChaosMode::Swarm),
        Chaos::Storage(ChaosMode::Swarm),
        Chaos::Attrition { config: attrition, mode: ChaosMode::Swarm },
    ])
    .chaos_duration(Duration::from_secs(60))
    .invariant(NoSplitBrain::new())
    .run()?;
if report.is_failure() { report.eprint(); std::process::exit(1); }

No builder method accepts a NetworkConfiguration or StorageConfiguration. Without Chaos::Network the network still has FoundationDB’s always-on faults (random close, clock drift, buggified delay, probabilistic connect failure). Without Chaos::Storage storage has no faults.

SimulationReport

report.eprint() prints it. success_rate(), average_wall_time(), average_simulated_time().

Assertions

Limits: 2048 sites, 256 buckets. Only sometimes-style assertions create discoveries. Full table: Assertion Reference.

Invariants and Traces

tracing::info!(target: "raft", term, leader = %my_ip, "leader_elected");   // emit, in process code

impl Invariant for NoSplitBrain {
    fn name(&self) -> &'static str { "no_split_brain" }
    fn observe(&self, q: &dyn TraceQuery, _sim_time_ms: u64) {
        for e in q.since("leader_elected", &self.cursor) {       // cursor: Cell<usize>
            let term = e.u64("term"); let leader = e.str("leader");
            assert_always!(.., "one leader per term");
        }
    }
    fn reset(&mut self) { .. }                                    // per seed
}

Capture rule: INFO and above, constant message, inside a process or workload task. TraceEvent { seq, time_ms, source, target, level, name, fields } with u64, i64, f64, bool, str accessors. TraceQuery { len(name), since(name, &cursor), snapshot(name) }. Sim faults appear as "sim_fault" events with a kind field. init_sim_tracing(Level) in a binary’s main.

Buggify and Hints

All are no-ops in production. moonpool-journal shows the pattern: four hint! moments per batched commit (one after a failed sync writes the batch again) (an ordered commit’s two durable writes are named inside the simulated write_at_durable, “storage: durable write landed, not synced”), and two per roll onto a recycled spare segment.

Chaos Configuration

Every default and range: Configuration Reference. Every fault: Fault Reference.

FaultContext (custom injectors)

#[async_trait]
impl FaultInjector for Partitioner {
    fn name(&self) -> &str { "partitioner" }
    async fn inject(&mut self, ctx: &FaultContext) {
        while !ctx.chaos_shutdown().is_cancelled() {
            if let Some(ip) = ctx.reboot_random(RebootKind::Crash).ok().flatten() { .. }
            ctx.time().sleep(Duration::from_secs(5)).await.ok();
        }
    }
}

Exploration

.workload_factory(|| Box::new(Driver::default()))
.enable_exploration(ExplorationConfig { workers: 0, max_runs_per_seed: 2400, branching_factor: 4, max_frontier: 512, max_recipe_len: 32 })

workers: 0 is deterministic and fork-free: use it in tests. workers > 0 only in a standalone binary run through cargo xtask sim run <name>. Bugs land in report.exploration.bug_recipes as BugRecipe { seed, recipe }; replay with replay_timeline(seed, recipe). format_timeline / parse_timeline convert "151@8837201 -> 80@1293847". Guidance comes from sometimes-style assertions only. See Multiverse Exploration.

Low-Level SimWorld

For provider tests without the runner: SimWorld::new_with_seed(seed), new_with_network_config(cfg), set_storage_config, set_process_storage_config(ip, cfg), storage_provider(ip), network_provider(ip), time_provider(), step(), has_pending_events(), run_until_empty(), take_faults(), partition_pair(from, to, d), corrupt_file(path, sectors), fail_file_with_eio(path, sectors, EioTarget), simulate_crash_for_process(ip, close_files), set_fault_focus, publish_layout. Drive futures with moonpool_sim::executor::Executor::new(seed).block_on(..) and step the world on Pending (the drive helper in the root AGENTS.md).

Staged Adoption

1. Boundary. Make the code generic over P: Providers. Replace every forbidden call (Sim Compatibility Checklist). Run one seed with no chaos. Add check_determinism() and keep it green.
2. Driver and oracle. Write the Workload with a small reference model (BTreeMap of expected state, not a copy of the server). Reconcile ambiguous results: a timed-out write may have committed.
3. Facts and invariants. Emit tracing events for every protocol decision. Write cursor-based invariants for agreement, validity, persistence, prefix, client semantics.
4. Guidance. Add assert_sometimes! on every retry, timeout and recovery path. Add watermarks for depth. Keep bucket vocabularies coarse.
5. Chaos. Enable Chaos::Network(Swarm), then Storage(Swarm), then Attrition, then Outage, Cpu, GrayFailure. Add hint! moments in storage code. Add buggify! at dangerous choices. Mask what the harness cannot observe.
6. Campaign. Keep the default UntilCoverageStable. Register the binary in xtask (Running Simulation Binaries with xtask). Add exploration when a bug needs a sequence of rare events.
7. Debug. Pin the seed with set_debug_seeds, set_iterations(1), trace_level(LevelFilter::ERROR). Read the event trace. Fix the root cause. Re-enable chaos.

Code to Read First