The Five Providers
Moonpool abstracts every interaction between your code and the outside world into five provider traits. Each trait covers one category of I/O. Together, they form a complete boundary around your application, giving the simulator full control over every source of non-determinism.
The sim runs single-threaded on the moonpool deterministic executor, but every provider trait is Send + Sync + 'static. One OS thread runs everything for determinism, yet the types are Send-bounded so customer code stays normal: Arc<RwLock<…>>, DashMap, Arc<AtomicBool>, and Send-bounded task spawning all just work. The async methods use native AFIT (async fn in trait) with explicit -> impl Future<…> + Send desugarings to propagate the Send bound, so no #[async_trait] and no ?Send anywhere in the provider layer.
TimeProvider
Time is the most pervasive dependency in distributed systems. Every timeout, backoff, heartbeat, and lease check goes through TimeProvider.
#![allow(unused)]
fn main() {
pub trait TimeProvider: Clone + Send + Sync + 'static {
/// Sleep for the specified duration.
fn sleep(
&self,
duration: Duration,
) -> impl Future<Output = Result<(), TimeError>> + Send;
/// Get exact current time.
fn now(&self) -> Duration;
/// Get drifted timer time (simulates clock drift between nodes).
/// Defaults to `now()`; the simulation overrides it.
fn timer(&self) -> Duration {
self.now()
}
/// Run a future with a timeout.
fn timeout<F, T>(
&self,
duration: Duration,
future: F,
) -> impl Future<Output = Result<T, TimeError>> + Send
where
F: Future<Output = T> + Send,
T: Send;
}
}The distinction between now() and timer() is borrowed from FoundationDB’s sim2. timer() is a default method that returns now(), which is what production keeps; only the simulation overrides it. In simulation, timer() can drift up to 100ms ahead of now(), testing how your code handles clock skew between processes. Use now() for event scheduling. Use timer() for application-level time checks like lease expiry and heartbeat deadlines.
Production: TokioTimeProvider delegates sleep to tokio::time::sleep, timeout to tokio::time::timeout, and now to std::time::Instant::elapsed.
Simulation: Sleep schedules a Timer in the global Scheduler<Event>.
The scheduler owns monotonic logical time and same-time FIFO sequence order.
When all tasks are blocked, the simulator performs “time travel” by popping the
next scheduled event. This compresses hours of simulated cluster time into
seconds of wall-clock time.
NetworkProvider
#![allow(unused)]
fn main() {
pub trait NetworkProvider: Clone + Send + Sync + 'static {
type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;
type TcpListener: TcpListenerTrait<TcpStream = Self::TcpStream> + 'static;
/// Create a TCP listener bound to the given address.
fn bind(
&self,
addr: &str,
) -> impl Future<Output = io::Result<Self::TcpListener>> + Send;
/// Connect to a remote address.
fn connect(
&self,
addr: &str,
) -> impl Future<Output = io::Result<Self::TcpStream>> + Send;
}
pub trait TcpListenerTrait: Send + Sync + 'static {
type TcpStream: AsyncRead + AsyncWrite + Unpin + Send + 'static;
/// Accept a single incoming connection.
fn accept(
&self,
) -> impl Future<Output = io::Result<(Self::TcpStream, String)>> + Send;
/// Get the local address this listener is bound to.
fn local_addr(&self) -> io::Result<String>;
}
}The associated types TcpStream and TcpListener let each implementation provide its own concrete types. Production gives you tokio::net::TcpStream. Simulation gives you an in-memory stream backed by buffers with controllable latency, reordering, and connection failures.
The API deliberately matches what you would expect from tokio networking. bind, connect, accept behave like their tokio counterparts. The streams implement AsyncRead + AsyncWrite + Send, so they work with any tokio-compatible codec or framing layer and they cross task boundaries cleanly.
Production: TokioNetworkProvider wraps tokio::net.
Simulation: NetworkSimulation owns listeners, connections, topology,
faults, pending operation results, and network wakers. Bind, connect, and accept
park until their scheduled latency expires. Established streams use in-memory
buffers with deterministic delivery delays, TCP half-close simulation, and
fault injection such as connection drops, partitions, and corruption.
For numeric socket addresses, binding port zero assigns a distinct dynamic port;
listener.local_addr() returns that resolved address for clients to connect to.
Opaque logical addresses, such as the process IP alone, remain valid in the
simulated provider.
Dropping a stream with unread received bytes resets the peer, as TCP does when
the application discards acknowledged data. A drained stream closes gracefully;
AsyncWrite::close shuts down only the write half so the stream can still read
a reply.
Each simulated listener has a bounded accept backlog (128 unaccepted
connections by default). A full backlog parks new connects in FIFO order until
an accept returns a stream; a delayed accept’s reservation still occupies a
slot. SimulationBuilder::accept_backlog_capacity
sets the capacity for each run.
TaskProvider
#![allow(unused)]
fn main() {
pub trait TaskProvider: Clone + Send + Sync + 'static {
/// Join handle returned by `spawn_task`. `Detach` provides explicit
/// fire-and-forget: `spawn_task(...).detach()` leaves the task running
/// without keeping the handle.
type JoinHandle: Future<Output = Result<(), JoinError>> + Detach + Send + 'static;
/// Spawn a named task.
fn spawn_task<F>(&self, name: &str, future: F) -> Self::JoinHandle
where
F: Future<Output = ()> + Send + 'static;
/// Yield control to allow other tasks to run.
fn yield_now(&self) -> impl Future<Output = ()> + Send;
}
}Spawned futures are Send + 'static. The runtime still pins everything to one OS thread for determinism, but the bound matches what tokio::spawn expects, so customer code reads exactly like normal tokio code. The name parameter is diagnostic only. In simulation the executor stores it with the task and traces every poll under it; in production TokioTaskProvider merely emits a tracing::trace! event when the task starts and when it completes.
Production: TokioTaskProvider uses plain tokio::spawn. The name feeds those two trace events and nothing else — it is not attached to the tokio task.
Simulation: SimTaskProvider spawns onto the deterministic executor, so scheduling order is a seeded-random, fully reproducible function of the iteration seed.
RandomProvider
#![allow(unused)]
fn main() {
pub trait RandomProvider: Clone + Send + Sync + 'static {
/// Generate a random value of type T.
fn random<T>(&self) -> T
where
StandardUniform: Distribution<T>;
/// Generate a random value within a specified range (start..end).
fn random_range<T>(&self, range: Range<T>) -> T
where
T: SampleUniform + PartialOrd;
/// Generate a random f64 between 0.0 and 1.0.
fn random_ratio(&self) -> f64 {
self.random()
}
/// Generate a random bool with the given probability of being true.
fn random_bool(&self, probability: f64) -> bool {
self.random_ratio() < probability
}
}
}Only random and random_range are required; random_ratio and random_bool are default methods derived from them.
RandomProvider is fully synchronous. The other four providers expose async methods via native AFIT, but random number generation never needs to suspend, so its trait has no async fn at all. The supertrait shape (Clone + Send + Sync + 'static) stays consistent with the rest of the provider family.
Production: TokioRandomProvider uses rand::rng() (thread-local, non-deterministic).
Simulation: Uses the seeded ChaCha8Rng from the simulation’s RNG system. Every call draws from the same deterministic stream, maintaining reproducibility.
StorageProvider
#![allow(unused)]
fn main() {
pub trait StorageProvider: Clone + Send + Sync + 'static {
type File: StorageFile + 'static;
fn open(
&self,
path: &str,
options: OpenOptions,
) -> impl Future<Output = io::Result<Self::File>> + Send;
fn exists(&self, path: &str) -> impl Future<Output = io::Result<bool>> + Send;
fn delete(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
fn rename(
&self,
from: &str,
to: &str,
) -> impl Future<Output = io::Result<()>> + Send;
fn create_dir_all(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
fn sync_dir(&self, path: &str) -> impl Future<Output = io::Result<()>> + Send;
}
pub trait StorageFile: AsyncRead + AsyncWrite + AsyncSeek + Unpin + Send + Sync + 'static {
fn constraints(&self) -> IoConstraints;
fn is_direct_io(&self) -> bool;
fn read_at(
&self,
offset: u64,
buf: &mut [u8],
) -> impl Future<Output = io::Result<usize>> + Send;
fn write_at(
&self,
offset: u64,
buf: &[u8],
) -> impl Future<Output = io::Result<usize>> + Send;
fn sync_all(&self) -> impl Future<Output = io::Result<()>> + Send;
fn sync_data(&self) -> impl Future<Output = io::Result<()>> + Send;
fn size(&self) -> impl Future<Output = io::Result<u64>> + Send;
fn set_len(&self, size: u64) -> impl Future<Output = io::Result<()>> + Send;
}
}Storage is the newest provider, and the one with the richest fault model. The
split is that the provider owns the filesystem namespace — open, exists,
delete, rename, and the directory sync that makes those durable — while the
file owns already-open bytes. The two never cross: a delete, or a
rename over a name, removes the name and nothing else, so a file opened
before it keeps reading and writing the same bytes, exactly as on Unix, and
the bytes are freed only when the last name and the last handle are both
gone. Log rotation and atomic replacement lean on that in production, and
the simulator holds to it too. And the namespace is per process: a
provider is created for one process IP, so two processes opening data.db
open two files on two disks, one node’s delete or rename never reaches
another’s, and a sync_dir commits only the syncing node’s directory
entries, as it would on two machines.
OpenOptions mirrors std::fs::OpenOptions with read, write, create,
truncate, and append, plus the one thing a database needs that
std::fs::OpenOptions does not name portably: direct_io(DirectIo). That is
deliberately an option on opening an ordinary file rather than a second
provider stack — there is no BlockProvider and no DirectIoProvider, and a
journal or pager is written directly against StorageFile. The flags obey
std’s rules too, on both backends: truncate, create and create_new
need write or append, append and truncate contradict each other, and a
combination std refuses with InvalidInput (OpenOptions::validate) is
refused by the simulator before it touches the file, so a read-only
truncate destroys nothing in simulation that it would not destroy in
production.
read_at/write_at are positioned: they take &self, never touch the stream
cursor, and return the number of bytes moved, so non-overlapping ranges can be
read and written concurrently and partial transfers stay visible to the
caller. On a file with I/O constraints they are the only way to transfer:
the stream half of the trait is refused there, because a shared cursor cannot
be kept aligned. BlockFile<F> wraps one open file to add block arithmetic and the
loops that turn those partial transfers into whole ones; it holds no path and
opens nothing.
Production: TokioStorageProvider wraps tokio::fs.
Simulation: StorageEngine owns an in-memory filesystem with fault
injection inspired by TigerBeetle and FoundationDB patterns: read and write
corruption, EIO, crash and torn writes, misdirected I/O, sync failures, short
transfers, unsynced directory-entry loss, and IOPS/bandwidth timing. There is
exactly one simulated file implementation behind every API — a positioned
write is visible to a stream read, and a block write is visible to both. Persistent file contents are separate from open-handle
state, so two handles have independent cursors, access options, and pending
operations. Each delayed operation has an exact ID, explicit pending or
completed result, and its own waker. Crash and shutdown complete pending work
with errors rather than treating absence as success.
Each SimStorageProvider is scoped to a process IP
(SimStorageProvider::new(sim, ip)), so the engine resolves the correct
per-process configuration and disk-degradation episode for every operation.
The Providers Bundle
All five come together in the Providers trait:
#![allow(unused)]
fn main() {
pub trait Providers: Clone + Send + Sync + 'static {
type Network: NetworkProvider;
type Time: TimeProvider;
type Task: TaskProvider;
type Random: RandomProvider;
type Storage: StorageProvider;
fn network(&self) -> &Self::Network;
fn time(&self) -> &Self::Time;
fn task(&self) -> &Self::Task;
fn random(&self) -> &Self::Random;
fn storage(&self) -> &Self::Storage;
}
}TokioProviders bundles all five production implementations. SimProviders bundles all five simulation implementations and requires an IP address at construction (SimProviders::new(sim, seed, ip)) so that the storage provider is scoped to the correct process. Your application code sees P: Providers and nothing else.