Consolidated quick-reference of every fault moonpool-sim can inject, organized by category. For detailed explanations and examples, see Network Faults , Storage Faults , and Attrition: Process Reboots .
Every fault listed below is automatically emitted to the "sim:faults" event timeline as a SimFaultEvent. Invariants can read these to correlate application behavior with infrastructure faults.
All defaults below refer to the values in ChaosConfiguration::default() and StorageConfiguration::default(). When using random_for_seed(), these values are randomized per seed within documented ranges.
Configured via ChaosConfiguration (nested under NetworkConfiguration::chaos).
Fault Config Field Default Real-World Scenario
Random connection close random_close_probability0.001% Reconnection logic, message redelivery, connection pooling Asymmetric close random_close_explicit_ratio30% explicit (FIN), 70% silent (RST) Half-closed sockets, FIN vs RST handling Close cooldown random_close_cooldown5s Prevents cascading failures after a close event Connect failure connect_failure_modeProbabilistic (50% refused, 50% hang)Connection establishment retries, timeout handling Connect failure probability connect_failure_probability50% Ratio of failed vs hanging connections
Fault Config Field Default Real-World Scenario
Latency distribution latency_distributionUniformP99/P99.9 tail latency testing Slow latency spike slow_latency_probability0.1% (bimodal mode only) GC pauses, cross-datacenter hops Slow latency multiplier slow_latency_multiplier10x normal Magnitude of tail latency spikes Write clogging clog_probability / clog_duration0%, 100-300ms Backpressure handling, flow control Clock drift clock_drift_enabled / clock_drift_maxenabled, 100ms Lease expiration, distributed consensus, TTL handling Buggified delay buggified_delay_probability / buggified_delay_max25%, 100ms Race conditions, timing-dependent bugs Handshake delay handshake_delay_enabled / handshake_delay_maxenabled, 10ms TLS negotiation, connection startup overhead
Fault Config Field Default Real-World Scenario
Random partition partition_probability0% Split-brain, quorum loss, leader election Partition duration partition_duration200ms-2s Recovery time after network heal Partition strategy partition_strategyRandomRandom / UniformSize / IsolateSingle patterns
Manual partition methods are also available on SimWorld: partition_pair(), partition_send_from(), partition_recv_to().
Fault Config Field Default Real-World Scenario
Bit flips bit_flip_probability0.01% CRC/checksum validation, data corruption detection Flip range bit_flip_min_bits / bit_flip_max_bits1-32 bits Power-law distribution of corruption severity Flip cooldown bit_flip_cooldown0 (no cooldown) Rate-limiting corruption events Partial writes partial_write_max_bytes1000 bytes TCP fragmentation, message framing
Fault Method Real-World Scenario
Peer crash simulation simulate_peer_crash()TCP keepalive, heartbeat detection, silent failures Half-open error detection should_half_open_error()Timeout-based failure detection Stable connection exemption mark_connection_stable()Exempt supervision channels from chaos
Configured via StorageConfiguration. All fault probabilities default to 0% and must be enabled explicitly or via random_for_seed(). Storage faults are scoped per process: StorageState holds a global config plus optional per-process overrides in per_process_configs. Use SimWorld::set_process_storage_config(ip, config) to assign different fault profiles to individual processes.
Fault Config Field Default Real-World Scenario
Read corruption read_fault_probability0% ECC failures, DRAM bit flips, media degradation Write corruption write_fault_probability0% Bad sectors, controller bugs, disk full Crash fault (torn writes) crash_fault_probability0% Power loss mid-I/O, crash consistency Misdirected write misdirect_write_probability0% Firmware bugs, wrong block written Misdirected read misdirect_read_probability0% Controller errors, wrong block read Phantom write phantom_write_probability0% Drive lies about durability Sync failure sync_failure_probability0% fsync fails, disk full
Method Parameters Description
SimWorld::set_process_storage_config(ip, config)IpAddr, StorageConfigurationSet per-process fault config (overrides global) SimWorld::simulate_crash_for_process(ip, close_files)IpAddr, boolSimulate power loss: torn writes, optional file close SimWorld::wipe_storage_for_process(ip)IpAddrDelete all storage owned by the process SimWorld::storage_provider(ip)IpAddrCreate a SimStorageProvider scoped to this process
Storage also simulates realistic performance characteristics independent of fault injection.
Parameter Config Field Default Description
IOPS iops25,000 I/O operations per second limit Bandwidth bandwidth150 MB/s Maximum throughput Read latency read_latency50-200us Per-read operation delay Write latency write_latency100-500us Per-write operation delay Sync latency sync_latency1-5ms Per-sync/flush delay
Configured via AttritionFaultInjector
Fault Mechanism Behavior
Graceful reboot RebootKind::GracefulSignal shutdown token, wait grace period (default 2-5s), force kill, restart after recovery delay (default 1-10s) Crash reboot RebootKind::CrashImmediate task abort, all connections reset, restart after recovery delay Crash + wipe RebootKind::CrashAndWipeCrash behavior + immediate wipe of all persistent storage owned by the process (scoped by IP) Continuous attrition Attrition configRandom reboots during chaos phase with weighted prob_graceful/prob_crash/prob_wipe and max_dead limit
Preset Description
NetworkConfiguration::random_for_seed()All chaos parameters randomized per seed for comprehensive testing NetworkConfiguration::fast_local()1-10us latencies, all chaos disabled ChaosConfiguration::disabled()Zero probability for every fault category StorageConfiguration::random_for_seed()Randomized faults (0.001%-0.1%), varied IOPS (10K-100K), varied bandwidth (50-500 MB/s) StorageConfiguration::fast_local()1M IOPS, 1 GB/s bandwidth, 1us latencies, all faults disabled
See Configuration Reference for the complete builder API and all configuration types.