Application Metrics

Your service is probably already instrumented. There is a prometheus::Registry somewhere, a requests_total counter, a latency histogram, a gauge for the queue depth. That instrumentation exists because it is how you understand the system in production.

Moonpool can report on it. Register a metrics source per simulated node and the counters, gauges and histograms your code already keeps show up in the simulation report — aggregated across seeds, attributed per node, with an exact time series of every mutation.

use std::sync::Arc;
use moonpool_prometheus::PrometheusSource;

let report = SimulationBuilder::new()
    .metrics_factory(|_ip| Arc::new(PrometheusSource::default()))
    .processes(3, || Box::new(MyNode::new()))
    .workload(MyWorkload::default())
    .set_iterations(10)
    .run();

report.eprint();

━━━ Metrics (15 series) ━━━━━━━━━━━━━━━━━━━━━━━━━━
  kv_keys_stored{instance="10.0.1.1"}                  15.81 avg        1 min       16 max
  kv_request_latency_seconds{instance="10.0.1.1"}     130.83 total  6,667 obs     0.02 avg
  kv_requests_in_flight{instance="10.0.1.1"}            2.22 avg         0 min        3 max
  kv_requests_rejected_total{instance="10.0.1.1"}      8,620 total     862 per seed
  kv_requests_served_total{instance="10.0.1.1"}        6,667 total  666.70 per seed

Reading your metrics from process code

The source is reachable from any process or workload through its context:

let metrics = ctx.metrics::<PrometheusSource>()
    .ok_or_else(|| SimulationError::InvalidState("no metrics factory".into()))?;

let served = metrics.counter("requests_served_total", "Requests served")?;
let in_flight = metrics.gauge("requests_in_flight", "Requests running")?;
let latency = metrics.histogram_with_buckets(
    "request_latency_seconds",
    "Time to serve one request",
    vec![0.001, 0.005, 0.010, 0.050, 0.100],
)?;

served.inc();

These are get-or-register, which matters because a source outlives process reboots (see below). A rebooted process looks its metrics up instead of failing with AlreadyReg.

To wrap a registry your application already built, use PrometheusSource::new(my_registry).

Why per node, and why per iteration

Every simulated node shares one OS process. A single shared registry would merge three nodes’ counters into one number, so the factory is keyed by IP and each node gets its own registry. Each sample is stamped with an instance label naming the node — Prometheus’ own label for a scrape target.

The factory also runs afresh every iteration. Most registries have no reset, so a reused instance would make seed 50 report the sum of fifty runs. This is the same reason a custom fault injector is registered through fault_factory rather than as an instance.

A reboot is the exception: a source is keyed by IP, so a crashed and restarted process keeps its counters, exactly as a real node’s /metrics does across a restart on the same host.

Time series, not samples

Metrics created through the source hand back instrumented handles. Every inc(), set() and observe() records a point, stamped with the simulated clock. There is no scrape interval and no sampling: the series is exact, and it replays bit-for-bit for a given seed.

let metrics = report.individual_metrics[0].as_ref().expect("seed passed");

for (series, points) in &metrics.app_series {
    for point in points {
        println!("{series} {} at {}ms", point.value, point.time_ms);
    }
}

That is what makes a latency spike line up with the partition that caused it. The aggregated view on report.app_metrics uses this series for min, max and mean, which is why a gauge reports the range it actually moved over rather than whatever value it happened to rest at when the run ended.

Series are capped (10,000 points each by default; see PrometheusSource::set_series_capacity) so a hot loop cannot grow a test’s memory without bound. SimulationMetrics::dropped_metric_points is non-zero when a series was truncated.

Asking questions: metric queries

A raw series is data, not an answer. After five hundred seeds the question is usually “what throughput did this hold, and which seed was the worst?” — so the runner declares what it wants to know, and the report answers it.

use std::time::Duration;
use moonpool_sim::{Fill, Mean, MetricQuery, Percentile};

let report = SimulationBuilder::new()
    .metrics_factory(|_ip| Arc::new(PrometheusSource::default()))
    .metric(
        MetricQuery::select("requests_total")
            .label("operation", "write")
            .rate()
            .bucketize(Duration::from_secs(60), Mean)
            // A minute with no requests is zero per second, not missing data.
            .fill(Fill::Value(0.0))
            .reduce(Mean)
            .named("write_throughput"),
    )
    .metric(
        MetricQuery::select("request_latency_seconds")
            .bucketize(Duration::from_secs(60), Percentile(0.99))
            .named("write_p99"),
    )
    .processes(3, || Box::new(MyNode::new()))
    .workload(MyWorkload::default())
    .set_iterations(500)
    .run();

━━━ Metric Queries (2) ━━━━━━━━━━━━━━━━━━━━━━━━━━
  write_throughput
    requests_total{operation="write"} → rate → 60s buckets (mean) → reduce (mean)  ·  scalar per run  ·  runs: 500
    min            5,421   seed=9182
    mean           5,973
    max            6,102   seed=224
    across runs:  p50 5,991  p95 6,071  p99 6,094

min and max name the seed that produced them, so the worst run goes straight into set_debug_seeds(vec![9182]).

Queries are declared before run() and their results travel with the report, in SimulationReport::metric_queries. Nothing reconstructs them afterwards from raw snapshots: the runner already knew what mattered.

The operations

The aggregators are Min, Mean, Max and Percentile(p) — nothing else.

Counter semantics are explicit. .rate() is the only place a series is read as a counter; bucketize(_, Mean) on a raw counter honestly reports the mean cumulative value, not a throughput. A drop in a counter’s value is read as a reset, Prometheus style. Because a fresh source is built per iteration, every counter is zero at simulated time zero, so the very first increment is rated too.

Empty buckets

bucketize omits a bucket nothing landed in, because a gap and a zero are different facts: a workload that stopped reporting is not a workload reporting zero. Two ops say which one this metric means, split the way Warp 10 splits them — .fill(policy) repeats a value the series already has, while .interpolate() computes one it never had:

Reach for a fill when the quantity holds its value between observations (or is simply zero when nothing happened), and .interpolate() when it moves continuously — a queue depth, a replication lag.

bucket                     0 |    1 |    2 |    3 |    4
no fill                   10 |    - |    - |   40 |    -
.fill(Fill::Previous)     10 |   10 |   10 |   40 |   40
.fill(Fill::Next)         10 |   40 |   40 |   40 |    -
.fill(Fill::Value(0.0))   10 |    0 |    0 |   40 |    0
.interpolate()            10 |   20 |   30 |   40 |    -

They compose, in call order: .interpolate().fill(Fill::Value(0.0)) draws the line through the interior gaps and zeroes the ends interpolation cannot reach.

Either way the grid runs from bucket zero to the bucket holding the run’s end, so a workload that went quiet halfway through gets buckets for the silence instead of the series just stopping.

That matters for more than tidiness. Without a fill, a seed’s bucket set depends on when that seed happened to be busy, so the across-run summary splits one window into several — each with a fraction of the runs in it, and min/max no longer comparing like with like:

# unfilled: two seeds busy in different minutes
[0s,60s)     runs: 1
[120s,180s)  runs: 1

# Fill::Value(0.0): one grid, every window sees every seed
[0s,60s)     runs: 2   min 0  max 5
[60s,120s)   runs: 2   min 0  max 0
[120s,180s)  runs: 2   min 0  max 5

Neither op collapses nor restores a distribution, so both leave the stage — and with it whether Percentile still applies — unchanged. Both read the recorded buckets only, never the ones they just synthesized, so a carried value is the last real observation and a run of interpolated gaps is one straight line rather than a curve that drifts. A series that recorded nothing at all stays empty: a policy alone must not conjure a metric the run never touched.

Series identity is name plus label set, canonically ordered, so two label sets that differ only in insertion order are the same series. Matching is by subset: a query naming operation="write" also matches a series carrying instance="10.0.1.1".

Moonpool’s own metadata stays out of your labels. Every result row carries seed (the replayable identity of one iteration) and run_id (the identity of the whole run() invocation) as moonpool-side fields, never injected into the application’s Prometheus labels.

Percentiles that mean something

mean(p99(node_a), p99(node_b)) is not a p99, and p99(p99(a), p99(b)) is not one either. Once observations are collapsed into a percentile, the percentile of that is a number without a meaning.

The builder enforces this at compile time. Each stage tracks what its values are, and Percentile only applies where the individual observations survive:

// Compiles: the raw histogram observations are still there.
MetricQuery::select("request_latency_seconds")
    .bucketize(Duration::from_secs(60), Percentile(0.99))
    .named("p99");

// Does NOT compile: p99 of per-bucket p99s.
MetricQuery::select("request_latency_seconds")
    .bucketize(Duration::from_secs(60), Percentile(0.99))
    .reduce(Percentile(0.99))
    .named("nonsense");

Min, Mean and Max apply anywhere — “the mean of each node’s p99” is a useful number as long as nobody calls it a p99, which is why the report prints each query’s provenance (observations, scalar or percentile-derived).

The percentiles in the summary block are a different dimension: they are taken across runs, where every seed contributes one equally-weighted value. p95 there names the 95th-percentile run, which is why the report labels it across runs.

Worked example: watching a metastable failure

Totals tell you what a run did; the recorded series tells you when it went wrong, which is the only way to see a failure that outlives its cause. crates/moonpool/examples/metastable_grpc_retry_storm.rs is the demonstration: a real gRPC service with a bounded worker pool, an open-loop client with an aggressive timeout and retries, and exactly one temporary slowdown in the middle of the run.

cargo run --release --example metastable_grpc_retry_storm \
    --features hyper,prometheus -- --seed 4

It reads back the series the run recorded, runs the query pipelines above over them — .rate().bucketize(1s, Mean).fill(Fill::Value(0.0)) for counters, .bucketize(1s, Mean).fill(Fill::Previous) for gauges — and prints the buckets as an ASCII graph on stdout, one column per simulated second, with the trigger window marked. No Grafana, no browser, no export step.

What the graph shows is a metastable failure. The slowdown lasts 650 ms. The service never recovers from it: retries provoked by that excursion keep the server saturated for the remaining 29 seconds, at exactly the offered load it carried comfortably before. The trigger is gone; the bad state is not.

--search 0..64 runs a range of seeds and prints one summary line each. The trigger sits on the tipping point, so some seeds absorb it and return to the healthy operating point while others fall into the storm and stay there — same code, same configuration, different seed. That is the clearest evidence available that the bad state is a second stable operating point rather than a broken setting, and it is exactly the kind of question the recorded series is there to answer.

Timing on the simulated clock

Use start_timer from this crate, not prometheus’ own:

let timer = latency.start_timer(ctx.time());
let response = client.request(req).await?;   // observed even on an early return
timer.stop_and_record();

prometheus::Histogram::start_timer() reads the wall clock. Inside a simulation that records how fast your laptop happened to be, not the latency the simulation modeled, and it gives a different answer on every replay of the same seed. SimHistogram::start_timer reads the provider clock instead.

Pick bucket bounds that bracket what your simulation produces. Prometheus’ defaults are tuned for real HTTP latency; simulated latencies often live on a different scale, and a distribution that lands entirely in the last bucket tells you nothing.

What metrics are not

Metric values are reported, never used to steer the simulation. Not everything in a registry is deterministic — a metric fed from the wall clock, or from a library’s own internal timing, is not — so the runner treats a scrape as an observation. If you want a property enforced, assert on it: metrics tell you the shape of a run, assertions tell you whether it was correct.

Under fork-based exploration, only root timelines are scraped. Explored timelines report back through the shared assertion table, which carries no metric values.

Beyond Prometheus

PrometheusSource is one implementation of MetricsSource, a registry-agnostic trait in moonpool-core:

pub trait MetricsSource: Send + Sync + 'static {
    fn collect(&self) -> Vec<MetricSample>;
    fn set_clock(&self, clock: Arc<dyn MetricClock>) {}
    fn series(&self) -> BTreeMap<String, Vec<MetricPoint>> { BTreeMap::new() }
    fn dropped_points(&self) -> u64 { 0 }
}

collect alone is enough for a scrape-only adapter over any registry. The other three are what an adapter implements to deliver an exact time series (dropped_points reports how many points a capped series discarded). An OpenTelemetry adapter slots in the same way, with no change to the simulation runner.