Prometheus Metrics

SwarmLLM exposes a Prometheus-compatible metrics endpoint at GET /metrics. It answers without a key only to a request made on the node's own machine, and not even then when api.metrics_auth_required = true; from anywhere else send Authorization: Bearer <api key> (see Monitoring).

Available Metrics

Core Metrics

route is one of local, split, distributed, relayed, cloud; outcome is ok, error, cancelled or pending. Both are closed sets, so this metric is 20 series regardless of how large the swarm grows. Per-peer, per-model and per-shard breakdowns are deliberately not exported here — that label set grows with the swarm and would eventually break the scrape. Fetch them from GET /api/admin/performance instead, which is served on request and retains nothing.

OpenTelemetry GenAI Metrics

Named to the OpenTelemetry GenAI semantic conventions so an OTel collector and the community Grafana dashboards work without a translation layer.

These are the two figures that separate a backed-up queue from slow generation; end-to-end latency alone cannot. swarmllm_inference_latency_seconds is the same measurement as the conventions' gen_ai.server.request.duration under a local name, and both are exported while dashboards migrate.

Only requests that emitted an incremental token contribute to these histograms. A non-streaming path never stamps a first token, so there is no honest way to split decode out of its total and it is omitted rather than counted as zero.

Serving-Side Metrics

Work this node performed for other peers. Every metric above measures requests this node made; these measure what it gave.

rate(swarmllm_segment_serve_seconds_total[5m]) / rate(swarmllm_layers_served_total[5m]) gives seconds per layer served — the figure other peers' schedulers actually rank this node on.

Channel Metrics

Internal channel health metrics for monitoring backpressure:

Emitted for the channels whose send paths actually count: network_out, rebalance and pool_cmd. A channel absent from the output is one nothing increments, which is deliberate — a perpetual zero cannot be told apart from a healthy channel, and an alert on it can never fire.

swarmllm_channel_dropped_total rising is worth an alert, but the log is where the severity is: a drop line carries nothing_accepted_for_secs, and that is what separates a momentary burst from a consumer that has stopped. Anything above a few seconds on network_out means this node is not receiving from the swarm at all (docs/FUTURE_WORK.md #90). The line is rate-limited to one per 30 s per channel and reports how many it swallowed.

Histogram Buckets

swarmllm_inference_latency_seconds uses these bucket boundaries (in seconds): 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, +Inf

The two gen_ai_server_* histograms use the boundaries published in the OpenTelemetry GenAI conventions rather than locally chosen ones, so buckets line up with every other GenAI server a collector scrapes: 0.001, 0.005, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, +Inf

For every histogram, _count and _sum come from monotonic counters rather than the in-memory sample ring. The ring is both size- and age-bounded, so its length falls when it wraps, which would break rate() and increase().

Scraping Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: "swarmllm"
    static_configs:
      - targets: ["localhost:8800"]

Example Queries

# Request rate (requests per second over 5 minutes)
rate(swarmllm_inference_requests_total[5m])

# P50 latency
histogram_quantile(0.50, rate(swarmllm_inference_latency_seconds_bucket[5m]))

# P99 latency
histogram_quantile(0.99, rate(swarmllm_inference_latency_seconds_bucket[5m]))

# Average latency
rate(swarmllm_inference_latency_seconds_sum[5m]) / rate(swarmllm_inference_latency_seconds_count[5m])

# P95 time to first token — is the queue backed up, or is generation slow?
histogram_quantile(0.95, rate(gen_ai_server_time_to_first_token_seconds_bucket[5m]))

# P95 per-token decode cost
histogram_quantile(0.95, rate(gen_ai_server_time_per_output_token_seconds_bucket[5m]))

# Share of requests that left this machine
sum(rate(swarmllm_inference_requests_by_route_total{route=~"distributed|relayed"}[5m]))
  / sum(rate(swarmllm_inference_requests_by_route_total[5m]))

# Error rate by route
sum by (route) (rate(swarmllm_inference_requests_by_route_total{outcome="error"}[5m]))

# Seconds of compute contributed per layer served
rate(swarmllm_segment_serve_seconds_total[5m]) / rate(swarmllm_layers_served_total[5m])

Health Check

GET /health/ready

Readiness probe returning subsystem status. Returns 200 when ready, 503 otherwise. No auth required.

{
  "ready": true,
  "subsystems": {
    "network": true,
    "inference_router": true,
    "api_server": true,
    ...
  }
}