Skip to main content

metrics

Spindle exposes Prometheus metrics on a separate listener, off unless you configure it:

[metrics]
# Loopback on purpose. The exposition names the peers this server talks to
# and the volume it carries; it is an operator's surface, not a public one.
# Put a reverse proxy or your scraper's network in front of it deliberately.
bind = "127.0.0.1:9090"

Then GET http://127.0.0.1:9090/metrics. Absent bind, there is no listener and no port — the same opt-in shape as [auth.delegated] and [auth] builtin_oidc.

What is exported today​

Each metric below is here because SPEC §17.2 names it and because a test drives the operation and asserts the counter moved. A metric that cannot be shown to move does not ship: a gauge stuck at 0 is indistinguishable on a dashboard from a healthy system, and the difference surfaces during the incident it was supposed to explain.

MetricTypeLabelsWhat it tells you
spindle_build_infogaugeversionWhich build the dashboard is watching. Always 1.
spindle_events_appended_totalcounterorigin = local|federatedEvents that reached a room log. The denominator below.
spindle_fork_resolutions_totalcountercase = 1|2|3Which of SPEC §9.2's three cases carried each append.
spindle_append_duration_secondshistogramdurabilityTime to commit one event, timed around the commit itself.
spindle_http_requests_totalcounterroute, method, statusRequests served.
spindle_http_request_duration_secondshistogramrouteTime to serve one request, in the wide buckets (0.1 ms to 120 s). A sync that was allowed to long-poll -- classic /sync or sliding sync with a since/pos and a non-zero timeout -- is timed under its route with (long-poll) appended (for example route="/_matrix/client/v3/sync (long-poll)"), so its intentional wait does not become the route's p95. An initial sync, or one with timeout=0, stays under the plain template. spindle_http_requests_total keeps the plain template for both.
spindle_federation_queue_depthgaugedestinationEvents waiting to go out, per peer.
spindle_federation_destinations_backing_offgaugebackoff = short|longDestinations whose last delivery failed. short ones are retried at least once a minute. long ones have failed for ten minutes or more: each waits as long as it has been down, up to a day, or until it sends this server a request. Many long destinations is normal on a server in large public rooms; it is the dead servers in them.
spindle_federation_catch_up_dropped_totalcounter—Queued events that a destination down for ten minutes or more will not get: its queue is cut to the newest ten events of each room, and the peer fetches older ones if it wants them. To-device messages and device-list updates are never dropped.
spindle_federation_outbox_capped_totalcounter—Queued events dropped because the queue of a destination that has been down a long time held 5,000 events. The oldest go first, and the peer fetches what it missed when it returns. A peer with peers.max_backoff_ms is never capped.
spindle_federation_outbox_body_missing_totalcounter—Queued events dropped because an admin purge had deleted their stored body. The outbox keeps one body for each event, in the room, and each destination's row refers to it.
spindle_federation_edus_received_totalcounteredu_type = m.typing|m.receipt|m.presence|m.direct_to_device|m.device_list_update|m.signing_key_update|other, result = accepted|ignored|malformed|unsupportedEDUs peers sent in transactions. ignored is well-formed but nothing applied (a user not on the origin, not in the room, an origin the room's server ACL denies, an unknown event). unsupported is a type this server does not act on: unknown types, and m.presence when [presence] enabled = false, so its volume shows up here before anyone switches it on. For to-device, device-list and signing-key EDUs accepted means handed to the handler, which runs its own checks.
spindle_federation_edus_sent_totalcounteredu_type (as above)EDUs put into outbound transactions. Read receipts are coalesced per destination, so one m.receipt carries every waiting reader.
spindle_federation_receipts_received_totalcounterresult = accepted|foreign_user|not_joined|denied|unknown_event|unsupported_type|malformed|over_limitRead receipts inside inbound m.receipt EDUs. Only m.read federates; unsupported_type counts anything else, m.read.private included. denied is an origin the room's server ACL denies. over_limit is past the 500-per-EDU bound.
spindle_sync_subscribersgauge—Clients currently blocked in a long-polling /sync.
spindle_sync_lag_secondshistogram—Age of the newest event a /sync delivered.
spindle_room_registry_acquisitions_totalcountermode = exclusive|sharedAcquisitions of the registry that hands rooms out.
spindle_retention_runs_totalcounterPasses of the history retention job (#817). Zero while [retention] is off.
spindle_retention_rooms_totalcounterresult = truncated|refused|failedRooms the job handled. refused means a hub, partial-state or forked room.
spindle_retention_deleted_totalcounterkind = entries|bodiesLog rows and event bodies the job deleted.
spindle_room_lock_acquisitions_totalcountermode = exclusive|sharedAcquisitions of a room's own lock: exclusive is the write path.
spindle_last_successful_backup_timestamp_secondsgauge—Unix time of the last backup that POST /_spindle/admin/v1/backup wrote. 0 until one succeeds after a start.
spindle_federation_gaps_remaininggauge—Recorded federation gaps whose history is not backfilled yet (SPEC §6.6). Should fall to 0 within minutes of a gap acceptance.
spindle_federation_recovery_attempts_totalcounterresult = recovered|budget_exceeded|peer_error|rate_limited|timeout|invalid|negative_cachedTries to get the missing dependencies of a received event from one peer. negative_cached is an event that this server refused at once, with no request to a peer: it names a dependency that failed verification in the last [federation] dependency_failure_ttl_secs. A high rate tells you that peers send many events that depend on one event that this server cannot verify.
spindle_federation_recovery_admissions_totalcounterresult = started|joined|busyReceived events that need a dependency recovery, by what this server did. started starts a recovery. joined is the same event again, for example in a transaction that the peer sent again: it waits for the recovery that runs. busy is an event that this server refused at once, because its room already has two recoveries. /send waits 20 s at most for a recovery. After that, the recovery continues and the event gets an error that says that it is pending. spindle_federation_pdus_received_total{result="recovery_pending"} counts these events.
spindle_federation_owed_eventsgauge—Pushed events this server refused for missing dependencies and has not applied yet. The backfill loop retries each one. A value that stays above 0 for hours means a room may be diverged from its peers.
spindle_federation_backfill_chunks_totalcounterresult = filled|completed|peer_error|rate_limited|invalid|truncatedGap backfill chunks. invalid is a peer serving history that failed verification; nothing of that chunk was stored.
spindle_federation_backfill_events_totalcounterresult = fetched|inserted|rejected|skippedEvents a gap backfill fetched, stored, kept out of the timeline for failing its auth checks, or stepped over because no participating server would serve them (a peer leaves out events it rejected).
spindle_federation_backfill_duration_secondshistogramresultTime one backfill chunk took: the page, its auth events, its /state_ids and the write.
spindle_federation_key_fetches_totalcountersource = direct|notary|cache, result = ok|error|invalid|throttled (direct, notary), hit|miss (cache)Lookups of a peer's signing keys. notary ok climbing means peers this server cannot reach are being verified through [federation] trusted_key_servers; invalid is a document that failed its signature checks and was refused.
spindle_federation_signature_failures_totalcounterreason = no_key|expired_key|bad_signature|missing_signature|malformedReceived events refused for their signatures. bad_signature means a key under the right ID did not verify the bytes judged; each one logs the redacted form it checked.
spindle_federation_listed_refusals_totalcounterdirection = inbound|outboundRequests refused because [federation] allow or deny does not permit the server. inbound is a signed request from a listed-out server; outbound is a request this server did not make to one.
spindle_auth_logins_totalcountermethod = password|oidc|oidc_session|account|upstream, result = success|bad_password|rate_limited|errorSign-in attempts: the client API's password login, the built-in provider's authorization page (with a password, or continuing a browser session), and the account pages. An unknown user counts as bad_password, as everywhere else. rate_limited also counts a login that found the password-hash queue full (#571).
spindle_oauth_token_grants_totalcountergrant = authorization_code|refresh_token, result = success|invalid_grant|invalid_request|errorBuilt-in provider token-endpoint grants. A climbing invalid_grant on refresh_token is clients holding rotated-away tokens.
spindle_password_resets_totalcounterstage = requested|completedForgot-password requests (counted per submission, whether or not the address is known) and resets completed with a valid emailed link.
spindle_emails_sent_totalcounterkind = password_reset|verification, result = sent|failedMail handed to the SMTP transport. failed is the relay refusing or unreachable.
spindle_account_actions_totalcounteraction = profile|password_change|session_end|deactivate|email_add|email_verify|email_remove|cross_signing_reset|recovery_codesChanges made through the account-management pages (recovery_codes: a fresh set generated).
spindle_password_recoveries_totalcountermethod = reset_link|recovery_code, result = success|rejected|rate_limitedForgotten-password recoveries: a reset link (mailed or administrator-issued) or a recovery code. rejected is a dead link or a code that did not match.
spindle_rooms_residentgauge—Rooms held open in memory. Each one keeps its whole log index until it is evicted, so this is the number that bounds resident memory.
spindle_rooms_evicted_totalcounter—Rooms dropped from memory because more than [storage] max_resident_rooms were open. An evicted room loads from the store the next time anything asks for it. A fast climb means the budget is below the working set and rooms reload often (see spindle_room_cold_loads_total).
spindle_reset_links_issued_totalcounter—Reset links issued through the admin API (the offline CLI cannot count into a running server).

Alert rules, a scrape target and a dashboard​

deploy/ carries the pieces an operator would otherwise write from this page (#325):

  • deploy/prometheus/spindle-alerts.yaml: the case-3 alert above, the SPEC §18.3 latency targets as p50 and p99 alerts against group durability, a federation backlog that is not draining, sync lag, server errors, and the listener going away. Thresholds from the SPEC where it states one; the rest are starting points.
  • deploy/kubernetes/servicemonitor.yaml: a headless Service naming the metrics port, and a ServiceMonitor and a PodMonitor for the Prometheus Operator, either of which sets the job="spindle" label the rules select on.
  • deploy/grafana/spindle.json: one dashboard, the targets first: append p50/p99 by durability, the case-3 ratio, HTTP rate and latency by route, the deepest federation destinations, sync subscribers and lag.

scripts/check-observability-pack.py runs in CI and refuses a rule or a panel that names a metric this file or metrics.rs does not have, so a renamed metric cannot leave a rule that never fires.

The one that matters​

spindle_fork_resolutions_total is not a throughput metric. It is the instrument the architecture is falsified by.

Spindle counts every append in one of three cases, cheapest first (SPEC §9.2):

  • case 1 — a non-state event whose parents hold one state. No state resolution.
  • case 2 — a state event whose parents hold one state. One apply(); no state resolution.
  • case 3 — an event whose parents' states differ. The expensive path: the room version's state resolution (ADR 0005).

SPEC §18.3 then states the target that makes the design testable rather than merely asserted:

Case-3 fork resolutions as a fraction of federated events: < 0.1%

So the query that matters is the ratio, not the raw count:

rate(spindle_fork_resolutions_total{case="3"}[1h])
/ ignoring(case) rate(spindle_events_appended_total{origin="federated"}[1h])

If that stays under 0.001, "no state resolution on the hot path" is holding for your traffic. If it climbs, the claim is not holding for your deployment, and that is a finding worth reporting upstream — the whole point of publishing a falsifiable target is that someone can falsify it.

An alert worth having:

- alert: SpindleForkCase3Elevated
expr: >
rate(spindle_fork_resolutions_total{case="3"}[1h])
/ ignoring(case) rate(spindle_events_appended_total{origin="federated"}[1h])
> 0.001
for: 6h
annotations:
summary: Case-3 state resolutions above the SPEC §18.3 target

The six-hour window is deliberate: this is a design-health signal, not a pager. A brief spike during a federation catch-up is expected.

One caveat, stated plainly​

Case 3 counts appends whose parents' states differ. The room version's algorithm resolves each of them (ADR 0005). Spindle does not defer a fork, and a fork does not stop a local send. The counter records a fork once, at the event that names its tips, local or received. Later events build on that event. Spindle also resolves the room's current state over several extremities, but that is not an append, so this counter ignores it.

Checking the latency targets​

SPEC §18.3 states local send at p50 < 2 ms, p99 < 10 ms against group durability, which is what spindle_append_duration_seconds measures — the commit, not the whole request, because that is what the target describes. Buckets are weighted to straddle those numbers (0.5 ms, 1 ms, 2 ms, 5 ms, 10 ms, …) rather than using the default set most libraries ship, which starts at 5 ms and would put every one of these appends in the first bucket and answer nothing.

histogram_quantile(0.99,
rate(spindle_append_duration_seconds_bucket{durability="group"}[5m]))

The HTTP histogram answers the same question one layer out, per route. route is the router's matched path — /_matrix/client/v3/rooms/{room_id}/send/{event_type}/{txn_id}, never the path that was requested. That is what keeps the label set bounded by the code (101 routes today) rather than by the room and user IDs a caller happens to use; a test asserts that a room ID appears nowhere in the exposition. Requests that match no route are counted under a single unmatched label, so a scanner walking random URLs is not an unbounded source of series.

Federation backlog and sync lag​

These are two of the four things #19's exit criteria say dashboards must cover (the other two being durability — the append histogram's durability label — and unexpected fork paths, above).

spindle_federation_queue_depth is set from the delivery loop's own view of the outbox, so it cannot disagree with what is actually being delivered. The twenty deepest destinations get their own series and the rest are summed into other: a room full of fabricated server names must not be able to mint a series each and turn the scrape into the attack. The tail is added up, never dropped.

The gauge counts PDUs only. The EDUs that a peer must get (to-device messages, device-list updates and key updates) wait in a different outbox. They go in the same transactions. Thus a peer that waits for only those EDUs shows 0 here. The admin API federation/destinations shows its failures and its backoff.

A server that stays down does not keep its queue. After ten minutes of failures, its wait grows with the outage, up to a day. The outbox also cuts its queue to the newest ten events of each room (spindle_federation_catch_up_dropped_total). A request from that server ends the wait at once. Thus a queue depth that stays high for one destination is a live peer that is slow, not a dead one. spindle_federation_destinations_backing_off{backoff="long"} counts the dead ones.

topk(5, spindle_federation_queue_depth)

spindle_sync_lag_seconds needs its definition stated, because "watermark lag" can mean several things. Here it is the age of the newest event a /sync actually delivered, measured at delivery. A client keeping up sees milliseconds; a server falling behind sees this climb. Syncs that deliver nothing are not counted — an empty sync is a client that is up to date, not a lagging one, and scoring it zero would flatten the average that matters.

What is not here yet​

Per #166: OpenTelemetry traces (slice 4).

State-trie cache hit rate, which SPEC §17.2 also names, is absent for a reason worth writing down: there is no state-node cache to instrument. Nodes are read from the store through the ordinary read path. Exporting a hit rate would mean inventing one, and a metric that reports on something that does not exist is worse than a missing metric — the same argument the rest of this page runs on. It arrives when the cache does.

Per-room series are deliberately absent and will stay that way. A server with 10,000 rooms would mint tens of thousands of mostly-idle series, and the scrape cost would grow with the room count rather than with traffic. Per-room questions are answered by the admin API's room endpoints, which already have an authorization model; /metrics answers "how is the server doing", with label sets bounded by config.