Skip to main content

lifecycle

The commands an operator uses between installing a server and retiring one: backing it up, restoring it, checking its media, and moving its store forward when the on-disk format changes. Part of #20.

Every one of these opens the store directly, and fjall holds an exclusive lock, so they run with the server stopped. For a restore that is not just a limitation: it is the only way to be sure nothing is writing behind you.

Sync positions have a durable checkpoint before classic sync, sliding sync, or dehydrated-device responses issue their tokens. The checkpoint survives mailbox drains and receipt-only activity. A restart must allocate future positions above tokens clients already hold, even when push is disabled.

The server keeps the state of each MSC4186 connection in memory, and writes a copy to the store every 15 seconds and at a graceful stop (#897). After a restart, a client continues its connection with its pos, and its room list does not reload. A connection stays for seven days without a request.

After seven days, the server answers M_UNKNOWN_POS. A crash can lose the positions of the last 15 seconds, and the server answers the same error for them. The client then starts its connection again, and the server sends one initial response for its visible window. To-device messages have their own acknowledgement token, which is not pos. Thus the client loses no messages.

The copies are in two new keyspaces, so an upgrade needs no migration. A server from before #897 does not read them. A row in a format that the server cannot read counts as an unknown pos, not as an error.

Backup​

spindle backup <config> <file>

Writes a consistent snapshot of the store. It refuses to overwrite an existing file — a backup that silently replaces the good copy with a bad one is discovered at restore time, which is the worst possible moment.

The archive carries rows, not media blobs. Media lives outside the store (on disk or in S3) and is copied separately. This is deliberate — a backup file that inlined a terabyte of media would be a file nobody takes — but it means a restore is only half done when the rows are in.

The command writes the rows from the snapshot to the file one at a time. Thus a backup needs memory for one row, not for the full store (#743).

Backup while the server runs​

Two admin endpoints take a backup while the server runs. Both need an admin token. Both read one store snapshot, so the backup shows one moment, whatever writes land while it runs. Both write the same format as spindle backup. spindle restore reads either.

POST /_spindle/admin/v1/backup writes a file to [storage] backup_dir and returns its path. Use it for a nightly job on the disk of the server. It sets the spindle_last_successful_backup_timestamp_seconds metric. The alert for a late backup covers it.

GET /_spindle/admin/v1/backup/stream sends the backup as the response body (application/octet-stream). Use it to pull the backup to another host with no file on the server. For example: curl -H "Authorization: Bearer $TOKEN" -o spindle.backup .../backup/stream.

It needs no backup_dir. It does not set the metric, because the server cannot know that the client kept the bytes. The body ends with a digest. A body that stops early is not a backup. spindle restore checks the digest before it writes anything, so it rejects a short file.

The stream uses bounded memory. A background task writes through a channel of four 64 KiB chunks. A slow client slows the backup and does not fill memory. If the client disconnects, the backup stops and frees its slot.

One backup runs at a time, file or stream. A second request gets 409. Both endpoints write an audit record when they succeed. Synapse has no equivalent, so the stream exists only under /_spindle.

Take the store backup first, then copy the media. The store names the blobs. A blob that arrives after the snapshot is in your media copy but not in the store backup, which is harmless. A copy taken before the snapshot could miss a blob that the store names, and spindle verify-media would report it as missing.

Incremental backups are not available. Each backup holds the whole store.

Restore​

spindle restore <config> <file>

Refuses a store that already holds rows. A restore into a populated store is a merge: rows the target holds and the backup does not survive it, so the result matches neither the backup nor the original, and it looks like a success. Restore into an empty store, or clear the old one first.

The restore reports which media the rows it wrote still need. A row-only restore leaves a server that is indistinguishable from a healthy one until somebody asks for a file, so the report is how you find out before your users do.

Verifying media​

spindle verify-media <config>

The same audit a restore prints, on its own. Blobs go missing without a restore in sight — a bucket lifecycle rule, a half-copied directory, a disk that came back smaller. Exits non-zero when anything is absent, and names the media IDs rather than only the content hashes: the hash is what the backend calls it, the ID is what you have to look for in your other copy.

Verifying the store​

spindle verify <config>

This command checks every room. It reads each record, the whole chain and every state root. Run it after a crash, a restore or a disk fault. It names each room that fails and exits non-zero. A read error is reported as an error, not as a missing node. The server must be stopped.

Truncating a room's history​

A truncation removes the log of a room below a position (#817). It removes the log entries, the event bodies, the stream rows, the state-delta rows, the relation rows and the old membership rows. After it, spindle gc can free the state nodes that only the removed entries used.

The hash chain needs the value before the first kept entry. So the server first writes a chain checkpoint for the room: the first position that is kept and the chain value just before it. A load, spindle verify and a restore start the chain from that checkpoint. A backup holds the checkpoint as an ordinary row.

The order of the writes keeps a crash safe. One atomic batch writes the checkpoint, the purge watermark and the entries that stay. Every delete comes after it, and the log rows go last. A crash leaves a room that already reads as truncated. The next truncation of the room finishes the deletes.

The server lowers the position to what is safe. It never passes the head or a forward extremity. It never passes a federation gap or the anchor of a filled gap. It refuses a room in hub mode, a room in a partial-state join and a room with several forward extremities. It keeps the entry of each event in the current state and of each event that a kept entry names as a parent. It keeps the body of each event in the auth chain of a kept event.

A server from before this change cannot load a truncated room. Its chain check fails at the first kept entry. Do not run an older binary on a store that has a truncated room.

Retention policy​

The server can truncate history on a schedule (#817). It is off by default, because it deletes history. Turn it on in [retention]:

[retention]
enabled = true
max_lifetime_days = 365
min_lifetime_days = 7
interval_minutes = 60

Each pass reads every room and finds the newest event older than the room's lifetime. It truncates the room below the entry after that event. The floor rules above still apply, so a pass can truncate less than the lifetime says.

A room sets its own maximum with an m.room.retention state event (MSC1763). Its max_lifetime and min_lifetime are in milliseconds. A room without the event takes max_lifetime_days. A room can shorten its lifetime and cannot pass max_lifetime_days. The lifetime is never below min_lifetime_days or the room's own min_lifetime. Without a server maximum, a room that sets none keeps everything.

A pass logs its totals and moves the spindle_retention_* metrics. A room that fails is logged and counted, and the pass goes on. A pass opens each room, so it reads every room into memory once.

An operator can also truncate one room at once. Send "truncate": true in the body of purge_history. Without it, the call keeps the spine as before. The answer has a truncated object that names the floor that took effect and the reason, if the server refused.

Rebuilding the search index​

spindle reindex-search <config>

This command rebuilds the word index for /search from the logs. Use it after you set [search] index = true on a store that holds rooms. Use it again after you turned the index off for a while. Stop the server first.

You can always rebuild the index, because the log is the truth. The command deletes the old index rows, reads each stored message body and writes the rows again. It prints the number of rooms, log entries and rows. A room without an index is still searched correctly. The search scans it.

A truncation removes the index rows of the entries it deletes.

Collecting unreachable state​

spindle gc <config> [--dry-run] [--max-marks N]

This command deletes the state-trie nodes that no stored record reaches (#742). The state of a room is a trie of content-addressed nodes. One node can occur in the state of many entries and rooms. To find a dead node, the command must examine all records. Before this command, nothing deleted nodes, and unused nodes stayed on disk. Examples are the intermediate states of a backfilled gap, the rows of an interrupted import, and the trie of a deleted room.

The command marks each node that a stored root can reach. Log entries, sidelined events and backfilled gap events hold these roots. The command reads a shared node only once. Then it deletes all other nodes.

The command prints the number of rows that hold roots, the reachable nodes and their bytes, and the deleted nodes and their bytes. --dry-run prints the same report and deletes nothing. A second run deletes nothing.

It does not guess. It stops before any delete if a reachable node is missing or has the wrong hash. It also stops if it cannot read the root of a record. A sweep from an incomplete mark deletes live state. The one exception is a current-state cache row that does not decode. A room load already ignores such a row, so the command deletes it.

The sweep deletes in batches of 4096 nodes. It does not hold all dead nodes in memory.

The set of reachable nodes has a memory bound. --max-marks N sets how many addresses stay in memory, 4194304 by default, about 200 MB. When that many are held, the command moves them to a scratch database in a directory next to the store, named like the storage path with .gc-scratch added. The command removes the directory when it ends, and clears one that a crashed run left before it starts. Keep free disk for it: it holds one 32-byte address for each reachable node that did not fit in memory. The answer is exact for every N. A Bloom filter would not be: one false positive keeps a live node out of the mark and the sweep then deletes it. The report shows the peak number of addresses held in memory and how many went to disk.

Intern rows stay. The command does not delete StateKeyIntern rows. The server loads the whole table at start, and a room that sends many unique state keys makes it grow. The rows cannot be removed one by one. The table must stay dense, because the store refuses to open onto an id sequence with a gap. The ids sit inside the content of state nodes, so renumbering would rewrite every node and change its address. That is a migration, not a sweep. #818 stays open for it.

Offline only, and not available as a background task. The key of a node is its content. When a server makes a node again, it writes the same key that the sweep can delete. The two writes look the same.

Thus a sweep that runs during an append can delete a node that the append needs. A snapshot does not help, because the snapshot already holds that row. Make a backup first if you want a way back.

Importing password hashes​

spindle set-password-hash <config> <localpart> < hash
spindle set-password-hash <config> < hashes

Stores password hashes that another system made (#611): Argon2 in PHC form, or bcrypt. This is usually the user_passwords.hashed_password of a Matrix Authentication Service. Those users then sign in to Spindle with the passwords they already have. The hashes come from stdin, never the command line, so they do not show in a process listing or shell history.

With a localpart, stdin is the hash of that one account. Without one, each line is <localpart> <hash> (@user:server works too, and the command does not read blank lines and # comments).

The command validates each hash before it writes it. An Argon2 hash must be argon2id, argon2i or argon2d, version 16 or 19, with exactly the m/t/p parameters. m is 256 MiB at most, and t and p are 16 at most. A bcrypt hash must be $2a$, $2b$ or $2y$, with a cost of 14 at most. The command refuses all other hashes (pbkdf2, a keyed keyid=/data= hash). If this server cannot verify a stored hash, nobody can enter that account.

A hash made with a pepper that the string does not record (the per-scheme secret of MAS) looks ordinary. This command accepts it, but it never verifies. For MAS, use spindle import-mas instead: it reads the schemes from the MAS configuration and stores each secret with its hash (delegated-auth.md).

The accounts must already exist (the Synapse importer makes them). The command names a refused line by line number and localpart, never with its hash. If it refused a line, it exits non-zero. Like each command here, it works offline. For a live server, use POST /_spindle/admin/v1/users/{user_id}/password_hash with {"password_hash": "...", "logout_devices": false}. This also works while authentication is still delegated, so hashes can land before a cutover.

spindle issue-reset-link <config> <localpart> [--ttl 24h]

This is for a user who cannot sign in, on a server that sends no mail (or for any user). It prints a link to the choose a new password page of the built-in provider, on the configured issuer. The link works one time. Give it by a channel you trust. The person who opens it can set the password of the account, and that signs each device out.

--ttl takes 30m, 24h, 2d or seconds: a day by default, a week at most. Only the newest link of an account works. A new link (or a mailed one that the user asks for) retires this one. The server keeps only a digest of the token.

The command needs auth.builtin_oidc and works offline, like the other commands here. For a live server, use POST /_spindle/admin/v1/users/{user_id}/reset_link (optional body {"ttl": "2h"}). It records the link in the audit log without the token. Users can also recover on their own with the recovery codes of the account pages. See delegated-auth.md.

A full disk​

When the disk fills, the store refuses the write that does not fit. The client gets a 500 for that request. From then on the store refuses every write until the process restarts, because fjall locks its database after a failed journal write. Nothing the server acknowledged is lost.

What the probes say:

  • /ready answers 503 from the first refused write. A load balancer then sends no clients to a server that cannot write.
  • /health stays at 200. The process is good and the disk is not. A restart onto the same full disk only adds a crash loop.
  • Reads continue. /sync and /messages serve the history up to the last acknowledged event.

To recover, make space on the disk, then restart the server. Make a real amount of space, not a few megabytes: at start, the store must flush what it recovered before it accepts a write. In the drill, a 64 MiB disk with 2 MiB free did not recover, and with 16 MiB free it did.

just drill-disk-full runs the drill. It mounts a 64 MiB tmpfs with sudo, then fills it under the store and under a server. CI runs it on each pull request. The two tests are crates/spindle-store/tests/disk_full.rs and crates/spindle-server/tests/disk_full.rs.

Migration​

spindle migrate <config> [--dry-run] [--progress text|json]
spindle migrate --list [<config>]

Moves a store forward to the schema the current binary speaks.

The server never migrates on its own. A binary that rewrites the store the moment it starts is a change you cannot back out of: by the time you know it happened, the old bytes are gone. So a store at another schema makes the server refuse to start, naming this command, and the rewrite waits until you ask for it — with the chance to take a backup first.

--dry-run prints the plan and writes nothing. Use it. It is where you find out whether the plan contains a step you cannot undo.

--list prints each step of this binary: its id, the schema it starts from, the schema it goes to, and if you can undo it. Without a config, the list tells you which schemas this release can move a store from. With a config, it also gives the status of each step on that store. The status is applied (with the time and the number of rows), pending, pending, interrupted (with the position), or not needed.

Take a backup first​

Not a formality. migrate is the one lifecycle command that rewrites data in place, and a backup taken beforehand is the only route back from a step marked irreversible.

Rollback limits​

Reversibility is declared per step and reported before anything runs.

step ismeans
reversiblethe older binary can still read the store afterwards. No step is reversible today, and an older binary refuses a store with a newer marker.
irreversibleit cannot. Going back means restoring the backup taken beforehand, and losing everything written since.

A plan containing one irreversible step is irreversible as a whole, and migrate says so on its first line, before the steps.

Two limits worth stating plainly, because neither is obvious:

  • There is no migrate --down. A reversible step is reversible because the older binary can still read what the newer one wrote, not because Spindle can walk the step backwards. Nothing undoes a migration in place.
  • A binary older than its store cannot be helped. migrate reports no path and touches nothing. The steps that would be needed have not been written, because the version they lead to did not exist when that binary was built. Upgrade the binary, or restore a backup from before the migration.

Steps​

Each step has an id. The id does not change, and no other step gets it. When a step is complete, migrate writes a ledger row with the id in the store. The same commit moves the schema marker to the schema that the step goes to. Backups copy the ledger with the other rows.

idfromtorollbackwhat it does
intern-state-keys1/1/21/1/3irreversiblerewrites the state tries of version 2 with interned key ids (#79)

The schemas are key schema/record schema/content digest, as in Format versions. A new step must have a row in this table. crates/spindle-store/tests/schema_migration.rs fails if a step id is not here.

Progress​

A step tells you its phase and its count while it runs, on stderr (#656). Each line has the step id and the phase, and counts in a stated unit: records, rooms or nodes.

  • When a phase knows its total, the line shows done/total and a percentage. After one second, it also shows a rate and an ETA.
  • When a phase does not know its total, the line shows the count and "total unknown". The count changes while the work continues. There is no percentage and no ETA.

Each phase writes a line when it starts and when it ends. Between these, it writes one line each 5 seconds at most, so a long step does not fill the log.

--progress json writes each line as one JSON object, for scripts and Kubernetes jobs. The event field is started, progress, applied or error. A progress object has step, phase, stage (started/running/finished), unit, done, total, percent, resumed, elapsed_seconds, rate_per_second and eta_seconds. A field has the value null when the step cannot calculate it. The report on stdout does not change.

The intern-state-keys step has three phases:

phaseunittotal
countrecordsunknown: this phase finds the total
convertrecordsknown
delete-v2-nodesnodesknown

What a failure leaves behind​

The schema marker moves only at the end of a step, after the step is complete. It never moves during a step.

If the marker of a store claims a schema that its data does not have, no check refuses the store. Then each read can be wrong. The marker exists to stop this silent failure, and migrate must not cause it. Thus a failed step leaves the marker at the schema that the step starts from. The store continues to name the version that a binary can read.

You can repair the cause and run migrate again, without a restore from the backup. Steps that are complete stay complete. The next run starts at the step that failed.

A long step also writes a progress record in the store when a part of its work is durable. The intern-state-keys step writes one after each room. If the step stops (an error, a kill, a power failure), the next migrate reads the record and continues after the last room in it. --dry-run and --list show this position. migrate locks the store while it runs. Thus, during the run, stderr is where you see the progress.

A step continues from a record only if the record has its id. If the record has a different id, migrate deletes it, and the step starts again from the start. Each step must give the same result when it runs again from the start or from its record.

Format versions​

"The storage format" is four formats stacked, each with its own number, and a bug report or a migration step has to say which one it means:

layernamed bynowwhat a bump means
key schemaKEY_SCHEMA_VERSION in spindle-core1the byte layout of keys: a keyspace discriminant reused or reordered, or a key's fields moved. Adding a keyspace is not a bump: every existing key keeps its bytes and its meaning.
record schemaRECORD_VERSION in spindle-store1the encoding of the values under those keys (log entries and bodies), which carry their version as their first byte.
content digestCONTENT_DIGEST_VERSION in spindle-core3how content addresses are derived: a domain tag, a length width or a field order in the hashing. Keys and records are untouched by such a change, which is why it has its own number (#78): a store under the old derivation would open and misread every node address.
engine formatthe major version of fjall in Cargo.toml3the on-disk layout of the store itself: journal, segments, the engine's own version file. Spindle does not choose it and cannot rewrite it.

The first three are what the schema marker records and what migrate moves between; all three are pinned by crates/spindle-store/tests/format_compatibility.rs, whose fixtures are bytes written under the versions they name, so a bump fails a test until fixtures for the new version and a decision about data written under the old one exist. The fourth is what the marker cannot describe, because the marker is stored inside it.

The transitions so far, numbered so they can be referred to:

#layerfromtowherehow a store crosses it
1engine formatfjall 2fjall 3#193it does not. fjall 3 refuses a v2 directory with InvalidVersion(Some(V2)) and has no in-process upgrade. Restore from a backup taken under fjall 3, or start a fresh store.
2content digest12#330it does not, yet. The store is refused as content_digest 1 (a store from before the marker existed counts as 1 if it holds any row), and no migration step re-derives the addresses. Start a fresh store.
3content digest23#79spindle migrate runs step intern-state-keys: it rewrites state tries with interned keys and persists their intern rows. Irreversible; restore the pre-migration backup to return to version 2.

The first two were accepted deliberately, because no Spindle had data to lose. They are kept on the books rather than forgotten because each is an example of what the machinery below does not do: transition 1 is the case it cannot help with, and crates/spindle-store/tests/backend_compatibility.rs holds fixtures written under fjall 2 to prove the refusal is clean and explains itself; transition 2 is a step the table could hold and does not, because nobody has needed it. Key and record schemas have never moved.

The current state of the table​

The first data rewrite is the version-2 to version-3 content-digest step (#79), intern-state-keys. It converts roots in the linear log, sidelined records and gap history, and writes the intern rows with their new nodes. It deletes the old nodes after all rooms have converted, because rooms can share a content address. Take a backup before this irreversible step.

A partial run keeps the old marker. The next run continues after the last converted room and finishes the remaining work. The old nodes of the rooms that the next run does not convert again stay on disk without a reference. spindle gc deletes them.

The migration machinery is covered by crates/spindle-store/tests/schema_migration.rs; the actual rewrite and reopening are covered by tests/state_migration.rs in the same crate. The intern table is part of the backup: losing its rows loses the meaning of the ids in state nodes. Version 1 still has no migration path.