matrix rtc
What a call needs from a deployment, in the order a client asks for it, and the two ways to provide the piece this server does not carry: the media. Both are supported; ADR 0004 records why both exist.
The pieces
A MatrixRTC call (Element Call, Element X, Element Web) touches four things:
- The homeserver — room state for the call membership, delayed events (MSC4140) to expire it, sticky events (MSC4354) so a membership can lapse without becoming permanent state, to-device signalling, and transport discovery (MSC4143). All served here; docs/dashboard.md's M7 row is the inventory.
- A LiveKit SFU — carries the media. Not bundled (#4 lists media servers under what not to build); run livekit-server.
- A JWT service — mints the token a client presents to the SFU.
Either the built-in one (
[rtc.livekit]) or lk-jwt-service deployed beside this server. - A TURN relay, optionally, for the client-side leg that cannot
reach the SFU directly.
[turn]inspindle.example.toml; not MatrixRTC-specific.
The flow a client drives, once it is in a room and has decided to call:
client ──GET /_matrix/client/v1/rtc/transports──▶ homeserver
◀── [{type: livekit, livekit_service_url: S}] ──
client ──POST /user/{me}/openid/request_token──▶ homeserver
◀── {access_token, matrix_server_name, expires_in} ──
client ──POST S/sfu/get {room, openid_token, device_id}──▶ JWT service
│
(external service; built-in for other ├─GET /_matrix/federation/v1/openid/userinfo─▶ token's homeserver
servers' tokens) │◀────────────── {sub: @me:server} ───────────
◀────────────────── {url: wss://sfu, jwt} ─────┘
client ──websocket, jwt──▶ SFU
The OpenID token is the credential in that exchange. It is short-lived (an hour), it opens nothing on this server, and the JWT service — built in or external — is what it is for.
In a call across servers, each participant publishes on the SFU of their own server and subscribes on the SFUs of the others. So a user of another server also asks this server's JWT service for a token, with an OpenID token from their own server.
Option A: the built-in JWT service
One binary, one secret to keep in step with the SFU.
[server]
name = "example.org"
public_base_url = "https://matrix.example.org"
[rtc.livekit]
url = "wss://livekit.example.org"
key = "APIxxxxxxxx"
secret = "..." # the SFU's matching API secret
# secret_file = "/run/secrets/livekit-secret" # instead of secret
# token_ttl_seconds = 900
# full_access_servers = []
key and secret are the pair the SFU was started with (--keys APIxxxxxxxx: ..., or keys: in livekit.yaml). With the section set,
this server:
- advertises itself as a
livekittransport, first in the list, athttps://matrix.example.org/_spindle/rtc/livekit— on/rtc/transportsand in.well-known/matrix/clientalike; - answers
POST /_spindle/rtc/livekit/sfu/getandPOST /_spindle/rtc/livekit/get_token(the Matrix 2.0 request) in the shape oflk-jwt-service0.7, so a client cannot tell the two apart; - mints only for a Matrix room the user is joined to at that moment,
checked against its own membership index. Never joined, only invited,
or since left: refused with
M_FORBIDDEN, and a room that does not exist is refused identically; - redeems its own users' OpenID tokens itself. It sends another server's
token to that server (
openid/userinfoover federation) and accepts it only for a user of that server; - rate limits minting per user, and per server for other servers' tokens (docs/rate-limits.md).
The token's grants are the ones lk-jwt-service gives:
| Who | Join, subscribe | Publish | Room on the SFU |
|---|---|---|---|
a user of this server, or of a server in full_access_servers | yes | yes | created through the SFU's API before the token is sent |
| a user of any other server | yes | no | not created |
So the users of another server see and hear a call on this SFU, and
their own media goes through their own server's SFU. full_access_servers
names each server whose users may also publish here. It has no
wildcard, because publishing uses the bandwidth of this SFU.
The LiveKit room name is a hash of the Matrix room ID and the call slot.
The Matrix 2.0 identity is a hash of user, device and member ID. Both are
the hashes of lk-jwt-service, so a call lands in the same LiveKit room
with either service. The legacy sfu/get identity is
{user_id}:{device_id}.
No token gets roomCreate, because in LiveKit it also permits deleting
the room, which ends the call for everyone. This server creates the room
itself, with the SFU's CreateRoom API at the http(s) address of
url. Thus the SFU can keep auto_create: false, which is the setting
lk-jwt-service recommends: a user of another server cannot open a room
on it. This server must reach the SFU at that address. If it cannot, the
token request fails with 500 M_UNKNOWN.
Element Call can give the transport its delayed leave on a token request
(delay_id). It does this when a delegation probe does not get a 404.
Then it schedules a leave an hour away and stops restarting it. This
server holds that leave for its own users, and the SFU webhook sends it
when the participant leaves the SFU. A delay that is not the caller's
gets 400 M_INVALID_PARAM. A user of another server gets
400 M_BAD_JSON, because this server cannot send their delayed events.
Clients then ask again without the delay.
Revocation. A minted token cannot be revoked: it is stateless, and
the SFU never asks this server again. A user who leaves the room after
minting holds their token until it expires. token_ttl_seconds is the
whole of that guarantee, which is why it defaults to fifteen minutes and
why the default is not an hour. lk-jwt-service issues an hour; a
client that needs that gets it by setting 3600 here, on purpose.
The secret is LiveKit's and is shared with nothing else. It is not this server's signing key and is not derived from it; the two rotate apart, and a leak of one is not a leak of the other. It is never logged.
To keep the secret out of spindle.toml, put it in a file and set
secret_file to its path instead of secret. For example, use a
Compose or Kubernetes secret. This is how lk-jwt-service reads
LIVEKIT_SECRET_FROM_FILE. Set exactly one of the two keys.
The server reads the file once, at startup, and ignores the trailing
newline. If the file is missing or empty, or if both keys or neither key
is set, the server does not start, and the error names rtc.livekit.
To use a new secret, restart the server. With a supervised SFU, the
generated livekit.yaml gets the secret from the file.
Option B: lk-jwt-service beside the server
The reference shape, and what Element and Tuwunel document. Run
lk-jwt-service with the SFU's key and secret, put it behind your
reverse proxy at some public path, and name that path here:
[server]
name = "example.org"
public_base_url = "https://matrix.example.org"
[rtc]
foci = [
{ type = "livekit", livekit_service_url = "https://matrix-rtc.example.org/livekit/jwt" },
]
The service redeems each OpenID token against this server's
GET /_matrix/federation/v1/openid/userinfo, resolved through
.well-known/matrix/server like any federation request, so the
federation listener has to be reachable from wherever the service runs.
Nothing else on this server is involved: the service mints for whatever
room the client names, with no membership check on this side.
Recent lk-jwt-service releases (0.6 and later; Element Server Suite
ships 0.6.0) also take over a client's delayed leave event when the
client asks (/delegate_delayed_leave, or a delay_id on /get_token):
while the participant stays connected to the SFU the service restarts the
delay, and when they drop it sends it. It does that with
POST /_matrix/client/unstable/org.matrix.msc4140/delayed_events/{delay_id}/{restart|send}
and no access token, the delay ID being the capability, so that route has
to reach this server from wherever the service runs, like openid/userinfo.
Delegation lengthens the leave, so the webhook matters. Element Call
asks the transport (POST {livekit_service_url}/delegate_delayed_leave,
unauthenticated) whether it can hold the leave; any answer but a 404 is a
yes, and Element Call then schedules its leave an hour out instead of
eighteen seconds and stops restarting it every four. From then on the only
thing that removes a participant whose browser died is lk-jwt-service
noticing them leave the SFU -- which it learns from LiveKit's webhooks
(webhook.urls in livekit.yaml pointing at the service's
/sfu_webhook, webhook.api_key equal to the service's LIVEKIT_KEY), or,
as a fallback, from LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS. A deployment
that serves /delegate_delayed_leave but whose SFU cannot reach
/sfu_webhook leaves crashed participants in the call for up to an hour.
Releases from 0.7 drop the token-less calls: the service runs as an
application service (option C) and calls the same endpoint authenticated
as the user. And they treat a 409 from it as final; this server answers
409 only on the stable /_matrix/client/v1/delayed_events/{id}/{action}
and keeps the 404 on the unstable paths, which is what older releases and
the matrix-js-sdk in shipping Element Call were built against (a 409 is a
retry loop for the first and an unrecoverable error for the second).
The two options compose. With both configured, the built-in service is
listed first and the operator's foci follow in the order written;
clients read the list as a priority order.
The SFU program: switch, models, sidecar
The built-in service above is one half of an SFU program with an
operator switch. spindle sfu on|off|status <config> flips and reads it;
GET/PUT /_spindle/admin/v1/rtc/sfu does the same behind admin auth. The
switch is stored server-side and survives restarts; it never rewrites the
config file. When off, everything reads as unconfigured: sfu/get
answers 404 M_UNRECOGNIZED, the transport is unadvertised on both
discovery surfaces, and no sidecar runs — off costs zero.
Two models serve the media, and the switch plus the minter behave identically for both:
- Remote (default): the SFU is separately hosted at
[rtc.livekit] url. This server only mints. - Local:
[rtc.livekit] binarynames the operator-suppliedlivekit-serverbinary (pinned at v1.13.5, Element Server Suite parity), and this server supervises it as a child — generating its config (keys from[rtc.livekit],auto_create: false, webhooks pointing at this server), watching it, restarting it. Upgrades are decoupled from spindle releases: replace the binary and restart, or flip the switch off and on. The only file supervision writes is the generated sidecar config (<storage.path>/livekit.yamlunlesssidecar_config_pathsays otherwise).
The minter gaps the "What is not here" section used to list are closed:
MSC4195's homeserver token endpoint (POST .../rtc/livekit/get_token, stable and unstable) mints for local users
over their access token; the federation twin mints for a remote
participant whose server vouches for them by X-Matrix signature, with
the same grants as any other remote user's token; the delegation probe
(POST .../delegate_delayed_leave, anything but 404 meaning "held")
records the hold; and the SFU webhook (POST /_spindle/rtc/livekit/sfu/webhook) releases held leaves when
participants drop off the media — fed by the supervised sidecar's
generated webhook config, or by the operator's hand on the remote model.
The webhook accepts only a request that LiveKit signed: an HS256 JWT
under the API secret in Authorization, with the SHA-256 of the body.
Option C: lk-jwt-service as the homeserver's sidecar
From 0.7, lk-jwt-service is built to sit behind the homeserver as an
application service rather than beside it as a web service, and Element
Call's own test rig runs it that way. The homeserver answers MSC4195's
endpoints by forwarding them (MSC4512): the client's request under
/_matrix/client/…/rtc/livekit/ is authenticated as usual and handed to
the service with the service's hs_token and the caller's user ID in
X-Matrix-User-Identifier; the federation twin, under
/_matrix/federation/…/rtc/livekit/, is checked against its X-Matrix
signature and handed over with X-Matrix-Origin. The service asks this
server whether the caller is in the room (/rooms/{id}/is_joined,
MSC4502) instead of trusting the client, and sends its own federation
requests through fed_proxy, signed as this server. And because Element
Call's delegation probe now goes to the homeserver path, this is the shape
in which the homeserver itself can hold a delayed leave.
The registration, as lk-jwt-service's README writes it:
id: "LiveKit JWT service"
as_token: "<snip>"
hs_token: "<snip>"
sender_localpart: "_lk_jwt_service"
namespaces:
users:
- exclusive: false
regex: ".*" # it checks membership and acts for every local user
url: null # no event traffic
io.element.msc4502.scopes: [ "urn:matrix:client:io.element.msc4502:rooms:is_joined" ]
io.element.msc4512.proxy_prefix: "rtc/livekit"
io.element.msc4512.proxy_url: "http://127.0.0.1:8080"
[appservices]
registrations = ["/etc/spindle/lk-jwt-service.yaml"]
with the service started with LIVEKIT_AS_REGISTRATION_FILE pointing at
the same file and LIVEKIT_HS_SERVER_NAME set to this server's name.
proxy_prefix and proxy_url go together or not at all, and two services
may not claim overlapping prefixes; either mistake stops the server at
startup rather than half-routing calls. is_joined answers only services
granted the scope, and server admins. Advertise the transport as the
service's public URL in [rtc] foci, as in option B; the homeserver path
is what Element Call probes and what clients post to once they move to
MSC4195's homeserver endpoint.
crates/spindle-server/tests/appservice_proxy.rs pins the forwarding,
the authentication on both APIs, the scope, and fed_proxy's confinement
to the service's prefix.
What to check
curl https://matrix.example.org/.well-known/matrix/clientnames the transport underorg.matrix.msc4143.rtc_foci. If it does not, neither[rtc.livekit]nor[rtc] fociis set.GET /_matrix/client/versionslistsorg.matrix.msc4140,org.matrix.msc4143andorg.matrix.msc4354underunstable_features; Element Call checks them before it will rely on the server.GET /_matrix/client/v3/capabilitiesnames both delayed- event limits underm.delayed_events(and the unstableorg.matrix.msc4140.delayed_events), from[delayed_events].curl -X POST https://matrix.example.org/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leaveis a 404 unless option C is configured, and a 401 when it is. Element Call reads anything but a 404 as "this homeserver holds my leave".- A send with
?org.matrix.msc4354.sticky_duration_ms=30000comes back fromGET .../event/{id}carryingmsc4354_sticky.duration_ms, and a client that joins the room afterwards finds it underrooms.join.{room}.msc4354_sticky.eventsin its first/sync, withunsigned.msc4354_sticky_duration_ttl_mscounting down. Durations above an hour are capped to it. - For option A: a joined user's
POST .../sfu/getreturns ajwtwhose decodedvideo.roomis the hash of the Matrix room ID and the call slot (not the room ID) and whoseexp - nbfistoken_ttl_seconds. A user who has left gets403 M_FORBIDDEN. A joined user of another server getscanPublish: false. - For option B:
GET /_matrix/federation/v1/openid/userinfo?access_token=…with a fresh token returns{"sub": "@you:example.org"}; with an expired or invented one,401 M_UNKNOWN_TOKEN.
The tests that pin each of these: crates/spindle-server/tests/openid.rs,
livekit_jwt.rs, livekit_remote.rs, rtc_sfu.rs, rtc_transports.rs,
rtc_membership.rs, delayed_events.rs, delayed_events_merged.rs,
appservice_proxy.rs and sticky_events.rs.
Moving from lk-jwt-service to the built-in minter
This moves a deployment from option B (or C) to option A. The SFU stays. Calls in progress continue, because both services give the same LiveKit room names.
Before you start:
- Make sure that this server can reach the SFU at the
http(s)form of itswssURL. The minter creates rooms there. Acurlofhttps://livekit.example.org/from the homeserver host must connect. - Read
LIVEKIT_FULL_ACCESS_HOMESERVERSfrom the lk-jwt-service environment. Put each name except this server's own intofull_access_servers. If the value is*, decide which servers to name: the built-in minter has no wildcard. - Keep the SFU's
auto_create: false. The minter does not needauto_create.
Roll out:
- Add
[rtc.livekit]with the SFU'surl,keyandsecret(the values ofLIVEKIT_URL,LIVEKIT_KEYandLIVEKIT_SECRET). Remove the lk-jwt-service entry from[rtc] foci. For option C, also remove its registration from[appservices]; otherwise this server sendsget_tokenanddelegate_delayed_leaveto the service. - Point the SFU's
webhook.urlsathttps://<public_base_url>/_spindle/rtc/livekit/sfu/webhook, withwebhook.api_keyequal tokey. Restart the SFU. - Restart this server.
GET /_matrix/client/v1/rtc/transportsnow nameshttps://<public_base_url>/_spindle/rtc/livekitfirst. - Publish the new transport where clients find it. If a static
.well-known/matrix/clientnames the lk-jwt-service URL, change it to the URL from step 3 or let this server serve the document. - Make a call to a user of another server, in both directions. Check
that each side sees and hears the other. In the server log, look for
the SFU refused to create a call's room: it means step 1 of the preparation is not done. - Stop lk-jwt-service when no client uses its URL. Clients that cached the old transport ask it until they refresh it.
Roll back:
- Put the lk-jwt-service entry back in
[rtc] foci(and the registration in[appservices]for option C), and start the service. - Turn the built-in minter off without editing its section:
spindle sfu off <config>, orPUT /_spindle/admin/v1/rtc/sfuwith{"enabled": false}. The built-in transport disappears from discovery at once, andsfu/getanswers 404. - Point the SFU's
webhook.urlsback at the service's/sfu_webhook. - Restore the static
.well-known/matrix/clientif you changed it.
Tokens that either service minted stay valid until they expire, and the room names are the same, so a call in progress continues in both directions.
Running Element Call's own suite
contrib/element-call/run.sh starts Element Call's Playwright stack with
two homeservers. LiveKit, lk-jwt-service, Element Web and nginx are
upstream's, at a pinned revision. Spindle takes the first homeserver
seat: docker-compose-spindle.yml puts it there, with spindle.toml in
place of upstream's backend/playwright_homeserver.yaml. The script then
runs upstream's specs against the stack. scripts/element-call-check.py
compares the results with contrib/element-call/allowlist.txt, as the
Complement check does with its allowlist. The element-call-e2e job in
.github/workflows/compliance.yml runs the suite nightly and on demand.
SPINDLE_PEER selects the server in the second seat:
spindle, the default: a second Spindle, fromdocker-compose-spindle-othersite.ymlandspindle-othersite.toml.synapse: upstream's Synapse, with no change.docker-compose-dead-peers.ymladds three more Spindles for the last ring test below. This variant has its own allowlist,allowlist-synapse-peer.txt. Theelement-call-synapse-peerjob runs it nightly, on demand, and on pull requests that change federation or MatrixRTC code.
The rig also runs specs of this server, from contrib/element-call/specs/.
The script copies them into the checkout as playwright/spindle/. They
use upstream's fixtures with no change. With two Spindles they run:
- a five-party call with joins, a clean leave, a rejoin, and a participant that stops mid-call;
- a homeserver restart mid-call in both membership modes. The call must stay up, and a crash must still expire;
- a ring that the caller abandons;
- a call across the two Spindles in both membership modes.
With Spindle and Synapse they run:
- a call in each direction, in both membership modes.
- a DM call in each direction. The ring must get to the callee in 15 seconds, and then the call must connect.
- the same ring from Spindle, while the DM also holds users of three servers that do not answer. Each request to these servers runs into its timeout. Before #794, the outbox sent to one destination at a time, and this ring did not arrive.
SPINDLE_RTC selects who mints the SFU tokens for each Spindle seat:
lk-jwt-service, the default: upstream's service beside each homeserver, as in the Element Call rig.builtin: the built-in minter of the Spindle, against upstream's SFU.docker-compose-builtin-rtc.ymlgives the first seatspindle-builtin-rtc.toml, anddocker-compose-builtin-rtc-othersite.ymlgives a second Spindlespindle-othersite-builtin-rtc.toml.run.shwrites copies of upstream's nginx and LiveKit configs that send/_spindle/rtc/livekitand the SFU webhooks to the Spindles. A Synapse in the second seat keeps its lk-jwt-service. The results go againstallowlist-builtin-rtc.txt, orallowlist-builtin-rtc-synapse-peer.txtwith Synapse. Theelement-call-e2ejob runs both minters nightly; a push to main runs only the first.
Calls with no SFU
A room of a handful of people can call peer to peer, full mesh, with no media server in the path; that is a different client build and its own page, docs/p2p-calls.md, with the rig that proves it against this server.
What is not here
- Remote users without the program on. The built-in minter and the
federation twin mint for a remote participant while the SFU program
is on. While it is off, a federated caller needs
lk-jwt-service, external (option B) or as the sidecar (option C), which mints on its own. - The transport's own delegation endpoint.
lk-jwt-servicealso serves{livekit_service_url}/delegate_delayed_leave. The built-in minter does not, so that probe gets a 404. Delegation still works throughdelay_idon a token request and through the homeserver path. - Delegated leave without SFU events. The probe records the hold.
The release learns the drop from the SFU webhook. The supervised
sidecar points its webhooks here on generation. A remote deployment
must point its SFU at
/_spindle/rtc/livekit/sfu/webhookby hand. Without that pointer, crashed participants keep an hour-long leave. - The SFU and the relay themselves. Their docs cover them. This server mints for one and supervises the local deployment of the other. It never carries media.