ci and iso pipeline
How Tromsø gets from .bst elements to a bootable, installable live ISO —
and how to debug it when it breaks. (Same architecture as tuna-os/xfce-linux;
patterns originate in projectbluefin/dakota and dakota-iso.)
Secure Boot key guard
OCI builds for both architectures and ISO builds check signature inputs before
the build and publication. They use files/boot-keys/check-release-keys.py --release.
Public snakeoil modes/identities and malformed certificates fail the job.
Never trust these development keys.
The pipeline signs artifacts with cosign keyless, not the legacy Secure Boot Makefile. Read the key policy before you enable Secure Boot. This guard prevents the regression in #219 that accepted public keys as signature inputs. It does not audit signatures inside existing artifacts.
Build chain
elements/** ──► Build Tromso (Multi-Runner) ──► ghcr.io/tuna-os/tromso:latest
│ (workflow_run)
▼
Build local ISO → sign → UEFI boot gate → R2 dated objects → latest aliases
│
▼ (boot gate: TROMSO_LIVE_READY on serial + screenshot)
LUKS Install End-to-End Test (PR round-robin / weekly / dispatch)
Multi-runner build (build-tromso-multirunner.yml)
The plan, core, and parallel dependency chunks (steps 1-3) use the shared
tuna-os/bst-ci reusable workflow. Each
BuildStream desktop repository uses this workflow, with different image names,
targets, and chunk counts. Step 4 uses the build-final.yml workflow of this
repository. Both architectures use this one copy of the steps (#314).
The sign_final jobs of build-tromso-multirunner.yml sign the pushed
digest. They use cosign without a key. The Fulcio certificate thus keeps the
identity of build-tromso-multirunner.yml. The verification instructions in
README.md use that identity. build-final.yml has no id-token permission.
Free GitHub runners can't hold the whole KDE build, so it's split:
- plan — The
scripts/ci-build-matrix.pyscript from bst-ci runsbst show. It splits uncached elements into a core set (firstCORE_SPLIT) andNUM_CHUNKSround-robin chunks. Each chunk has a composite cache key. The job checks out the script from bst-ci at run time. This repository no longer contains a copy. - build_core — builds the bootstrap set, pushes the CAS as
ghcr.io/…/cache-tromso-core:latest(zstd tarball via oras). - build_deps (matrix) — each chunk restores core + its own previous CAS,
builds, pushes
cache-tromso-<chunk>:{latest,<cache-key>}. The job skips a chunk when GHCR already has its exact cache key. - build_final — merges all chunk CAS tarballs, builds the final target,
just export(squash + OCI labels + chunkify),just lint(bootc container lint), pusheslatest+ date + sha tags (main only).scripts/resolve-image-tags.shsets the tags. Only x86_64 gets thelatest,nightlyandstabletags without a suffix. Other architectures must use a suffix, for examplelatest-aarch64. The aarch64 path is manual only. It does not use remote execution. It does not run the x86_64 checks for kwin capabilities and overlay whiteouts.
BuildStream settings CI uses live in the checked-in buildstream-ci.conf.
Remote execution (tromso#311)
The whole plan, core, chunk and merge design exists because this repository
has no remote executor. Other repositories with one finish the same work in
18 to 22 minutes. RazorfinOS-org/cosmic-build-meta is the clearest proof,
because it is one repository before and after the change.
build_final now takes its config from
.github/actions/generate-bst-ci-config, a port of razorfin's action. The
action reads two credentials:
| name | kind | purpose |
|---|---|---|
CASD_CLIENT_CERT | repository variable | mTLS identity for cache.projectbluefin.io:11002 |
CASD_CLIENT_KEY | repository secret | the matching private key |
Neither exists yet, and nobody in this repository can create them. Someone
must ask the Bluefin maintainers for a client certificate. Until then the
action writes the committed buildstream-ci.conf byte for byte, so the build
behaves as it did before. tests/pytest/test_bst_ci_config.py asserts that
equality on every pull request.
Two rules protect the change:
- The action stops the job when a caller asks for remote execution and the credentials are absent.
- The build step greps the console for the
Remote Execution Configurationbanner. A green cache hit does not prove that BuildStream loaded the executor. A quiet fall back to local builds looks like a slow success.
Two steps remain. First, get the certificate. Second, keep the junction pins near the pins of the other repositories. Cache keys are a function of the ref, the patch queue, the options and the overrides. This repository pins freedesktop-sdk 25.08.17, the same ref as razorfin. Dakota and zirconium use 26.08, so a move to 26.08 is a separate decision.
Delete the chunk machinery from the caller only after a remote build passes.
Cache-key invalidation warning: a change to the cache key of every element
causes a full world rebuild. A change to name: in project.conf is one example.
Expect chunk jobs to run for hours or reach their six-hour limit once. They
recover after they refresh the GHCR caches.
Live ISO (iso.justfile + tromso/)
just iso-sd-boot tromso (see iso.justfile, imported from Justfile):
just container tromso— 3-stagetromso/Containerfile: ghcr payload (kernel modules) → Debian stage builds a dmsquash-live initramfs (incl. the95tromso-isofileVentoy dracut module) → final stage installs flatpaks (src/install-flatpaks.sh) and configures the live env (src/configure-live.sh).iso-sd-bootgives this recipe thetromso/payload_refimage. Thus the live kernel and the install payload come from one image.- The recipe squashes the payload image and puts it in the squashfs as an
OCI layout.
/etc/bootc-installer/media-input.jsonbinds fisherman to this layout for the offline installation. tromso/src/build-iso.shassembles a systemd-boot UEFI ISO.
The live session automatically logs in to Plasma as liveuser. It starts
org.tunaos.InstallerKde from the OCI flatpak remote for tuna-os. A symbolic
link at /usr/local/bin/fisherman points to fisherman. It uses the shared
org.tunaos.Installer.install polkit action. See INSTALLER-FRONTENDS.md in the
organization workspace.
Publication fails closed: the local candidate must pass the UEFI ready-token
assertion before any R2 upload. A separate publish job signs the ISO and
uploads it. It starts only after the build job boots the ISO. It has no
checkout, so it runs no repository code. Only this job has OIDC and the R2
secrets.
All four dated objects (ISO, checksum, signature,
certificate) must upload before the workflow updates latest aliases.
Dated names include the checked-out source SHA, the workflow run ID, and its execution number;
rclone --immutable rejects replacement. Checkout uses an immutable action pin and does not
persist credentials. The signer identity and R2 secret names remain unchanged.
The workflow also publishes the latest checksum and updates the ISO alias last. R2 has no transaction across these aliases: a failed promotion can leave mixed sidecars, but fails the workflow. Dated objects remain the consistent download set.
LUKS end-to-end test (test-luks-install.yml)
Local equivalent:
just debug=1 iso-sd-boot tromso # debug=1 enables SSH (liveuser/live)
just luks-test-qemu tromso # boot → fisherman LUKS install → reboot → unlock
tromso/src/luks-unlock.py drives the QEMU monitor. It checks screen dumps
until Plymouth appears, types the passphrase with sendkey, and verifies the
installed system boot. Screenshots and serial logs remain in Actions artifacts.
The install jobs use read-only tokens and do not write screenshot branches
or PR comments.
Source updates
- Renovate (
renovate.json) — GitHub Actions, container tags. Automerge on green CI, majors included. track-bst-sources.yml— Renovate can't parse.bst; this runsbst source trackdaily across the repo-local element groups. The workflow uses its matrix to split updates into focused PRs. It excludes orphaned elements that cannot build in the offline sandbox of BuildStream. There is nokde-build-meta.bstjunction or separate junction-update PR. PRs made with the defaultGITHUB_TOKENdon't trigger CI — set aBOT_TOKENsecret to fix that.
Troubleshooting log (symptom → root cause → fix)
| Date | Symptom | Root cause | Fix |
|---|---|---|---|
| 2026-11-13 | Sign ISO stops with exit code 137 after Using payload from (run 43385946786) | cosign 3 reads the whole ISO into memory. The 9 GB Tromsø ISO peaks above the 16 GB of runner RAM, so the kernel stops the job | Add a 16 GB swap file before signing; test_iso_signing_has_swap_for_whole_iso_reads |
| 2026-11-13 | ISO signing fails on every run: must specify --bundle with --new-bundle-format (run 43353984640) | cosign-installer v4.1.3 installs cosign 3, which writes bundles by default and rejects the detached --output-signature/--output-certificate flags | Select cosign's legacy detached mode (--use-signing-config=false --new-bundle-format=false), require nonempty .sig/.cert, and run the README cosign verify-blob command before any upload; test_iso_signing_preserves_detached_outputs_with_cosign3 |
| 2026-11-03 | Python Tests fails with No module named 'yaml' (#449) | The workflow guard test imports PyYAML, but CI installs only pytest | Install PyYAML in the Python Tests job; test_python_ci_installs_workflow_parser checks this dependency |
| 2026-11-03 | Public development signing keys could be selected for release inputs (#219) | Legacy key helper had no release policy or complete-import check | Reject public modes and snakeoil certificate identities before build/publish; workflow coverage in test_release_workflows_enforce_guard |
| 2026-11-02 | STE fails with 493 findings against a zero budget (#428) | Seven existing documents contain long sentences, passive voice, gerunds, and noun clusters | Rewrite the flagged prose; preserve code examples and the zero budget |
| 2026-10-24 | Unverified ISO could replace public latest (#334) | Upload preceded boot verification and ignored R2 failures | Boot gate precedes publication; immutable dated set completes before latest aliases; upload failures fail the job |
| 2026-10-24 | CI executes Actions and reusable workflows through mutable tags/branches (#375) | Moving references can change executable code without a repository review | Pin remote uses: to full commit SHAs and Docker Actions to SHA-256 digests; test_github_actions_use_immutable_references prevents regressions |
| 2026-10-24 | Build workflows store GITHUB_TOKEN in plaintext .git/config during build jobs (#382) | actions/checkout defaults to persist-credentials: true, writing tokens readable by subsequent steps and containers | Add persist-credentials: false to all actions/checkout steps in build-iso.yml and build-tromso-multirunner.yml; enforced by test_iso_invariants.py |
| 2026-10-24 | Four workflows inherited repository token defaults (#321) | Job-level permissions did not constrain other jobs | Add top-level contents: read; retain job overrides and require explicit workflow permissions in the invariant tests |
| 2026-07-19 | Every just call in CI fails: "multiple candidate justfiles" | justfile + Justfile both at root; just ≥1.30 hard-errors | ISO recipes moved to iso.justfile, imported from Justfile |
| 2026-07-19 | just iso-sd-boot tromso in build-iso.yml never worked | recipe + tromso/Containerfile + dracut module never existed in this repo | ported from xfce-linux/dakota-iso (PR #74) |
| 2026-07-19 | tromso/Containerfile missing from git after commit | .gitignore had unanchored Containerfile rule | anchored to /Containerfile |
| 2026-07-19 | All 10 chunk jobs building for 5+ h | project.conf name: aurora→tromso changed every cache key → world rebuild | expected one-time cost; caches repopulate |
| 2026-07-19 | Installer flatpak never launched in live session | ISO baked org.bootcinstaller.Installer but autostart/symlink pointed elsewhere | both sides now use org.tunaos.InstallerKde |
| 2026-07-19 | Multi-runner never went green since May; every run "cancelled" at ~6.5 h | chunk jobs killed by job-level timeout-minutes — a cancelled job never reaches the CAS-push step, so 6 h × 10 chunks of build work was discarded daily (≈720 runner-hours; zero chunk cache packages ever existed on GHCR) | build bounded inside the step (timeout 270m), push steps if: always() — partial CAS salvaged, builds converge across days |
| 2026-07-19 | Failed chunks could publish their exact-cache-key tag and be skipped forever | for i in 1 2 3 … done retry loop exits 0 on total failure (status of last sleep) | retry loop removed (bst retry-failed/network-retries already cover it); rc propagated |
| 2026-10-03 | build_final (run 37069338183) fails integrating oci/kde-linux/filesystem.bst and oci/layers/tromso-runtime.bst: setcap cap_sys_nice+ep /usr/bin/kwin_wayland exit 1 | oci/kde-linux/stack.bst ran setcap as a public integration command; BuildStream's sandbox cannot write security.capability xattrs, so every compose that stages the stack failed. Present since May, first hit once chunks stopped timing out | setcap moved to the fakecap-preloaded OCI assembly scripts (oci/kde-linux/image.bst, already in oci/tromso.bst), the freedesktop-sdk pattern (see its components/shadow.bst); build_final now getcaps kwin_wayland in the exported image |
| 2026-10-03 | build_final (run 37102603548) fails in oci/kde-linux/image.bst: glib-compile-schemas exit 127 | the element never declared components/glib.bst; first reached once the setcap fix let the compose elements integrate | freedesktop-sdk.bst:components/glib.bst added to its build-depends (as oci/tromso.bst already has) |
| 2026-10-04 | First published tromso:latest breaks the ISO/E2E container recipe: buildah commit --squash fails with lstat /etc/alsa/conf.d/50-arcam-av-ctl.conf: no such file or directory; 65 c????????? entries in the mounted image | freedesktop-sdk build-oci analyze_lowers() dropped only the directory entry on a .wh.<dir> whiteout, not its descendants. The kde-linux layer whites out /etc/alsa (and /etc/fonts, /usr/share/licenses/freedesktop-sdk); the tromso layer recreates them and build-oci then emitted whiteouts for files that no longer exist. Over a whiteout the recreated directory is not merged, so overlayfs lists those whiteouts as unstattable char devices | patches/freedesktop-sdk/0012 fixes analyze_lowers(); build_final fails on any char device in the exported image; .gitignore no longer swallows new patches/ files |
| 2026-07-20 | Multi-runner build_deps chunk jobs queued for 20+ min when tromso and xfce-linux built simultaneously | Simultaneous schedule triggers and push builds across repos reached free-tier org concurrency cap (~20 jobs) | Removed push triggers; staggered daily crons (xfce-linux at 23:30 UTC, tromso at 00:30 UTC) and accepted residual manual-dispatch contention as free-runner trade-off (tromso#93) |
| 2026-08-12 | Daily chore(deps): track element sources PR red on Build changed elements (tromso/glow.bst: go: download go1.26.5 … lookup proxy.golang.org … connection refused) | glow.bst/gum.bst run go mod download in build-commands, but the BuildStream sandbox has no network; they are orphaned (absent from tromso/deps.bst), so the world build never built them and only a ref bump touching the file exposes it. glow v3.0.0 additionally wants a Go toolchain newer than freedesktop-sdk 25.08 ships | Both excluded from track-bst-sources.yml via TRACK_EXCLUDE so a broken element can't block buildable ref bumps. #180 merged red, so glow stays at v3.0.0 on main: reverting the ref would touch the file and trip the same gate. Re-include (and repair the ref) once they vendor modules (-mod=vendor, as uupd.bst and kde-linux-deps/toolbox.bst do) |
| 2026-07-20 | Runner agent process crashes mid-build (System.IO.IOException: No space left on device) on heavy chunks (util-linux-full, cryptsetup, pwquality), bypassing if: always() CAS salvage | ublue-os/remove-unwanted-software@v9 freed insufficient disk space compared to jlumbroso/free-disk-space (which removes tool-cache), leaving ~25GB instead of ~45GB free | Upstreamed jlumbroso/free-disk-space@v5 (tool-cache: true) to build_core in tuna-os/bst-ci (matching build_deps and build_final), increasing free disk space for heavy dependency closures (tromso#96) |
Add rows to this table as you change CI. See the organization ci-fix-loop
skill. The docs/ci-troubleshooting.md file in tuna-os/tunaos shows the format.
Channels: nightly (main) and stable
- main is the nightly trunk: the daily scheduled multi-runner build
publishes
:latest,:nightly,:nightly-YYYYMMDD,:<sha>; the ISO lands at R2tromso/. - stable is a release bookmark branch. A weekly schedule or manual dispatch
starts
promote-stable.yml. It selects one candidate from main. The candidate has an immutable digest. See the next section. The:stabletag gets the digest of that candidate, and the stable ISO under R2tromso/stable/embeds that digest. - Update PRs from source trackers or Renovate target only main. Stable moves only through promotion.
Candidate-bound promotion (tromso#309)
Each x86_64 build that pushes and signs an image also uploads a
release-candidate artifact. This artifact is release-candidate.json with
the commit SHA and the pushed digest. It stays for 30 days.
promote-stable.yml has these jobs:
candidateselects the newest successful main run ofbuild-tromso-multirunner.ymland downloads itsrelease-candidateartifact. The SHA must be the head SHA of the run. The digest must besha256:with 64 hex characters. If the artifact is missing, expired, or not correct, the promotion stops. Start a new main build to get a new candidate.iso,plain, andlukscallbuild-iso.yml,test-plain-install.yml, andtest-luks-install.ymlwithcandidate_shaandcandidate_digest. Each checks out the candidate SHA.scripts/pin-candidate-payload.shwritesghcr.io/tuna-os/tromso@sha256:...totromso/payload_ref. The script stops if the checkout is not the candidate or the digest is not correct. Theisocall setsskip_upload, so it does not sign or publish.promoteruns only if the other four jobs succeed. It checks that the candidate is on main and thatstablecan fast-forward to it. Then it copies the digest to:stablewithskopeo copy --preserve-digests. Then it movesstablethrough the GitHub API withforce=false. There is no force input.publish-stable-isocallsbuild-iso.ymlagain withrelease_channel: stable. It builds and boots the ISO from the same candidate. Then thepublishjob ofbuild-iso.ymlsigns it and puts it under R2tromso/stable/.
Only promote has write permission. It has no checkout. The jobs that run
candidate code have read-only tokens and get no secrets. The iso caller
gives id-token: write only because GitHub needs it for the publish job of
build-iso.yml. That job does not run when skip_upload is true.
When the payload ref names a digest, scripts/embed-oci-payload.sh writes
that ref as image in media-input.json. It also writes it to both installer
catalogs and to recipe.json, because fisherman uses the media only when the
recipe asks for that image. targetImgref keeps the ref from the shipped
recipe for updates after installation.
The registry copy and the branch update are two steps. If the branch update
fails, :stable already has the new digest. Run the promotion again, or use
rollback-stable.yml. The aarch64 images are not part of this gate.
tests/pytest/test_release_candidate.py runs the selection and promotion
shell with fake gh and skopeo commands. tests/pytest/test_media_input.py
runs the embed script on a real OCI archive. It needs skopeo and jq.
Release-linked sources
Local elements use upstream release tags (globs such as v[0-9]*) instead
of development branches. Thus, the daily bst source track selects releases.
Some content repositories use branches, such as aurora common, docs,
wallpapers, and the xfwl4 development repositories. Junctions also use a fixed
branch. Do not set track: to one exact tag, because the tracker cannot move it.
Guard rails (what stops a bad commit)
Branch protection on main needs pre-merge checks. These checks include
shellcheck, yamllint, actionlint, BATS, pytest, the 52-test luks-unlock suite,
and test_iso_invariants.py. Each invariant represents a class of defect that
reached a release. The BuildStream graph gate runs bst show --deps all on the
release target with resolved junctions.
Just Parse is also a gate. Add the multi-runner result to branch protection
after the world rebuild converges (tromso#80). The scheduled or manual
multi-runner workflow is the only path to build and publish an image. This
design gives one shared CAS and one final signer.
Post-merge: salvage-enabled nightly world build → ISO boot gate (ready-marker + screenshot artifact) → weekly LUKS install e2e (screenshots and serial logs in Actions artifacts). A cloud routine ("tromso + xfce-linux CI babysitter", every 3 h) diagnoses completed failures from logs and pushes fixes.
Rules that keep this healthy: do not add paths: filters to workflows
with required checks. If a required check does not report, automerge cannot
continue. If you rename a required job, update the branch-protection contexts
in the same PR. Do not wrap a gate in || echo; this error hid failures in
bst-validate and pytest for months.
Pull request CI (test.yml)
All pre-merge checks are in one workflow, test.yml. Its changes job lists
the changed files and starts only the jobs that those files can affect. A
skipped job reports success, so each required check always reports. If the
changes job fails, all checks run. A change to test.yml runs all checks.
| Job | Runs when the change touches |
|---|---|
| shellcheck | *.sh, *.bash |
| yamllint | *.yml, *.yaml |
| actionlint | .github/workflows/, .github/actions/ |
| Renovate Config Validate | renovate.json |
| lint .bst files | *.bst |
| BuildStream Validate | a file outside docs, tests, scripts, tromso/, .github/ and the justfiles |
| BATS Unit Tests | tests/bats/, tromso/src/ |
| Python Tests | a file that is not *.md or LICENSE* |
| Just Parse | Justfile, iso.justfile, just/ |
The plain and LUKS install tests build the same ISO and use different install
paths. A change to their shared inputs (tromso/, the justfiles,
scripts/lib-install-qemu.sh, scripts/fisherman-install.sh) runs one of
the two tests. The run number selects the test. Set the repository variable
E2E_ROUND_ROBIN_SIZE to 2 to run both. A change to the install script or
workflow of one test always runs that test. The two tests run fully on their
weekly schedules and on manual dispatch.
Both install tests run PR-supplied recipes with read-only repository tokens (tromso#270). Checkout does not persist credentials. They do not log in to GHCR: the payload image is public. Screenshots, serial logs, and job summaries remain available in Actions. No privileged job reports results, and no write boundary connects the untrusted artifacts to the repository. Any future change to reports needs its own review and behavioral security tests.
Rollback
rollback-stable.yml runs only after a manual dispatch, and dry_run defaults
to true. It reverses a promotion. The workflow verifies that the target
:<sha> image exists. Then, skopeo copy --preserve-digests puts it on
:stable and a dated stable-rollback-* tag. It force-pushes the stable branch
to the same commit, so the branch and tag cannot diverge. It shares the
concurrency group for promotion and cannot run at the same time.
This workflow does not restore a full release. It restores only the x86_64
:stable tag and the git branch. The -aarch64 tags and R2 ISO under
tromso/stable/ stay on the bad build. The workflow does not use
cosign verify on the target before the new tag. Signatures are available
because build-tromso-multirunner.yml signs each pushed digest. The
rollback runbook gives the
manual steps for other artifacts and the final verification.