Skip to main content

ci and iso pipeline

How Tromsø gets from .bst elements to a bootable, installable live ISO — and how to debug it when it breaks. (Same architecture as tuna-os/xfce-linux; patterns originate in projectbluefin/dakota and dakota-iso.)

Secure Boot key guard​

OCI builds for both architectures and ISO builds check signature inputs before the build and publication. They use files/boot-keys/check-release-keys.py --release. Public snakeoil modes/identities and malformed certificates fail the job. Never trust these development keys.

The pipeline signs artifacts with cosign keyless, not the legacy Secure Boot Makefile. Read the key policy before you enable Secure Boot. This guard prevents the regression in #219 that accepted public keys as signature inputs. It does not audit signatures inside existing artifacts.

Build chain​

elements/** ──► Build Tromso (Multi-Runner) ──► ghcr.io/tuna-os/tromso:latest
│ (workflow_run)
▼
Build local ISO → sign → UEFI boot gate → R2 dated objects → latest aliases
│
▼ (boot gate: TROMSO_LIVE_READY on serial + screenshot)
LUKS Install End-to-End Test (PR round-robin / weekly / dispatch)

Multi-runner build (build-tromso-multirunner.yml)​

The plan, core, and parallel dependency chunks (steps 1-3) use the shared tuna-os/bst-ci reusable workflow. Each BuildStream desktop repository uses this workflow, with different image names, targets, and chunk counts. Step 4 uses the build-final.yml workflow of this repository. Both architectures use this one copy of the steps (#314).

The sign_final jobs of build-tromso-multirunner.yml sign the pushed digest. They use cosign without a key. The Fulcio certificate thus keeps the identity of build-tromso-multirunner.yml. The verification instructions in README.md use that identity. build-final.yml has no id-token permission.

Free GitHub runners can't hold the whole KDE build, so it's split:

  1. plan — The scripts/ci-build-matrix.py script from bst-ci runs bst show. It splits uncached elements into a core set (first CORE_SPLIT) and NUM_CHUNKS round-robin chunks. Each chunk has a composite cache key. The job checks out the script from bst-ci at run time. This repository no longer contains a copy.
  2. build_core — builds the bootstrap set, pushes the CAS as ghcr.io/…/cache-tromso-core:latest (zstd tarball via oras).
  3. build_deps (matrix) — each chunk restores core + its own previous CAS, builds, pushes cache-tromso-<chunk>:{latest,<cache-key>}. The job skips a chunk when GHCR already has its exact cache key.
  4. build_final — merges all chunk CAS tarballs, builds the final target, just export (squash + OCI labels + chunkify), just lint (bootc container lint), pushes latest + date + sha tags (main only). scripts/resolve-image-tags.sh sets the tags. Only x86_64 gets the latest, nightly and stable tags without a suffix. Other architectures must use a suffix, for example latest-aarch64. The aarch64 path is manual only. It does not use remote execution. It does not run the x86_64 checks for kwin capabilities and overlay whiteouts.

BuildStream settings CI uses live in the checked-in buildstream-ci.conf.

Remote execution (tromso#311)​

The whole plan, core, chunk and merge design exists because this repository has no remote executor. Other repositories with one finish the same work in 18 to 22 minutes. RazorfinOS-org/cosmic-build-meta is the clearest proof, because it is one repository before and after the change.

build_final now takes its config from .github/actions/generate-bst-ci-config, a port of razorfin's action. The action reads two credentials:

namekindpurpose
CASD_CLIENT_CERTrepository variablemTLS identity for cache.projectbluefin.io:11002
CASD_CLIENT_KEYrepository secretthe matching private key

Neither exists yet, and nobody in this repository can create them. Someone must ask the Bluefin maintainers for a client certificate. Until then the action writes the committed buildstream-ci.conf byte for byte, so the build behaves as it did before. tests/pytest/test_bst_ci_config.py asserts that equality on every pull request.

Two rules protect the change:

  • The action stops the job when a caller asks for remote execution and the credentials are absent.
  • The build step greps the console for the Remote Execution Configuration banner. A green cache hit does not prove that BuildStream loaded the executor. A quiet fall back to local builds looks like a slow success.

Two steps remain. First, get the certificate. Second, keep the junction pins near the pins of the other repositories. Cache keys are a function of the ref, the patch queue, the options and the overrides. This repository pins freedesktop-sdk 25.08.17, the same ref as razorfin. Dakota and zirconium use 26.08, so a move to 26.08 is a separate decision.

Delete the chunk machinery from the caller only after a remote build passes.

Cache-key invalidation warning: a change to the cache key of every element causes a full world rebuild. A change to name: in project.conf is one example. Expect chunk jobs to run for hours or reach their six-hour limit once. They recover after they refresh the GHCR caches.

Live ISO (iso.justfile + tromso/)​

just iso-sd-boot tromso (see iso.justfile, imported from Justfile):

  1. just container tromso — 3-stage tromso/Containerfile: ghcr payload (kernel modules) → Debian stage builds a dmsquash-live initramfs (incl. the 95tromso-isofile Ventoy dracut module) → final stage installs flatpaks (src/install-flatpaks.sh) and configures the live env (src/configure-live.sh). iso-sd-boot gives this recipe the tromso/payload_ref image. Thus the live kernel and the install payload come from one image.
  2. The recipe squashes the payload image and puts it in the squashfs as an OCI layout. /etc/bootc-installer/media-input.json binds fisherman to this layout for the offline installation.
  3. tromso/src/build-iso.sh assembles a systemd-boot UEFI ISO.

The live session automatically logs in to Plasma as liveuser. It starts org.tunaos.InstallerKde from the OCI flatpak remote for tuna-os. A symbolic link at /usr/local/bin/fisherman points to fisherman. It uses the shared org.tunaos.Installer.install polkit action. See INSTALLER-FRONTENDS.md in the organization workspace.

Publication fails closed: the local candidate must pass the UEFI ready-token assertion before any R2 upload. A separate publish job signs the ISO and uploads it. It starts only after the build job boots the ISO. It has no checkout, so it runs no repository code. Only this job has OIDC and the R2 secrets.

All four dated objects (ISO, checksum, signature, certificate) must upload before the workflow updates latest aliases. Dated names include the checked-out source SHA, the workflow run ID, and its execution number; rclone --immutable rejects replacement. Checkout uses an immutable action pin and does not persist credentials. The signer identity and R2 secret names remain unchanged.

The workflow also publishes the latest checksum and updates the ISO alias last. R2 has no transaction across these aliases: a failed promotion can leave mixed sidecars, but fails the workflow. Dated objects remain the consistent download set.

LUKS end-to-end test (test-luks-install.yml)​

Local equivalent:

just debug=1 iso-sd-boot tromso # debug=1 enables SSH (liveuser/live)
just luks-test-qemu tromso # boot → fisherman LUKS install → reboot → unlock

tromso/src/luks-unlock.py drives the QEMU monitor. It checks screen dumps until Plymouth appears, types the passphrase with sendkey, and verifies the installed system boot. Screenshots and serial logs remain in Actions artifacts. The install jobs use read-only tokens and do not write screenshot branches or PR comments.

Source updates​

  • Renovate (renovate.json) — GitHub Actions, container tags. Automerge on green CI, majors included.
  • track-bst-sources.yml — Renovate can't parse .bst; this runs bst source track daily across the repo-local element groups. The workflow uses its matrix to split updates into focused PRs. It excludes orphaned elements that cannot build in the offline sandbox of BuildStream. There is no kde-build-meta.bst junction or separate junction-update PR. PRs made with the default GITHUB_TOKEN don't trigger CI — set a BOT_TOKEN secret to fix that.

Troubleshooting log (symptom → root cause → fix)​

DateSymptomRoot causeFix
2026-11-13Sign ISO stops with exit code 137 after Using payload from (run 43385946786)cosign 3 reads the whole ISO into memory. The 9 GB Tromsø ISO peaks above the 16 GB of runner RAM, so the kernel stops the jobAdd a 16 GB swap file before signing; test_iso_signing_has_swap_for_whole_iso_reads
2026-11-13ISO signing fails on every run: must specify --bundle with --new-bundle-format (run 43353984640)cosign-installer v4.1.3 installs cosign 3, which writes bundles by default and rejects the detached --output-signature/--output-certificate flagsSelect cosign's legacy detached mode (--use-signing-config=false --new-bundle-format=false), require nonempty .sig/.cert, and run the README cosign verify-blob command before any upload; test_iso_signing_preserves_detached_outputs_with_cosign3
2026-11-03Python Tests fails with No module named 'yaml' (#449)The workflow guard test imports PyYAML, but CI installs only pytestInstall PyYAML in the Python Tests job; test_python_ci_installs_workflow_parser checks this dependency
2026-11-03Public development signing keys could be selected for release inputs (#219)Legacy key helper had no release policy or complete-import checkReject public modes and snakeoil certificate identities before build/publish; workflow coverage in test_release_workflows_enforce_guard
2026-11-02STE fails with 493 findings against a zero budget (#428)Seven existing documents contain long sentences, passive voice, gerunds, and noun clustersRewrite the flagged prose; preserve code examples and the zero budget
2026-10-24Unverified ISO could replace public latest (#334)Upload preceded boot verification and ignored R2 failuresBoot gate precedes publication; immutable dated set completes before latest aliases; upload failures fail the job
2026-10-24CI executes Actions and reusable workflows through mutable tags/branches (#375)Moving references can change executable code without a repository reviewPin remote uses: to full commit SHAs and Docker Actions to SHA-256 digests; test_github_actions_use_immutable_references prevents regressions
2026-10-24Build workflows store GITHUB_TOKEN in plaintext .git/config during build jobs (#382)actions/checkout defaults to persist-credentials: true, writing tokens readable by subsequent steps and containersAdd persist-credentials: false to all actions/checkout steps in build-iso.yml and build-tromso-multirunner.yml; enforced by test_iso_invariants.py
2026-10-24Four workflows inherited repository token defaults (#321)Job-level permissions did not constrain other jobsAdd top-level contents: read; retain job overrides and require explicit workflow permissions in the invariant tests
2026-07-19Every just call in CI fails: "multiple candidate justfiles"justfile + Justfile both at root; just ≥1.30 hard-errorsISO recipes moved to iso.justfile, imported from Justfile
2026-07-19just iso-sd-boot tromso in build-iso.yml never workedrecipe + tromso/Containerfile + dracut module never existed in this repoported from xfce-linux/dakota-iso (PR #74)
2026-07-19tromso/Containerfile missing from git after commit.gitignore had unanchored Containerfile ruleanchored to /Containerfile
2026-07-19All 10 chunk jobs building for 5+ hproject.conf name: aurora→tromso changed every cache key → world rebuildexpected one-time cost; caches repopulate
2026-07-19Installer flatpak never launched in live sessionISO baked org.bootcinstaller.Installer but autostart/symlink pointed elsewhereboth sides now use org.tunaos.InstallerKde

| 2026-07-19 | Multi-runner never went green since May; every run "cancelled" at ~6.5 h | chunk jobs killed by job-level timeout-minutes — a cancelled job never reaches the CAS-push step, so 6 h × 10 chunks of build work was discarded daily (≈720 runner-hours; zero chunk cache packages ever existed on GHCR) | build bounded inside the step (timeout 270m), push steps if: always() — partial CAS salvaged, builds converge across days | | 2026-07-19 | Failed chunks could publish their exact-cache-key tag and be skipped forever | for i in 1 2 3 … done retry loop exits 0 on total failure (status of last sleep) | retry loop removed (bst retry-failed/network-retries already cover it); rc propagated | | 2026-10-03 | build_final (run 37069338183) fails integrating oci/kde-linux/filesystem.bst and oci/layers/tromso-runtime.bst: setcap cap_sys_nice+ep /usr/bin/kwin_wayland exit 1 | oci/kde-linux/stack.bst ran setcap as a public integration command; BuildStream's sandbox cannot write security.capability xattrs, so every compose that stages the stack failed. Present since May, first hit once chunks stopped timing out | setcap moved to the fakecap-preloaded OCI assembly scripts (oci/kde-linux/image.bst, already in oci/tromso.bst), the freedesktop-sdk pattern (see its components/shadow.bst); build_final now getcaps kwin_wayland in the exported image | | 2026-10-03 | build_final (run 37102603548) fails in oci/kde-linux/image.bst: glib-compile-schemas exit 127 | the element never declared components/glib.bst; first reached once the setcap fix let the compose elements integrate | freedesktop-sdk.bst:components/glib.bst added to its build-depends (as oci/tromso.bst already has) | | 2026-10-04 | First published tromso:latest breaks the ISO/E2E container recipe: buildah commit --squash fails with lstat /etc/alsa/conf.d/50-arcam-av-ctl.conf: no such file or directory; 65 c????????? entries in the mounted image | freedesktop-sdk build-oci analyze_lowers() dropped only the directory entry on a .wh.<dir> whiteout, not its descendants. The kde-linux layer whites out /etc/alsa (and /etc/fonts, /usr/share/licenses/freedesktop-sdk); the tromso layer recreates them and build-oci then emitted whiteouts for files that no longer exist. Over a whiteout the recreated directory is not merged, so overlayfs lists those whiteouts as unstattable char devices | patches/freedesktop-sdk/0012 fixes analyze_lowers(); build_final fails on any char device in the exported image; .gitignore no longer swallows new patches/ files |

| 2026-07-20 | Multi-runner build_deps chunk jobs queued for 20+ min when tromso and xfce-linux built simultaneously | Simultaneous schedule triggers and push builds across repos reached free-tier org concurrency cap (~20 jobs) | Removed push triggers; staggered daily crons (xfce-linux at 23:30 UTC, tromso at 00:30 UTC) and accepted residual manual-dispatch contention as free-runner trade-off (tromso#93) | | 2026-08-12 | Daily chore(deps): track element sources PR red on Build changed elements (tromso/glow.bst: go: download go1.26.5 … lookup proxy.golang.org … connection refused) | glow.bst/gum.bst run go mod download in build-commands, but the BuildStream sandbox has no network; they are orphaned (absent from tromso/deps.bst), so the world build never built them and only a ref bump touching the file exposes it. glow v3.0.0 additionally wants a Go toolchain newer than freedesktop-sdk 25.08 ships | Both excluded from track-bst-sources.yml via TRACK_EXCLUDE so a broken element can't block buildable ref bumps. #180 merged red, so glow stays at v3.0.0 on main: reverting the ref would touch the file and trip the same gate. Re-include (and repair the ref) once they vendor modules (-mod=vendor, as uupd.bst and kde-linux-deps/toolbox.bst do) | | 2026-07-20 | Runner agent process crashes mid-build (System.IO.IOException: No space left on device) on heavy chunks (util-linux-full, cryptsetup, pwquality), bypassing if: always() CAS salvage | ublue-os/remove-unwanted-software@v9 freed insufficient disk space compared to jlumbroso/free-disk-space (which removes tool-cache), leaving ~25GB instead of ~45GB free | Upstreamed jlumbroso/free-disk-space@v5 (tool-cache: true) to build_core in tuna-os/bst-ci (matching build_deps and build_final), increasing free disk space for heavy dependency closures (tromso#96) |

Add rows to this table as you change CI. See the organization ci-fix-loop skill. The docs/ci-troubleshooting.md file in tuna-os/tunaos shows the format.

Channels: nightly (main) and stable​

  • main is the nightly trunk: the daily scheduled multi-runner build publishes :latest, :nightly, :nightly-YYYYMMDD, :<sha>; the ISO lands at R2 tromso/.
  • stable is a release bookmark branch. A weekly schedule or manual dispatch starts promote-stable.yml. It selects one candidate from main. The candidate has an immutable digest. See the next section. The :stable tag gets the digest of that candidate, and the stable ISO under R2 tromso/stable/ embeds that digest.
  • Update PRs from source trackers or Renovate target only main. Stable moves only through promotion.

Candidate-bound promotion (tromso#309)​

Each x86_64 build that pushes and signs an image also uploads a release-candidate artifact. This artifact is release-candidate.json with the commit SHA and the pushed digest. It stays for 30 days.

promote-stable.yml has these jobs:

  1. candidate selects the newest successful main run of build-tromso-multirunner.yml and downloads its release-candidate artifact. The SHA must be the head SHA of the run. The digest must be sha256: with 64 hex characters. If the artifact is missing, expired, or not correct, the promotion stops. Start a new main build to get a new candidate.
  2. iso, plain, and luks call build-iso.yml, test-plain-install.yml, and test-luks-install.yml with candidate_sha and candidate_digest. Each checks out the candidate SHA. scripts/pin-candidate-payload.sh writes ghcr.io/tuna-os/tromso@sha256:... to tromso/payload_ref. The script stops if the checkout is not the candidate or the digest is not correct. The iso call sets skip_upload, so it does not sign or publish.
  3. promote runs only if the other four jobs succeed. It checks that the candidate is on main and that stable can fast-forward to it. Then it copies the digest to :stable with skopeo copy --preserve-digests. Then it moves stable through the GitHub API with force=false. There is no force input.
  4. publish-stable-iso calls build-iso.yml again with release_channel: stable. It builds and boots the ISO from the same candidate. Then the publish job of build-iso.yml signs it and puts it under R2 tromso/stable/.

Only promote has write permission. It has no checkout. The jobs that run candidate code have read-only tokens and get no secrets. The iso caller gives id-token: write only because GitHub needs it for the publish job of build-iso.yml. That job does not run when skip_upload is true.

When the payload ref names a digest, scripts/embed-oci-payload.sh writes that ref as image in media-input.json. It also writes it to both installer catalogs and to recipe.json, because fisherman uses the media only when the recipe asks for that image. targetImgref keeps the ref from the shipped recipe for updates after installation.

The registry copy and the branch update are two steps. If the branch update fails, :stable already has the new digest. Run the promotion again, or use rollback-stable.yml. The aarch64 images are not part of this gate.

tests/pytest/test_release_candidate.py runs the selection and promotion shell with fake gh and skopeo commands. tests/pytest/test_media_input.py runs the embed script on a real OCI archive. It needs skopeo and jq.

Release-linked sources​

Local elements use upstream release tags (globs such as v[0-9]*) instead of development branches. Thus, the daily bst source track selects releases. Some content repositories use branches, such as aurora common, docs, wallpapers, and the xfwl4 development repositories. Junctions also use a fixed branch. Do not set track: to one exact tag, because the tracker cannot move it.

Guard rails (what stops a bad commit)​

Branch protection on main needs pre-merge checks. These checks include shellcheck, yamllint, actionlint, BATS, pytest, the 52-test luks-unlock suite, and test_iso_invariants.py. Each invariant represents a class of defect that reached a release. The BuildStream graph gate runs bst show --deps all on the release target with resolved junctions.

Just Parse is also a gate. Add the multi-runner result to branch protection after the world rebuild converges (tromso#80). The scheduled or manual multi-runner workflow is the only path to build and publish an image. This design gives one shared CAS and one final signer.

Post-merge: salvage-enabled nightly world build → ISO boot gate (ready-marker + screenshot artifact) → weekly LUKS install e2e (screenshots and serial logs in Actions artifacts). A cloud routine ("tromso + xfce-linux CI babysitter", every 3 h) diagnoses completed failures from logs and pushes fixes.

Rules that keep this healthy: do not add paths: filters to workflows with required checks. If a required check does not report, automerge cannot continue. If you rename a required job, update the branch-protection contexts in the same PR. Do not wrap a gate in || echo; this error hid failures in bst-validate and pytest for months.

Pull request CI (test.yml)​

All pre-merge checks are in one workflow, test.yml. Its changes job lists the changed files and starts only the jobs that those files can affect. A skipped job reports success, so each required check always reports. If the changes job fails, all checks run. A change to test.yml runs all checks.

JobRuns when the change touches
shellcheck*.sh, *.bash
yamllint*.yml, *.yaml
actionlint.github/workflows/, .github/actions/
Renovate Config Validaterenovate.json
lint .bst files*.bst
BuildStream Validatea file outside docs, tests, scripts, tromso/, .github/ and the justfiles
BATS Unit Teststests/bats/, tromso/src/
Python Testsa file that is not *.md or LICENSE*
Just ParseJustfile, iso.justfile, just/

The plain and LUKS install tests build the same ISO and use different install paths. A change to their shared inputs (tromso/, the justfiles, scripts/lib-install-qemu.sh, scripts/fisherman-install.sh) runs one of the two tests. The run number selects the test. Set the repository variable E2E_ROUND_ROBIN_SIZE to 2 to run both. A change to the install script or workflow of one test always runs that test. The two tests run fully on their weekly schedules and on manual dispatch.

Both install tests run PR-supplied recipes with read-only repository tokens (tromso#270). Checkout does not persist credentials. They do not log in to GHCR: the payload image is public. Screenshots, serial logs, and job summaries remain available in Actions. No privileged job reports results, and no write boundary connects the untrusted artifacts to the repository. Any future change to reports needs its own review and behavioral security tests.

Rollback​

rollback-stable.yml runs only after a manual dispatch, and dry_run defaults to true. It reverses a promotion. The workflow verifies that the target :<sha> image exists. Then, skopeo copy --preserve-digests puts it on :stable and a dated stable-rollback-* tag. It force-pushes the stable branch to the same commit, so the branch and tag cannot diverge. It shares the concurrency group for promotion and cannot run at the same time.

This workflow does not restore a full release. It restores only the x86_64 :stable tag and the git branch. The -aarch64 tags and R2 ISO under tromso/stable/ stay on the bad build. The workflow does not use cosign verify on the target before the new tag. Signatures are available because build-tromso-multirunner.yml signs each pushed digest. The rollback runbook gives the manual steps for other artifacts and the final verification.