ci boot gate
If a bootc OS image builds, that proves that it assembles. It does not prove that
it boots. Bootloader installs, initramfs contents, display-manager wiring and
compression formats all fail in ways podman build cannot see. Corral turns
"does this image boot?" into one command with an exit code. That makes it a
publish gate: the image builds, corral boots it, and only then does the
pipeline promote a tag or upload an ISO.
This page documents that use case on both backends. The reference consumer is
tuna-os/tunaOS. It gates every promotion
of a GHCR tag on a QEMU boot of the image (see its docs/PIPELINE.md). The
same checks run against a KubeVirt cluster for local development.
Start with corral vmtest, below. It is the full harness. It customises the
image, boots it, collects the evidence, and hands the VM over for your tests.
corral create --wait-ssh is the smaller gate: one exit code, no artifacts. It
is still there for a pipeline that only asks "did it boot".
corral vmtest — the whole job in one command
corral vmtest builds the disk, boots it, waits for the guest, and runs your
assertions. It then leaves the VM up, so the next step can test it. It also
adds what a test needs and the published image does not have: accounts,
passwords, packages, files, and a hook for the first boot.
corral vmtest gate --bootc ghcr.io/tuna-os/yellowfin:gnome-testing \
--user tester --password hunter2 --sudo-user \
--package jq \
--check 'systemctl is-active sshd' \
--check 'systemctl --failed --no-legend' \
--video
# exit 0 → the image booted, the hook passed, every check passed
# With --keep, use the SSH command printed in the summary. The private key
# is kept outside corral-vmtest-out/ and is not uploaded with the evidence.
corral delete gate
Add --rm for a pure gate, where nothing runs after it.
What it writes
Every run fills an artifact directory (--artifacts, default
corral-vmtest-out/). Upload it from CI; a failure is diagnosable without a
second run. The generated SSH identity is kept in a private temporary directory
outside the artifact directory. With the default --keep, use the SSH command
printed in the summary.
| File | What it tells you |
|---|---|
result.json | the whole run: verdict, boot time, each check, each frame |
serial.log | the guest console, from the firmware to the readiness marker |
frames/f*.png | one screenshot per --screenshot-interval during the boot |
ready.png / failure.png | the screen at the moment the run ended |
timelapse.webm | the boot as a video (--video, needs ffmpeg) |
diagnostics/ | failed units, bootc status, journal warnings |
Exit codes
One code per failure class, so a pipeline can branch on the answer:
| Code | Meaning |
|---|---|
| 0 | passed |
| 1 | the spec cannot run (no image, a bad regular expression) |
| 2 | this host cannot run it (no podman, no qemu, no loop device) |
| 3 | the layer build failed (a package that does not exist) |
| 4 | the disk build failed (bootc install) |
| 5 | the VM did not start |
| 6 | the guest never became ready |
| 7 | the post-boot hook failed |
| 8 | a check failed |
| 9 | the guest painted nothing, and --require-paint was set |
Code 2 is the important one. It separates a broken runner from a broken image, which is the distinction a red pipeline usually hides.
Users, passwords and packages: the derived layer
Anything the spec asks for goes into a thin image layer built on top of the reference under test. The published image is never modified. With no customisation asked for, corral builds no layer at all and boots the image exactly as published.
The layer adds:
-
Accounts.
--user tester --password hunter2 --sudo-usercreates the account, sets the password, and grants sudo with no password. The home directory lands in/var/home. On a bootc system/homeis a symlink into/var, and only the image's/varreaches the installed disk. corral hashes every password on the host, so no plain password reaches the image. -
Packages.
--package jqinstalls with the base image's own package manager. corral reads the image filesystem to find out which one it is — dnf, zypper, apt, pacman, or apk. -
A hook for the first boot.
--post-boot ./firstboot.shruns in the booted guest as a systemd oneshot unit. Its exit code is the run's verdict. Its output reaches the artifact directory, and its markers reach the serial console. A guest that never answers SSH therefore still reports.The hook is part of the boot, so it must never wait for the boot to finish.
systemctl is-system-running --waitinside one deadlocks: systemd waits for the hook, and the hook waits for systemd. corral bounds the unit at 10 minutes, so you get exit 7 instead of a hang. Assert a state (systemctl is-active sshd); do not wait for one.
Where a host has remora, corral asks
remora to generate the layer's Containerfile. remora is the same project's tool
for local layers. It knows six package managers. It resolves a package
lockfile, so an unchanged rebuild costs nothing. It also lints the result. Pick
one engine with --layer-engine remora|builtin.
An image that the same job built is a normal case. Give --bootc the
localhost/ tag that the build wrote. corral reads such a tag from local
podman storage. It does not pull it, because no registry can serve it. The
layer builds on it in the usual way, so users:, packages: and provision:
all apply to a locally built image.
Images with no sshd
A production desktop image ships sshd in a disabled state. No SSH probe can gate such an image. Gate on the console instead:
corral vmtest desk --bootc "$IMAGE" --ready-marker 'Reached target Graphical' \
--require-paint
--ready-marker waits for a regular expression on the guest's serial console.
--require-paint fails the run when the last frame is blank: the standard
deviation of its luminance is at or under 0.02. A guest that boots and never
draws is the failure an SSH probe hides. Without this flag, no exit code
catches it and somebody has to look at a picture.
Paint is a point-in-time measurement, not proof of guest initialization or that a later screenshot will remain painted. QEMU's initial host-rendered “Guest has not initialized the display (yet).” message is nonuniform too. For guest boot gates, paint must accompany a readiness milestone. A host message does not prove that firmware or an OS drew.
CI checks the native lifecycle of QEMU with an empty disk.
This proves neither an OS boot nor a desktop boot.
The ARM guest first waits for one of these EDK2 messages:
BdsDxe: loading Boot…, BdsDxe: starting Boot…, or
BdsDxe: failed to load Boot….
EDK2 prints these messages after it initializes the console.
An empty disk can produce failed-load output.
This shows firmware progress, not an OS boot.
The CLI then runs wait --paint within the same 30-second budget.
The 100 ms interval stays unchanged.
The separate screenshot --require-paint captures the current framebuffer.
Its 0.02 threshold stays unchanged.
The check never uses an earlier image.
Artifacts retain the milestone receipt, the serial output, and the live screenshot. On failure, cleanup saves the serial output before it stops or deletes the VM.
An image that checks itself on boot can report a failure as well as a pass.
Give that line to --fail-marker, and the run fails (exit 6) when the line
appears. It does not wait for the timeout. corral checks the failure marker
first, so a console that shows both lines fails:
corral vmtest desk --bootc "$IMAGE" --rm \
--ready-marker '(?s)BASE_CONTRACT_OK.*DESKTOP_CONTRACT_OK|DESKTOP_CONTRACT_OK.*BASE_CONTRACT_OK' \
--fail-marker 'CONTRACT_FAIL|Entering emergency mode|Kernel panic'
corral matches the markers against the whole console, not one line at a
time. Thus one (?s) pattern can need two lines in either order.
You can also drive the console keyboard and pointer directly. This is the only way into a LUKS passphrase prompt, or into a greeter on an image with no sshd:
corral screenshot desk -o greeter.png
corral key desk ret
corral type desk 'correct horse battery staple' --enter
corral key desk ctrl alt f2
corral pointer desk 1236 12 --click left
corral screenshot also captures a KubeVirt VM, through KubeVirt's VNC
screenshot subresource (virtctl vnc screenshot). It reports the same frame
size and deviation, and --require-paint fails a blank KubeVirt frame the same
way.
type and key work on local QEMU and KubeVirt VMs. pointer works on
KubeVirt VMs that have a tablet input device. See
Console keyboard and pointer.
Kernel arguments and the root filesystem
--karg (repeatable, kargs: in the file) puts a kernel argument into the
installed boot entry. The argument is there from the first boot, which a
command that runs over SSH later cannot do. Use it to turn off telemetry in
test guests, or to pick the boot target. corral always adds
console=ttyS0,115200n8 first, so that the console log exists.
--filesystem (filesystem:) replaces the root filesystem that the image's
bootc backend implies. Without it, ostree images get xfs, and composefs images
get ext4 (or btrfs when the image has no mkfs.ext4). Use it for an image
whose bootloader cannot read the default.
The spec file (Lima-shaped)
Everything above fits in a file, and corral reads the Lima field names it shares:
# verify.yaml
bootc: ghcr.io/tuna-os/yellowfin:gnome-testing
cpus: 4
memory: 4GiB
disk: 32GiB
timeout: 20m
users:
- name: tester
password: hunter2
sudo: true
packages: [jq, htop]
extraRun:
- dnf config-manager --set-enabled crb
files:
- path: /etc/corral-test.conf
content: |
test=1
mode: "0644"
provision:
- mode: image # runs at build time, in the layer
script: systemctl mask systemd-resolved
- mode: system # runs in the booted guest (Lima's own meaning)
script: |
systemctl is-system-running --wait
checks:
- systemctl is-active sshd
- bootc status --format json
screenshots:
interval: 5s
video: true
requirePaint: false
corral vmtest gate -f verify.yaml
A flag beats the file, and only when you pass it. An unknown field in the file
is an error, not a warning: a misspelled packages: that installs nothing
wastes the whole run.
Guest behavior assertions
The guest suite is optional. It adapts the boot, identity and graphical scenarios from
Frostyard's smoke features.
It uses the same spec and SSH checks: machinery, not a BDD runner or a
container test. For a headless image, omit desktop:
bootc: quay.io/fedora/fedora-bootc:41
guestAssertions:
id: fedora # exact /etc/os-release ID
versionId: "41" # optional exact VERSION_ID
For a desktop image, specify the installed session executable and the display-manager unit that systemd must show as active (names differ by distribution):
bootc: ghcr.io/tuna-os/yellowfin:gnome-testing
guestAssertions:
id: fedora # replace with the image's actual ID
desktop:
sessionCommand: gnome-shell
displayManagerUnit: gdm.service
screenshots:
requirePaint: true
The suite checks the system bus, journal, systemd service inventory and
os-release identity and human-readable names. A desktop check also needs a
graphical default target, the specified executable and an active manager.
Missing tools and failed scenarios fail the run; their commands and
output are in result.json alongside the serial log and other artifacts.
Other checks: still run. A headless configuration makes no claim about the
absence of a desktop.
Add image-specific checks (including any failed-unit allowlist or provenance requirements) separately; the suite does not assume Frostyard's distribution or packages. A painted frame is useful evidence but not proof of a successful user login.
These checks run only after the QEMU guest reports ready and SSH answers. A container-only layer test is not evidence that the VM booted. For real-boot regression coverage on a prepared KVM host, configure both images and run:
export CORRAL_VMTEST_HEADLESS_IMAGE=quay.io/fedora/fedora-bootc:41
export CORRAL_VMTEST_HEADLESS_ID=fedora CORRAL_VMTEST_HEADLESS_VERSION=41
export CORRAL_VMTEST_DESKTOP_IMAGE=your-desktop-bootc-image
export CORRAL_VMTEST_DESKTOP_ID=your-os-id
export CORRAL_VMTEST_DESKTOP_COMMAND=gnome-shell CORRAL_VMTEST_DESKTOP_UNIT=gdm.service
export CORRAL_REQUIRE_TOOLS=bootc-local-build,kvm,guest-images
export CORRAL_VMTEST_ARTIFACTS="$PWD/vmtest-guest-out"
sudo -E "$(command -v go)" test -tags e2evmtest -timeout 90m -count=1 -v ./pkg/vmtest/ -run 'TestE2E_GuestAssertion'
This also boots the headless image with a deliberately incorrect identity and
asserts an ExitCheck failure with check output, serial log and result file.
The desktop case uses a 32 GiB disk and builds the existing test layer to enable
SSH. Its desktop configuration does not change. When a case passes, it also
checks for nonempty serial and result evidence. CORRAL_VMTEST_ARTIFACTS preserves each case's artifacts under
its VM name; without it, Go removes them with its temporary test directories.
CORRAL_REQUIRE_TOOLS makes missing images/identities (guest-images), local
build tools or KVM fail these acceptance cases. Without this setting, missing
prerequisites skip the case for local development. A skipped case is not boot
evidence.
The existing VM test harness Actions workflow exposes the same cases on
manual dispatch. To use them, enable guest_assertions and configure image,
headless_id and headless_version. Then supply desktop_image, desktop_id,
and the desktop command/unit (plus an optional desktop version). The workflow runs all three
cases with required prerequisites and uploads vmtest-guest-out/ alongside its
usual evidence, even on failure. Guest acceptance is opt-in; scheduled and PR
runs keep their existing scope. A successful job for the container layer does not
satisfy guest acceptance.
GitHub Actions
jobs:
boot-gate:
runs-on: ubuntu-24.04 # hosted runners have KVM
steps:
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm
- name: Install corral
run: go install github.com/tuna-os/corral@latest
- name: Boot gate
run: |
sudo -E "$(which corral)" vmtest gate --bootc "$IMAGE" \
--check 'systemctl --failed --no-legend' \
--video --rm
- name: Upload the evidence
if: always()
uses: actions/upload-artifact@v7
with:
name: boot-gate
path: corral-vmtest-out/
sudo is not optional: bootc install partitions a disk and installs a
bootloader. Run corral as root, or pass --sudo to let it call podman through
sudo itself.
The smaller gate: corral create --wait-ssh
corral create gate --bootc ghcr.io/tuna-os/yellowfin:gnome-testing \
--wait-ssh --timeout 900
# exit 0 → disk built via `bootc install to-disk`, VM booted, root SSH answers
# exit ≠0 → it didn't; fail the pipeline
corral delete gate
--wait-ssh starts the VM, then blocks until a root SSH probe over the
forwarded port succeeds. bootc install injects the SSH key with
--root-ssh-authorized-keys, and nothing changes the published image.
Kernel arguments
--karg (repeatable) puts a kernel argument into the installed boot entry,
as it does for corral vmtest. The argument is there from the first boot:
corral create gate --bootc ghcr.io/tuna-os/yellowfin:gnome-testing \
--karg tunaos.countme=0 --wait-ssh --timeout 900
Each value must be one argument with no spaces. corral bootc create also
takes --karg.
Locally built images
Images in root podman storage work directly — no registry round-trip:
sudo podman build -t localhost/myos:test .
corral create gate --bootc localhost/myos:test --wait-ssh
(The install runs from containers-storage: when the ref is present
locally; localhost/ refs error early with a podman save | sudo podman load hint if the image is only in your rootless store.)
Declarative form (Lima-style YAML)
Note that corral create runs provision: scripts offline, chrooted into
the installed disk. corral vmtest follows Lima's own meaning and runs them in
the booted guest, unless you mark one mode: image.
Corral reads Lima YAML natively. bootc: plus provision: covers the
common CI need. You can enable sshd or drop test hooks chrooted into the
installed disk before first boot, and the published image stays unchanged:
# verify.yaml
bootc: ghcr.io/tuna-os/yellowfin:gnome-testing
cpus: 4
memory: 4GiB
disk: 32GiB
provision:
- mode: system
script: |
#!/bin/sh
systemctl enable sshd
corral create gate -f verify.yaml --wait-ssh --timeout 900
GitHub Actions recipe
jobs:
boot-gate:
runs-on: ubuntu-latest # hosted runners have KVM
steps:
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm
- name: Install corral
run: go install github.com/tuna-os/corral@latest
- name: Boot gate
run: |
corral create gate --bootc "$IMAGE" --wait-ssh --timeout 900
corral delete gate
Beyond "it boots": health checks over SSH
Once --wait-ssh returns, the VM is a normal SSH target — assert whatever
"it works" means for your image:
check() { corral ssh gate -u root -c "$1"; }
check "systemctl is-active graphical.target" # desktop reached
check "systemctl is-active gdm" # right display manager
check "systemctl --failed --no-legend" # empty, or a known allowlist
check "bootc status --format json | jq -r '.status.booted.image.image.image'"
Each check runs one time. To wait until a condition is true, use corral wait.
It works on local QEMU and on KubeVirt, and it fails with the name of the
condition that is not true:
corral wait gate --boot-complete --unit gdm.service --paint --timeout 15m
KubeVirt backend (clusters, heavy images, local dev parity)
corral bootc create gate --image ghcr.io/tuna-os/yellowfin:gnome -n corral-vms
# or: corral create gate --kubevirt --bootc <image> [-s <storage-class>]
corral start gate # bootc creates the VM stopped
corral wait gate --unit gdm.service --paint --timeout 15m
To test a desktop image with a real login account from its first boot:
corral bootc create gate --image ghcr.io/tuna-os/yellowfin:gnome \
--user tester --password-prompt
For automation, pipe the password to --password-stdin. Corral reads the
first line of stdin:
printf '%s\n' "$PASSWORD" | corral bootc create gate \
--image ghcr.io/tuna-os/yellowfin:gnome --user tester --password-stdin
You can also use --password-hash with a SHA-512 crypt value. Do not put a
plaintext password on a command line. The account flags work on local and
KubeVirt backends. The account goes into the installed deployment's /etc
and /var, so the greeter shows it at the first boot.
The build runs in a builder VM on the cluster (bootc install to-disk
onto a PVC). This is the only way to install images whose filesystems the
node kernel can't handle (btrfs/composefs on Talos, for example). Notes
that matter in practice:
- Storage: disk PVCs are Filesystem-mode file-backed disks, so any
provisioner works — including
local-path. Block-mode provisioners are not required. - Registry cache: deploy
deploy/registry-cache.yaml, a pull-through cache for ghcr.io. Builders then use it on their own. A desktop image of several gigabytes pulls at LAN speed after the first fetch.CORRAL_REGISTRY_MIRROR=offturns the detection off. Each instance serves only one upstream. A registry:2 aimed at quay is not able to serve content from ghcr. - Interrupted builds: sometimes the builder finishes but the final VM step
does not. Then
corral bootc create --resume <name>reuses the disk PVC that the builder completed. It does not build the disk again, so it cannot add a kernel argument. With--karg, the resume continues only if the builder installed each argument that you give. If not, it stops with an error. Then build the disk again without--resume.
Troubleshooting the gate
| Symptom | Cause / fix |
|---|---|
| VM lands in the UEFI setup menu (UiApp) | disk has no portable bootloader — installs must use --generic-image; NVRAM entries written in a builder VM never reach the final VM |
501 Unsupported client range during pull | zstd:chunked partial pulls need multi-range HTTP; GHCR's CDN refuses them and some podman versions don't fall back. Builders set enable_partial_images = "false"; if you hit this elsewhere, do the same in storage.conf |
no space left on device in /var/tmp mid-pull | full pulls stage blobs in $TMPDIR before committing to storage — point TMPDIR at a big disk (builders stage on the scratch disk) |
ostree: min-free-space-percent '3%' would be exceeded | target disk too small for the extracted image + reserve — desktop images generally want ≥ 32G |
| SSH never answers but the build reported OK | KubeVirt VMs are created stopped — corral start <name> first; then check the DM/sshd actually exist in the image |
Cluster ssh works, CI ssh refused at 127.0.0.1 | expected: without a tailnet the hostfwd binds loopback, which is where --wait-ssh probes; interactive corral ssh needs the tailnet |
vmtest exits 2 | the runner cannot host a VM at all: no podman, no qemu, no /dev/loop-control, or corral is not root. The message names the missing one |
Re-exec in host mountns: ... Permission denied | bootc install needs the host's mount namespace, and a container inside a container does not have one. Run corral on the runner, not in a container on it |
vmtest exits 6 and serial.log is empty | the guest never reached the bootloader, or the VM predates console capture. Recreate it — the console karg is installed by vmtest itself, so a VM built another way may not have one |
vmtest exits 6 and the console stops in dracut | the root filesystem or the boot files are wrong for the image's backend. A composefs image needs a filesystem with fs-verity (ext4, else btrfs), and its kernel and initrd restored on the ESP after the install. The local builder does both — check the build log for composefs: restoring the kernel and initrd. An image that needs another filesystem takes --filesystem |
vmtest exits 7 | the first-boot hook failed. Its own output is in result.json under hook.log, and on the console between the CORRAL_POSTBOOT_FAIL and CORRAL_VM_READY markers |
vmtest exits 7 and the console shows the hook starting and nothing after | the hook waits for something that waits for the hook. systemctl is-system-running --wait is the usual one. Assert a unit's state instead of waiting for the boot |
vmtest exits 9 | the guest booted and painted nothing. Look at ready.png and the last frames: a greeter that crashed looks exactly like this |
Every row above was hit for real while gating TunaOS images — this table is field notes, not speculation.