Skip to main content

rotation promotion

Rotation (hive-rotate.sh) and pacing (hive-pace.sh) move into the HiveSpoke controller together, gated by one field: spec.rotationMode.

  • Shadow (every spoke today) computes the full plan every reconcile — .status.rotationPlanText, .status.contributorPlanText (primary only), and, every 20 minutes, the fleet-wide .status.pacePlanText — and applies nothing.
  • Enforce applies the same plan for that spoke's agents only: one atomic PUT /api/config/agent/{name}/models per placement (X-Hive-Internal token), pause/resume for strands, at most once per spec.rotation.intervalMinutes (20, the CronJob cadence). Pace notches are applied for that spoke's agents, with one notch per provider per 20 minutes across all Enforce spokes (the pool's .status.pace.lastActuation).

Promotion is per spoke: hanthor first, then reef, then school (the primary, which also owns the contributor replicas and the hive/hive-provider-usage publication). Each step changes the spoke and retires that spoke's share of the bash jobs in the same change. Two controllers actuating the same agents fight: that is how reef's sec-check was restarted ~25 times in one evening when pace and rotate disagreed.

Optional minimum quality score​

ModelLadder.spec.minimumScore is an opt-in, inclusive floor on each exact model and reasoning effort, using the existing rung score strings. For example, minimumScore: "40.2" admits a rung scored "40.2" but not "39.7". Only supplied finite numeric scores count: missing/invalid scores, implicit or conflicting effort declarations and invalid floor values fail closed. The operator does not infer scores from model families, contributor variants, aliases or other efforts. Omitting the field retains legacy placement and hardcoded pacing behavior.

Deploy the operator and ModelLadder CRD schema before setting the floor; older schemas prune unknown fields. Populate scored, inventory-supported exact rungs at every required tier, then enable the floor. status.effective and status.dropped expose eligibility and rejection reasons. With a floor enabled, consumers require a current-generation Ready condition and recheck scores. Removed, newly downgraded or unscored rungs in old status cannot authorize a launch.

Rotation corrects active off-policy placements, including old pace demotions, but preserves pins, declared holds, intentional pauses and on-demand agents. Pinned off-policy placements require an explicit operator update; the controller does not override the pin. No eligible capacity means an active off-policy agent is stranded/paused rather than knowingly started below the floor. A qualifying model does not bypass provider thresholds, cadence guards or hot/over-budget pool controls. Pacing remains enabled: hardcoded down/up notches and journal restorations must be eligible, otherwise saturation/capacity is reported. Watchdog repairs/rotate-offs use the same ladder; restarting or nudging an off-policy placement is held for rotation to correct. Release snapshot restoration also checks the current floor: an old disallowed snapshot is reported as held, keeping the post-boot placement for rotation to re-evaluate rather than falsely failing or rolling back an otherwise healthy release. Non-spoke release targets have no ModelLadder policy.

For agy, -high/-medium/-low requires the matching explicit launch effort; for pi, :thinking is launch syntax and authoritative. For omp, use selectors such as google-antigravity/gemini-3.8-flash:high or anthropic/claude-sonnet-5:max: placement atomically clears the separate reasoning_effort flag so it cannot override the encoded thinking level. Conflicting declared rung efforts fail closed. Encoded selectors also preserve Anthropic's trailing -5 through the current Hive model normalizer and do not depend on the dashboard's potentially stale per-model thinking catalog. Other backends carry explicit effort in the same atomic placement request. Backend/provider routing and authentication remain deployment prerequisites; adding a scored rung alone does not add a new backend.

What moves, what stays​

bashafter promotion of spoke X
hive-rotate-X (apply, */20)suspended — operator rotation for X
hive-pace (5,25,45, every namespace in HIVE_PACE_NAMESPACES)X removed from HIVE_PACE_NAMESPACES; suspended with the last spoke
hive-watchdog-X (*/5)stays until X's livenessMode is promoted (liveness-promotion.md) — see caveats
hive-nudge (:13,:43, every namespace in HIVE_NUDGE_NAMESPACES)stays; narrowed per spoke by liveness promotion, not rotation
journals on the hive-ops-state PVC (stranded, pace-demoted, kiro-evict, canary-cool-*)X's rows live in HiveSpoke X .status.journal; pace-demoted rows are seeded from the live placement at promotion (condition PaceJournalSeeded)
hive/hive-provider-usage publication (primary rotate)the operator republishes it once school is in Enforce
contributor replica scaling (primary rotate)the operator, once school is in Enforce

Prerequisites (once)​

  1. Deploy this operator build with the new CRDs and the pools. The hive-usage (ccusage) sidecar is not needed — do not apply config/usage/hive-usage-patch.yaml (it restarts the hive pods):

    kubectl apply -f config/crd -f config/rbac -f config/manager
    kubectl apply -f config/usage/usagepools.yaml # kiro anthropic openai google meta
    kubectl apply -f config/samples/fleet.yaml # every spoke Shadow
  2. Let it soak in Shadow. Each pool needs history before it has a verdict: ≥ 1 h of samples for the generic fit, ≥ 20 min for the Kiro budget. Check:

    kubectl get usagepools -o custom-columns=NAME:.metadata.name,READING:.status.rotationReading.note,PACE:.status.pace.verdict
    kubectl get usagepool kiro -o jsonpath='{.status.pace.kiroBudget.line}{"\n"}'
    kubectl -n hive logs job/<newest hive-pace job> | grep '^kiro budget'
  3. Shadow diff must be clean on consecutive ticks (at least 24 h, including at least one tick where pace or rotation actually moved something):

    export KUBECONFIG=~/.kube/config-aws-migration
    go run ./cmd/hive-shadow-diff --live # read-only; exit 0 = identical
    # or per spoke against the deployed operator's own status:
    kubectl -n hive logs job/<newest hive-rotate-hanthor job> > bash.log
    kubectl get hivespoke hanthor -o json > spoke.json
    go run ./cmd/hive-shadow-diff --bash bash.log --spoke spoke.json

    Classify every difference before promoting (DESIGN.md §7): snapshot skew (/api/status lags; --live undoes a newer pace job's moves for you), ladder (the hive-tiers AA cache on school), or logic. Only logic blocks.

Promote hanthor​

One change, applied together (and committed to the dotfiles manifests — talos-k8s/hive/ops/cronjobs.yaml — in the same PR, or the next kubectl apply of the ops manifests reverts it):

kubectl patch hivespoke hanthor --type merge -p '{"spec":{"rotationMode":"Enforce"}}'
kubectl -n hive patch cronjob hive-rotate-hanthor -p '{"spec":{"suspend":true}}'
kubectl -n hive set env cronjob/hive-pace HIVE_PACE_NAMESPACES="hive hive-reef"

Verify within one reconcile (2 min) and one rotation interval (20 min):

kubectl get hivespoke hanthor -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
# RotationPlanned=True, RotationEnforced=True Applied, PaceJournalSeeded=True (if any agent sits demoted)
kubectl get hivespoke hanthor -o jsonpath='{.status.rotationPlanText}' | tr ',' '\n'
kubectl get hivespoke hanthor -o jsonpath='{.status.journal}'
kubectl get hivespoke hanthor -o jsonpath='{.status.pacePlan}'
kubectl -n hive-system logs deploy/hive-operator --since=30m | grep -i hanthor

RotationEnforced=False NoActuator means the hive API was unreachable that reconcile; it plans as Shadow and retries.

Promote reef​

Same step:

kubectl patch hivespoke reef --type merge -p '{"spec":{"rotationMode":"Enforce"}}'
kubectl -n hive patch cronjob hive-rotate-reef -p '{"spec":{"suspend":true}}'
kubectl -n hive set env cronjob/hive-pace HIVE_PACE_NAMESPACES="hive"

Promote school (last)​

school is the primary: its rotate job also scales the contributor Deployments and publishes hive/hive-provider-usage; the operator takes both over. With school promoted no namespace is left for bash pacing, so hive-pace is suspended rather than narrowed:

kubectl patch hivespoke school --type merge -p '{"spec":{"rotationMode":"Enforce"}}'
kubectl -n hive patch cronjob hive-rotate -p '{"spec":{"suspend":true}}'
kubectl -n hive patch cronjob hive-pace -p '{"spec":{"suspend":true}}'

Then check kubectl -n hive get cm hive-provider-usage -o jsonpath='{.data.measured_by} {.data.updated_at}' moves every 20 min and .status.contributorPlanText matches what the bash job used to print.

Rollback (per spoke, same shape in reverse)​

kubectl patch hivespoke hanthor --type merge -p '{"spec":{"rotationMode":"Shadow"}}'
kubectl -n hive patch cronjob hive-rotate-hanthor -p '{"spec":{"suspend":false}}'
kubectl -n hive set env cronjob/hive-pace HIVE_PACE_NAMESPACES="hive hive-reef hive-hanthor"
# school: also kubectl -n hive patch cronjob hive-pace -p '{"spec":{"suspend":false}}'

What the bash side then sees:

  • Agents the operator stranded are paused with trigger dashboard-api; bash has no stranded row for them, so its auto-resume resumes them, and its placement re-strands them if the provider is still exhausted. Self-healing.
  • Agents the operator's pacer demoted have no row in bash's pace-demoted: bash rotate reads a demoted T1 agent as off-tier and moves it back onto a T1 rung on its next tick (an effective restore). To keep the demotion, append <ns>/<agent>|<backend>|<model> rows from .status.journal.paceDemoted to /state/hive-rotate/pace-demoted first.
  • .status.journal stays on the spoke; a later re-promotion reuses it (rows whose agent is no longer below the original rung are ignored).

Known caveats while spokes are mixed​

  • hive-watchdog-X keeps running until X's liveness is promoted (spec.livenessMode, liveness-promotion.md). Its healing is unaffected, but two of its paths place agents: the renewal wake-up runs a full hive-rotate.sh apply for X when a provider reset passes, and the auth/shell/approval rotate-off uses choose_rung_healthy. The planners agree (shadow diff), so the renewal apply makes the operator's placement a second time; the rotate-off is a watchdog decision the operator then keeps (a healthy in-tier rung is sticky). Promote rotation first, then liveness: the operator's own watchdog applies rotate-offs and renewal wake-ups only when rotationMode is Enforce too.
  • Kiro budget during the mixed window: bash pace computes the budget over its namespaces' agents, the operator over the whole fleet but applies only its own spoke's share. Both see the same Kiro burn, so in one tick each may take its share of demotions; the next tick is settling for both (the burn is never fitted across the last actuation). At most one extra notch.
  • Generic pace: one notch per provider per 20 min across operator Enforce spokes; bash pace may take its own notch for its namespaces in the same tick.
  • The operator does not read the bash PVC: hive-tiers' AA cache rows (tiers.tsv, anthropic/openai only) are not in the ModelLadder. They affect only agents placed on claude/codex rungs on school; add them to ModelLadder.spec.builtin (or set benchmarkURL) if school gains such agents before promotion.