diff --git a/.claude/skills/backend-seeding/SKILL.md b/.claude/skills/backend-seeding/SKILL.md index fff24847..9eaff362 100644 --- a/.claude/skills/backend-seeding/SKILL.md +++ b/.claude/skills/backend-seeding/SKILL.md @@ -5,8 +5,8 @@ description: or changing hackathons, users, projects, teams, submissions, phases and votes through the in-process gRPC harness, the step helpers in steps.go, capabilities that must be on when they are used, synthetic users who cannot - log in, idempotency and re-seeding (locally and on the dev cluster), and how - to verify a change without wiping local state. Use when asked to add or change + log in, idempotency and re-seeding (locally and on dev or staging), and how to + verify a change without wiping local state. Use when asked to add or change dev/test data, make a scenario reproducible locally, pick which seeded hackathon to test something in, or fix a seeded hackathon nobody can act in. --- @@ -151,9 +151,12 @@ To use the new fixture locally for real: `just clean::state && just start`, policy before the seed wrote any, and until it restarts, owners are refused their own hackathons. -On the dev cluster, `just cluster::reseed ` backs up, wipes the app -database, seeds from your checkout and restarts the backend. It refuses unless -`components/backend` matches `origin/main`, so it runs after merging. +On the deployed instances, `just cluster::reseed dev` or +`just cluster::reseed staging` backs up, wipes the app database, seeds and +restarts the backend. The seed is built from the code the instance serves — +`origin/main` for dev, the newest `origin/release/*` for staging — not from your +checkout, so a seed change reaches dev only after merging and staging only once +it is on a release branch. ## After changing the seed diff --git a/CLAUDE.md b/CLAUDE.md index bdce84dc..5f0e3a3e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -113,10 +113,10 @@ just helm::template [args] # render the chart to stdout just helm::check-bump # fail if helm-chart/ changed without a version bump just helm::publish # push the chart if Chart.yaml's version is unpublished -# Deployed dev instance only — takes the kube context, refuses any other cluster -just cluster::backup # pg_dumpall to ~/hackagon-backups -just cluster::wipe # backup, then drop + recreate the app DB; dev restarts empty -just cluster::reseed # backup, wipe, then seed; refuses if components/backend differs from origin/main +# Deployed dev or staging only — set HACKAGON_KUBECONFIG to the cluster's kubeconfig first +just cluster::backup # pg_dumpall to ~/hackagon-backups +just cluster::wipe # backup, then drop + recreate the app DB; the instance restarts empty +just cluster::reseed # backup, wipe, then seed built from origin/main (dev) or the newest origin/release/* (staging) ``` Backend listens on **:3000**, frontend on **:8081**. Dev users (Keycloak @@ -169,10 +169,12 @@ These hold across the whole codebase; the skills explain the mechanisms. focused pull request per change, each carrying its manually added `CHANGELOG.md` entry. Each released minor line has a `release/X.Y` branch that takes hotfixes only, released from its own tags and cherry-picked back to - `main`. Both clusters install the same chart from the GitOps setup in - `sdsc-ordes/cloud-infra` and differ only in image overrides: **dev** runs - `temporary/*:latest` with `pullPolicy: Always`, so every push to `main` - reaches it on the next container restart; **prod** takes the chart defaults, + `main`. All three instances install the same chart from the GitOps setup in + `sdsc-ordes/cloud-infra` and differ only in image overrides: **dev** + (namespace `hackagon-dev`) runs `temporary/*:latest` with + `pullPolicy: Always`, so every push to `main` reaches it on the next container + restart; **staging** (namespace `hackagon-staging`, same cluster as dev) runs + the latest release, hotfixes included; **prod** takes the chart defaults, `release/*` at `appVersion`, so nothing merged can reach it until someone bumps `appVersion` and installs the new chart version. See `RELEASING.md`. - **The chart versions itself; `VERSION` never touches it.** diff --git a/RELEASING.md b/RELEASING.md index eba8cb06..7e9efe6c 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -15,7 +15,7 @@ main ──●──●──●──●──●──●──●──●─ │ v0.12.1 + its chart bump │ - └── release/0.12 ──●── v0.12.2 ──●── v0.12.3 hotfixes only; prod runs these tags + └── release/0.12 ──●── v0.12.2 ──●── v0.12.3 hotfixes only; staging and prod run these tags ▲ hotfix/… PRs ``` @@ -44,14 +44,15 @@ itself builds nothing; only its tags do. ## Deployment -Two clusters, both installed from the Helm chart by the GitOps setup in -`sdsc-ordes/cloud-infra` — a base `values.yaml` plus per-cluster overrides. The +Three instances, all installed from the Helm chart by the GitOps setup in +`sdsc-ordes/cloud-infra` — a base `values.yaml` plus per-instance overrides. The only thing that differs between them is which images they pull. -| Cluster | Images | Moves when | -| ------- | -------------------------------------------- | -------------------------------- | -| dev | `temporary/*:latest`, `pullPolicy: Always` | anything is merged to `main` | -| prod | chart defaults — `release/*` at `appVersion` | a new chart version is installed | +| Instance | Images | Moves when | +| -------- | -------------------------------------------- | -------------------------------- | +| dev | `temporary/*:latest`, `pullPolicy: Always` | anything is merged to `main` | +| staging | the latest release, hotfixes included | a new release or hotfix is out | +| prod | chart defaults — `release/*` at `appVersion` | a new chart version is installed | ### dev follows `main` @@ -60,13 +61,6 @@ merge to main ──► CI rebuilds temporary/*:latest ──► next container serves the new code ``` -A restart is not always something you chose: an eviction or a node drain -restarts the container too. To restart it on purpose: - -```bash -kubectl rollout restart deploy/hackagon-frontend -``` - To see which build it is serving, read the version at the bottom of the page. `pullPolicy: Always` is what makes this work. The chart defaults to diff --git a/components/backend/cmd/seed/README.md b/components/backend/cmd/seed/README.md index 69e41c2d..3f817db1 100644 --- a/components/backend/cmd/seed/README.md +++ b/components/backend/cmd/seed/README.md @@ -53,10 +53,10 @@ cannot log in — see [Users](#users). ## When to re-seed **Nothing re-seeds automatically.** The backend never runs `cmd/seed`: a deploy, -a restart or a new image only migrates the schema and keeps every row. So the -dev cluster holds whatever the last manual re-seed wrote, plus everything people -have done to it since; production is never seeded at all. CI does run the seed, -but against a throwaway stack, to prove it still works — nothing it writes +a restart or a new image only migrates the schema and keeps every row. So dev +and staging hold whatever the last manual re-seed wrote, plus everything people +have done to them since; production is never seeded at all. CI does run the +seed, but against a throwaway stack, to prove it still works — nothing it writes survives the job. Re-seed when one of these is true: @@ -95,10 +95,12 @@ just deploy::down && just deploy::up # reload the casbin policy, see below # Or all of that at once, regenerating code too just schema-change -# The dev cluster — backs up, wipes the app database (Keycloak is kept), -# seeds from your checkout and restarts the backend. Refuses unless -# components/backend matches origin/main. -just cluster::reseed +# Dev or staging — backs up, wipes the app database (Keycloak is kept), seeds +# and restarts the backend. The seed is built from the code the instance: +# origin/main for dev, the newest origin/release/* for staging, +# needs HACKAGON_KUBECONFIG to be set. +just cluster::reseed dev +just cluster::reseed staging ``` ## Running the seed diff --git a/tools/just/cluster.just b/tools/just/cluster.just index fe082d98..e18a04ef 100644 --- a/tools/just/cluster.just +++ b/tools/just/cluster.just @@ -4,7 +4,6 @@ set shell := ["bash", "-cue"] root_dir := `git rev-parse --show-toplevel` # Names the chart gives a release called `hackagon` (see helm-chart/templates). -namespace := "hackagon" backend_deploy := "hackagon-backend" backend_config := "hackagon-backend-config" backend_db_secret := "hackagon-backend-db" @@ -14,13 +13,28 @@ postgres_svc := "hackagon-postgresql" app_db := "hackagon" app_db_owner := "hackagon" -# Refuse any cluster whose ingress does not serve this domain, so a recipe -# pointed at the prod context by mistake stops before touching anything. -dev_domain := "hackagon-dev.dscompute.ch" - backup_dir := env("HOME") + "/hackagon-backups" local_pg_port := "15432" +# Dev and staging share one cluster, reached through the kubeconfig that +# HACKAGON_KUBECONFIG points at, and its current context. No default: where +# that file lives is each operator's business, and _guard refuses when unset. +kubeconfig := env("HACKAGON_KUBECONFIG", "") + +# Sourced by every recipe: `resolve_instance ` sets the namespace and the +# host its ingress must serve. Only these two names exist, so a recipe can never +# be pointed at prod, and the host check catches a kubeconfig that reaches some +# other cluster. +instances := ''' +resolve_instance() { + case "$1" in + dev) ns=hackagon-dev; host=hackagon-dev.sck-sit-dev.dscompute.ch ;; + staging) ns=hackagon-staging; host=hackagon-staging.sck-sit-dev.dscompute.ch ;; + *) echo "✗ Unknown instance '$1': expected dev or staging." >&2; exit 1 ;; + esac +} +''' + # Runs inside the postgres pod: finds the superuser password the bitnami image # was started with (env var or mounted file) so it never leaves the pod. pg_as_superuser := ''' @@ -45,53 +59,59 @@ help: reset="\033[0m" echo "" - echo -e " ${bold}Dev Cluster Commands${reset}" - echo -e " ${dim}Act on the deployed dev instance. Every recipe takes the kube context${reset}" - echo -e " ${dim}explicitly and refuses a cluster not serving {{dev_domain}}.${reset}" + echo -e " ${bold}Cluster Commands${reset}" + echo -e " ${dim}Act on a deployed instance — dev or staging, never prod — through the${reset}" + echo -e " ${dim}kubeconfig HACKAGON_KUBECONFIG points at.${reset}" + echo -e " ${dim}Every recipe refuses a namespace whose ingress does not serve that instance's host.${reset}" echo "" - echo -e " ${cyan}just cluster::backup${reset} " - echo -e " ${dim}pg_dumpall the dev postgres to {{backup_dir}}${reset}" + echo -e " ${cyan}just cluster::backup${reset} " + echo -e " ${dim}pg_dumpall the instance's postgres to {{backup_dir}}${reset}" echo "" - echo -e " ${cyan}just cluster::wipe${reset} " + echo -e " ${cyan}just cluster::wipe${reset} " echo -e " ${dim}Back up, then drop and recreate the app database. Keycloak is untouched.${reset}" - echo -e " ${dim}The backend recreates the schema on startup, so dev comes back empty.${reset}" + echo -e " ${dim}The backend recreates the schema on startup, so the instance comes back empty.${reset}" echo "" - echo -e " ${cyan}just cluster::reseed${reset} " - echo -e " ${dim}Back up, wipe, then run cmd/seed (backend code must match origin/main).${reset}" + echo -e " ${cyan}just cluster::reseed${reset} " + echo -e " ${dim}Back up, wipe, then run cmd/seed built from the code the instance serves —${reset}" + echo -e " ${dim}origin/main for dev, the newest origin/release/* for staging — whatever is checked out.${reset}" echo "" - echo -e " ${dim}e.g. just cluster::wipe sck-sit-dev${reset}" + echo -e " ${dim}e.g. just cluster::reseed staging${reset}" echo "" -# pg_dumpall the dev postgres (app and keycloak databases) to ~/hackagon-backups. +# pg_dumpall the instance's postgres (app and keycloak databases) to ~/hackagon-backups. [group('cluster')] -backup context: +backup instance: #!/usr/bin/env bash set -euo pipefail - just -f "{{source_file()}}" _guard "{{context}}" + {{instances}} + resolve_instance "{{instance}}" + just -f "{{source_file()}}" _guard "{{instance}}" mkdir -p "{{backup_dir}}" - out="{{backup_dir}}/{{context}}-$(date +%Y%m%d-%H%M%S).sql" + out="{{backup_dir}}/$ns-$(date +%Y%m%d-%H%M%S).sql" discard_partial() { rm -f "$out.partial"; } trap discard_partial EXIT echo "==> Backing up to $out" - kubectl --context "{{context}}" -n "{{namespace}}" exec "sts/{{postgres_sts}}" -- \ + kubectl --kubeconfig "{{kubeconfig}}" -n "$ns" exec "sts/{{postgres_sts}}" -- \ sh -c '{{pg_as_superuser}}' sh pg_dumpall -h 127.0.0.1 -U postgres > "$out.partial" mv "$out.partial" "$out" echo "✓ $(du -h "$out" | cut -f1) written" -# Back up, then drop and recreate the app database; dev restarts empty. +# Back up, then drop and recreate the app database; the instance restarts empty. [group('cluster')] -wipe context: +wipe instance: #!/usr/bin/env bash set -euo pipefail - just -f "{{source_file()}}" _confirm "{{context}}" "WIPE the app database" - just -f "{{source_file()}}" backup "{{context}}" + {{instances}} + resolve_instance "{{instance}}" + just -f "{{source_file()}}" _confirm "{{instance}}" "WIPE the app database" + just -f "{{source_file()}}" backup "{{instance}}" - kc=(kubectl --context "{{context}}" -n "{{namespace}}") + kc=(kubectl --kubeconfig "{{kubeconfig}}" -n "$ns") # A backend found at 0 (e.g. left there by an interrupted run) comes back - # at 1: both recipes promise a running dev, and wipe needs it to recreate - # the schema. + # at 1: both recipes promise a running instance, and wipe needs it to + # recreate the schema. replicas="$("${kc[@]}" get "deploy/{{backend_deploy}}" -o jsonpath='{.spec.replicas}')" [ "${replicas:-0}" -gt 0 ] || replicas=1 @@ -103,20 +123,22 @@ wipe context: # however this ends, including when stopping it or the wipe fails halfway. trap restore_backend EXIT - just -f "{{source_file()}}" _backend-down "{{context}}" - just -f "{{source_file()}}" _wipe-db "{{context}}" + just -f "{{source_file()}}" _backend-down "{{instance}}" + just -f "{{source_file()}}" _wipe-db "{{instance}}" -# Back up, wipe, then seed the dev fixture from this checkout. +# Back up, wipe, then seed the dev fixture, built from the code the instance serves. [group('cluster')] -reseed context: +reseed instance: #!/usr/bin/env bash set -euo pipefail + {{instances}} + resolve_instance "{{instance}}" cd "{{root_dir}}" - just -f "{{source_file()}}" _backend-matches-main - just -f "{{source_file()}}" _guard "{{context}}" + ref="$(just -f "{{source_file()}}" _seed-ref "{{instance}}")" + just -f "{{source_file()}}" _guard "{{instance}}" - kc=(kubectl --context "{{context}}" -n "{{namespace}}") + kc=(kubectl --kubeconfig "{{kubeconfig}}" -n "$ns") tmp="$(mktemp -d)" pf_pid="" backend_down="" @@ -126,19 +148,23 @@ reseed context: echo "==> Scaling backend back to $replicas" "${kc[@]}" scale "deploy/{{backend_deploy}}" --replicas="$replicas" } - # However this ends: close the tunnel, delete the temp dir, and bring the - # backend back if it was taken down. + # However this ends: close the tunnel, delete the temp dir and the worktree + # in it, and bring the backend back if it was taken down. cleanup() { [ -n "$pf_pid" ] && kill "$pf_pid" 2>/dev/null || true rm -rf "$tmp" + git -C "{{root_dir}}" worktree prune [ -z "$backend_down" ] || restore_backend } trap cleanup EXIT # Everything that can fail without changing the cluster happens first, so - # a failure here leaves dev exactly as it was. - echo "==> Building the seed" - (cd components/backend && go build -o "$tmp/seed" ./cmd/seed/) + # a failure here leaves the instance exactly as it was. + # The seed runs the backend's handlers in-process and creates the schema, so + # it is built from the code the instance serves, not from this checkout. + echo "==> Building the seed from $ref" + git worktree add --quiet --detach "$tmp/src" "$ref" + (cd "$tmp/src/components/backend" && go build -o "$tmp/seed" ./cmd/seed/) echo "==> Fetching backend config and database password from the cluster" "${kc[@]}" get cm "{{backend_config}}" -o jsonpath='{.data.config\.yaml}' > "$tmp/config.yaml" @@ -150,17 +176,17 @@ reseed context: replicas="$("${kc[@]}" get "deploy/{{backend_deploy}}" -o jsonpath='{.spec.replicas}')" [ "${replicas:-0}" -gt 0 ] || replicas=1 - just -f "{{source_file()}}" _confirm "{{context}}" "WIPE and RESEED the app database" - just -f "{{source_file()}}" backup "{{context}}" + just -f "{{source_file()}}" _confirm "{{instance}}" "WIPE and RESEED the app database" + just -f "{{source_file()}}" backup "{{instance}}" # The backend stays off until the seed is done: started on the empty # database it would create the tables at the same time as the seed, and it # would load its casbin policy before the seed has written any. backend_down=yes - just -f "{{source_file()}}" _backend-down "{{context}}" + just -f "{{source_file()}}" _backend-down "{{instance}}" # Wipe the app database, leaving keycloak untouched. The seed creates the schema. - just -f "{{source_file()}}" _wipe-db "{{context}}" + just -f "{{source_file()}}" _wipe-db "{{instance}}" echo "==> Port-forwarding postgres to localhost:{{local_pg_port}}" "${kc[@]}" port-forward "svc/{{postgres_svc}}" "{{local_pg_port}}:5432" >/dev/null & @@ -175,7 +201,7 @@ reseed context: || { echo "✗ The port-forward to postgres did not open within 10s." >&2; exit 1; } echo "==> Seeding" - cd components/backend + cd "$tmp/src/components/backend" HACKAGON_DATABASE_HOST=127.0.0.1 \ HACKAGON_DATABASE_PORT="{{local_pg_port}}" \ HACKAGON_DATABASE_PASSWORD="$db_password" \ @@ -186,13 +212,15 @@ reseed context: # the caller's job, in an EXIT trap set before calling, so it happens even when # this or a later step fails halfway. [private] -_backend-down context: +_backend-down instance: #!/usr/bin/env bash set -euo pipefail + {{instances}} + resolve_instance "{{instance}}" - # Guard the context, so a recipe pointed at the prod context by mistake stops before touching anything. - just -f "{{source_file()}}" _guard "{{context}}" - kc=(kubectl --context "{{context}}" -n "{{namespace}}") + # Guard the instance, so a kubeconfig reaching the wrong cluster stops before touching anything. + just -f "{{source_file()}}" _guard "{{instance}}" + kc=(kubectl --kubeconfig "{{kubeconfig}}" -n "$ns") backend_pods() { "${kc[@]}" get pods -o name | grep "{{backend_deploy}}-" || true; } @@ -210,13 +238,15 @@ _backend-down context: # Drop and recreate the app database and run _backend-down first, # so the backend does not try to connect to it while it is being replaced. [private] -_wipe-db context: +_wipe-db instance: #!/usr/bin/env bash set -euo pipefail + {{instances}} + resolve_instance "{{instance}}" - # Guard the context, so a recipe pointed at the prod context by mistake stops before touching anything. - just -f "{{source_file()}}" _guard "{{context}}" - kc=(kubectl --context "{{context}}" -n "{{namespace}}") + # Guard the instance, so a kubeconfig reaching the wrong cluster stops before touching anything. + just -f "{{source_file()}}" _guard "{{instance}}" + kc=(kubectl --kubeconfig "{{kubeconfig}}" -n "$ns") echo "==> Recreating database {{app_db}}" "${kc[@]}" exec "sts/{{postgres_sts}}" -- sh -c '{{pg_as_superuser}}' sh \ @@ -225,47 +255,57 @@ _wipe-db context: -c 'CREATE DATABASE {{app_db}} OWNER {{app_db_owner}};' echo "✓ Database {{app_db}} is empty" -# Refuse unless this checkout's backend is the code dev serves. +# Fetch and print the ref whose code the instance serves: dev follows +# origin/main, staging the newest origin/release/* branch. [private] -_backend-matches-main: +_seed-ref instance: #!/usr/bin/env bash set -euo pipefail cd "{{root_dir}}" - git fetch --quiet origin main - if ! git merge-base --is-ancestor origin/main HEAD; then - echo "✗ This checkout is behind origin/main. Rebase or merge main first." >&2 - exit 1 - fi - if ! git diff --quiet origin/main -- components/backend; then - echo "✗ components/backend differs from origin/main, which dev is serving." >&2 - exit 1 - fi - if [ -n "$(git status --porcelain components/backend)" ]; then - echo "✗ components/backend has uncommitted changes." >&2 - exit 1 - fi - echo "✓ components/backend matches origin/main" - -# Refuse a context whose hackagon ingress does not serve the dev domain. + case "{{instance}}" in + dev) + git fetch --quiet origin main + ref=origin/main + ;; + staging) + git fetch --quiet --prune origin '+refs/heads/release/*:refs/remotes/origin/release/*' + ref="$(git for-each-ref --sort=-version:refname --count=1 \ + --format='%(refname:short)' refs/remotes/origin/release/)" + [ -n "$ref" ] || { echo "✗ origin has no release/* branch." >&2; exit 1; } + ;; + *) echo "✗ Unknown instance '{{instance}}': expected dev or staging." >&2; exit 1 ;; + esac + echo "$ref" + +# Refuse unless the kubeconfig reaches a namespace whose ingress serves the instance's host. [private] -_guard context: +_guard instance: #!/usr/bin/env bash set -euo pipefail - hosts="$(kubectl --context "{{context}}" -n "{{namespace}}" get ingress \ + {{instances}} + resolve_instance "{{instance}}" + [ -n "{{kubeconfig}}" ] \ + || { echo "✗ HACKAGON_KUBECONFIG is not set. Point it at the kubeconfig for dev and staging." >&2; exit 1; } + [ -f "{{kubeconfig}}" ] \ + || { echo "✗ HACKAGON_KUBECONFIG points at {{kubeconfig}}, which is not a file." >&2; exit 1; } + hosts="$(kubectl --kubeconfig "{{kubeconfig}}" -n "$ns" get ingress \ -o jsonpath='{.items[*].spec.rules[*].host}')" - if [[ " $hosts " != *"{{dev_domain}}"* ]]; then - echo "✗ Context '{{context}}' does not serve {{dev_domain}} (found: ${hosts:-none})." >&2 - echo " Refusing — this does not look like the dev instance." >&2 + if [[ " $hosts " != *" $host "* ]]; then + echo "✗ Namespace $ns via {{kubeconfig}} does not serve $host (found: ${hosts:-none})." >&2 + echo " Refusing — this does not look like the {{instance}} instance." >&2 exit 1 fi -# Guard, then make the operator type the context name back. +# Guard, then make the operator type the namespace back, which names the +# instance; dev and staging share a cluster. [private] -_confirm context action: +_confirm instance action: #!/usr/bin/env bash set -euo pipefail - just -f "{{source_file()}}" _guard "{{context}}" - echo "About to {{action}} on context '{{context}}' (namespace {{namespace}})." + {{instances}} + resolve_instance "{{instance}}" + just -f "{{source_file()}}" _guard "{{instance}}" + echo "About to {{action}} in namespace $ns (via {{kubeconfig}})." echo "Everything in the app database is lost; a backup is taken first." - read -r -p "Type the context name to continue: " answer - [ "$answer" = "{{context}}" ] || { echo "Aborted."; exit 1; } + read -r -p "Type the namespace to continue: " answer + [ "$answer" = "$ns" ] || { echo "Aborted."; exit 1; }