Scion Hub Bootstrap, Upgrade & Reconciliation (`scion-hub-setup`)

This skill provides an end-to-end operational guide and idempotent Python reconciliation script for standing up, upgrading, or reconciling a self-hosted Scion Hub instance (combo Hub + Runtime Broker + Web UI) on Google Cloud Compute Engine (GCE).

---

1. Prerequisites

1. gcloud CLI authenticated with access to the target GCP project and GCE instance.
2. Passphrase-less SSH key for non-interactive GCE automation: If ~/.ssh/google_compute_engine is passphrase-protected, generate an ephemeral key so non-interactive gcloud compute ssh commands do not hang on passphrase prompts:

   ssh-keygen -t ed25519 -f /tmp/gce_tmp_key -N "" -q

3. Local checkout of GoogleCloudPlatform/scion (./scripts/starter-hub/gce-start-hub.sh).

---

2. Helper Scripts

• scripts/hub-reconcile.py — Idempotent Python reconciliation script that executes on the Hub VM (or via SSH stdin) to:
  1. Ensure the bootstrap admin user exists in ~/.scion/hub.db and holds the super-admin system role binding (role_bindings).
  2. Deterministically derive the HS256 user_signing_key from SCION_SERVER_SESSION_SECRET (~/.scion/hub.env) and mint a 365-day CLI JWT token.
  3. Configure server-config operational defaults (default_harness_config, default_template).
  4. Create the per-broker max_agents_per_broker entitlement binding (e.g. 100 or 200, overriding the upstream default ceiling of 12).
  5. Set Hub-scoped Vertex AI global region environment variables (GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_REGION=global, GOOGLE_CLOUD_LOCATION=global, CLOUD_ML_REGION=global, ANTHROPIC_VERTEX_PROJECT_ID).
  6. Update global harness configs (claude → claude-opus-5-5@default, antigravity → Gemini 3.8 Flash (Medium), opencode), write their config.yaml files, and optionally prune unused seeded harness configs.
  7. Batch-import all 13 global agent-team templates (coordinator, eng-manager, developer, code-reviewer, architect, investigator, qa-tester, test-engineer, doc-writer, researcher, security-auditor, release-notes, web-builder).

---

3. End-to-End Reconciliation Workflow

Phase 1: Hub Server Upgrade & Optional Clean Database Reset

1. Back up the existing SQLite database and point the VM's checkout to upstream:

   CLOUDSDK_CORE_PROJECT=<gcp-project> gcloud compute ssh scion@<vm-name> \
     --zone=<zone> --ssh-key-file=/tmp/gce_tmp_key --command='
       cp ~/.scion/hub.db ~/.scion/hub.db.bak-$(date +%Y%m%d) || true
       cd /home/scion/scion && git remote set-url origin https://github.com/GoogleCloudPlatform/scion.git
     '

2. Deploy the latest main build (add --reset-db only when wiping an unused/drifted database):

   CLOUDSDK_CORE_PROJECT=<gcp-project> PROJECT_ID=<gcp-project> \
     HUB_NAME=<hub-short-name> CERT_DOMAIN=<hub-domain> HUB_DOMAIN=<hub-domain> \
     ./scripts/starter-hub/gce-start-hub.sh --full --reset-db --branch main

---

Phase 2: Build & Push Harness Container Images

Build core-base, scion-base, and the desired agent harnesses (claude, antigravity, opencode) directly on the GCE VM and push to Artifact Registry:

CLOUDSDK_CORE_PROJECT=<gcp-project> gcloud compute ssh scion@<vm-name> \
  --zone=<zone> --ssh-key-file=/tmp/gce_tmp_key --command='
    cd /home/scion/scion
    for target in core-base scion-base claude antigravity opencode; do
      ./image-build/scripts/build-images.sh \
        --registry us-central1-docker.pkg.dev/<gcp-project>/scion \
        --target "$target" \
        --push
    done
  '

---

Phase 3: Run Idempotent Hub Reconciliation (hub-reconcile.py)

Pipe scripts/hub-reconcile.py over SSH to bootstrap the admin user, derive the CLI token, configure quotas, environment variables, harness configs, and global agent-team templates:

CLOUDSDK_CORE_PROJECT=<gcp-project> gcloud compute ssh scion@<vm-name> \
  --zone=<zone> --ssh-key-file=/tmp/gce_tmp_key \
  --command='python3 - --admin-email ghchinoy@gmail.com --gcp-project <gcp-project> --registry us-central1-docker.pkg.dev/<gcp-project>/scion --broker-ceiling 100 --prune-unused-harnesses' \
  < ./scripts/hub-reconcile.py

Save the emitted SCION_ADMIN_TOKEN=<jwt> in ~/.scion/credentials.json under both "https://<hub-domain>" and "https://<hub-domain>/".

---

Phase 4: Verify Agent Service Account IAM Bindings

Ensure the Hub VM's service account (<vm-sa>@<gcp-project>.iam.gserviceaccount.com) holds roles/iam.serviceAccountTokenCreator on each agent runner service account so the Hub can mint GCP tokens for agent containers:

for SA in scion-agent-runner@<gcp-project>.iam.gserviceaccount.com sa-scion-warmup@<gcp-project>.iam.gserviceaccount.com; do
  gcloud iam service-accounts add-iam-policy-binding "$SA" \
    --project="<gcp-project>" \
    --member="serviceAccount:<vm-sa>@<gcp-project>.iam.gserviceaccount.com" \
    --role="roles/iam.serviceAccountTokenCreator" \
    --condition=None \
    --quiet
done