Operations Runbook

For SREs running the Vertex endpoint, the Cloud Run GPU service and the gateway described in From Laptop to Production. Placeholders: <PROJECT>, <REGION>, <ENDPOINT_ID>, <your-dgem-gateway>.

Health signals

GET /health on the serving container (/invoke/health on Vertex) returns HTTP 200 once the container is up, with readiness details:

Field	Meaning
phase	mounting_gcs → staging_tmpfs → loading_vllm_siglip → ready
bytes_staged_gb	Weight copy progress (17.53 when done)
vllm_ready	The engine answers; decisions will succeed
warmed, warmup_s	The self-warmup (1-sample, 4-sample and image decisions) finished

The gateway exposes GET /health, GET /api/status?backend=vertex|cloudrun|local and GET /api/backend-config (live Vertex state: deployed, replica count, machine type), and the MCP tool get_health_and_gpu_status. Each backend’s readiness is tracked separately, so a Vertex probe never marks Cloud Run as warm.

Routine checks

• Vertex: availableReplicaCount ≥ 1 on the deployed model (gcloud ai endpoints describe <ENDPOINT_ID>), and a periodic /invoke/health (the gateway sends one every 4 minutes; keep the gateway at min 1 instance).
• Cloud Run GPU: scales to zero by design; verify it can wake with an occasional decision.
• Gateway: one decision per backend with X-DGem-Backend and check X-DGem-Backend-Used.
• Costs: only intended Vertex deployed models and GCE VMs exist (gcloud ai endpoints list, gcloud compute instances list).

Promote a new serving image

A serving image bundles the vLLM runtime and the structured server; small changes can alter answers or break the API. Validate every new image next to production, in the same session, before moving traffic. The procedure we used for the current image is in the image-parity run; it found four regressions that single-image testing would have missed.

1. Build with an immutable tag: GCP_PROJECT=<PROJECT> IMAGE_TAG=$(git rev-parse --short HEAD) ./scripts/build_cloudrun_image.sh.

2. Canaries: a second Vertex endpoint (VERTEX_ENDPOINT_NAME=dgemma-canary-<TAG> VERTEX_MAX_REPLICAS=1) and/or a Cloud Run service (CLOUDRUN_SERVICE_NAME=dgemma-canary) on the new image.

3. Compare with the regression matrix, tier T1, production as the baseline in the same session (tier T2 as well for a release or a vLLM change):

   scripts/bench_matrix.py run --tier T1 --target prod=<PRODUCTION_URL> --target new=<CANARY_URL> \
     --baseline prod --label image-<TAG>

   It covers the API contract (including multilingual cases with known answers), readiness, accuracy (calibration and JevBench ×3 on both prompt paths, intents) judged against the noise floor it measures, a multilingual spot check, and latency and load (modes, sweep at 16 and 32 workers, 0 errors). Promote only when every gate is PASS or each REVIEW is explained; a FAIL blocks. On Cloud Run also run scripts/coldstart_probe.py (cold start comparable to the current image): the matrix does not create revisions.

4. Switch without downtime: Vertex with VERTEX_NEW_TRAFFIC=0 then a traffic-split update (steps); Cloud Run and the gateway with a --no-traffic tagged revision then update-traffic.

5. Clean up: delete canaries; after a soak period, undeploy the old Vertex model (keep its model resource for rollback).

Per-item answers can shift a few percent between vLLM versions even when accuracy is unchanged: don’t compare individual answers across images in experiments.

When changing deploy/cloudrun/server/structured_server.py, keep it as vLLM’s upstream example plus dgem.patch: edit the file, regenerate the patch with diff -u against structured_server.upstream.py, and check it re-applies. The patch must keep the /health readiness fields, HTTP/1.1 keep-alive with backlog 256, MAX_INFLIGHT, DEFAULT_SAMPLES and the /predict / /rawPredict aliases.

Release a new version

Serving images, the gateway and the CLI share one version (vMAJOR.MINOR.PATCH, see CHANGELOG.md). A running service reports it: /health returns version and revision, images carry org.opencontainers.image.version, and dgem --version prints it.

1. Add a ## vX.Y.Z section to CHANGELOG.md, commit it on main and push (the release tag must point at a commit already on origin).
2. make release VERSION=vX.Y.Z: checks a clean main, runs the tests, tags git, builds dgem and dgem-weights with the version baked in, and adds the vX.Y.Z image tags. It prints the digests.
3. Validate the release image next to production (Promote a new serving image): matrix tiers T1 and T2.
4. make publish-latest VERSION=vX.Y.Z to move :latest, git push origin vX.Y.Z, and pin the new digests in Public container images.
5. Deploy to your own endpoints by digest; record the version in your deployment log.

If only gateway, MCP or CLI code changed (nothing under deploy/cloudrun/), use make release-gateway VERSION=vX.Y.Z instead of step 2: it tags the release without rebuilding the serving images, then redeploy the gateway from the tag. Serving endpoints keep the previous image, and the CHANGELOG says which serving image a release uses.

Builds between releases report git describe versions such as v0.1.0-3-gabc1234.

Roll back

Component	How	Time
Vertex (old model still deployed)	Move the traffic split back	Seconds
Vertex (old model undeployed)	Redeploy the old model resource with VERTEX_NEW_TRAFFIC=0, then move traffic	~15 min
Cloud Run GPU service / gateway	gcloud run services update-traffic <SERVICE> --to-revisions=<PREVIOUS>=100	Seconds (GPU service then cold-starts)

Tear down

• Vertex: GCP_PROJECT=<PROJECT> make vertex-teardown (or undeploy the model to keep the endpoint and its URL).
• Cloud Run: idle services cost nothing; gcloud run services delete <SERVICE> removes them.
• GCE benchmark VMs: make gce-teardown immediately after use.

Alerting

Alert policies, the scheduled health check, what each alert means and how to change thresholds: Monitoring and alerts. scripts/setup_cloud_monitoring.sh also creates an operational dashboard.

Incident playbooks

Symptom	Likely cause	Action
Vertex 429 ... scale-up from zero / “not yet ready”	Replica scaled to 0 or still starting	Check availableReplicaCount; the gateway fails over to Cloud Run meanwhile; confirm min replicas ≥ 1 and keepalive running
Many 503 server busy	Load above MAX_INFLIGHT for 30 s	Add replicas / raise max replicas; ask clients to back off
503 Model server early terminated the request	Serving image without HTTP keep-alive	Upgrade to an image with the keep-alive fix
All traffic on Cloud Run with multi-minute latencies	Vertex down and Cloud Run cold	Restore Vertex; Cloud Run warms in ~2.5 min
/health stuck in staging_tmpfs	Slow or blocked Cloud Storage access	Check Direct VPC egress and Private Google Access (Cloud Run §4)
Replica restarts during boot	Out of host memory	Larger machine / RTX PRO 6000 instance, or DISABLE_MM=1
IAP 401/403 for a user	Not in the allowed group	Add to <GROUP>