Deploy on Your Own Cloud GPU

A Cloud Run GPU service is the walk stage: your own endpoint that costs nothing while idle and wakes in about 2.5 minutes. It is also the failover tier behind a production Vertex endpoint; for whole datasets see To batch or not to batch. Everything below uses scripts/deploy_cloudrun_vllm.sh, which encodes the configuration we measured; placeholders are <PROJECT>, <REGION> and <GROUP>.

What you get

	Measured on 1× RTX PRO 6000 (receipts)
Warm latency, 3 questions, 1 sample	61 ms GPU / 153 ms end to end (p50)
Warm latency, 4 samples / image	101 / 76 ms GPU
Cold start (idle → warmed)	135–159 s (3 runs)
Cost while idle	$0 (scales to zero)

1. Prerequisites

A project with billing, gcloud logged in, and these APIs enabled: run.googleapis.com, artifactregistry.googleapis.com, cloudbuild.googleapis.com, storage.googleapis.com, secretmanager.googleapis.com, compute.googleapis.com.
Cloud Run GPU quota for RTX PRO 6000 in your region (new projects often have 0):
Terminal window
```
gcloud beta quotas info describe NvidiaRtxPro6000GpuAllocNoZonalRedundancyPerProjectRegion \
  --service=run.googleapis.com --project=<PROJECT> --format=json \
  | jq -r '.dimensionsInfos[] | select(.dimensions.region == "<REGION>") | .details.value'
```
Request an increase in the console (IAM & Admin → Quotas) if it is 0. If Cloud Run GPU quota is unavailable but Vertex AI serving quota exists, go straight to Production on Vertex AI.
A Google group (or user) that may call the service: <GROUP>.

2. Stage the weights in Cloud Storage

The service reads the 17.53 GiB NVFP4 checkpoint from a regional bucket in your project. The deploy script stages it automatically on first run; to do it explicitly:

GCP_PROJECT=<PROJECT> ./scripts/stage_model_gcs.sh      # creates gs://dgem-weights-<PROJECT>/dgemma/ via Cloud Build

3. (Optional) Hugging Face token in Secret Manager

The public checkpoint does not need a token. If you use a gated model or hit Hugging Face rate limits, store the token as a secret, never as a plain environment variable; the deploy script mounts dgemma-hf-token as HF_TOKEN when it exists:

printf %s "$HF_TOKEN" | gcloud secrets create dgemma-hf-token --project=<PROJECT> --data-file=-
gcloud secrets add-iam-policy-binding dgemma-hf-token --project=<PROJECT> \
  --member=serviceAccount:dgemma-gpu-sa@<PROJECT>.iam.gserviceaccount.com --role=roles/secretmanager.secretAccessor

Rotate by adding a new version and disabling the old one; new instances read :latest.

4. Networking: Direct VPC egress (the biggest cold-start win)

On a cold start the container copies the weights from Cloud Storage into memory with 64 parallel range requests while vLLM initialises. Over the default public egress that copy runs at ~46 MiB/s (~6.5 minutes); routed through a VPC subnet with Private Google Access it runs at ~410 MiB/s (64–83 s). The script enables it by default on the default network; make sure the subnet has Private Google Access:

gcloud compute networks subnets update default --region=<REGION> --enable-private-ip-google-access

With --vpc-egress=all-traffic and no Cloud NAT the service has no public internet access. The serving container only needs Cloud Storage and the metadata server; add Cloud NAT if you need the internet (for example to download weights from Hugging Face at boot). Use another network with CLOUDRUN_NETWORK/CLOUDRUN_SUBNET, or disable with CLOUDRUN_VPC_EGRESS=off.

5. Deploy

GCP_PROJECT=<PROJECT> GCP_REGION=<REGION> ALLOW_GROUP=<GROUP> \
CLOUDRUN_IMAGE=us-central1-docker.pkg.dev/dgem-diffusiongemma/dgem/dgem:v0.3.7@sha256:6153fe7079620d7ad608d8409ce55b295c362de808fce2fc28368cfb6b64c19f \
./scripts/deploy_cloudrun_vllm.sh

Use the lean image pinned by digest (current digests), or your own build (GCP_PROJECT=<PROJECT> IMAGE_TAG=$(git rev-parse --short HEAD) ./scripts/build_cloudrun_image.sh).

What the script configures:

Setting	Value	Why
GPU	1× `nvidia-rtx-pro-6000` (`CLOUDRUN_GPU_TYPE`), 20 vCPU, 80 GiB	Native NVFP4; room for the vision tower and the in-memory weight copy
Scaling	min 0 (`CLOUDRUN_MIN_INSTANCES`), max 1, concurrency 32	Scale to zero; one GPU per service
Identity	service account `dgemma-gpu-sa`, read-only on the weights bucket	Least privilege
Access	`--no-allow-unauthenticated`; invoker for `<GROUP>` and the gateway service account	Callers need an ID token
Weights	bucket mounted at `/mnt/gcs`, copied to memory at boot (`COPY_TO_SHM=1`)	Fast sequential load instead of random reads over FUSE
Network	Direct VPC egress, `default`/`default`, all traffic	5–6× faster weight copy
Serving	`DEFAULT_SAMPLES=1`, `MAX_INFLIGHT=8`, `DISABLE_MM=0`	Fast default reads; queue bursts instead of overloading the engine; images on
Startup	probe on `/health` for up to 10 minutes; self-warmup before serving	First real request doesn’t pay kernel warmup

The script injects the repository’s deploy/cloudrun/entrypoint.sh at deploy time, so entrypoint changes take effect on redeploy without rebuilding the image.

L4 instead of RTX PRO 6000: CLOUDRUN_GPU_TYPE=nvidia-l4 DISABLE_MM=1 (24 GB GPU, 32 GiB RAM, no vision tower, weights streamed from the bucket). It is about 3× slower per request.

6. Verify

URL=$(gcloud run services describe dgemma --project=<PROJECT> --region=<REGION> --format='value(status.url)')
TOKEN=$(gcloud auth print-identity-token)

curl -s -H "Authorization: Bearer $TOKEN" "$URL/health"
# during boot: {"phase": "staging_tmpfs", "bytes_staged_gb": ..., "vllm_ready": false}
# ready:       {"status": "ok", "vllm_ready": true, "phase": "ready", "warmed": true, "warmup_s": 14.6}

./bin/dgem decide -u "$URL/v1" --gcp-auth -t templates/support_triage.json.tmpl \
  -v 'ticket=Database connection pool exhausted' --stats

/health answers 200 as soon as the container is up; wait for vllm_ready and warmed before sending load.

7. Cold start: what to expect and your options

Seconds from a new instance starting (cold-start receipts):

Configuration	Weight copy	Idle → warmed	Extra cost
Lean image + Direct VPC egress (default)	64–83 s	2.3–2.7 min (4 runs)	None
Lean image, public egress	386–403 s	7.5–7.7 min	None
Weights baked into the image, first pull	none (image pull ~12.4 min)	14.6 min	None
Weights baked into the image, image cached	none	~3 min	None
Minimum 1 instance (`CLOUDRUN_MIN_INSTANCES=1`)	—	0	One GPU billed continuously
Vertex AI endpoint (min replicas ≥ 1)	—	0	Per replica-hour

The remaining ~2.5 minutes are vLLM start-up (overlapped with the copy), loading weights onto the GPU and building caches (~45 s), and the self-warmup (~15–20 s).
The baked-weights image does not help on Cloud Run: every new image or revision can pay the ~12-minute pull. Keep it for offline hosts.
To hide cold starts from users, put a gateway in front that routes to an always-warm Vertex endpoint first and uses this service for failover and batch work.

8. Update safely (blue/green) and roll back

# Deploy a new image as a tagged revision with no traffic
gcloud run services update dgemma --project=<PROJECT> --region=<REGION> \
  --image=<NEW_IMAGE> --no-traffic --tag=candidate

# Compare it with the serving revision, then switch
python3 scripts/contract_diff.py --target old=$URL --target new=https://candidate---<SERVICE_HOST>
gcloud run services update-traffic dgemma --project=<PROJECT> --region=<REGION> --to-tags=candidate=100

# Roll back
gcloud run services update-traffic dgemma --project=<PROJECT> --region=<REGION> --to-revisions=<PREVIOUS_REVISION>=100

For a new serving image, run the full comparison first: Promote a new serving image.

9. Troubleshooting

Symptom	Cause	Fix
`429` / connection refused right after deploy or idle	Instance still loading (vLLM not ready)	Poll `/health` until `vllm_ready`; retry with backoff (`--http-retries 3`)
Container restarts ~10 minutes into boot	Out of memory (vision tower + in-memory weights on a 32 GiB instance)	Use RTX PRO 6000 (80 GiB), or `DISABLE_MM=1` on L4
Weight copy takes 6+ minutes	Public egress	Enable Direct VPC egress with Private Google Access (§4)
Weights download from Hugging Face fails	No internet with all-traffic VPC egress and no NAT	Stage weights in Cloud Storage (§2) or add Cloud NAT
`503 server busy`	More than `MAX_INFLIGHT` decisions queued for 30 s	Reduce client concurrency, raise `MAX_INFLIGHT`, or add capacity (Vertex replicas)
`403` from the service	Caller lacks `run.invoker` or sends no ID token	Add the caller to `<GROUP>`; use `--gcp-auth`
Deploy fails with GPU quota error	No Cloud Run GPU quota in the region	Request quota, or use Vertex AI

10. Tear down

gcloud run services delete dgemma --project=<PROJECT> --region=<REGION> --quiet

An idle service already costs nothing; delete it when you no longer need the URL. The weights bucket bills for storage (~18 GB) until deleted.

Production on Vertex AI for zero cold start, or Gateway and routing to put one URL in front of both.