Skip to content
DiffusionGemma
dgem has moved to github.com/GoogleCloudPlatform/dgem. Docs are moving to googlecloudplatform.github.io/dgem; this site stays up during the move.

Path to Production: Serving dgem at Scale

This track takes dgem from your laptop to a production endpoint and keeps it running. Every number on these pages is measured (sample sizes and receipts are linked); identifiers such as <PROJECT>, <REGION> and <ENDPOINT_ID> are placeholders.

The journey

Section titled “The journey”
StepPageWhat you get
1Run on your laptopA first decision in minutes on Apple Silicon or a local NVIDIA GPU
2Use a remote GPUPoint the CLI, Studio and MCP at any hosted endpoint
3Deploy on Cloud RunYour own scale-to-zero GPU service: $0 when idle, ~2.5 min cold start
4Production on Vertex AIAn always-warm endpoint with autoscaling replicas and zero cold start
5Gateway and routingOne URL for users, automatic failover from Vertex to Cloud Run, IAP
6Latency and capacityRequest design, engine settings and how to measure your own
7Operations runbookHealth, promotion, rollback, teardown and troubleshooting
8Monitoring and alertsScheduled health check, alert policies, what to do when they fire

Deciding a whole dataset rather than one request at a time? To batch or not to batch compares a warm endpoint, Cloud Run Jobs, Dataflow and Vertex AI batch prediction, with measured cost per 1,000 decisions.

Hosting it yourself? Evaluate dgem on your own GPU covers GPU requirements, checking the install against known benchmark ranges, and a custom evaluation on your own labelled data.

Crawl → walk → run

Section titled “Crawl → walk → run”
StagePlatformUse it forCost modelCold startWarm latency, 1 sample (p50)
CrawlApple Silicon (diffgemma, Metal, q4) or a local NVIDIA GPUDevelopment, writing policies, offline and private dataYour machineNone~0.9 s on Metal (different engine; numbers don’t transfer to GPU serving)
WalkCloud Run GPU, 1× RTX PRO 6000Batch evaluation, research, bursty internal use, failoverPer instance-second; $0 when idle2.3–2.7 min (lean image + Direct VPC egress)61 ms GPU / 153 ms end to end
RunVertex AI dedicated endpoint, g4-standard-48 + 1× RTX PRO 6000Production: always warm, IAM, autoscaling, monitoring, imagesPer replica-hour while deployedNone (min replicas ≥ 1)55 ms GPU / 150 ms end to end

Latency: 3-question decision, 30 sequential requests, same-region client, serving image 504638d (receipts). The same image, policies, CLI, HTTP API and MCP server work on every stage.

TL;DR recommendations

Section titled “TL;DR recommendations”
  • GPU: use a Blackwell GPU (RTX PRO 6000). The checkpoint is NVFP4 (4-bit); Blackwell runs FP4 natively, and the L4 was about 3× slower on every request type we measured. Other GPUs are untested (requirements).
  • Production: a Vertex AI dedicated endpoint with minimum replicas ≥ 1, autoscaling on GPU duty cycle.
  • Failover: a scale-to-zero Cloud Run GPU service using the lean image and Direct VPC egress.
  • Bulk decisions: Cloud Run Jobs with the batch server settings above ~10,000 rows, or your warm endpoint if it has spare capacity (to batch or not to batch).
  • One gateway (dgem serve) in front of both, routing vertex_first with automatic failover.
  • Requests: "samples": 1 by default; opt in to 4 only when you need agreement/standard error; never "auto" on latency-sensitive paths. Put all questions about one input in one request.
  • Change safely: validate every new serving image next to production before switching traffic (runbook).

Method

Section titled “Method”