EXP-23: dgem vs Strands Decider 2B (Model Comparison)
The first comparison run under the standard process in
Comparing dgem with Another Decision Model. Evidence level E2 (lab head-to-head)
on every suite, E3 on the frozen sets, and E1 for Strandsâ own published JevBench run.
dgem is more accurate on general decisions. The two are level on tasks Strands was trained on.
JevBench public (231 items, 3 runs): dgem 194 / 194 / 195, Strands 169 / 169 / 169 (McNemar p = 3.5e-4). On the
hard tier (111 items) dgem scores 67% and Strands 50%.
On items from datasets in Strandsâ training mix (n = 142): Strands 85.9%, dgem 85.2%.
On items with no known relation to its training data (n = 2,346): dgem 72.5%, Strands 62.7%.
Frozen sets (one scored run each): MASSIVE (51 languages) dgem 82.8% vs 67.2%; XNLI dgem 70.0% vs 63.7%;
typed-decisions dgem 71.8% vs 60.2%.
The comparison found a dgem defect.
Two yes/no questions in one request whose correct answers have opposite polarity tend to get the same answer.
On a 51-case tool-call gate built from Strandsâ own agent example, dgemâs two answers matched in 34â40 cases (gold: 7) and it scored 58%, against 86% for Strands.
Asked one question per request, dgem scores 100 of 102.
Now PROP-19 and the matrix v2 suite gate_mixed_noul.
Calibration: Strands is cautious, dgem is sharp but overconfident off its development suites.
On JevBench, Strands has the lower ECE (0.048 vs 0.071). dgem has the lower Brier score (0.224 vs 0.348) and the higher AUROC (0.877 vs 0.806).
With one held-out temperature (T â 1.5), dgemâs pooled ECE falls from 0.069 to 0.028. Strandsâ falls from 0.037 to 0.031.
Out of domain, Strands rarely reaches 0.9 confidence (XNLI 0.1% of answers), while raw dgem is 74% correct at â„ 0.9 on XNLI. See PROP-20.
Latency: Strands is faster on short inputs, dgem on long inputs, many questions and under load.
Server p50 for one question over about 200 tokens: Strands 34 ms, dgem 54 ms.
Over about 3,800 tokens: Strands 118 ms, dgem 62 ms. The crossover is about 1,700 tokens.
Peak throughput with 5 questions per request: dgem 321 decisions/s, Strands 66.
Determinism and cold start favour Strands.
Strands returned identical probabilities on all 1,509 repeated requests. dgem changed 6.4% of answers.
Strands cold-starts in 38 s on Cloud Run.
Strandsâ own numbers reproduce.
The official JevBench harness at the commit Strands pinned matches its published per-task predictions on 229 of 231 tasks (169 correct vs 167 published).
Rank-16 LoRA plus a ~1M-parameter pointer head on 21 public short-task datasets plus multi-step and generated data
None (zero-shot, policy templates)
Readout
Pointer head; each question scored independently over a shared state prefix
Letter readout per slot; all questions read jointly on one canvas
Option limit
255
26 natively; more through dgem systemone serve bracket routing
Calibration as served
Temperature per question type, fitted on its own held-out short tasks
Raw restricted-softmax probabilities
Hardware here
Cloud Run, 1Ă RTX PRO 6000
Vertex AI G4, 1Ă RTX PRO 6000
Same requests: both models received the same /v1/systemone bodies.
Same scoring: one scorer recomputes every metric from the returned distributions. We never use the serversâ
confidence fields.
Runs: core suites ran 3 times per model, in parallel. The frozen sets ran once on Strands (it is deterministic)
and twice on dgem.
Latency client: in the same region, with keep-alive HTTPS.
Preflight finding. The stock strands-decider serve returned different answers under concurrent requests: one
item moved by 0.82 probability, while serial runs were identical. All Strands results therefore come from the stock
server plus a lock around the engineâs evaluate(); the lock changes no numerics. This is why the process now
starts with scripts/compare/preflight.py.
dgem wants T â 1.5 overall and a larger temperature for yes/no questions (1.6â1.9 in pool A, about 3 with the gate
suite included).
Strandsâ shipped temperatures transfer well to JevBench. A per-bucket refit still helps it on long option lists
(T â 1.5) and score items (T = 0.5).
5. The tool-call gate and dgemâs cross-slot coupling
The gate suite generalizes Strandsâ agent example (examples/strands/tool_call_intervention.py) to 51 authored
cases, using their exact question wording, true/false criteria and state rendering. There are three kinds of case:
Kind
Meaning
args_grounded
premature
G
grounded and ready to call
yes
no
U
the tool argument was guessed
no
yes
A
grounded, but something is left to confirm
yes
yes
Mean correct of 3 runs
Strands
dgem
args_grounded G (21) / U (23) / A (7)
20 / 23 / 3
10.3 / 23 / 3.7
premature G (21) / U (23) / A (7)
14 / 22 / 6
21 / 1.0 / 0.7
both answers the same label (gold 7 of 51)
11
34â40
Post-hoc diagnostics (one run each, designed after the result):
Variant
dgem correct of 102
dgem: same label in both answers
both questions in one request
59
36
one question per request
100
5
both in one request, no true/false criteria
68
31
dgem systemone serve --noul-mode choice
95
10
On typed-decisions (200 yes/no pairs in one request), dgemâs rate of matching answers equals gold (138 vs 139). The
defect needs related wording with opposite answers.
Strands is immune by construction: each question is evaluated independently. The flip side is that it cannot use a
real dependency between questions.
38 s (21 s of it in the first request, which looks like kernel compilation)
none on the dedicated endpoint; about 2.5 min on Cloud Run
Strandsâ time grows linearly with input length. A request with more than one question pays a second forward pass,
after which extra questions are nearly free.
dgemâs time is dominated by fixed denoising steps.
Strandsâ stock server serves one request at a time; dgem batches inside vLLM.
On the same GPU, dgemâs cost per decision at saturation is 3â5Ă lower. Strands can also run on much cheaper hardware
(CPU, Mac), which was not measured here.
Measurement note. A first latency pass caught transient load on the shared dgem endpoint (about 100â120 ms on
five modes). Those modes were repeated, and both passes are in the study receipts.
Size class. 1.9B dense vs 26B-A4B MoE: this is not a same-size comparison. The in-domain parity shows what
fine-tuning buys a small model; the out-of-domain gap shows what a large zero-shot base buys.
Hardware. Strands targets RTX 3090, Mac and CPU deployments. Both models here ran on an RTX PRO 6000.
The gate suite is ours: n = 51, labelled by the author, not independently reviewed. The coupling finding rests
on the diagnostics, not on the headline count.
dgem nondeterminism (6.4% of answers change between identical requests) means single dgem runs carry about
±3 items of noise on JevBench.
The receipts in benchmarks/runs/20261003-compare-strands-v19 were produced by the study harness that tier TC
replaces, then exported into matrix format. A confirming TC run is queued in the
register.