Mizan | Blog

Developing for Mizan: extending the core

Wed, 11 Nov 2026 00:00:00 GMT

The previous piece made a promise: you can extend Mizan without touching the core. A template pack is data, so authoring one is a YAML file and a pull request against a separate repository, not a change to the CLI. That covers most of what people want to add.

This piece is for the other case. Sometimes the thing you want to add is not a new template but a new kind of check, a new place results can live, or a new behavior in the engine that runs the judge. That is a code change to mizan itself, and it is the deep end of the series. The good news is that the core was built to be extended along a small number of named seams, and once you can see them, the change you want to make is usually smaller than you expect.

The shape of the codebase

Mizan is a single Go module. The entry points live under cmd/mizan/, the domain logic lives under internal/, and one rule holds the whole thing together: the command layer depends on a very small set of types and nothing else. Concretely, cmd/mizan imports registry.Service, eval.Engine, and config, and never imports the SQLite store, the sync code, the pack codec, or the Vertex AI protos directly. The architecture reference calls this the dependency direction, and it is enforced as an acceptance check rather than left to good intentions.

The place where concrete backends get attached to those abstract types is a single file, internal/wire/wire.go, the composition root. It is the one spot that knows a registry.Service is backed by SQLite, that the eval engine talks to a regional Vertex client and a separate global one, and that the results store is a SQLite file. The comment at the top of the package states the payoff plainly: swapping SQLite for a future store, or changing the eval transport, “is a one-line change here and nowhere else.” When you add something to the core, you are almost always doing one of two things: implementing an interface, then wiring the concrete type in wire.go, or adding a branch to a dispatch that already exists.

The metric registry and its three seams

The registry is where metric templates are stored, authored, imported, and exported. Everything the CLI and the eval engine touch goes through one facade, registry.Service. Underneath it sit three separate interfaces, and the split is the load-bearing design decision of the collaboration layer (collaboration-design.md holds the rationale):

Store is local runtime persistence, the thing the eval engine reads at eval time. Its methods are Get, List, Put, Delete, and a sync-friendly ListChangedSince. SQLite implements it today; a future Firestore store is a drop-in.
Codec is serialization, MetricTemplate to and from bytes, with a single Marshal / Unmarshal / Ext shape. YAML implements it today.
SyncBackend is the contribution channel, where shared templates come from and go to. Its methods are Load, Save, and Describe. The git pack backend implements it today.

These are not aspirational interfaces from a design doc. They are the exact signatures shipping in internal/registry/store.go, codec.go, and sync.go. The reason to keep them apart is that each answers a different question, and a new contributor usually only needs to change one. If you want templates to persist somewhere other than a local SQLite file, you implement Store and change one constructor in wire.OpenService. If you want a new on-disk format, you implement Codec. Nothing in the command layer or the eval engine has to change, because neither of them names the concrete type.

The evaluation engine and dispatch by kind

The engine turns a stored MetricTemplate into a real evaluation. It depends only on the registry domain model and two narrow, mockable client seams: EvaluationClient over the native Vertex AI EvaluateInstances RPC, and GenaiClient over the single GenerateContent call the strict custom-schema path uses. Because both are interfaces, the engine’s spec building and result mapping are unit-tested with fakes and no network at all.

The heart of the engine is one method, Engine.dispatch, and it is worth reading in full because it is the seam every metric kind attaches to:

func (e *Engine) dispatch(ctx context.Context, tmpl registry.MetricTemplate, inst Instance, model string, rc runConfig) (Result, error) {
  switch tmpl.Kind {
  case registry.KindPointwise:
    return e.runPointwise(ctx, tmpl, inst, model)
  case registry.KindRubric:
    // ... rubric-detail routing ...
    return e.runRubric(ctx, tmpl, inst, model)
  case registry.KindCustomSchema:
    return e.runCustomSchema(ctx, tmpl, inst, model)
  case registry.KindPairwise:
    return e.runPairwise(ctx, tmpl, inst, model)
  case registry.KindHeuristic:
    return runHeuristic(tmpl, inst)
  default:
    return Result{}, fmt.Errorf("eval: unknown metric kind %q", tmpl.Kind)
  }
}

A metric kind is a value of registry.MetricKind plus a branch in this switch. The shared work, resolving which autorater model to use, timing the run, validating that every supplied field maps to a placeholder, and recording the applied autorater, all happens once in Engine.Run before dispatch. Each per-kind runner only has to produce a Result. That is why adding a kind is a bounded change: you are filling in one case, not rewriting the engine.

A concrete walkthrough: how `kind: heuristic` was added

The newest metric kind is a good worked example, because it landed as a real change against exactly the seams above. A heuristic is a deterministic, credential-free check: does this text contain a string, match a regex, equal a value, parse as JSON, or validate against a JSON Schema. There is no model and no Vertex call. Here is everything it took to add it.

The registry got a new kind and its config. internal/registry/model.go declares KindHeuristic alongside the four existing kinds, teaches NormalizeKind to accept it, and adds a HeuristicSpec field to MetricTemplate that holds the check type and its operand. The check type is a small closed enum (contains, regex, equals, json-valid, json-schema-valid) validated at the authoring boundary so a typo fails when you create the template, not when you run it.

The engine got a new case. The KindHeuristic branch in dispatch calls runHeuristic. One detail here is the most important interface decision in the whole change: runHeuristic is a free function, not a method on *Engine. That is deliberate. Because it is not a method, it structurally cannot reach e.client, e.globalClient, or e.genai, so the “no network, no credentials” guarantee is enforced by the compiler rather than by a comment. Engine.Run also skips model resolution entirely for a heuristic, so a heuristic run stamps no applied autorater and never validates a model it will not use.

The command layer got authoring flags and an honest pre-flight. registry create gained --heuristic-type, --heuristic-target, --heuristic-value, and the JSON-Schema operands, and eval run prints a pre-flight line that says no autorater is involved.

That is the whole surface area: one enum value and a struct in the registry, one switch case and a free function in the engine, and a handful of flags in the CLI. Now watch it run, with no project and no credentials configured:

$ mizan registry create \
    --id checks/ends-with-next \
    --name "Ends with a Read next section" \
    --kind heuristic --modality text \
    --input "response:text:true" \
    --heuristic-type contains \
    --heuristic-target response \
    --heuristic-value "Read next"

$ mizan eval run --metric checks/ends-with-next \
    --field response="The mechanics hold steady. Read next"
mizan: heuristic: no autorater (deterministic contains check, no network)
Score:        1
Explanation:  matched: text contains "Read next"

$ mizan eval run --metric checks/ends-with-next \
    --field response="An abrupt stop with no pointer onward."
mizan: heuristic: no autorater (deterministic contains check, no network)
Score:        0
Explanation:  no match: text does not contain "Read next"

The verdict maps to a Score of 1.0 for pass and 0.0 for fail, on purpose. By reusing the same *float32 score every other kind returns, a heuristic member flows unchanged through eval-set aggregation, gating thresholds, and the results store. It is a first-class metric, not a special case bolted onto the side. If you pass --model to a heuristic run, the CLI tells you it is ignored rather than silently pretending to honor it:

$ mizan eval run --metric checks/ci --field response="all OK here" --model gemini-2.5-pro
mizan: warning: --model is ignored for kind:heuristic metrics (deterministic, no autorater)

The invariant that this path never touches the network is not a claim to take on faith. The end-to-end test in cmd/mizan/eval_heuristic_cli_test.go drives the real Cobra commands with no PROJECT_ID and no Application Default Credentials set, for every check type, and asserts the exact score. If the path ever reached Vertex, that test would fail:

$ go test ./cmd/mizan/ -run TestCLIHeuristic -count=1
ok    github.com/ghchinoy/mizan/cmd/mizan  0.511s

Where results live

Runs persist by default, and the results store follows the same seam pattern as the registry. wire.OpenResultService returns a results.Service over a backend selected by config. Today that is SQLite. A Firestore leg is a visible deferred dependency, so an unknown backend returns a clear error rather than a silent stub. The lesson for a contributor is that “add a place results can live” is the same shape of change as “add a place templates can live”: implement the store interface, wire the one constructor, leave everything above it alone.

The contribution gates

Before you open a pull request, run the checks CI runs. They all have Makefile targets, and the CONTRIBUTING guide maps each CI gate to its local command:

make fmt-check   # gofmt cleanliness
make vet         # go vet ./...
make test        # unit tests
make lint        # golangci-lint (v2)
make vuln        # govulncheck

Two gates are worth knowing before they surprise you. The coverage floor is a soft nudge: CI prints total coverage and warns if it dips, but never fails the build on coverage alone. The doc-drift guard is not soft. If your pull request changes cmd/mizan/ or internal/eval/ but updates none of the tracked docs, the job fails. That is the guard doing its job: a change to the engine or the command surface is expected to move a guide with it. If the docs genuinely do not apply, you override it explicitly with a docs: N/A label or a docs: N/A line in the pull request body. The build is CGO-free and the toolchain is pinned in go.mod, so local and CI builds use the same compiler, and the author is never the reviewer.

Turn a writing skill into an eval: grading this series with Mizan

Wed, 28 Oct 2026 00:00:00 GMT

Every hands-on piece in this series closes with a “graded by Mizan” sidebar. This piece is about that sidebar. The previous piece showed that a metric template is data you can author, validate, and share without touching the core. Here we point that machinery back at the series and answer a fair question: what is the rubric in the sidebar, where did it come from, and what does its score actually mean?

The short version is that the editorial standard these posts are written to is itself a set of criteria, and criteria are exactly what a Mizan rubric metric holds. So the standard can stop living in a reviewer’s head and become a template you run. This is the strongest form of dogfooding the series has: the product grades the writing that documents the product.

The standard is already a rubric

The posts are held to a small, named editorial standard. Five dimensions carry most of it. Directness: state a claim plainly instead of announcing that you are about to make it. Rhythm: vary sentence length instead of marching in a metronome. Density: make every sentence load-bearing, with no cuttable filler. Authenticity: read as someone who actually did the thing. Trust: state facts and let the reader conclude, without hype.

Written out like that, the standard is a list of yes-or-no questions about a draft. That is a rubric group. Encoding it is one registry create, exactly the shape piece four used for a support reply:

mizan registry create --id docs-quality/technical-explanation --kind rubric \
  --name "Technical explanation quality" \
  --description "Grades a technical blog draft on the series editorial standard." \
  --prompt 'Evaluate this technical blog draft against the editorial criteria. Draft: {{response}}' \
  --input 'response:text:true' \
  --rubric-group 'editorial=The draft states its claims plainly instead of announcing them.;Sentence lengths vary instead of marching in a metronome.;Every sentence is load-bearing, with no cuttable filler.;It reads as written by someone who did the thing.;It states facts and lets the reader conclude, without hype.' \
  --tag docs-quality --tag editorial \
  --license Apache-2.0 --author "Build Evals with Mizan" \
  --model gemini-2.5-flash

That is the whole metric. It is a prompt, one declared input, and five named criteria. Nothing in it is executable, so anyone can read it, diff it, and reason about what it scores before they run it.

Let the model draft the criteria

Writing five criteria by hand is fine when you already know the standard. When you are deriving a rubric from a longer guidance document, Mizan has an authoring aid: rubric generate asks Gemini to draft candidate criteria aligned to a representative prompt, then writes a draft YAML for you to review. It touches the registry for nothing; it only proposes. You keep your non-negotiables by unioning them in with --add-criterion:

mizan rubric generate \
  --sample 'A technical blog post that explains a Mizan feature to a Google Cloud developer, in direct, dense, plain-spoken prose.' \
  --id docs-quality/technical-explanation-draft \
  --out draft-rubric.yaml \
  --recipe text_quality_v1 --group-name editorial \
  --add-criterion 'It reads as written by someone who did the thing.' \
  --add-criterion 'It states facts and lets the reader conclude, without hype.'

When I ran that, Gemini drafted a dozen candidate criteria (directness, density, plain-spokenness, relevance, accuracy, and so on) and printed them in a table whose ORIGIN column marks each row adaptive-generated, followed by the two --add-criterion rows marked hand-authored. That separation is the point: the draft keeps the model’s suggestions apart from your non-negotiables, and it records a rubricProvenance block noting the generator model, the recipe, and a hash of the sample prompt, so a reviewer can see later what came from the model and what came from you. Adaptive generation is an authoring aid, not a source of truth: Gemini drafts, you review and edit, and the moment you freeze the criteria the template is an ordinary, reproducible static rubric. If you would rather generate and freeze in one step, eval adaptive does the whole thing and, with --save-as, writes the result straight into the registry:

mizan eval adaptive \
  --prompt 'A technical blog paragraph that explains a Mizan concept, in direct, dense prose.' \
  --response '<a paragraph to score>' \
  --recipe text_quality_v1 --group-name editorial \
  --save-as docs-quality/technical-explanation-adaptive

That single command makes one generation call and one eval call, prints the score and a mizan: froze generated rubric as docs-quality/technical-explanation-adaptive line, and leaves a reusable template behind. The adaptive rubrics section of the user guide covers the recipes and the provenance record in full, and the shipped rubric-generate-from-brand-book skill drives exactly this loop when you ask an agent to derive a rubric from a guidance doc.

Grade a real draft

With the hand-authored metric frozen, run it against a real draft. Adding --rubric-detail returns a verdict per criterion instead of one blended number. The draft below is the actual opening of piece one:

You shipped a generated image, a marketing line, or a model response, and you asked yourself whether it was good enough. You answered by eye, once, and moved on. The next person on your team answered differently. Neither of you wrote down what “good enough” meant, so neither answer travels past the moment you made it.

Scoring it produced the following, pasted exactly as the CLI printed it:

$ mizan eval run --metric docs-quality/technical-explanation \
    --field response="You shipped a generated image, a marketing line, ..." --rubric-detail
mizan: autorater → project=ghchinoy-genai-sa (src=env-file) location=global (src=global-path) model=gemini-2.5-flash (path=genai)
mizan: samplingCount=4 is ignored on the genai/global structured path (it has no sampling-count concept); samplingCount applies only to the native evaluation path
mizan: flipEnabled is ignored on the genai/global structured path (it has no flip concept); flipEnabled applies only to the native pairwise evaluation path
Score:        5
Explanation:  This is an exceptionally strong opening for a technical blog post. It immediately engages the reader by presenting a highly relatable and common problem in a clear, concise, and authentic voice. The language is direct, every sentence serves a purpose, and it effectively sets the stage for further discussion without any unnecessary fluff or hype. The slight similarity in length of two sentences is a minor point in an otherwise excellent draft.
Per-criterion:
GROUP      CRITERION                                                        SCORE  RATIONALE
editorial  The draft states its claims plainly instead of announcing them.  5      The draft immediately presents a common scenario and problem without any introductory announcements or preambles.
editorial  Sentence lengths vary instead of marching in a metronome.        4      There is some variation in sentence length (23, 9, 9, 26 words), though two consecutive sentences are quite short and similar, slightly impacting flow.
editorial  Every sentence is load-bearing, with no cuttable filler.         5      Every sentence directly contributes to establishing the problem and its implications; there is no discernible filler or redundant phrasing.
editorial  It reads as written by someone who did the thing.                5      The scenario described is highly relatable and specific to real-world challenges in evaluating outputs, suggesting direct experience from the author.
editorial  It states facts and lets the reader conclude, without hype.      5      The draft presents a common problem in a straightforward, factual manner without any hype or exaggerated language, allowing the reader to infer the significance.

Two things about that run are worth noticing. The autorater line reports location=global and path=genai: --rubric-detail uses the structured generative path rather than the native sampling path, which is why the run also notes that samplingCount and flipEnabled do not apply here. And the per-criterion table is the reason to reach for the flag at all. A single “5” tells you little. The Rhythm row scoring a 4, with the judge counting the sentence lengths (23, 9, 9, 26 words) and flagging the two short middle sentences, is the kind of specific, actionable note that makes the score useful as revision feedback rather than a grade you file away.

Because Mizan persists every run by default, that feedback compounds. Score a draft, take the Rhythm note, rewrite the two short sentences, and rerun the same metric. mizan results list and mizan results show <run-id> let you look back at what each revision scored and why, so you can see a dimension climb as you work rather than guessing whether an edit helped. The standard is the same each time, which is the whole reason to write it down: the check is repeatable, so the comparison across drafts is honest.

Ship it so others can run it

A rubric that only grades this series is a curiosity. The value is that anyone can run the same standard on their own writing, and that is the contribution loop from piece four, applied to a rubric instead of a support metric. Scaffold a pack, add the metric, and validate it with no credentials:

mizan pack init packs/docs-quality --name docs-quality
mizan pack add packs/docs-quality --from docs-quality/technical-explanation
mizan pack validate packs/docs-quality

initialized pack "packs/docs-quality" (namespace "docs-quality")
1 written, 0 skipped (dest: packs/docs-quality)
  written: docs-quality/technical-explanation -> templates/technical-explanation.yaml
OK: no defects found.

0 error(s), 0 warning(s)

The validated pack is a directory of data you commit and open as a pull request, the same way the google-brand pack travels. On the consuming side, importing it is one command, previewable with --dry-run first:

mizan registry import packs/docs-quality

1 inserted, 0 updated, 0 skipped, 0 conflicted, 0 unchanged, 0 forked (source: packs/docs-quality)
  inserted: docs-quality/technical-explanation

Now the reader has the exact rubric the series is graded by, ready to point at their own drafts. The editorial standard stopped being tribal knowledge and became an artifact that travels with the work.

What the score does not measure

Here is the honest part, and it is not a footnote. This rubric grades surface style and clarity. It says nothing about whether the substance is correct.

A draft can score a clean five while claiming a command that does not exist, citing a flag that was renamed, or building an argument that does not hold. The judge is reading prose, not running the CLI and not checking the reasoning. The rubric would happily award full marks to a fluent, confident, completely wrong paragraph. Style conformity is necessary, not sufficient: it is real signal about readability and it catches the obvious problems, but it is silent on the question that matters most, which is whether the piece is right.

So the score sits alongside the checks that do test substance, and does not replace them. Every command in this series was run live and its output pasted verbatim. Every capability claim was verified against the shipped binary. A human read each draft for whether the argument earns its conclusion. The rubric is one repeatable check in that stack, not the top of it. Read a high score as “this reads cleanly,” never as “this is correct.” Overselling a rubric number as objective quality is the one failure mode this piece exists to warn against.

Contribute a template pack: extend Mizan without touching the core

Wed, 14 Oct 2026 00:00:00 GMT

The asset manager playbook ended on a promise. The checks a manager curates for a class of assets are just metric templates, so the presets a team settles on can travel as a template pack that creators import once and reuse. This piece keeps that promise. It walks the whole loop: author a pack, validate it, and share it so anyone can import it, without changing a line of Mizan’s code.

That last part is the point. A metric template is data, not code. It is a YAML file that states a prompt, the inputs it expects, and the criteria a judge should weigh. A pack is a directory of those files plus a small manifest. Contributing one means writing data and opening a pull request, which is the lowest-barrier way to extend Mizan. You are not compiling anything, and you are not touching the engine that runs your template.

A pack is a directory of data

Start by scaffolding the pack. mizan pack init writes the layout for you:

mizan pack init packs/acme-support --name acme-support

initialized pack "packs/acme-support" (namespace "acme-support")

That creates three things: a mizan-pack.yaml manifest, an empty templates/ directory, and an empty evalsets/ directory. The templates/ directory holds one YAML file per metric. The evalsets/ directory holds eval-set manifests, the grouped, runnable sets part one introduced; a pack can carry templates, eval-sets, or both. One file per template is deliberate: it keeps merge conflicts small when several contributors add to the same pack at once.

The manifest itself is short. Its metadata.name is the namespace, and every template id in the pack begins with it:

apiVersion: mizan.dev/v1alpha1
kind: Pack
metadata:
    name: acme-support
    version: 0.1.0
spec:
    requiresApiVersion: mizan.dev/v1alpha1

Author a template into the pack

Author the metric in your local registry first, exactly as the earlier pieces did, then write it into the pack. Here is a rubric metric that scores a customer-support reply on three separate concerns:

mizan registry create --id acme-support/reply-quality --kind rubric \
  --name "Support reply quality" \
  --description "Scores a customer-support reply on clarity, tone, and whether it resolves the issue." \
  --prompt 'Evaluate this support reply against the criteria. Customer message: {{customer_message}} Reply: {{reply}}' \
  --input 'customer_message:text:true' \
  --input 'reply:text:true' \
  --rubric-group 'clarity=Is the reply easy to follow?;Does it avoid jargon the customer would not know?' \
  --rubric-group 'tone=Is the reply courteous and free of blame?' \
  --rubric-group 'resolution=Does the reply actually resolve or advance the issue?' \
  --tag support --tag customer-comms \
  --license Apache-2.0 --author "ACME Support Guild" \
  --model gemini-2.5-flash

Now add it to the pack. pack add writes one schema-valid file under templates/:

mizan pack add packs/acme-support --from acme-support/reply-quality

1 written, 0 skipped (dest: packs/acme-support)
  written: acme-support/reply-quality -> templates/reply-quality.yaml

The file it wrote is the whole metric, and it is readable enough to review in a pull request:

apiVersion: mizan.dev/v1alpha1
kind: MetricTemplate
metadata:
    id: acme-support/reply-quality
    name: Support reply quality
    description: Scores a customer-support reply on clarity, tone, and whether it resolves the issue.
    version: 0.1.0
    authors:
        - name: ACME Support Guild
    license: Apache-2.0
    tags:
        - support
        - customer-comms
spec:
    kind: rubric
    modalities:
        - text
    inputs:
        - name: customer_message
          modality: text
          required: true
        - name: reply
          modality: text
          required: true
    metricPromptTemplate: 'Evaluate this support reply against the criteria. Customer message: {{customer_message}} Reply: {{reply}}'
    rubricGroups:
        clarity:
            - Is the reply easy to follow?
            - Does it avoid jargon the customer would not know?
        resolution:
            - Does the reply actually resolve or advance the issue?
        tone:
            - Is the reply courteous and free of blame?
    autorater:
        model: gemini-2.5-flash
        samplingCount: 4
        flipEnabled: true

Nothing here is executable. It is a prompt, a list of declared inputs, and named rubric groups. Anyone can read it, diff it, and reason about what it scores before they ever run it. That is what “templates are data” buys you.

The validate gate

Before you share a pack, you check it. mizan pack validate runs a set of structural, identity, semantic, placeholder, and lint checks over every manifest in the pack, and it does so with no credentials. It never calls Vertex AI, so it costs nothing and needs no project:

mizan pack validate packs/acme-support

OK: no defects found.

0 error(s), 0 warning(s)

The reason this matters is that it is the same gate a shared templates repository runs in continuous integration on every pull request. Because the checks are credential-free, they run identically on your laptop and in CI. You catch the problem before you publish, not after a reviewer’s build turns red.

Here is the gate earning its place. Say you edit the prompt to reference an account tier but forget to declare it as an input:

mizan pack validate packs/acme-support

templates/reply-quality.yaml:
  [ERROR] prompt references undeclared placeholder {{account_tier}} (add it to spec.inputs)

1 error(s), 0 warning(s)
error: pack validate: 1 error(s) found

The command exits non-zero and names the file, the defect, and the fix. Declare the input (or drop the placeholder) and it passes again. Warnings behave differently from errors: a missing license or a missing description is reported as a warning and never fails the gate, so a pack can be accepted with advisories you choose to act on later. Branch on the exit code for the accept-or-reject decision, and read the trailing count line for the summary.

This is exactly the loop the shipped author-and-validate-a-template-pack skill drives when you ask an agent to package or check a pack: it scaffolds, authors, and runs pack validate, reading the exit code and the text report rather than parsing JSON. The user guide documents each check the gate applies.

A validated pack directory is ready to share. registry export writes your local templates into a pack directory (pack add is a thin convenience over it), and from there the workflow is ordinary git. You commit the pack and open a pull request against a templates repository; Mizan writes files but never pushes:

mizan registry export --out packs/acme-support --namespace acme-support

1 written, 0 skipped (dest: packs/acme-support)
  written: acme-support/reply-quality -> templates/reply-quality.yaml

The community repository, github.com/ghchinoy/mizan-templates, is the default source, and the google-brand pack that piece two imported is the worked model to copy: a manifest, a templates/ directory, and one file per metric.

On the consuming side, someone imports your pack. Preview first with --dry-run, which computes the reconciliation without writing anything:

mizan registry import packs/acme-support --dry-run

dry run (no changes written): 1 inserted, 0 updated, 0 skipped, 0 conflicted, 0 unchanged, 0 forked (source: packs/acme-support)
  inserted: acme-support/reply-quality

Drop the flag to commit it:

mizan registry import packs/acme-support

1 inserted, 0 updated, 0 skipped, 0 conflicted, 0 unchanged, 0 forked (source: packs/acme-support)
  inserted: acme-support/reply-quality

Import reconciles by template id, so re-importing an unchanged pack does nothing:

0 inserted, 0 updated, 0 skipped, 0 conflicted, 1 unchanged, 0 forked (source: packs/acme-support)
  unchanged: acme-support/reply-quality (unchanged (same content))

The imported template records where it came from, so a consumer always knows the provenance of a metric in their registry:

mizan registry get acme-support/reply-quality

ID:                       acme-support/reply-quality
...
Version:                  0.1.0
Kind:                     rubric
...
Source:                   pack:acme-support@packs/acme-support
...

By default, import takes the higher version and never clobbers a template you have edited locally; --strategy lets you choose skip, overwrite, or fork when you want different behavior. The shipped discover-and-import-templates skill wraps this half of the loop, previewing, reconciling, and browsing what landed, and the user guide covers the import strategies in full. A source can be a local directory, a single pack, or a git URL, so importing from your teammate’s checkout and importing from the community repository are the same command.

You have now extended Mizan the easy way: you wrote data, validated it for free, and shared it through a pull request, with the engine untouched. The next piece turns this contribution loop on the series itself. It builds a docs-quality pack (the rubric that grades every entry, including this one, in the sidebar below) and is honest about what such a rubric does and does not measure. After that, the final piece moves from contributing data to extending the code, and shows the seams where a new metric kind or behavior attaches to the core.

mizan registry create --id docs-quality/technical-explanation --kind rubric \
  --name "Technical explanation quality" \
  --rubric-group "quality=Is the explanation direct, stating claims not announcing them?;\
Is it dense with no cuttable filler?;Is it accurate and correctly scoped?;\
Does it read as written by someone who did the thing?" \
  --model gemini-2.5-flash

mizan eval run --metric docs-quality/technical-explanation \
  --field response="<draft of this article>" --rubric-detail

Be clear about what that score does and does not mean. The rubric grades surface style and clarity. It does not verify that the substance is correct, that the commands run as written against your project, or that the pack you author is the right one for your team. A clean style score sits alongside the live-command checks and human review that catch those things; it does not replace them. Read the scorecard as a repeatable check that catches the obvious problems, not as a measurement of whether the piece is right.

(Any scorecard numbers shown in this series are manual review estimates unless labeled as measured; the automated readability tooling was unavailable at the time of writing.)

Using Mizan in your role: per-persona playbooks, part 2 (Genmedia configurator, Brand Lab)

Wed, 30 Sep 2026 00:00:00 GMT

Part one handed the two roles closest to the work a playbook each: the asset creator running the shortest path from one eval to a scorecard, and the asset manager curating a set that keeps every concern on its own line. This part picks up the other two personae, both a step back from the raw output. The genmedia configurator does not grade one asset by hand; they wire evaluation into an application so every asset it produces is checked the same way. The Brand Lab user does not write criteria by hand; they hold a brand book and want the criteria drafted from it. Different jobs, the same machinery, and each role’s standard ends up as a reusable Mizan artifact.

Genmedia configurator: the eval-set is the calling shape

You build or configure a generative-media application, and you want evaluation available inside it rather than as a manual step someone runs afterwards. What you need from Mizan is not a single metric but a named, reusable way to call a set of metrics: point at one thing, get back one scorecard your application can act on.

That thing is the eval-set manifest. Part one ran eval-sets from the creator’s and manager’s side; here the manifest matters as the calling shape, the stable unit your application embeds. It names its members by template id and says how to aggregate them:

spec:
  inputs:
    prompt: prompt
    response: response
  members:
    - metric: quickstart/response-helpfulness
      weight: 2
    - metric: quickstart/response-conciseness
      weight: 1
  aggregation:
    method: weighted-mean
    threshold: 3.0
    gate: false

Scope this honestly. Phase 1 is library-first: you embed the set by pointing the CLI or the Go library at a manifest on disk. There is no hosted service and no network API to call. The calling shape is fixed now, and it doubles as the integration spec for a future API surface; that service is not built, and this piece does not promise one.

Two properties make the manifest safe to depend on from inside an application.

First, it validates without credentials. Before your application commits to a set, pack validate checks the manifest structure and every member with no Vertex call, so it runs in CI as a gate:

$ mizan pack validate docs/examples/evalset-quickstart
evalsets/answer-quality-badmember.yaml:
  [warn ] spec.members[0].metric "quickstart/does-not-exist" resolves to no template in this tree (ok if it lives in another pack)

0 error(s), 1 warning(s)

The one warning is deliberate: that example manifest references a member that does not exist in the tree, and the validator flags it rather than failing, because the member could live in another pack. A structural error, an empty member list, or a malformed id would exit non-zero and block the change.

Second, running the set returns one scorecard, not a pile of loose scores. Import the member templates once, then run the set:

$ mizan registry import docs/examples/evalset-quickstart
2 inserted, 0 updated, 0 skipped, 0 conflicted, 0 unchanged, 0 forked (source: docs/examples/evalset-quickstart)
  inserted: quickstart/response-conciseness
  inserted: quickstart/response-helpfulness

$ mizan eval run --set docs/examples/evalset-quickstart/evalsets/answer-quality.yaml \
    --field prompt="What is the capital of France?" \
    --field response="The capital of France is Paris, a major European city on the Seine."
EvalSet: quickstart/answer-quality (v1.0.0)  asset-class: text-answer

MEMBER                           STATUS  WEIGHT  SCORE  NOTE
quickstart/response-helpfulness  ok      2       5.00
quickstart/response-conciseness  ok      1       3.00

Aggregate (weighted-mean over 2 scored): 4.33   threshold: 3   PASSED

For an application the table is the wrong surface, so add --output json and the same run returns a structured object: a Members array where each entry carries a flat Status and Score plus a nested Result object holding that member’s Explanation, and an Aggregate block with the method, the numeric score, the threshold, and a boolean Passed, alongside a top-level Verdict. Your application branches on Passed, logs each member’s Result.Explanation, and surfaces the aggregate to the user. That is the whole embedding contract: one manifest in, one parseable verdict out.

Because the members are ordinary metric templates, the whole set travels as a template pack. You curate the evaluation your application enforces once, ship it in a pack, and every instance of the application imports and runs the same standard. The eval-set runner section of the user guide documents the manifest fields, the partial-failure behavior, and the gate exit code in full.

Brand Lab: draft the rubric from the brand book

You are a Brand Lab user. You hold a brand book, and you want evaluations generated from it rather than authored by hand. Writing rubric criteria from scratch for every brand rule is exactly the work you are trying to avoid.

The shipped piece of this is adaptive rubric generation: Gemini drafts rubric criteria from a sample prompt, and you review and freeze them. This is the suggest-first shape the Brand Lab journey asks for. The model proposes criteria from your source material, and you decide what to keep. There are two entry points, one for each moment you reach for it.

The first is the authoring aid. Feed mizan rubric generate a sample prompt that encodes the brand book, and it drafts criteria for review, writing a draft YAML and touching nothing in your registry:

$ mizan rubric generate \
    --sample "Write a one-line product announcement that follows our brand book: warm and concise, never salesy, name the product exactly once, no superlatives, no competitor comparisons, plain language over jargon." \
    --id brand-lab/announcement-voice \
    --name "Announcement brand voice" \
    --recipe general_quality_v1 \
    --out brand-lab-draft.yaml
GROUP            CRITERION                                                               TYPE                                                   IMPORTANCE
general_quality  The response is in English.                                             LANGUAGE:PRIMARY_RESPONSE_LANGUAGE                     HIGH
general_quality  The response is a single line of text.                                  FORMAT_REQUIREMENT:SINGLE_LINE                         HIGH
general_quality  The response functions as a product announcement.                       CONTENT_REQUIREMENT:PURPOSE:PRODUCT_ANNOUNCEMENT       HIGH
general_quality  The announcement has a warm tone.                                       TONE_REQUIREMENT:WARM                                  MEDIUM
general_quality  The announcement is concise.                                            STYLE_REQUIREMENT:CONCISE                              MEDIUM
general_quality  The announcement avoids salesy language.                                TONE_REQUIREMENT:NON_SALESY                            HIGH
general_quality  The announcement mentions a product name exactly once.                  CONTENT_REQUIREMENT:PRODUCT_NAME_MENTION:EXACTLY_ONCE  HIGH
general_quality  The announcement contains no superlative adjectives or adverbs.         STYLE_REQUIREMENT:NO_SUPERLATIVES                      HIGH
general_quality  The announcement avoids direct or indirect comparisons to competitors.  CONTENT_REQUIREMENT:NO_COMPETITOR_COMPARISONS          HIGH
general_quality  The announcement uses plain language and avoids jargon.                 STYLE_REQUIREMENT:PLAIN_LANGUAGE_NO_JARGON             MEDIUM

Read the criteria: the brand book decomposed into checkable statements, each tagged with a type and an importance. The draft is a file, not a registry entry. You review and edit it, then wrap it in a pack and import it as an ordinary template. From that moment it is a static, reproducible rubric, not a fresh generation on every run.

The second entry point is for when you have an asset in front of you and want feedback now. mizan eval adaptive generates the rubric from the prompt and scores the response against it in one step, and --save-as freezes the rubric it actually used into the registry so you do not lose it:

$ mizan eval adaptive \
    --prompt "Write a one-line product announcement that follows our brand book: warm and concise, never salesy, name the product exactly once, no superlatives, no competitor comparisons, plain language over jargon." \
    --response "Meet Aria, the smart home hub that makes your mornings a little easier." \
    --recipe general_quality_v1 \
    --save-as brand-lab/announcement-voice
Score:        10
Explanation:  All ten criteria from the rubric were satisfied, as the announcement is in English, a single line, functions as an announcement, is warm and concise, avoids salesy language, names the product once, contains no superlatives or competitor comparisons, and uses plain language.

The on-brand line scores full marks. The value of --save-as shows up on the next run: the rubric is now frozen as brand-lab/announcement-voice, an ordinary rubric metric you rerun with mizan eval run against any candidate, grading each one against the same criteria. Hand it an off-brand line, and the same frozen template catches every violation:

$ mizan eval run --metric brand-lab/announcement-voice \
    --field prompt="Write a one-line product announcement that follows our brand book: warm and concise, never salesy, name the product exactly once, no superlatives, no competitor comparisons, plain language over jargon." \
    --field response="Introducing Aria, the world's #1 smartest home hub, way better than anything else, buy now!"
Score:        5
Explanation:  The response is in English, a single line, functions as a product announcement, names the product once, and uses plain language. However, it fails on being warm and concise, contains salesy language and a call to action, uses a superlative ('#1 smartest'), and makes a comparison ('way better than anything else'), missing 5 out of 10 criteria.

Same template, same criteria, a different asset, and the score drops from 10 to 5 with a reason that names the superlative, the competitor comparison, and the sales pitch. That is the Brand Lab standard encoded as a Mizan metric: generated once from the brand book, frozen, and reusable. Curate a few of these and the brand book itself travels as importable templates. The adaptive rubrics section of the user guide covers the recipes, --save-as, and the generation provenance Mizan records. Keep one caveat in view: generation is a drafting aid, so review the criteria before you freeze them, exactly as you would review any generated draft.

That closes the per-persona playbooks. Across both parts the four roles ran the same machinery and differed only in what they wrote down: the creator ran one eval and then a set, the manager curated a set that keeps concerns apart, the configurator embedded a set as a calling shape, and the Brand Lab user generated a rubric from a brand book and froze it. Each standard ended up as the same kind of artifact, a metric template or an eval-set, and each of those travels as a template pack. Piece four is about that last step: authoring a pack, validating it, and sharing it so other people import your standard without touching the core.

mizan registry create --id docs-quality/technical-explanation --kind rubric \
  --name "Technical explanation quality" \
  --rubric-group "quality=Is the explanation direct, stating claims not announcing them?;\
Is it dense with no cuttable filler?;Is it accurate and correctly scoped?;\
Does it read as written by someone who did the thing?" \
  --model gemini-2.5-flash

mizan eval run --metric docs-quality/technical-explanation \
  --field response="<draft of this article>" --rubric-detail

Be clear about what that score does and does not mean. The rubric grades surface style and clarity. It does not verify that the substance is correct, that the commands run as written against your project, or that the two playbooks fit your role. A clean style score sits alongside the live-command checks and human review that catch those things; it does not replace them. Read the scorecard as a repeatable check that catches the obvious problems, not as a measurement of whether the piece is right.

(Any scorecard numbers shown in this series are manual review estimates unless labeled as measured; the automated readability tooling was unavailable at the time of writing.)

Using Mizan in your role: per-persona playbooks, part 1 (Asset creator, Asset manager)

Wed, 16 Sep 2026 00:00:00 GMT

The first piece turned one by-eye judgment into a check you can rerun. The second placed Mizan’s four users on a single spectrum and showed that the same machinery serves all of them. This piece gets practical: it hands two of those users a playbook for their actual job. Pick your role, and here is the shortest path to value.

Mizan has four canonical personae. This part covers the two closest to the work itself. The asset creator produces candidate output; the asset manager decides what a class of assets must clear before it ships. Part 2 picks up the other two.

Asset creator: one eval, then a set, then a scorecard

You make assets and you want to know they align with guidance before you send them on. Your path has three steps, each shipped today.

Run one eval. Piece one already walked this: author a metric, hand it a response, read back a score and a reason. That is the atom, and it covers the day you have exactly one question about one asset.

Run a set. Your work rarely raises one question. A tagline needs to be concise and on-voice and free of competitor names. Rather than run three commands and reconcile three outputs by hand, bundle the metrics into an eval-set and score the asset against all of them in a single run. In Phase 1 the runner is path-based: eval run --set takes a filesystem path to an EvalSet manifest and scores its members together. The set’s members reference metric template ids, so import their templates first, then run the set:

$ mizan registry import docs/examples/evalset-quickstart
2 inserted, 0 updated, 0 skipped, 0 conflicted, 0 unchanged, 0 forked (source: docs/examples/evalset-quickstart)
  inserted: quickstart/response-conciseness
  inserted: quickstart/response-helpfulness

$ mizan eval run --set docs/examples/evalset-quickstart/evalsets/answer-quality.yaml \
    --field prompt="What is the capital of France?" \
    --field response="The capital of France is Paris, a major European city on the Seine."

Read the scorecard. The run prints one table, a row per member, then an aggregate line with the method, the count of scored members, the threshold, and the always-computed verdict:

EvalSet: quickstart/answer-quality (v1.0.0)  asset-class: text-answer

MEMBER                           STATUS  WEIGHT  SCORE  NOTE
quickstart/response-helpfulness  ok      2       5.00
quickstart/response-conciseness  ok      1       4.00

Aggregate (weighted-mean over 2 scored): 4.67   threshold: 3   PASSED

Read the member rows first, since they tell you which criteria the asset met, then read the aggregate line for the single go/no-go. A member that can’t resolve is reported and the run continues, so a broken template never silently drops a concern; add --fail-fast when you would rather stop at the first problem. Every run persists by default, so mizan results list and mizan results show <run-id> let you look back at what you scored and why. The eval-set runner walkthrough in the user guide covers the manifest fields, --output json, and partial failures in full.

Asset manager: curate a set that keeps concerns apart

You do not run assets one at a time. You decide what a class of assets must clear, curate the checks once, and hand creators something they can reuse. The eval-set is that artifact. Its most valuable property for you is that it reports each concern on its own line rather than blending everything into one opaque number.

Say a campaign asset has to satisfy five separate concerns: it aligns with the prompt, adheres to brand, follows campaign guidelines, meets the advertising channel’s requirements, and passes safety. Curate each as its own member so each gets its own verdict, and aggregate with min so the set only passes when the weakest concern does:

apiVersion: mizan.dev/v1alpha1
kind: EvalSet
metadata:
  id: campaign/spring-launch-readiness
  version: 1.0.0
  assetClass: display-ad
spec:
  inputs:
    prompt: prompt
    response: response
  members:
    - metric: campaign/prompt-alignment
    - metric: brand/adherence
    - metric: campaign/guidelines
    - metric: channel/display-requirements
    - metric: safety/content
  aggregation:
    method: min            # mean | weighted-mean | min
    threshold: 3.0
    gate: true             # non-zero exit only when the verdict is FAILED

Run it and the scorecard names each concern independently, so a stakeholder can see exactly which dimension an asset fails, not just that it failed somewhere:

MEMBER                        STATUS  WEIGHT  SCORE  NOTE
campaign/prompt-alignment     ok      1       5.00
brand/adherence               ok      1       4.00
channel/display-requirements  ok      1       4.00
campaign/guidelines           ok      1       2.00
safety/content                ok      1       5.00

Aggregate (min over 5 scored): 2.00   threshold: 3   FAILED

Two things make this a manager’s tool rather than a creator’s. First, because each member persists as its own result, you can track a single concern’s trend over time. mizan results list --metric campaign/guidelines --since 2026-10-01 shows only that dimension, run after run, so a slipping category is visible before it becomes a pattern. Second, gate: true makes a failing verdict exit non-zero, which is exactly what wires the set into a CI check that blocks a release. The members themselves are just metric templates, so the industry presets you curate travel as a template pack that creators import once and reuse, the contribution loop piece four is built around. The testing guide has the eval-set recipe end to end, including the gate exit-code behavior.

Keep the runner’s scope in mind: Phase 1 is a library-first, path-based surface. There is no hosted service or API to call. You point the CLI (or the Go library) at a manifest on disk. Store-backed resolution by set id is a documented fast-follow, not something to build a pipeline around yet.

That is the operational end of the spectrum: the creator running the shortest path from one eval to a scorecard, and the manager curating a set that keeps every concern on its own line. Part 2 picks up the other two personae: the genmedia application configurator embedding an eval-set’s calling shape, and the Brand Lab user auto-generating a rubric from a brand book.

mizan registry create --id docs-quality/technical-explanation --kind rubric \
  --name "Technical explanation quality" \
  --rubric-group "quality=Is the explanation direct, stating claims not announcing them?;\
Is it dense with no cuttable filler?;Is it accurate and correctly scoped?;\
Does it read as written by someone who did the thing?" \
  --model gemini-2.5-flash

mizan eval run --metric docs-quality/technical-explanation \
  --field response="<draft of this article>" --rubric-detail

Be clear about what that score does and does not mean. The rubric grades surface style and clarity. It does not verify that the substance is correct, that the commands run as written against your project, or that the two playbooks fit your role. A clean style score sits alongside the live-command checks and human review that catch those things; it does not replace them. Read the scorecard as a repeatable check that catches the obvious problems, not as a measurement of whether the piece is right.

(Any scorecard numbers shown in this series are manual review estimates unless labeled as measured; the automated readability tooling was unavailable at the time of writing.)

One spectrum, four users: from brand alignment to technical metrics

Tue, 15 Sep 2026 00:00:00 GMT

The first piece took one judgment you used to make by eye and turned it into a check you can rerun and hand to a colleague. That piece stayed narrow by design: one pointwise metric, one response, one score. This piece widens the frame to the people doing the judging, because who you are changes what you ask a judge to check, and it is easy to assume that a different question needs a different tool.

It does not. The question a brand reviewer asks and the question a platform engineer asks land on the same command. The criteria they write down separate them, not the machinery that runs them.

Four users, one spectrum

Mizan has four canonical users. Line them up by the kind of question each one brings to a generated asset, and they form a spectrum rather than four separate worlds.

At one end sits the Brand Lab user. They hold a brand book and a set of assets, and they want evaluations that check alignment: does this ad carry the approved voice, show the logo where the guideline requires, stay off competitor colors. Their criteria are qualitative and human-authored, and a good answer reads like a careful reviewer’s note.

At the other end sit the asset manager and the genmedia application creator or configurator. The asset manager curates groups of evaluations for a class of assets so that creators can reuse them, and wants each concern reported as its own verdict rather than blended into one number. The genmedia configurator wires evaluation into a generative-media application and wants a result the application can parse and act on. Both live at the technical, metric end: they want structured, gateable output that a pipeline can read.

The asset creator spans the middle. They run an eval against an asset to check it against guidance, and they run sets of evals as their work matures. Some days their question is a brand question; other days it is an accuracy question. They move along the spectrum depending on the asset in front of them.

Read left to right, the spectrum runs from “is this on-brand?” to “is this grounded and accurate?” The claim of this piece is that a single toolset covers the whole line.

The tool is fixed; the criteria move

Recall the four metric kinds from piece one. A pointwise metric scores one response against one question. A rubric metric scores several named criteria at once. A custom_schema metric returns typed fields you define. A pairwise metric picks the stronger of two candidates.

None of those kinds is a “brand” tool or a “technical” tool. A pointwise metric does not know whether the number it produces measures brand voice or factual grounding. You decide that when you write the prompt and the criteria. The workflow is identical at both ends of the spectrum: you author a template, you hand it a response, and you read back a verdict with the judge’s reasoning. The brand reviewer and the platform engineer follow the same three steps. They differ only in what they put in the template and what shape of answer they ask for.

The two worked examples below prove it. The first sits at the brand end and uses a shared template pack. The second sits at the technical end and returns a typed verdict. Both are the same author-and-run loop.

Brand end: import a pack and score alignment

The Brand Lab user rarely writes brand criteria from scratch. Brand rules are shared property, so they travel as a template pack: a versioned folder of metric templates that anyone can import. The community pack repository ships a worked example, the google-brand pack, which contains a video-brand-alignment template.

Import it into your local registry. With no argument, registry import pulls from the configured default source, github.com/ghchinoy/mizan-templates:

mizan registry import --namespace google-brand

1 inserted, 0 updated, 0 skipped, 0 conflicted, 0 unchanged, 0 forked (source: https://github.com/ghchinoy/mizan-templates)
  inserted: google-brand/video-brand-alignment

The template is now a first-class metric in your registry, addressable by its stable id:

mizan registry get google-brand/video-brand-alignment

Under the hood it is a pointwise metric: it scores one asset on a 1-to-5 scale against a supplied brand guideline. Its prompt asks the judge to weigh tone, visual identity, and messaging consistency, and it declares two inputs, the response (the video under evaluation) and a brand_guideline (the reference text). Run it by supplying both inputs. The video is a media asset, so it is passed with --gcs rather than --field; multimodal assets travel as gs:// URIs because the native Eval Service reads them from Cloud Storage rather than inline:

mizan eval run --metric google-brand/video-brand-alignment \
  --gcs response=gs://ghchinoy-genai-sa-veo/camping_bear.mp4 \
  --field brand_guideline="Our brand voice is warm, concise, and never salesy. \
Always show the logo in the final 3 seconds. Primary color is #1A73E8; avoid competitor colors."

The result comes back as a pointwise score in the declared 1-to-5 range with a free-text rationale that names what aligned and what did not:

Score:        2
Explanation:  The ad's warm, concise, and non-salesy tone is perfect, but it completely fails on the visual identity rules by omitting the logo and not using the primary brand color.

Here a generic clip pulled from Cloud Storage scores a 2: the judge credits the tone the guideline asks for and marks it down for the missing logo and the wrong palette. That is the whole brand end: someone authored the criteria once, shared them as a pack, and now any reviewer runs the same standard against their own footage and gets a comparable answer. Nobody re-litigated what “on-brand” means for this run.

Technical end: a typed verdict a pipeline can gate on

The asset manager and the genmedia configurator want the same author-and-run loop, but their downstream needs a machine to read the result, not a person. When the honest question is “does this response stay grounded and inside policy,” the custom_schema kind returns a typed JSON verdict instead of prose.

The repository ships an example schema, docs/examples/compliance-schema.json, that returns four fields: an overall_score, a boolean compliant, an array of flagged_issues, and an explanation. Author a metric against it and run it:

mizan registry create --id demo/custom-compliance --name "Compliance Check" --kind custom_schema \
  --prompt "Check if this response follows the policy: no medical advice. Response: {{response}}" \
  --response-schema-file docs/examples/compliance-schema.json

mizan eval run --metric demo/custom-compliance --field response="Drink plenty of water and rest."

Score:                         (none)
Explanation:
CustomOutput[compliant]:       true
CustomOutput[explanation]:     General wellness suggestions, not specific medical advice; complies with the policy.
CustomOutput[flagged_issues]:  []
CustomOutput[overall_score]:   9

Read that output next to the brand run. The brand reviewer got a score and a sentence for a human to weigh. Here the genmedia configurator gets a compliant boolean their application can branch on, an overall_score they can threshold, and a flagged_issues array they can log or surface, all as typed data returned with --output json. Same command, mizan eval run; the difference is that the criteria live in a schema the judge fills in rather than in a scale it reasons to.

The grounding and accuracy question follows the identical shape. Swap the prompt for one that asks whether every claim in the response is supported by a supplied source, add a source field to feed the reference text, and the same custom_schema template returns a typed verdict on factual grounding. You did not change tools to move from a compliance concern to a grounding concern. You changed the sentence in the prompt.

Where you sit changes what you write, not what you run

Put the two examples side by side and the spectrum collapses into one workflow. The Brand Lab user imported a pack of qualitative criteria and read a scored, explained verdict. The genmedia configurator authored a schema of structured criteria and read a typed, parseable verdict. The asset creator in the middle does both on different days. The asset manager’s job is to curate these templates into reusable sets and report each concern independently, which is the same loop run over a group; Mizan ships a path-based eval-set runner (eval run --set) for exactly that, and the next piece walks it end to end per role.

The LLM-as-a-Judge scenarios guide is the map from “here is the question I am asking” to “here is the command that answers it,” across all four kinds and both ends of this spectrum; the custom_schema scenario covers the typed-verdict path in full, and the user guide documents importing and inspecting packs. Batch evaluation and the desktop app are not built yet, and the docs say so rather than implying otherwise.

The lesson to carry into the per-persona playbooks in piece three: pick your place on the spectrum, and the only decision left is what to write down. The machinery is already the same.

mizan registry create --id docs-quality/technical-explanation --kind rubric \
  --name "Technical explanation quality" \
  --rubric-group "quality=Is the explanation direct, stating claims not announcing them?;\
Is it dense with no cuttable filler?;Is it accurate and correctly scoped?;\
Does it read as written by someone who did the thing?" \
  --model gemini-2.5-flash

mizan eval run --metric docs-quality/technical-explanation \
  --field response="<draft of this article>" --rubric-detail

Be clear about what that score does and does not mean. The rubric grades surface style and clarity. It does not verify that the substance is correct, that the two worked examples run as written against your project, or that the spectrum argument holds. A clean style score sits alongside the live-command checks and human review that catch those things; it does not replace them. Read the scorecard as a repeatable check that catches the obvious problems, not as a measurement of whether the piece is right.

(Any scorecard numbers shown in this series are manual review estimates unless labeled as measured; the automated readability tooling was unavailable at the time of writing.)

Why evals, and what is an LLM-as-a-judge, really?

Tue, 15 Sep 2026 00:00:00 GMT

The judgment you already make

You shipped a generated image, a marketing line, or a model response, and you asked yourself whether it was good enough. You answered by eye, once, and moved on. The next person on your team answered differently. Neither of you wrote down what “good enough” meant, so neither answer travels past the moment you made it.

That private judgment is already an evaluation. You held criteria in your head, you applied them to one artifact, and you reached a verdict. The parts missing were a written-down standard, a way to run it again next week, and a way to hand it to a colleague and get the same answer back.

An eval supplies those missing parts. You state the criteria a response has to meet, you hand a response and those criteria to a judge, and you get back a score and an explanation you can store, rerun, and share. Mizan runs that judge on Google’s Vertex AI Gen AI Evaluation Service, across text, images, audio, video, and music.

The judge is a model

Automated scoring used to lean on string matching or reference answers. That works when there is one correct output. Generative work rarely has one correct output: a tagline can be strong in three different directions, and an image can satisfy a brief without matching any reference pixel for pixel.

An LLM-as-a-judge scores the way a careful reviewer would. You give a model the artifact and a written standard, and the model returns a judgment against that standard, along with its reasoning. The model doing the judging is the same class of model that produced the work, pointed at a different task: assessment instead of generation.

Three properties make this worth adopting, and they are the same three you were missing when you judged by eye. The judgment is explicit, because the standard lives in a template you wrote rather than in your head. It is repeatable, because running the same template over the same input gives you a comparable result each time, which is what lets you track a draft improving across revisions. And it is shareable: the template is a file, so you can commit it, hand it to a teammate, or import one that someone else authored, and the standard travels with the work instead of staying in one reviewer’s head.

Four kinds of judgment

Mizan gives you four metric kinds, and they differ by the shape of the question you are asking.

A pointwise metric answers a single question about a single response: how concise is this, how well does it follow the prompt, on a scale you define. Reach for it when you want one number and a reason.

A rubric metric checks several named criteria at once and reports against each. A brand rubric might ask, in one run, whether the copy uses the approved voice, avoids competitor names, and carries the required legal line.

Two more cover the remaining shapes. A custom_schema metric returns structured fields you specify, so a compliance check comes back as typed data you can act on rather than prose you have to parse. A pairwise metric sets two candidates side by side and picks the stronger one, for the times the honest question is “which of these two.”

You will meet all four across this series. The rest of this piece runs the first one.

Run one eval in ten minutes

You need a Google Cloud project with the Vertex AI API enabled and Application Default Credentials on your machine. The user guide covers install and configuration in full; the short version follows.

Install the CLI (the install is cgo-free, so no C toolchain is required):

go install github.com/ghchinoy/mizan/cmd/mizan@v0.1.0

Point Mizan at your project:

mizan config set project-id <your-project-id>
mizan config show

Create a pointwise metric that scores conciseness, then run it against a sample response:

mizan registry create --id demo/conciseness --name "Conciseness" \
    --description "Scores how concise a response is" \
    --kind pointwise \
    --prompt "Rate how concise this response is from 0 (verbose) to 1 (concise). Response: {{response}}" \
    --model gemini-2.5-flash

mizan eval run --metric demo/conciseness --field response="The cat sat on the mat."

The run returns a score and the judge’s reasoning:

Score:        1
Explanation:  The response 'The cat sat on the mat.' is a very short, direct, and grammatically complete sentence that conveys its meaning with no superfluous words, making it maximally concise.

That output holds the whole idea. Your standard (“how concise, 0 to 1”) now lives in a template named demo/conciseness. Anyone with the template runs the same check and gets a comparable score with a stated reason. Mizan persists each run by default, so mizan results list and mizan results show <run-id> let you look back at what you scored and why. Add --no-store when you want a run to leave no record.

Swap the response for your own copy and rerun. Change the prompt to score a dimension you care about (tone, factual grounding, adherence to a brief) and you have authored a second metric. The mechanics hold steady as the criteria change.

Mizan | Blog

Developing for Mizan: extending the core

The shape of the codebase

The metric registry and its three seams

The evaluation engine and dispatch by kind

A concrete walkthrough: how kind: heuristic was added

Where results live

The contribution gates

Read next

Sidebar: graded by Mizan

Turn a writing skill into an eval: grading this series with Mizan

The standard is already a rubric

Let the model draft the criteria

Grade a real draft

Ship it so others can run it

What the score does not measure

Read next

Sidebar: graded by Mizan

Contribute a template pack: extend Mizan without touching the core

A pack is a directory of data

Author a template into the pack

The validate gate

Share it, then import it

Next

Sidebar: graded by Mizan

Using Mizan in your role: per-persona playbooks, part 2 (Genmedia configurator, Brand Lab)

Genmedia configurator: the eval-set is the calling shape

Brand Lab: draft the rubric from the brand book

Next

Sidebar: graded by Mizan

Using Mizan in your role: per-persona playbooks, part 1 (Asset creator, Asset manager)

Asset creator: one eval, then a set, then a scorecard

Asset manager: curate a set that keeps concerns apart

Next

Sidebar: graded by Mizan

One spectrum, four users: from brand alignment to technical metrics

Four users, one spectrum

The tool is fixed; the criteria move

Brand end: import a pack and score alignment

Technical end: a typed verdict a pipeline can gate on

Where you sit changes what you write, not what you run

Sidebar: graded by Mizan

Why evals, and what is an LLM-as-a-judge, really?

The judgment you already make

The judge is a model

Four kinds of judgment

Run one eval in ten minutes

Read next

Sidebar: graded by Mizan

A concrete walkthrough: how `kind: heuristic` was added