Skip to content
Mizan
Blog

Mizan — Strongly-Typed LLM-as-a-Judge Evals on Vertex AI

Turn the judgment you already make by eye into an explicit, repeatable, shareable check. Author typed decision primitives (boul, choice, score) and native Vertex AI metrics across text, images, audio, video, and music—stored locally in pure-Go SQLite and shared as Git-native template packs.
User GuideBuild Evals SeriesScenarios Guide

Choose Your Path

Section titled “Choose Your Path”

Run your first eval

For builders getting started: install the CGO-free CLI, point it at a Google Cloud project, walk the local SQLite registry, and run end-to-end recipes across every metric kind. Start with the User guide and Testing guide.

Pick the right judgment shape

For teams turning fuzzy review criteria into deterministic checks: map each evaluation question to boul, choice, score, rubric, pairwise, or custom_schema. Read the Scenarios guide or Three typed primitives.

Share packs & wire agents

For teams standardizing quality bars: package metrics into credential-free validated template packs, operate Mizan from coding agents, or export rubrics to Stax. Explore Contribute a template pack, Agent skills, and Stax interop.

Core Capabilities

Section titled “Core Capabilities”

Typed Decision Primitives (boul, choice, score)

Answer software’s three core evaluation questions without hand-rolling JSON schemas: boul for binary pass/fail propositions with confidence, choice for grammar-constrained categorical routing, and score for calibrated continuous grading. How the primitives work →

Pointwise, Rubric, Pairwise & Custom Schema

Run Vertex AI EvaluateInstances pointwise metrics, multi-criterion rubric scorecards (with --rubric-detail), head-to-head pairwise (BASELINE / CANDIDATE / TIE) comparisons, and arbitrary custom_schema structured output.

Multimodal Across Five Modalities

Evaluate text, images, audio, video, and music with the same template contract. Pass --file key=/path/to/asset (auto-staged to your configured GCS bucket) or --gcs key=gs://... for pre-staged objects.

SQLite Provenance, Packs & Agent Skills

Every run persists its inputs, template snapshot, score, and explanation to a pure-Go SQLite store (mizan results list|show). Share standards via Git-native packs (mizan pack init|add|validate, mizan registry import|export), mizan mcp, and Agent Skills.

Quickstart Workflows

Section titled “Quickstart Workflows”

Install Mizan, configure your Vertex AI project, and run a strongly-typed boolean check with zero custom JSON schema boilerplate:

Terminal window
# 1. Install the CGO-free Go binary and set your GCP project
go install github.com/ghchinoy/mizan/cmd/mizan@latest
mizan config set project-id <your-project-id>


# 2. Register a typed boolean proposition check (boul)
mizan registry create --id safety/pii-check --name "PII Check" \
  --kind boul \
  --prompt "Does this customer message contain unmasked PII or credit card numbers? {{message}}"


# 3. Evaluate an input (persisted automatically to the local results store)
mizan eval run --metric safety/pii-check \
  --field message="My phone is 555-0199 and card is 4111-2222-3333-4444"

Read & Reference

Section titled “Read & Reference”