Jev vs LLM-as-a-Judge

Most teams grade their agent’s outputs by handing them to an LLM and asking it to write a verdict. That pattern is called LLM-as-a-judge, and it works well enough that it has become the default. TypeSafe’s decision model, Jev, can grade the same outputs, and it does so in a different way that changes what you can do with the result. This post explains the difference, shows where it mattered in our measurements, and says which judge to use for which kind of rubric.

It assumes you know what LLM-as-a-judge is. If not, read LLM-as-a-Judge: Score AI Agent Outputs Automatically first.

TL;DR

An LLM judge writes a verdict. Jev gives a probability for a typed question. When the rubric is closed and the evidence is included in the request, Jev matched the LLM judge on accuracy. Jev has better-calibrated probabilities, is 1/5 as expensive, and has 1/10 the latency of the LLM judge. When the rubric is open and reading a whole article is required, the LLM judge follows expert ratings much more closely.

• Use Jev for closed criteria if the evidence is present, i.e., the answer is source-supported, on-policy, or matching a reference. It can also be used if a threshold or routing rule is needed, or if a judge is needed fast enough and cheap enough to fit into the request path.
• Use an LLM judge for open rubrics, long-context checks, e.g., a summary consistency check, and if someone needs to read why the answer failed.
• Use neither if the code can check the criterion exactly. When a decision is consequential and novel, a person should be used.

The rest of the post discusses where the numbers are coming from and why the two judges behave as they did.

Two different kinds of answer

An LLM judge is a generative model. You give it a rubric and an output, and it writes text. Ask for JSON and it writes JSON. Ask for a confidence number and it writes one. Every part of the verdict, including the confidence, is text the model chose to produce. The number was written, not measured.

Jev is a decision model. You do not ask it to write anything. You ask it a fixed question with a fixed set of possible answers, and it returns a probability for each answer. There are three question shapes:

Question type	You ask	Jev returns	Use it for
Noul	A yes/no question with true and false criteria	noul, the probability of yes	Pass/fail gates: is this answer supported, is this reply on-policy
Choice	Pick one of N labeled options	choice, probabilities per option, confidence	Pairwise preference, categorical rubrics
Score	Rate on an ordered scale you define	score, probabilities per level, legend, confidence	Likert-style rubrics: coherence 1 to 5, severity low/medium/high

The two requests look similar. Here is the same faithfulness rubric asked both ways.

// LLM judge: a prompt, and a verdict parsed out of the text that comes back.
const verdict = await chat.chat.send({
  chatRequest: {
    model: '~openai/gpt-luna-latest',
    temperature: 0,
    responseFormat: { type: 'json_object' },
    messages: [
      { role: 'system', content: `Grade the answer. acceptable = ${RUBRIC_TRUE} unacceptable = ${RUBRIC_FALSE} Reply with JSON: {"acceptable": true|false, "confidence": 0 to 1}` },
      { role: 'user', content: `knowledge: ${knowledge}\n\nquestion: ${question}\n\nanswer: ${answer}` },
    ],
  },
});
// -> {"acceptable": false, "confidence": 1}

// Jev: a typed question over the same state, and a probability back.
const decision = await decisions.alpha.decisions.create({
  decisionsRequest: {
    model: 'typesafe/jev-1.13',
    state: { knowledge, question, answer },
    questions: {
      acceptable: {
        type: 'noul',
        instructions: 'Does answer directly answer question using only facts that knowledge states?',
        criteria: { true: RUBRIC_TRUE, false: RUBRIC_FALSE },
      },
    },
  },
});
// -> answers.acceptable.noul === 0.01

The LLM judge’s 1 is a number it wrote. Jev’s 0.01 is a claim about frequency: across many answers like this one, about 1% should turn out to be acceptable. That claim can be checked. If it holds, you can set a threshold, route the uncertain middle to a person, and reason about error rates. If it does not, the probability is decoration. Full request and response shapes are in the Decisions API reference and the TypeSafe SDK guide.

How we compared them

We ran both judges on two public datasets through OpenRouter on 2026-09-21. The whole run cost $0.041.

Faithfulness set. We took 50 rows from the QA split of HaluEval (Li et al., 2023), each with a short passage, a question, a correct answer, and a made-up answer, for 100 answers in all. We had Devin, the coding agent that ran this experiment for us, review every item against its passage, drop six garbled or ambiguous rows, and relabel eight of the made-up answers as acceptable because the passage supported them. That left 88 items: 52 acceptable and 36 unacceptable. This is a closed rubric. The evidence is in hand, and the question is whether the answer sticks to it.

Summary set. We took 10 news articles from SummEval (Fabbri et al., 2021), each with 5 machine-written summaries. Human experts had rated each summary from 1 to 5 for coherence (does it read well) and consistency (does it stick to the facts in the article). This is a harder, more open rubric over a much longer text.

Judges. Jev ran as typesafe/jev-1.13, resolving to typesafe/jev-1.13-20260917. The LLM judge ran as ~openai/gpt-luna-latest, resolving to openai/gpt-5.6-luna, with temperature: 0 and JSON output. Both judges got the same rubric text, so any difference comes from the model and not the wording.

For each judgment we recorded the verdict, the probability, the latency, and the usage.cost from the response. For the LLM judge we turned “acceptable plus confidence” into a probability of acceptable so both judges sat on the same 0 to 1 scale.

Agreement was a tie

On the 88 faithfulness items, Jev agreed with the labels on 84 and the LLM judge on 83. Both had precision 0.98. Two of the misses were the same item for both judges, and in both cases the label was wrong. When both judges disagree with you on the same item, check your label.

If you only look at accuracy, there is nothing to choose between them. Accuracy is the wrong lens, though, because it only asks which side of 0.5 the judge landed on and treats 0.51 the same as 0.99.

Calibration is where they differ

To see whether the probabilities meant anything, we used two views. The Brier score is the mean squared error between the probability a judge gave and the true 0 or 1 label, so a judge that is wrong at 90% sure is penalized more than one wrong at 60% sure. A reliability table groups cases by the probability the judge gave and reports how often those cases were actually acceptable.

Jev’s Brier score was 0.043 and the LLM judge’s was 0.054. The reliability tables explain why.

jev:  bin 0.0-0.2: n=34 mean_p=0.04 observed=0.03
      bin 0.2-0.4: n=1  mean_p=0.34 observed=1.00
      bin 0.4-0.6: n=4  mean_p=0.48 observed=0.50
      bin 0.6-0.8: n=7  mean_p=0.74 observed=1.00
      bin 0.8-1.0: n=42 mean_p=0.93 observed=0.98
llm:  bin 0.0-0.2: n=39 mean_p=0.01 observed=0.10
      bin 0.8-1.0: n=49 mean_p=0.99 observed=0.98

Jev put four cases between 0.4 and 0.6 and got half of them right, which is what a 0.5 should do. Cases it scored above 0.8 were acceptable 98% of the time. The LLM judge had no middle ground. Across 88 cases its confidence took only six values (0.82, 0.84, 0.87, 0.98, 0.99, and 1), so every case landed near 0 or near 1, and the cases near 0 were acceptable 10% of the time. It was as sure about its mistakes as about everything else.

In practice this is the difference between a judge you can route on and a judge you can only read. Seven of Jev’s 88 cases fell between 0.3 and 0.7, and those are exactly the ones worth sending to a person. The LLM judge gave us no such cases to send.

Cost and latency differ by 5x and 10x

Jev cost $0.021 per 1,000 judgments and answered in 171 ms at the median. The LLM judge cost $0.114 per 1,000 and took 1,662 ms. For a nightly eval of 1,000 items, the cost gap is about $34 a year, which does not matter. If the judge sits inside a request path and blocks a response until it answers, the latency gap matters a lot, and it is the reason Jev fits as a gate in production where an LLM judge fits as an offline grader.

Both judges resisted the classic biases, Jev slightly more

The MT-Bench paper found two common biases in LLM judges. They prefer whichever answer is shown first, and they prefer the longer of two answers. Both are cheap to test, so we tested both.

Position. We showed each judge the same two answers twice, in reverse order the second time, across all 36 pairs of one acceptable and one unacceptable answer. For Jev this is a Choice question with one criterion per slot. Neither judge changed its pick when the order flipped. Both got 70 of 72 orderings right and missed the same pair in both orders, which points at the label rather than at position. If you run this test yourself, run it on your hardest pairs, where the two answers are close in quality, because that is where position bias shows up. The Choice criteria also only make sense when one answer is acceptable and the other is not. If both can be acceptable, write each criterion as a comparison (more specific, better supported, closer to the question) rather than a label, or Jev has no option that fits.

Verbosity. We took each unacceptable answer and padded it with two true sentences copied from its own passage. The padded answer is still wrong, but it grew from 69 to 376 characters on average. We kept only the pairs where padding left the answer wrong, since in a few cases the copied sentences happened to contain the correct answer and the padded version became acceptable.

Judge	Unpadded wrong answers accepted	Padded wrong answers accepted	Flips
Jev Noul	1 / 36	2 / 36	1 (probability 0.48 to 0.57)
LLM judge	1 / 36	4 / 36	3

Padding nudged Jev. Its average probability of acceptable on the wrong answers rose from 0.08 to 0.12, and the one case that crossed 0.5 was already sitting at 0.48. The LLM judge flipped three cases, and each moved from a confident reject to a confident accept. The pattern is the same as in the calibration table. When Jev is uncertain, it says so, and when the LLM judge is wrong, it is wrong confidently.

Where the LLM judge wins

The picture changes when the judge has to read a whole article. On the 50 SummEval summaries, Jev got one Score question per criterion, with the five levels written out as criteria, and the LLM judge got the same five level descriptions and returned an integer from 1 to 5. We compared both to the expert averages with Spearman correlation, where 1.0 is a perfect match in ranking.

Criterion	Jev Score, Spearman vs experts	LLM judge, Spearman vs experts
Coherence	0.42	0.44
Consistency	0.47	0.72

Coherence is a tie. Consistency, the criterion you care about when you are hunting hallucinations, is a clear win for the LLM judge. Our guess is that checking consistency means going claim by claim through the summary and finding where the article supports each one. That is a natural step-by-step process for a generative model, while Jev has to do it in one pass over roughly 1,000 tokens of state.

Jev was not useless here. Both judges caught all three summaries the experts rated below 4 on consistency, with no false alarms on the other 47, so Jev still works as a rough pass/fail gate. But if you want a ranking that tracks expert judgment, the LLM judge’s is the better signal. Jev cost $0.043 per 1,000 summaries at 168 ms, versus $0.261 at 2,902 ms for the LLM judge, so you are paying about 6x for that signal. Jev’s cost here is double its faithfulness cost because each request carried about twice the input tokens, the whole article plus two Score questions instead of one short passage and one Noul.

There are two more situations where a generative judge is the right tool, and we did not benchmark them because there is nothing to measure against.

• Open rubrics. If the question is “is this a good answer?” with no written criteria, you need a generative judge. Jev needs the criteria spelled out, so if you cannot write them down you cannot ask Jev.
• Written explanations. If a reviewer needs to know why an answer failed, the LLM judge’s reasoning is the product. Jev tells you how sure it is and nothing else.

Which judge for which rubric

An LLM judge has three alternatives, a deterministic check in code, a Jev question, or a human. Pick one per criterion, not one per project.

If the criterion is…	Use	Because
Something code can check (schema, arithmetic, forbidden strings)	Deterministic check	Exact, free, no bias to test
Closed, evidence in hand, needs a probability or a threshold (supported by source, on-policy, matches reference)	Jev Noul or Choice	Calibrated probabilities, ~170 ms, ~$0.02 per 1,000
Ordered scale where the levels are describable (severity, tone on a 1 to 5)	Jev Score	Distribution over levels plus confidence, same cost profile
Open, or needs reasoning across long context (consistency of a summary, quality of a plan)	LLM judge	0.72 vs 0.47 correlation with experts on consistency in our test
Needs a written explanation a human will read	LLM judge	Jev returns a probability and no explanation
Consequential and novel	A person	Neither judge is a substitute for one

The Jev verified cascade cookbook shows the middle rows in production. A cheap model drafts an answer, a Jev question verifies it, and anything in the uncertain band gets escalated. The gate tool calls with Jev cookbook applies the same idea to an agent’s tool calls.

Set your own threshold

Our numbers show the shape of the trade-off. To set a production threshold you need the same measurement on your own data.

1. Label 50 to 100 items that represent what you will run in production, with a written definition of what counts as acceptable.
2. Send each one to both judges and record the probability, the latency, and the usage.cost.
3. Build the reliability table and ignore the accuracy line. Pick your accept threshold as the lowest bin whose observed rate you would sign off on, and send everything between that and your reject threshold to a person or an LLM judge.
4. Rerun the table whenever the rubric or the model version changes. The model field in every Decisions response tells you which version answered.

The Jev model page has current pricing and limits, the TypeSafe SDK guide has the full Decisions API shape, and the LLM-as-a-judge post covers how to write the rubric in the first place.

FAQ

Is Jev a calibrated judge model?

Jev returns a probability with every judgment, and on our 88-item faithfulness set those probabilities were accurate. The Brier score was 0.043, and when Jev gave a probability above 0.8 the answer was acceptable 98% of the time. Calibration depends on the exact task you’re measuring, so measure it on your own labels before choosing a threshold.

What are the alternatives to LLM-as-a-judge?

When code is able to decide the answer, use a deterministic check. For anything consequential or novel, involve a person. For closed-rubric cases where having a probability is useful, use a decision model like Jev. Most production pipelines are a mix of all three. For open rubrics and cases where you want a written explanation, keep a generative judge around.

When should I still use an LLM as a judge instead of Jev?

Use an LLM judge when the rubric is open and you can’t write the criteria down, when the score needs a written explanation, or when the judgment has to proceed step by step over a long text. On factual consistency of SummEval summaries, the LLM judge’s correlation with expert scores was 0.72, while Jev’s was 0.47.

How do I calibrate Jev for my own evaluation?

Label 50 to 100 items, send each one to Jev as a Noul or Score question through the Decisions API, and record the probability, the latency, and the usage.cost from each response. Compute the agreement with your labels and the Brier score, build a reliability table grouped by probability bin, and pick your threshold from that table.