Jev vs LLMs vs Traditional ML for Classification

What does a classifier actually need to give you?

Two things, not one. The obvious output is the class: fraud or not, refund or billing, safe or block. The output that actually runs the business is the probability attached to it. Almost every real classification system is a threshold on that number — auto-approve above 0.95, send to a human between 0.6 and 0.95, auto-decline below. If the number is meaningless, the thresholds are meaningless, and the "accuracy" you measured on a test set says nothing about what happens at each cut-off in production.

A probability you can trust is a calibrated one. The definition is simple: take every case the model called about 90% likely to be fraud; if the model is calibrated, about 90% of them really are fraud. Same at 70%, same at 30%. You measure the gap with Expected Calibration Error (ECE) — bucket predictions by stated confidence, compare stated confidence to observed accuracy in each bucket, take the weighted average of the gaps — and you eyeball it with a reliability diagram, where a perfectly calibrated model sits on the diagonal.

Hold onto that definition. Everything that follows — why the LLM-as-classifier hack is a mistake, what Jev is for, and when a boring gradient-boosted tree still wins — comes down to which of the three approaches hands you a number that passes this test, and at what price.

What do traditional ML classifiers actually give you?

Logistic regression, gradient-boosted trees (XGBoost and friends), and small fine-tuned transformers all do the same thing at the output layer: they emit a class probability directly. That number is a real, differentiable function of the input, learned from labeled examples, and it is the same number every time you run the same input. Everything the operations side of a company loves about these models follows from that:

• Cheap and fast. A tree ensemble or a linear model scores a row in microseconds to milliseconds on a CPU. At millions of decisions a day, that is the difference between "free" and a line item.
• Deterministic and reproducible. Same model, same input, same output — which is what a regulator, an auditor, or a colleague debugging a complaint needs.
• Interpretable and auditable. Logistic weights and tree feature importances tell you why; a reliability diagram tells you how much to trust it.
• Calibratable. Raw scores from a boosted tree or an SVM are often not calibrated, so you fit a second-stage map on held-out data: Platt scaling (a logistic fit on the scores) or isotonic regression (a monotone, non-parametric fit). scikit-learn packages both as CalibratedClassifierCV.

One caveat that matters for the rest of this post: neural networks are not calibrated by default. Guo et al. 2017, "On Calibration of Modern Neural Networks", showed that as networks got deeper and more accurate they also got systematically overconfident, and proposed temperature scaling — a single learned scalar on the logits — as a cheap fix. So even a fine-tuned BERT-class classifier needs a calibration step. The point is that for this whole family, calibration is a known, solved, one-line problem. You just have to remember it exists.

The cost is the labeled data. A trained classifier only knows the labels it was trained on, only performs on the distribution it was trained on, and adding a category means going back to the labeling queue. That single limitation is what created the demand for the next two options.

Why is "LLM as classifier" usually the wrong call?

Here is the pattern, and if you have shipped anything with an LLM in the last three years you have either written it or reviewed it: "Classify the following as A, B, or C and give a confidence from 0 to 100." Parse the JSON, threshold on the confidence, done. It works in the demo. It is also the single most common misuse of a language model I see in production, for five reasons, and the first one is the one that matters.

The confidence is not a probability. "confidence: 0.9" is a string the model generated because your prompt asked for a number. It was not computed from anything. It is text.
Cost and latency. Every decision is an autoregressive generation — hundreds of milliseconds to seconds, and paid per token — against a classifier that is effectively free and instant. At scale, this is the difference between a rounding error and a budget line.
Non-deterministic, prompt-sensitive, and drifting. Rephrase the instruction, reorder the labels, or wait for the provider's next model version and the decisions move. There is no fixed artifact to version.
No real decision threshold or reliability curve. Because the number is not calibrated, the threshold you set is a guess, and you cannot draw the diagram that would tell you how wrong it is.
Inconsistent label adherence. Models invent labels, merge two, or answer in prose. You end up writing a parser and a retry loop around a classifier.

Reason one deserves the detail, because people push back on it: "surely the model has some internal probability." It does — token log-probabilities — but those are probabilities over the next token, not over your classes, and post-training distorts them. OpenAI's own GPT-4 Technical Report is unusually candid here: the pre-trained base model was well-calibrated on its evaluations, and RLHF post-training reduced that calibration. The alignment process that makes a model pleasant to talk to also makes it more confidently wrong. Tian et al. 2023, "Just Ask for Calibration", studied exactly the verbalized-confidence trick on RLHF models and found that the stated numbers need deliberate prompting strategies just to reduce their error — out of the box, they skew overconfident. So the "0.9" you are thresholding on carries the fingerprint of a training process that was optimizing for helpfulness, not for honesty about uncertainty.

Now the fair part. There is a real niche where an LLM is the right classifier, and it is bigger than the purists admit:

• Genuine zero-shot with no training data. You have a label set and zero examples, and you need something today.
• Brand-new or long-tail labels. Categories that change weekly, or that appear a handful of times a month, will never accumulate enough examples to train on.
• Tasks that need world knowledge or reasoning about the text. "Is this support ticket describing a bug that a recent regulation would make reportable?" is not a bag-of-features problem.
• Low volume. If you make a few hundred decisions a day, cost and latency are irrelevant and the flexibility is worth it.

In those cases, use the LLM — but use it for the label, and treat its confidence as a hint, not a probability. If you need a threshold you can defend, measure calibration yourself on a labeled sample, or reach for a model that was trained to give you one. That is the gap the third option is aimed at.

Get the weekly AI engineering brief

Classifiers, calibration, agents, and the tools worth using — one practical email a week. Plus the free roadmap PDF.

By subscribing you agree to receive emails from AI Engineer Insights. Unsubscribe anytime. See our Privacy Policy.

Where does Jev fit — the zero-shot calibrated middle?

Jev is TypeSafe AI's "System One" model, and the cleanest way to describe it is as a direct answer to the section above. Instead of generating text, it returns a typed decision — a Choice among up to 255 labels, a Score, or a calibrated probability — plus a confidence, in one non-autoregressive pass. No prose, no JSON to parse, no "give me a number from 0 to 100." (We have a fuller explainer on what Jev is and a head-to-head on Jev vs LLMs; this section is about where it sits relative to a trained classifier.)

The part that matters for this post is how it was trained. Jev uses RLCD — Reinforcement Learning for Calibrated Decisions — a post-training process built specifically so that the confidence it emits tracks how often it is actually right. That is the exact property RLHF erodes in a chat model. The stated goal is to close the LLM calibration gap while keeping the thing a trained classifier lacks: zero-shot flexibility. You define the labels at request time, with no training data, and change them next week without a retraining cycle. It is also priced like infrastructure rather than like a chat model — $0.042 per million input tokens, output free, 32k context — which, with the single-pass design, is why TypeSafe positions it as the cheap, fast option. (Those speed and cost figures are TypeSafe's own claims; treat them as vendor numbers.)

That is what makes it "more advanced" than the LLM hack, and the difference is structural rather than a matter of degree. The LLM produces a confidence because you asked; Jev produces one because that is the only thing it is built to produce, and it was optimized for that number to mean something. Behind it is a team with the pedigree to attempt that: TypeSafe AI was founded in 2024 in San Francisco with a $40M seed from DCVC, and its CEO, Diogo Almeida, is ex-OpenAI and Google Brain, a co-inventor of RLHF and InstructGPT, and a contributor to GPT-4 — which is to say, someone who knows precisely how RLHF breaks calibration, building the corrective.

The honest limits, from independent testing rather than the launch post:

• It is calibrated, not perfectly calibrated. An independent study measured an ECE of roughly 0.107 — about 4.4× a well-calibrated baseline — with confidence noticeably shakier in the 0.3–0.8 mid-range, which is exactly the range where your "send to a human" band usually lives.
• Trained in-distribution classifiers still beat it on raw accuracy. Zero-shot has a ceiling; a model that has seen ten thousand of your labeled examples knows things about your data that no runtime label description can convey.
• Prompt injection can move the verdict. VentureBeat covered an Octomind demo in which a block probability of 0.76 fell to 0.48 after a fake "user pre-approved" field was inserted into the input. A calibrated number is only as trustworthy as the input it was computed on — and if that input includes attacker-controlled text, it is not fully trustworthy.

None of that disqualifies it. An ECE of 0.107 with an actual reliability curve you can measure is a different universe from a verbalized "0.9" with no curve at all. But it does set up the real question. For the mechanics of calling it, see how to use Jev.

Can Jev or a decision model replace traditional classifiers?

No, not wholesale — and yes, in a specific and valuable set of cases. Both halves matter, so here they are separately.

No, for the workhorse case. If you have a stable, high-volume, in-distribution task and you already hold labeled data — fraud scoring on your own transaction history, spam on your own mail, routing between a fixed set of queues — a calibrated XGBoost or a fine-tuned small transformer still wins on every axis an operations team cares about: cost per decision, raw accuracy, reproducibility, and auditability. It is deterministic, its calibration is a solved one-liner, it has no prompt-injection surface because it has no prompt, and it is nearly free to run. Do not rip out a working classifier to put a zero-shot model in its place; you would be paying more to be less accurate and less explainable.

Yes, where it genuinely wins. The trained classifier's superpower is also its precondition: it needs the labeled data. Wherever that precondition fails, a decision model is not just competitive but the obviously better tool:

• Cold start. New product, new market, new policy — no labels yet, and the decision has to ship anyway.
• Fast-changing label sets. Moderation taxonomies, incident categories, campaign-specific intents. Retraining every time the list changes is a treadmill.
• True zero-shot and long-tail categories. Labels that will never see enough examples to train on.
• "Good enough, calibrated, this afternoon." When a measurable, thresholdable probability by end of day beats a better one after a six-week labeling project.

Put those together and the net effect is clear: a decision model mostly replaces the LLM-as-classifier hack, not a real trained classifier. Everywhere a team today is prompting a chat model for "a confidence from 0 to 100" and thresholding on it, a model built to emit a calibrated decision is a strict upgrade — cheaper, faster, typed, and with a number that at least attempts to mean something. Everywhere a team has a calibrated tree ensemble humming along on in-distribution data, nothing has changed. It is the same "not competitors, different jobs" framing we used for RAG vs fine-tuning: the tool that needs no training data and the tool that exploits it are answering different questions.

The one place the "no" softens: a decision model can be the bridge to a trained classifier. Ship Jev zero-shot on day one, log every decision alongside the eventual ground truth, and in three months you have the labeled dataset that lets a cheap in-distribution model take over the high-volume core — while the zero-shot model keeps handling the long tail. That is how the two coexist rather than compete.

Traditional ML vs LLM vs Jev: the three-way comparison

The whole argument in one table. Read the first column first — it is the one that decides most real deployments.

Approach	Calibrated probability?	Needs training data?	Cost & latency	In-distribution accuracy	Zero-shot flexibility	Interpretable / auditable?	Best for
Traditional ML classifier	Yes — outputs P(class); calibrate with Platt scaling or isotonic regression	Yes — labeled, in-distribution examples	Near-free, milliseconds, deterministic	Highest, when the data is representative	None — a new label means relabeling and retraining	Yes — features, weights, and a reliability curve you can inspect	Stable, high-volume tasks where you have labeled data
LLM as classifier	No — the 'confidence' is generated text; RLHF makes it worse, not better	No — a prompt is enough	Highest — autoregressive tokens per call, seconds of latency	Often good, but non-deterministic, prompt-sensitive, and drifts across versions	Best — plus reasoning and world knowledge about the text	Partly — it can explain itself, but the explanation is not the decision	Low-volume, long-tail, or reasoning-heavy labeling with no training data
Jev (decision model)	Trained for it (RLCD); an independent test still measured ECE ≈ 0.107, weakest in the 0.3–0.8 range	No — labels are defined at runtime	Low — one non-autoregressive pass; $0.042 per million input tokens, output free (TypeSafe's figures)	Below a trained in-distribution classifier	Yes — a Choice of up to 255 labels, a Score, or a probability	Typed decision plus confidence; no feature attribution	Cold-start, changing label sets, and replacing the LLM-as-classifier hack

Notice the shape of the middle row. The LLM is the best at exactly one thing — zero-shot flexibility with reasoning — and the worst at nearly everything else, which is why it belongs in a niche rather than in the default slot. The bottom row keeps most of that flexibility and fixes the cost, the typing, and (imperfectly) the calibration; what it gives up is the reasoning and world knowledge of a full language model, and the raw accuracy of a model trained on your data.

How do you choose between them?

Ask the questions in this order and stop at the first one that answers "yes."

Do you have labeled, in-distribution data and a stable label set? Train a classical model, calibrate it with Platt or isotonic, plot the reliability diagram, and ship. This is the default, and it is the default for a reason.
Does the decision need reasoning or world knowledge about the text, at low volume? Use an LLM for the label. Do not threshold on its confidence; either measure calibration on a labeled sample or route uncertain cases to a person.
Do you need a typed, thresholdable decision now, with no labels or a label set that keeps moving? Use a decision model like Jev. Verify its calibration on your own data — an ECE of about 0.107 in one independent test is not the same as zero — and keep attacker-controlled text out of the input, or at least out of the fields the decision hinges on.
Is it a mix? Then it is a pipeline: a decision model for the cold start and the long tail, logging into the dataset that trains the classical model for the high-volume core. If the decisions gate actions inside an AI agent, that calibrated number is also what tells the agent when to ask before acting.

Practical tips before you ship any of them

• Plot the reliability diagram, whatever the model. It takes a few hundred labeled examples and ten lines of code, and it is the only way to know whether the threshold you picked means what you think it means. If you cannot plot it, you do not have a probability; you have a number.
• Never threshold on verbalized LLM confidence. Use the label and route on something you have measured. If a colleague insists the model's "0.9" is meaningful, hand them the GPT-4 report's calibration section.
• Calibrate on held-out data, not training data. Platt and isotonic fits on the training set just re-learn the training set's overconfidence. CalibratedClassifierCV handles the cross-validation for you.
• Treat vendor speed and cost numbers as claims to verify. TypeSafe's $0.042 per million input tokens and single-pass latency are its own figures; benchmark them on your traffic before they go into a cost model.
• Assume the mid-range is the weak spot. Whether it is an LLM or a decision model, calibration tends to be worst between roughly 0.3 and 0.8 — exactly where "escalate to a human" bands live. Widen the band or add a second signal there.
• Log every zero-shot decision with its eventual outcome. That log is the labeled dataset that lets you graduate to a cheap trained classifier later. The zero-shot phase should be building toward its own replacement.
• Keep untrusted text out of the decision input. A 0.76 that becomes 0.48 after one injected field is the demonstration; the fix is structural — separate the fields the decision hinges on from anything a user or attacker can write.

Frequently Asked Questions

Can an LLM be used as a classifier?

Yes, and for genuine zero-shot work — no training data, brand-new or long-tail labels, low volume, or tasks that need world knowledge and reasoning about the text — it is often the right call. The mistake is treating it as a drop-in replacement for a trained classifier on a stable, high-volume task: it is slower, far more expensive per call, non-deterministic, prompt-sensitive, and the confidence number it reports is not a calibrated probability.

Is an LLM's confidence score a real probability?

No. When you prompt a model to 'give a confidence from 0 to 100', the number is generated text, not a measured probability. Token log-probabilities are not class probabilities either, and RLHF post-training distorts them further: OpenAI's GPT-4 Technical Report shows the pre-trained model was well-calibrated and post-training reduced that calibration. If you need a number to threshold on, you have to measure calibration (ECE, reliability diagram) rather than trust the model's self-report.

What is calibration in machine learning?

A classifier is calibrated when its probabilities match reality: among all the cases it calls about 90% likely, roughly 90% actually belong to that class. You measure it with Expected Calibration Error (ECE) and a reliability diagram. Traditional classifiers are calibrated after training with Platt scaling or isotonic regression (scikit-learn's CalibratedClassifierCV); modern neural networks tend to be overconfident, which Guo et al. 2017 showed temperature scaling can fix.

Can Jev replace traditional ML classifiers?

Not wholesale. On a stable, high-volume, in-distribution task where you already have labeled data, a calibrated XGBoost or fine-tuned classifier still wins on cost, accuracy, reproducibility, and auditability, so there is no reason to rip out a working one. Jev genuinely wins where there is no labeled data, where the label set changes often, for long-tail categories, or when you need a good-enough calibrated decision in an afternoon instead of a labeling project. In practice it mostly replaces the LLM-as-classifier hack, not a real trained classifier.

What is the difference between Jev and an LLM for classification?

An LLM generates text autoregressively and, if asked, verbalizes a confidence that is not a calibrated probability. Jev is TypeSafe AI's 'System One' model: it returns a typed decision — a Choice of up to 255 labels, a Score, or a calibrated probability — plus a confidence, in one non-autoregressive pass, with no text. It is trained with RLCD (Reinforcement Learning for Calibrated Decisions) so the confidence is meant to track accuracy, and it keeps the zero-shot flexibility of an LLM. Independent testing still found its calibration imperfect (ECE around 0.107) and its verdict movable by prompt injection.

When should I use a traditional ML classifier instead of an LLM?

Whenever the task is stable, the volume is high, and you have (or can cheaply get) labeled in-distribution data: fraud scoring, spam, churn, routing between a fixed set of queues. A logistic regression, gradient-boosted tree, or small fine-tuned transformer will be cheaper, faster, deterministic, reproducible, and easier to audit — and after Platt or isotonic calibration it gives you a probability you can actually set a threshold on.

Want 1:1 help? Book a session

5.0 · 18 reviews

Career guidance, resume & interview prep, or tech consulting — 1:1 with a Lead AI Engineer. Start with a free 30-min quick chat.

On This Page

Jev vs LLMs vs Traditional ML: The Right Way to Classify in 2026