Jev Architecture Explained: Open-Source Clones, Benchmarks, Calibration, and Production Trade-Offs

I originally planned to write about Gemini 3.8 Live Extended Thinking and the way it separates low-latency foreground voice interaction from slower background reasoning and tool execution. Then TypeSafe released Jev on September 15, 2026, and I changed the topic, because I think this release can materially change a fairly large class of enterprise AI systems and non-AI systems using workflows.

The reason is not that we suddenly rediscovered classic encoder-based classification. If I have one or two business-critical branches with a bounded input and output space, I would still seriously consider training a dedicated model — ModernBERT, another BERT-family encoder, CatBoost/xgboost with embedder features, whatever fits the problem and gives maximum quality with target latency. We do exactly that in AgentUnicorn.AI for one of the signals used to decide whether a user has finished speaking. It sits on the latency-critical path of every voice turn, so paying for the full ML cycle — data, training, evaluation, deployment, drift checks — is justified.

What I am interested in is the layer below that threshold. A real agent contains many decisions where a better classifier would improve the system, but not enough to justify a separate dataset, training pipeline, deployment, monitoring, recalibration and maintenance cycle. Those tasks usually stay in the backlog, become error-prone rules, or burn money with another general-LLM call with constrained decoding if latency and unit economics allow it.

You can build an AutoML-like pipeline around this: generate synthetic data, enrich it with production examples, retrain small classifiers, monitor quality and distributions. When I was Head of AI at a Series A AI startup, I could reasonably put such a platform on the roadmap. As a pre-seed founder, I consider that to be too expensive in engineering time and organizational complexity, delaying delivery of more business-critical modules.

This is the context in which Jev matters to me. TypeSafe positions it as the first public model in a System One class: fast, typed, probabilistic decisions instead of free-form text generation. If a general decision model can give sufficiently good quality, low inference cost and useful probabilities across many bounded decisions without a separate training cycle for every one of them, it opens up a wide class of improvements that today are usually not worth implementing.

The open-source reaction made the release even more interesting. Within days, teams were trying answer-token logits, custom attention masks, pointer heads, NLI, whole-option likelihood, ModernBERT, contrastive training, calibration objectives, conformal prediction, RL on verifiable outcomes and DiffusionGemma. It looked less like a normal clone race and more like a distributed search over different combinations of fairly familiar ML ideas.

The question I kept coming back to while reading those implementations was simple: how many decisions inside an AI system actually need long and expensive autoregressive text generation?

I separate four kinds of evidence below: what TypeSafe actually disclosed, what black-box work strongly suggests, what the open-source projects demonstrate in code and in value, and what is still inference. That matters here because the repos are rocket-moving daily and, in a launch cycle like this, a plausible hypothesis can become an internet “fact” after being repeated a few times.

Everything below is current as of September 23, 2026.

---

TL;DR

Why I care about this

I have been building ML/AI enterprise solutions for 20 years and shipped them in 9 domains with over 500+ models built by teams under my guidance, and now I build low-latency conversational AI, so I am looking at Jev less as a leaderboard entry and more as a possible change in the system architecture. The Jev-class model opens the possibility of local improvement in enormously wide number of enterprise workflow scenarios without the need of high-class ML-team and expensive infrastructure, bringing the financial effect compared to spread of vibe-coding.

In case of Voice AI solutions, a single voice turn can trigger endpointing, tool routing, retrieval, escalation, safety checks, response strategy and model routing while the user is waiting. A few hundred extra milliseconds in the wrong place are audible; an unnecessary frontier-model call repeated across millions of turns is also visible in the unit economics.

So the questions I care about are practical: latency, cost per successful workflow, reliability under concurrency, auditability, and how much model-specific infrastructure we have to build and own.

Jev takes shared state plus typed questions and returns probability distributions over bounded answers. TypeSafe publicly confirms a new architecture, a parallel sampler, and a training method called RLCD — Reinforcement Learning for Calibrated Decisions. It does not publicly disclose the exact backbone, attention topology, or decision head.

The strongest black-box reconstruction so far points toward a model with:

• direct probability readout instead of autoregressive text generation;
• shared computation over a common state;
• isolated question branches;
• listwise interaction between options within a question;
• a likely causal transformer backbone;
• possibly sparse MoE, although that part is much more speculative.

Open-source work shows that many of these ideas do not require a proprietary architecture:

• mini-Jev / reflex: read answer-token logits after prefill;
• Kev: shared state + isolated question branches + pointer head;
• SemIf: read native option logits from Qwen3.5 with runtime-defined criteria/options and shared-state reuse;
• AlexWortega/openjev and other NLI variants: score runtime candidates as premise/hypothesis pairs;
• daseinlabs/open-jev: score the likelihood of full candidate sequences with shared KV;
• Nimble: get large gains from contrastive single-fact-flip training;
• Laya / GLiClass / Verdict: return to bidirectional encoders for fast decision workloads;
• decider / poorjev: focus on proper scoring, calibration, and abstention;
• DiffusionGemma/OpenJev: use parallel diffusion slots instead of autoregressive decoding.

The open implementations also make it clear where the real work starts after the first fast argmax:

• probabilities that remain calibrated under domain shift;
• high-cardinality and long-context behavior;
• option-order robustness;
• multi-question isolation;
• production serving under concurrency;
• deciding when a specialized decision model is better than a simple classifier, rules engine, or ordinary LLM call.

---

1. What is Jev?

TypeSafe describes Jev as a model for structured decisions rather than strings. A request contains a shared state and one or more typed questions; the public interface supports three primitives:

• Choice — choose among caller-supplied alternatives;
• Score — return a distribution over ordered levels and an expected score;
• Noul — a yes/no probability.

Conceptually:

state + typed questions + candidate answers
                  ↓
                 Jev
                  ↓
typed probability distributions

rather than:

prompt
  ↓
LLM
  ↓
token → token → token → JSON

What I find important is that this separation is visible in the API itself. state, questions, answer type and options are separate objects instead of one long prompt plus an output schema. That does not tell us exactly how the model is built, but it makes shared state and typed decisions part of the interface rather than a prompt convention.

TypeSafe's launch post lists an input price of $0.042 per million tokens, with output described as effectively free, and vendor-reported end-to-end latency of roughly 70–500 ms. The product pitch is essentially “fuzzy decision rules” that can be called like ordinary software.

Primary source: TypeSafe — Introducing System One Models & Jev

“Cannot hallucinate” needs a precise interpretation

TypeSafe says Jev “can’t hallucinate.” I would read that narrowly: structural, not epistemic.

If a Choice question only allows:

billing
technical
sales

Jev cannot return:

legal

or malformed JSON or a paragraph of prose; the output space is constrained by construction. It can still be wrong. If the correct answer is missing from the option set, the model has to distribute probability over the alternatives it was given, and it can do that very confidently. TypeSafe makes the same distinction in the launch material: the “0% hallucination” figure refers to guaranteed schema matching, not empirical correctness. And there is no guarantee of absence of incorrect variant choice.

From the consumer's point of view, Jev looks like a well-pretrained for any domain encoder, giving similar latency and budget - but managed by input and not requiring fine-tuning for your specific task, what significantly distinguishes from ModernBert and similar things.

---

2. What TypeSafe has disclosed — and what it has not

TypeSafe publicly confirms only a small part of the design, but three pieces are important:

2.1 Typed outputs

The model is built around bounded decision primitives rather than generated strings.

2.2 Parallel sampling

TypeSafe now documents a stronger property than the original launch wording: Jev ingests the state once and evaluates every question against it in parallel. That moves shared-state reuse from the black-box reconstruction bucket into the officially documented interface/serving behavior. It still does not tell us the exact internal attention topology or whether TypeSafe implements the same block-causal mechanism inferred by Hume and reproduced in Kev.

2.3 RLCD

The company calls its training method Reinforcement Learning for Calibrated Decisions.

The point of RLCD, at least from the public description, is not just to choose the right label but to make the probability useful to software: if the model returns 0.95, that number should mean something close to a 95% empirical success rate on comparable traffic. The exact loss, reward and data-generation recipe are still undisclosed.

This creates a data problem that ordinary language-model pretraining does not have. Masked-token training and next-token prediction get supervision almost for free from raw text; a general decision model needs examples closer to:

state → question → outcome

and, if the returned probability is supposed to drive production policy, the training process has to teach the probability, not only the winning label. Somebody has to create the decision situations, outcomes, hard negatives and calibration cases. In practice that usually implies a separate synthetic-data pipeline — often itself driven by a large LLM — before the decision model is trained.

TypeSafe now also states explicitly that Jev is not fine-tuned or LoRA-adapted per customer: the same weights serve every account. Domain adaptation happens through state, instructions, criteria and decomposition of a broad judgment into smaller questions. That is important for the economics of the model class. The whole point is to amortize one expensive training process across many workflows instead of starting another training cycle for every customer decision.

What TypeSafe does not publicly specify:

• exact backbone architecture;
• whether the model is dense or MoE;
• exact attention topology;
• exact readout head;
• reward function;
• loss function;
• synthetic-data construction pipeline;
• which weights are updated during RLCD.

I am spelling this out because a lot of secondary discussion already treats the inferred architecture as if TypeSafe had published it. They have not.

---

3. The strongest black-box reconstruction of Jev

Disclaimer: I do not have a Jev account, and I did not do any query to it (I was stopped by strong license agreement, bounding development of similar-class solutions), and all of the subsequent analysis is done based on other people's Jev's testing, its disclosed documents/posts/interviews, and my hypothesis how such a system could have been built in 2026.

The most detailed reverse engineering I found is Archer Hume's September 17 essay, Jev's Architecture Unmasked, based on roughly 10,000 API calls. It gives a fairly specific computational picture, but the evidence is not equally strong for every part of it.

3.1 Direct readout instead of a decode loop

The strongest evidence concerns the absence of a normal decode loop. A conventional causal LLM performs:

1. prefill — process the known prompt;
2. decode — repeatedly generate the next token.

For a bounded decision such as “which queue?”, the second phase is not inherently necessary; the model can map an internal representation directly to logits:

h → W h + b → softmax

Hume found that Jev's latency does not scale with the reported output_tokens field in a way consistent with actual token-by-token decoding.

Two Hume probes are useful here. A 255-option response was reported as 2,714 output_tokens. Separately, a 200-option query — reported as 1,911 output tokens — returned about as quickly as a 2-option query. Taken together, that strongly suggests the output_tokens field is accounting/serialization metadata rather than neural decode steps.

Operationally, the distinction is:

generate "0.91" as text

versus:

read 0.91 from a predictive distribution

Both outputs can be wrong, but the second is at least a native predictive distribution rather than a number that had to be serialized through text.

3.2 Shared state

Jev accepts one shared state and many questions. TypeSafe now explicitly documents that the state is ingested once and every question is evaluated against it in parallel, so the shared-state part is no longer only an inference from black-box scaling:

shared state
    ↓
shared representation / KV
   ↙    ↓    ↘
 Q1    Q2    Q3

That matters most when the state is long and the questions are short.

Without sharing:

Q questions × prefill(state)

With sharing:

1 × prefill(state) + Q short branches

The branches are not free; the saving comes from avoiding another full pass over the same long context for every question.

There is an important system-design caveat, though: a larger shared state is not automatically better. TypeSafe's own Jev 1.13 jaggedness notes say that accuracy falls as the state fills with information unrelated to the decision. So the useful production pattern is not put everything into state once; it is closer to retrieve/filter relevant state once → fan out many decisions over that state. Shared-state reuse saves compute, but it does not remove context rot.

3.3 Question isolation

Hume also found behavioral evidence for question isolation: a fact placed in the shared state was visible to another question, while the same fact placed only inside a sibling question was not. One natural implementation is a block-causal attention mask:

• every question can attend to the shared state;
• a question can attend to its own tokens;
• sibling questions are masked.

That topology would give shared-state reuse without letting sibling questions leak into one another. The behavior supports isolation strongly; it does not prove that TypeSafe implemented this exact mask.

3.4 Listwise option interaction

Options inside one question do not appear to behave independently. In one replicated probe, adding an irrelevant fifth option changed the log-odds between two existing options from roughly +0.38 to +0.11.

If each candidate were scored independently with a fixed logit and the server merely applied a softmax, the relative odds between existing candidates should remain unchanged.

That shows list-dependent behavior somewhere in the scoring path. It is consistent with set-aware/listwise processing, but it does not prove a particular internal head: Hume explicitly notes that rounding, request noise, or list-dependent temperature could also contribute.

This kind of interaction is useful for options such as:

• “none of the above”;
• overlapping actions;
• context-dependent alternatives whose meanings depend on the rest of the set.

3.5 Special data preparation

BERT-like encoder models are pretrained by reconstructing masked tokens, while causal LLMs learn to predict the next token. In both cases, the supervision comes automatically from ordinary text: you do not need separate manual labels for every training example.

A decision model is different: you need to construct examples of the form:

state → question → outcome

which usually means additional cost for the LLM generating the training data. And if you want calibration, you also need to train the predicted probabilities to match the model’s actual empirical error rates.

This part is highly important for building such a system class and may require high budget for preparing training dataset. And using LLM as a trainer to produce training examples for decision model can significantly change the accuracy of trainee due to accuracy of the trainer itself.

3.6 Causal transformer: probable, not confirmed

Hume argues that a causal transformer is the most likely backbone because Jev appears to retain broad pretrained world knowledge, causal decoders dominate frontier-scale pretraining, and shared-prefix KV serving fits naturally.

But the black-box tests cannot distinguish a causal decoder from every possible bidirectional/encoder-style implementation. So I would keep causal transformer in the “probable” bucket, not the “confirmed” bucket.

3.7 Sparse MoE: plausible, but speculative

I would put the sparse-MoE claim in a much weaker bucket. Hume argues from throughput and latency that dense frontier-scale compute is hard to reconcile with the observed performance, while a large sparse model with a much smaller active parameter count is plausible. That is a useful hypothesis, not a measurement. My confidence ladder for the black-box reconstruction is:

---

4. Why this matters: the autoregressive tax

A lot of current agent architecture still treats the LLM as a universal conditional operator:

state
  ↓
prompt
  ↓
LLM
  ↓
{"action": "billing", "confidence": 0.91}

For an open-ended response, that is the right primitive. For RAG = yes/no, route = billing, or escalate = true, I am less convinced: we are paying for a generative interface even though the business decision is bounded.

The cost is not only output tokens. It is also:

• decode latency;
• repeated context prefills;
• schema/constrained-decoding machinery;
• parsing and retries;
• another place where the model can return something operationally awkward;
• another model call to observe, trace, rate-limit, and capacity-plan.

Autoregression is valuable when we need to construct a sequence. If the result is already known to be one of ten actions, it is often an expensive way to implement an if.

There is a useful analogy here with optimization. Constraints do not make every optimization problem easier, but reducing the feasible region can make the search dramatically cheaper when the algorithm can exploit those constraints. The same intuition applies here. A generative LLM is solving a much more general problem: it can produce an essentially arbitrary sequence of tokens. If the actual task is only to choose one of 20 actions, answer yes/no, or return a score in a fixed range, most of that output space is irrelevant. By making the admissible output space explicit, a decision model can avoid searching through sequences that could never be valid answers in the first place.

The interesting part is that this is not only a decoding constraint. In the stronger JEV-like designs, the restriction changes the computation itself: instead of generating and then constraining text, the model maps the representation directly into the bounded decision space.

That is the part of the Jev story I care about. Classifiers, NLI, rerankers, reward models, calibrated heads and retrieval pipelines are not new. And in previous years it WAS possible to build a universal reranker/classifier/NLI that can work in multiple domains - but its quality OUT OF THE BOX was low, that is why it was required to train it on your domain data - like many used to do with ModernBert. The difference now is the stronger and richer representation underneath them: current foundation models bring much stronger language understanding and world knowledge than previous-generation encoders. The habit that emerged in parallel, however, was to route almost every semantic branch through text generation.

Jev productizes the opposite choice — keep the representation, skip generation where it adds no value — and the open-source projects are useful because they test several concrete ways of doing it.

---

5. The open-source Jev ecosystem is an architecture search, not one clone

I would not group everything below under “Jev clones”; the projects are different enough that the label hides the useful part.

One of the things that fascinated me about ML long before foundation models was how often algorithms from completely different areas reused the same ideas in different combinations. When I ran the ATP department at MIPT, I liked describing an algorithm almost as a chain of concepts inside it. The Jev-clone open-source race feels very similar: representation learning, reranking, NLI, masking, calibration, retrieval, diffusion and classical classification, recombined quickly and in different proportions.

The easiest way I found to compare the projects is to separate four choices: backbone, readout, training and fan-out.

Backbone

• causal LLM;
• bidirectional encoder;
• diffusion model.

Readout

• answer-token logits;
• whole-candidate likelihood;
• hidden-state MLP;
• pointer/listwise head;
• fixed diffusion slots.

Training

• frozen model;
• temperature scaling;
• LoRA;
• contrastive data;
• proper-scoring objectives;
• calibration-aware RL;
• conformal abstention.

Fan-out

• independent candidate passes;
• shared KV;
• isolated block-causal branches;
• batched encoder inputs;
• diffusion slots.

With that decomposition, the current projects look roughly like this:

I would not choose among these families by benchmark score alone. What matters is whether the action space is stable or dynamic, how much per-customer adaptation is needed, whether the probabilities are good enough to drive policy, and whether the serving path still behaves well under real concurrency.

---

6. The simplest version: read answer-token logits

Projects such as mini-Jev and reflex show that removing decode does not require a new architecture at all. Take an ordinary causal LLM and, after prefill, read the logits for the allowed answer tokens instead of asking it to generate JSON.

If the options are encoded as A/B/C:

logits = LM_head(h_last)[A, B, C]
p = softmax(logits)

Qwen remains Qwen — same transformer, same LM head — but inference stops before autoregressive generation.

What this gives you

• no JSON generation;
• no decode loop;
• bounded output;
• easy batching;
• minimal implementation work.

What it does not give you

Calibration

The LM head was trained for next-token prediction, not to produce calibrated posterior probabilities for your business decision.

A softmax over three selected vocabulary logits is not automatically:

P(option is correct | state)

It is closer to:

P(model emits this answer token | prompt, answer restricted to A/B/C)

I would not treat those two quantities as interchangeable.

Verbalizer bias

The choice of A, B, C, yes, no, etc. can introduce priors.

Reordering candidate-to-token mappings can change results.

Multi-token candidate difficulty

This works most cleanly when every answer maps to one known token.

It becomes less natural when the options are:

• approve refund;
• request documents;
• escalate to fraud.

No automatic question isolation

Reading logits alone also does not solve the “many independent questions over one shared state” problem.

No automatic listwise semantics

You still need to decide how candidate descriptions interact before the readout.

I would treat answer-token readout as the cheapest baseline in an evaluation, not as proof that the problem is solved. It tells you how much of the value comes simply from removing decode before you invest in custom heads, training, or serving.

---

7. Kev: shared state + isolated branches + pointer scoring

Kev is one of the clearest open implementations of the architecture inferred from Jev, but the implementation has already evolved enough that one sentence no longer describes every Kev checkpoint.

The common pieces are:

• Qwen backbone + LoRA;
• shared state;
• exact isolation between questions;
• a pointer-style readout head that compares option-boundary representations with a decision representation.

On the earlier attention-only Qwen3 path, Kev packs state and questions into one sequence and uses a block-causal mask. On the current Qwen3.5 family, that exact packed-mask trick does not apply cleanly because Qwen3.5 contains Gated DeltaNet recurrent layers that do not honor the attention mask in the same way. Kev therefore runs questions as separate causal rows continuing from the same shared state representation. Isolation is exact by construction.

Conceptually, the useful property is the same:

                    ┌→ Q1 → option representations → pointer softmax
state → shared rep ─┼→ Q2 → option representations → pointer softmax
                    └→ Q3 → option representations → pointer softmax

For an agent this topology is useful because one conversation state can feed many decisions without another full prefill of the same context. For example:

intent
escalation
RAG/no-RAG
tool class
handoff
policy violation
response type

Instead of:

7 × prefill(large conversation state)

the shared state is computed once and reused across the short decision branches.

The important limitation: good topology is not good calibration

Kev has moved since the first version of this article. The current checkpoints now apply temperature calibration by default: each checkpoint stores a temperature fitted on its in-distribution development set and the pointer head applies it at load time. For Kev-9B, the repo reports calibration error on new sources improving from 0.106 to 0.042, while the share of wrong predictions made with p ≥ 0.9 falls from 8.7% to 4.0%, with accuracy unchanged. Jev was at 3.7% confident errors on the same development suite.

That is a meaningful improvement, but it still does not make the probabilities universally portable. A single fitted temperature cannot reorder examples by reliability, and the share of decisions Kev can automate at a 5% error budget is still reported at roughly 0.45–0.57, versus 0.70 for Jev. I would still calibrate and set thresholds on the actual production distribution rather than copy the repository's number.

There is a concrete older example of the same point. On early Kev-0.5B, simply reordering the options produced roughly 7% argmax flips. The current Qwen3.5 family is much better behaved, so I would not treat that number as a property of “Kev” in general. But it is a useful reminder that a clean computation graph does not automatically give you order robustness or trustworthy probabilities.

Kev also got a small but revealing training update on September 21. The authors added a short second pass with two kinds of generated examples: policy cases with explicit day counts, and cases where the deciding evidence was deliberately removed and the target distribution was made uniform. On new-source test data, Kev-9B moved from 0.837 to 0.852, Kev-4B from 0.832 to 0.837, and Kev-0.8B from 0.668 to 0.684.

I like this result because it is not another architecture trick. It is the same point that keeps appearing across this ecosystem: once the basic computation path works, how you formulate the task, construct hard cases and encode uncertainty in the target can move quality as much as another model change.

Serving is also still research-grade, although the repo has moved beyond the older “no cross-request cache” description. The current serve.py keeps a small state-prefix cache across requests, but access to the single loaded model is serialized under one lock. There is still no production-style continuous batching across independent callers.

And the custom-serving requirement did not disappear with the move from Qwen3 to Qwen3.5; it only changed shape. The old path needed the block-causal attention mask. The current path needs a serving layer that can compute a shared Qwen3.5 state once, carry the hybrid attention/recurrent state into multiple causal question rows, collect the relevant option/decision hidden states, and run the pointer head. A stock vllm serve for ordinary Qwen3.5 does not know that execution graph.

Source: Kev repository

So I would treat Kev as strong evidence for the computation graph, not yet as evidence for production economics. Isolation works; the harder engineering question is whether the same topology survives continuous batching, p95 load, cache eviction, upgrades and the rest of the inference stack without giving back the latency advantage.

---

8. Runtime-defined candidates: SemIf and NLI take different routes

There are two different approaches here that are easy to conflate because both accept runtime-defined textual candidates.

SemIf: native option logits with runtime-defined criteria

SemIf currently uses a frozen Qwen3.5 model and reads native option logits directly, without sampling answer text.

The request can still define criteria and option descriptions at runtime:

state + runtime criterion + typed options
                  ↓
             Qwen3.5
                  ↓
          native option logits
                  ↓
             probabilities

SemIf also experiments with shared-state reuse: prefill a long state once, then score many short criteria as suffixes. On its published 37-state × 21-criterion workload, the repo reports roughly 2.33 decisions/s for fresh scoring, 10.75 decisions/s for serial prefix reuse, and 20.03 decisions/s for parallel suffix reuse. The fast reuse path is explicitly marked experimental, and the authors report a handful of BF16 argmax changes across 777 decisions relative to fresh scoring.

SemIf is useful evidence for two fairly practical points:

• runtime-defined decision descriptions do not require a fixed classifier head;
• shared-prefix reuse can turn many decisions over one state into a much cheaper workload.

It is not, in its current form, an NLI cross-encoder.

AlexWortega/openjev: the NLI route

A separate implementation, AlexWortega/openjev, does use the NLI formulation.

For each candidate:

state     = premise
candidate = hypothesis

and the classifier predicts:

entailment
neutral
contradiction

The candidate can therefore be arbitrary text rather than a class fixed at training time, which maps naturally to agent routing where available actions change with customer, permissions, workflow state, product catalog or integration set.

AlexWortega/openjev: frozen representation + tiny task-specific head

The same project also explored a more classical point in the design space: let Qwen build the representation, freeze the backbone, and train a small MLP on top of the latent for (question, option):

frozen Qwen representation
          ↓
      d → 512 → 1
          ↓
       decision score

Architecturally this is the old pretrained features → task-specific classifier pattern, except that the features now come from a much stronger pretrained model. The head is cheap to train and the backbone stays frozen, but the usual limitation comes back immediately: the head is task-specific. Change the domain or the meaning of the decision and you may have to retrain it, and a small MLP cannot recover information that the frozen representation never encoded well.

The downside of independent NLI scoring

If candidates are scored independently, five mutually exclusive actions might receive:

0.80
0.75
0.70
0.60
0.55

Those entailment scores are not naturally one categorical P(action | state) distribution, so a policy such as:

if p > 0.9:
    execute_action()

does not have the same clean interpretation across candidate sets.

High cardinality naturally becomes retrieval + reranking

If there are 1,000 possible actions, scoring all 1,000 with a cross-encoder is still expensive. Current NLI implementations can reuse the shared premise/prefix and batch hypotheses, which removes a lot of repeated context compute, but candidate-side compute and memory still scale with the shortlist.

The natural production architecture becomes:

retrieval / embeddings
        ↓
shortlist 20–50
        ↓
stronger NLI / decision scorer

At that point the architecture looks very much like an IR stack where the “documents” are actions. I expect this to be common in agents with large tool catalogs, with the usual two-stage retrieval failure mode:

If the correct action does not make the shortlist, no reranker can recover it.

---

9. daseinlabs/open-jev: score the full candidate, not A/B/C

daseinlabs/open-jev takes a different middle point: instead of reading the logit of A, it scores the log-probability of the entire candidate sequence.

The context is prefetched once:

context
   ↓
shared KV
   ├─ "approve refund"
   ├─ "request documents"
   └─ "escalate to fraud"

Each candidate is teacher-forced through the decoder and its sequence likelihood becomes the score. There is still no free-form autoregressive generation because the candidate text is already known.

Why this is attractive

The obvious advantage is that the scorer sees the actual action text rather than an arbitrary A/B/C verbalizer, while the expensive context prefill can still be reused across candidates.

The project reports roughly a 4× improvement on one M5 Pro example when using cached shared context instead of re-encoding it separately for eight options.

The cost

Candidate tokens still have to pass through the transformer layers, so compute grows with:

• number of candidates;
• length of candidates.

Raw sequence likelihood also has a length bias:

score(option) = Σ log P(token_i | context, previous option tokens)

Longer options accumulate more negative log-probabilities.

The repo therefore supports:

• sum;
• mean length normalization;
• PMI normalization.

PMI costs an extra batched pass.

This exposes a deeper point: normalization changes the meaning of the score. None of these transformations automatically converts a generative LM likelihood into a calibrated probability that the action is correct.

The authors explicitly note that zero-shot Gemma probabilities can be extremely sharp and poorly calibrated.

Frozen features + a small listwise head

The same project also explores the next step: stop using the LM head and train a small cross-attention/listwise scorer over frozen Gemma hidden states. That gives a useful progression:

answer-token logits
        ↓
whole-option likelihood
        ↓
frozen representations + small head
        ↓
LoRA / backbone adaptation

As you move down the ladder:

• training cost increases;
• task-specific data requirements increase;
• dependence on arbitrary LM-head priors decreases.

I like this experiment because it makes the trade-off unusually explicit: zero-shot scoring buys flexibility; a learned head or LoRA buys more task-specific behavior. Moving along that curve changes training cost, inference semantics and calibration at the same time.

---

10. Nimble: sometimes the biggest gain is in the data

Bespoke Nimble is architecturally much more conventional. It uses:

• Qwen3.5-9B;
• LoRA;
• cross-entropy over allowed candidate logits.

The interesting part is the data construction. Training examples are built as contrastive single-fact-flip pairs:

Alice may approve refunds → approve
Bob may approve refunds   → reject

One decision-critical fact changes and the label must flip. It is a very classical ML recipe:

• hard negatives;
• minimal pairs;
• explicit decision boundaries;
• anti-shortcut training.

On Nimble's public 324-example holdout:

• base Qwen3.5-9B: 66.36%;
• Bespoke Nimble: 90.12%;
• Jev 1.13.0: 93.21%.

The gain is large despite the fairly conventional architecture.

But the benchmark is narrow

The holdout consists of:

• 324 examples;
• 162 contrastive pairs;
• six source families;
• synthetic reference labels.

The authors explicitly call it a narrow test.

I would read the result as:

an ablation on the value of contrastive data construction

not:

90% representative enterprise accuracy

This is very consistent with my own ML experience: task formulation, hard negatives and the loss/reward have repeatedly moved production quality more than another model-architecture iteration. If I had one engineering sprint for this problem, Nimble would push me to spend it on the data boundary before inventing a new block.

What Nimble does not change is just as important. It is still a 9B causal backbone, so the full shared state still has to go through prefill. The contrastive dataset sharpens the decision boundary; it does not give you JEV/Kev-style shared-state serving, question isolation, or calibrated probabilities under shift.

A newer open project, system-one-open, provides another useful data point for the same argument. Its Gemma-based replica reports 98.8% / ECE 0.003 on the demo task families it trained around, but only 74.8% on held-out task types it had never seen. I would not compare those numbers directly with JevBench, but the gap is informative: getting a decision model to look excellent on the task families represented in its synthetic generator is much easier than getting the same behavior to transfer to a new decision family.

---

11. The encoder comeback: Laya, GLiClass, and Verdict

Once the output is a bounded decision rather than generated text, the old encoder question comes back immediately: why use a decoder at all? Laya, GLiClass and Verdict explore that branch directly with bidirectional encoders specialized for classification.

Laya

Laya uses ModernBERT/mmBERT with a specialized decision head.

Candidates are included in the input and represented by marker tokens. The encoder processes state and candidates bidirectionally, and the head scores the candidate markers.

The resulting inference path is fast. The multilingual Laya variant reports around:

• 32.8 ms for one question on a T4;
• ~72 ms for ten;
• ~337 ms for fifty;
• 103–332 questions/s in batched tests.

The quality numbers need more context than the latency numbers. On typed-decisions:

laya-typed-decisions: 0.766
Jev published:        0.727

However, the strong Laya checkpoint is fine-tuned on the training split of that benchmark.

The base Laya checkpoints are only around:

0.36
0.35

against a majority baseline of 0.461.

(The current Laya README reports 0.362 and 0.352; an older benchmark file reported the multilingual checkpoint a point lower. The conclusion is unchanged: the strong 0.766 result comes from task-specific fine-tuning.)

The Laya authors explicitly say that the capability comes from fine-tuning, which is almost exactly the trade-off I started with. If a workflow is stable, high-volume and tied directly to revenue, risk or user experience, I would still seriously consider training a dedicated BERT-like model for it; the one-time ML work is justified because that branch matters enough to own.

Where Jev is different is the middle layer. A real product may have five decisions important enough for dedicated models and fifty more where a better classifier would help but nobody wants fifty separate training pipelines. Those fifty tend to become rules, prompts or another frontier-LLM call. Laya is a strong specialist after task-specific fine-tuning; a general decision model is trying to cover that larger set without a new training cycle for each decision.

Banking77 exposes a real architectural ceiling

On Banking77, the published comparison is directionally striking but not exactly apples-to-apples:

Laya: 0.425 on 77 labels
Jev published: 0.870 on 72 labels

So I would not read the raw delta as a clean head-to-head accuracy gap. The result is still useful because the Laya authors themselves use it to expose a real high-cardinality limitation.

The failure is an architecture design problem: Laya allocates a fixed token budget to candidate descriptions. With the default head budget, 77 labels leave only about 3–4 tokens per class description. That is the actual failure mechanism: long, similar labels begin to get truncated or become hard to distinguish. The repo describes this as a high-cardinality option-budget limitation and recommends keeping choice sets much smaller by default. It is not an absolute ceiling — head_max_len can be increased, at the cost of more compute/context budget, or a retrieval/embedding stage can reduce the set to a top-K shortlist before Laya scores it.

So the tradeoff is clear: for 5–20 labels + stable domain + ML team capacity for ML model lifecycle Laya (and other ModernBert-based solutions) is very attractive. For long state → many questions → 10-100+ candidates the absence of shared KV and default limitation of option budget start biting, high-cardinality can be partially compensated by increase of head_max_len or building shortlist candidates by auxiliary model.

There is also a benchmark-hygiene caveat I would keep in mind: Banking77 has been public since 2020. That does not invalidate the Jev result, but it means I would not treat the published Jev number as the same kind of clean zero-shot evidence as performance on a genuinely unseen taxonomy.

This limitation matters for workloads like:

one long state
→ many questions
→ 50–200 runtime actions

and much less for something like:

10 stable intents

There is another practical caveat: the base Laya checkpoints ship over-confident. The current README reports temperature refitting reducing mean ECE from 0.466 → 0.081 for English Laya and 0.314 → 0.106 for the multilingual checkpoint. That is another reminder that fast classification quality and trustworthy decision probabilities are separate problems.

GLiClass

GLiClass pushes the dynamic-label idea further.

GLiClass supports several architectures. In its default uni-encoder path, textual labels and text share one encoder input, allowing new class descriptions at runtime without a fixed K-class head. The project also supports bi-encoder and fused variants that make different scaling/interaction trade-offs.

The default uni-encoder path is powerful for semantic routing and zero-shot classification, but it brings familiar trade-offs:

• labels consume the shared input budget;
• attention cost grows with label text;
• zero-shot quality varies substantially by domain/taxonomy;
• raw scores are not automatically calibrated production probabilities.

The published spread makes that dependency concrete. gliclass-large-v3.0 is around 0.94 F1 on CR, but roughly 0.60 on 20 Newsgroups, 0.56 on Banking, and 0.45 on Emotion. Dynamic labels work; they do not make the classifier domain-independent.

Bi-encoder variants can scale label handling differently, but give up some of the full token-level interaction that makes the uni-encoder attractive.

For hundreds of long tool descriptions, the same pattern reappears:

retrieval → shortlist → GLiClass / reranker

Verdict

Verdict also uses ModernBERT-style encoder machinery but spends more effort on uncertainty and calibration.

There are two important regimes in the project:

• a more general Verdict line;
• a specialist Verdict 2.0 line built for typed workflow decisions, whose headline result is reported on the held-out typed-decisions benchmark.

I would not read the specialist number as evidence of zero-shot generality.

The project also exposes useful hard limits:

• Choice is bounded to 25 slots in the current design (24 substantive options plus abstention);
• reported top-1 accuracy falls from about 97% at K=3 to 72% at K=25 in its capacity test;
• the current JevBench hard-tier result is only about 36.9% accuracy;
• the context budget was tightened to 512 tokens because the training states were much shorter, exposing positional/OOD sensitivity.

On the current public 231-task JevBench split, that shows up clearly across tiers: roughly 87.5% on easy, 69.4% on standard, and 36.9% on hard.

There is another subtlety if uncertainty is implemented with a separate confidence head: that head is a learned model too. Under distribution shift you can get the unpleasant combination of a wrong class prediction and a still-confident confidence head. Separating “which option?” from “how sure am I?” can be useful, but it does not remove the need to evaluate calibration under shift.

Those are useful failure tests, not reasons to dismiss the approach: they show exactly where a small specialized encoder stops looking like a general decision engine.

My conclusion from the encoder branch is not that one family “wins.” Encoders are still excellent when the problem really is classification; the Jev-like generality becomes more valuable as the workflow becomes more dynamic, high-cardinality, long-context and multi-question.

---

12. Calibration is harder than argmax

Calibration is the part I would spend the most time on before letting a decision model touch money, permissions, customer access or a regulated workflow. But Jev's API terminology needs one important distinction.

For Choice and Score, probabilities is the model's distribution over the supplied outcomes. TypeSafe's separate confidence field is derived from the shape of that distribution: a concentrated distribution gets high confidence, a flat one gets low confidence. It is not itself the model's estimate that “this answer is correct with probability 0.95.” Noul does not return this derived confidence field at all.

So there are two different things a production system may want to calibrate or threshold:

P(selected option)

and

confidence(distribution shape)

They are related but not interchangeable. If I need a policy such as execute automatically above 95% expected correctness, I would validate that policy empirically on the exact output quantity I use rather than assume TypeSafe's confidence=0.95 means 95% correctness.

TypeSafe positions RLCD as the training method behind calibrated decisions, but the exact recipe is undisclosed; the open-source projects therefore give us more transparent calibration experiments to inspect.

decider

decider is a family rather than one frozen recipe. The current decider-2b v10 uses Qwen3.5-2B with one-pass readout, proper-scoring training, temperature fitting, and a later calibration-aware RL stage. A larger 35B-A3B variant exists as well, but the current 35B line does not use the same RL stage.

The 2B RL experiments use environments where outcomes are objectively verifiable:

• browser tasks;
• Minesweeper probabilities;
• controlled stochastic games.

Those environments make sense for training probabilities rather than merely labels because the outcomes are objectively verifiable. The repo's own results also show why I would not assume calibration transfers: improvements on one environment do not uniformly improve every benchmark.

The serving path exposes another trade-off that I find particularly instructive. In schema-first mode, a fixed set of questions/options can be prefetched and cached, while the changing state is processed against that cached schema. On short messages the repo reports up to roughly 4× speedup, and the gain can be larger on very high-cardinality choices.

But changing the information order is not free. In the project's evals, schema-first loses about 1.5 percentage points on average on fixed-label tasks, around 5 points when the options change per example, and on some long-context/high-cardinality workloads the gap reaches roughly 5–24 points. This is a nice example of something that is easy to forget in model benchmarks: serving optimization can become part of model quality.

The current model is also English-only, so calibration and quality still need to be re-established for another language rather than assumed to transfer.

poorjev

poorjev takes a very classical path:

NLI model
  ↓
temperature scaling
  ↓
conformal abstention

On its shipped 55-item / 160-decision hand-labelled evaluation set, with temperature fitted by 5-fold cross-validation, the repo reports ECE improving from:

0.170 → 0.071

For production, the more useful idea in poorjev is the risk/coverage trade-off rather than the lower ECE by itself. At a target error budget, the model answers only the subset of cases where it is sufficiently reliable and abstains on the rest:

easy / confident → cheap decision model
uncertain        → stronger LLM or human

The caveat is standard for conformal methods: guarantees depend on the calibration distribution remaining sufficiently representative of production traffic.

---

13. DiffusionGemma: parallel decision slots instead of causal decoding

OpenJev on DiffusionGemma is the implementation I would show to someone who thinks the only way to remove decode is to stop a causal LLM early.

DiffusionGemma is interesting here because it does not generate left-to-right like a conventional causal LLM. OpenJev lays out a fixed canvas with answer slots:

q1: [MASK]
q2: [MASK]
q3: [MASK]

A read-only denoising step produces distributions over the allowed labels in all slots, so for this workload the model behaves more like a parallel masked classifier than a normal text generator.

Why the architecture is interesting

Parallel output is native to the backbone

There is no “stop decoding early” trick here; the inference topology already updates multiple output positions in parallel.

Adaptive test-time compute is possible

Uncertain decisions can also be re-read with more noise samples or denoising steps and averaged, which gives a natural adaptive-compute pattern:

1 pass by default
→ extra compute only when uncertain

But extra stochastic reads do not automatically create calibration, and latency becomes input-dependent.

The implementation does not enforce Kev-style sibling isolation

Unlike Kev's explicit isolated branches, decision slots live in one diffusion canvas. The repo even notes that extra denoising steps let answers “settle against each other.”

I would not call that harmful cross-contamination without evidence; related decisions may even benefit from interaction. But strict sibling-question isolation is not an invariant of this implementation, so for independent decisions I would test it explicitly.

It is still largely a single-token label interface

Candidate descriptions can live in the prompt, but the Jev-compatible decision read still chooses compact labels such as A/B/C.

So DiffusionGemma does not eliminate every verbalizer issue solved by full-candidate scoring.

There are also concrete packing limits in the current implementation: Choice is capped at 128 options versus Jev's 255, and large question sets are answered in chunks of roughly 12 questions per read. The repo now exposes optional think and text-generation extensions too, but those are OpenJev additions; the core System One-style read path remains the bounded decision mechanism discussed here.

The backbone choice is interesting for another reason. DiffusionGemma is 26B-A4B: the total model is large, but only a much smaller subset of parameters is active per forward pass. That MoE structure is a natural fit for the “large pretrained representation, bounded parallel readout” idea — although it is separate from the question of whether the diffusion slots themselves are the right decision head.

Current serving reality

The serving path also shows how quickly an architecture experiment becomes an infrastructure problem. On NVIDIA, the current implementation depends on a custom/pinned vLLM fork with diffusion support rather than an ordinary stable vllm serve deployment.

The project does support concurrency and backpressure, but published load tests show the expected throughput/latency trade-off on an RTX PRO 6000:

Source: razorback16/openjev

For capacity planning, that curve is much more useful than the isolated “94 ms” number.

---

Another axis: how small can the production policy get?

A separate group of projects — minojev, NanoJev, PlayJev, reflexrl, and JevForge — explores a different question: once the task has been turned into a bounded decision problem, how far can the policy be compressed or distilled?

That matters because the large model does not necessarily have to survive into deployment. A large LLM/VLM can be the teacher that creates labels, representations, or trajectories, while the production policy becomes dramatically smaller and cheaper. In the multimodal variants, the same idea becomes pixels → bounded action rather than text → bounded action.

I find this branch important because it changes the economics one more time:

large teacher / expensive data generation
                ↓
      small production policy

The recurring pattern is very old ML: spend compute during training so you do not have to spend it on every production decision.

---

14. Benchmarks: what they show, and what they do not

The benchmark numbers around Jev are useful, but several of them measure different things and are easy to compare too literally.

JevBench

JevBench currently uses 534 frozen decisions and combines four axes:

• intelligence/capability;
• calibration;
• speed;
• cost.

The composite is the geometric mean of those axes, so a JevBench score is not an intelligence or accuracy score.

JevBench is changing every day. The current scoring is v1.3.0. Intelligence is now chance-corrected before being combined with Calibration, Speed and Cost, and systems below 50 Intelligence receive an additional (Intelligence / 50)^2 penalty. The benchmark still uses the same 534 frozen decisions, but the composite values are therefore not directly comparable to the v1.2 numbers quoted in earlier versions of this article.

As of September 23, the top of the current board is:

Jev 1.13.0                 74.4
SemIf Qwen3.5-4B           73.1
djev / DiffusionGemma      73.0
Winnow-12B Q8              71.2
reflex-4B                  70.3

There are now 48 ranked rows. I would still treat this as a dated systems benchmark rather than a model-IQ ranking: hardware, endpoint type, serving implementation, calibration method and cost model all contribute to the composite.

The v1.2 benchmark lineage is also being revised quickly as new systems are added, while the frozen items/answers remain fixed. I would cite the benchmark methodology and date rather than treating a transient leaderboard rank as a durable model fact.

The important methodological caveats are:

• systems run on different hardware;
• production and demo endpoints are mixed;
• non-production endpoint latency may be adjusted by benchmark assumptions;
• the score mixes model quality with economics;
• some rows use hosted APIs, others local inference;
• the v1.2 harness runs one request at a time, so it does not measure realistic concurrency;
• latency is measured from the benchmark runner rather than from one standardized deployment geography.

I would not use this table to claim that one model is 2.1 points “smarter” than another. What I do take from it is that very different architectures are already close enough that system design, calibration and serving can dominate the model-choice discussion.

Nimble's holdout is not JevBench

Nimble's famous:

66.36% → 90.12% → 93.21%

comparison is on its own 324-example contrastive synthetic holdout.

It is valuable as a training-data ablation, not as a broad model leaderboard.

Laya's typed-decisions result is specialist vs generalist

laya-typed-decisions was fine-tuned on the training split of the same workflow family.

The base Laya model performs much worse zero-shot.

The result shows that a specialized ModernBERT decision model can be excellent after task-specific training; it does not show that a 421M encoder is generally more capable than Jev.

This distinction is especially important for CTOs deciding between:

• rules;
• dedicated classifier;
• general decision model;
• general-purpose LLM.

---

A real-agent workload gives a different picture

Archestra tested Jev and several open models on 100 real tool calls from production Claude Code sessions. Each call produced four security-related decisions, so the initial set contained 400 labels. They then had three independent judge families — Claude Opus, GLM 5.3 Flash and Gemini 3.8 Flash — grade the answers blind and reported only the 337 decisions where all three agreed.

I think the 63 discarded decisions are almost as interesting as the model scores. The disagreements exposed underspecified product semantics: whether outbound reads count as an exfiltration channel, how to classify opaque IDs with no semantic context, and whether tool metadata is internal or public. In other words, part of the “model error” turned out to be a task-definition and eval-harness problem. Also, some of that 63 discarded decisions may show the potential errors of the 3 powerful LLMs used as LLM-as-a-Judge.

The class distribution also makes the headline accuracy misleading. 79% of the calls were the benign/default class, so a hardcoded return benign already scores 79%. Against that baseline:

The Laya number is a good example of why recall alone is not enough either: it caught all nine dangerous calls by flagging essentially everything, for only 12% precision on that decision.

The few-shot effect is also architecturally interesting. SemIf Qwen3.5-4B jumped from 63% to 84%, and the MiniCPM path from 54% to 80%, while Laya stayed essentially flat at 48% → 46%. That is one practical advantage of retaining a causal pretrained LLM backbone: runtime examples can substantially reshape behavior without a training job. The encoder path remains fast, but for a new decision boundary it may need actual fine-tuning rather than another nine examples in context.

For a production system, my conclusion would not be “Sonnet wins” or “Jev wins.” The test has only nine positive requires_trusted cases. What it does show very clearly is that overall accuracy is the wrong metric when false positives and false negatives have different business costs. At an execution boundary I want per-class precision/recall, the majority baseline, risk-weighted errors, coverage at a chosen error budget and inspection of the actual failures.

15. Serving is half the system

Serving is where most of the pretty single-request numbers stop being directly useful. A 4 ms kernel number is not a 4 ms product; production adds queues, cache misses, batching, memory pressure, failures and the actual traffic shape. For capacity planning I want to know:

• p50 / p95 / p99 under concurrency;
• continuous batching;
• queueing and backpressure;
• GPU utilization;
• KV-cache reuse and hit rate;
• memory pressure;
• large-candidate packing;
• tenant isolation;
• cold starts;
• version compatibility;
• observability and failure recovery;
• cost per successful decision, not cost per synthetic benchmark call.

The current open-source projects are at very different maturity levels on exactly these dimensions.

Kev

The computation graph is interesting, but the reference server is still designed more as a research/local sidecar than a production scheduler.

The current code now includes a small cross-request state-prefix cache, so the older “no cross-request KV cache” description is stale. At the same time, one loaded model is protected by a single lock, so independent requests are still serialized rather than continuously batched across callers.

That is enough to validate the architecture, but not enough to infer production economics.

daseinlabs/open-jev

A useful architecture experiment, but primarily MLX / Apple-Silicon oriented. I would not use its local numbers to plan an H100 fleet.

Laya

Laya batches questions efficiently, but it is an encoder rather than a causal shared-prefix architecture. There is no Jev-style shared-state KV cache to reuse across many different questions: the state/question/label input is encoded as part of each batched example. Batching amortizes hardware cost; it does not turn one long state into one reusable causal prefix.

DiffusionGemma/OpenJev

Real concurrency exists, but the NVIDIA path depends on a custom diffusion-aware vLLM branch. Published load tests also show the expected trade-off: throughput goes up with concurrency while latency rises sharply.

There is nothing surprising about that curve; it is simply why the ~94 ms single-request number tells us very little about a loaded service.

Maisa/djev

There is now also djev from Maisa, which is worth distinguishing from the other DiffusionGemma/OpenJev implementations rather than treating “DiffusionGemma” as one repo. JevBench describes djev as an Apache-2.0 self-hostable runtime over the open DiffusionGemma base weights, with no djev-specific model weights. It now sits near the top of JevBench v1.3.

That is useful evidence by itself: the diffusion decision pattern is no longer tied to one implementation. Independent runtimes are converging on the same basic idea — use the pretrained diffusion backbone as a parallel typed-decision engine without first training a separate decision model.

openjev-sglang

openjev-sglang is interesting for a different reason: it asks how far you can get without a custom decision-trained model or a custom transformer implementation.

The current server uses Qwen3.6-35B-A3B on SGLang. It renders the common prefix once, warms SGLang's radix cache, then sends the question suffixes concurrently and requests the first-token log-probabilities only for the allowed answer labels. The sampled token is discarded and the selected label logits are renormalized:

ordinary open MoE
+ shared prefix / radix cache
+ concurrent suffixes
+ selected first-token logprobs
→ Jev-compatible typed API

This is not the same computation graph as Jev or Kev. Radix reuse is opportunistic rather than a pinned per-request shared KV state, and the repo explicitly says its probabilities are conditioned on the supplied options, depend on prompt/label ordering and are not calibrated estimates of correctness. But as a serving baseline it is important: a surprisingly large part of the JEV-like latency pattern can now be assembled on top of a mature inference engine rather than by forking the model internals.

Winnow-12B: one set of weights, two inference modes

Text:

Winnow-12B adds another architecture/product pattern I did not have in the first version of this article. It fine-tunes Gemma 4 12B for typed decisions, but the same loaded model still exposes normal /v1/chat/completions and vision alongside /v1/systemone.

For decision requests, its llama.cpp-based server prefills the state once, forks question branches and reads answer-token logits without generating answer text. For ordinary requests, the same weights remain a generative multimodal model:

one foundation model
   ├─ generative chat / vision
   └─ direct typed-decision mode

That is potentially important for heterogeneous inference. Specialization does not necessarily mean loading a separate model for every primitive; one backbone can expose a cheap discriminative path for bounded decisions and keep generation available when the workflow actually needs it.

The released Q8 model is also unusually edge-oriented for this class: the authors report 64k context plus vision on a 16 GB RTX 5070 Ti, although the near-full-context cold prefill takes tens of seconds and the fast repeated numbers rely on the prefix already being cached.

decider

decider is more interesting from a systems perspective because it already includes:

• CUDA graphs;
• shape buckets;
• FP8;
• schema caching;
• continuous batching;
• HTTP load tests.

It also demonstrates a trade-off I expect to show up repeatedly in this model class: making the input easier to cache can improve speed while hurting quality on dynamic/high-cardinality/long-context workloads.

This is the kind of trade-off a platform team eventually has to own. One thing that surprised me while digging through the repos is that the implementations are already diverging more than the model architectures themselves. The neural ideas are mostly familiar; the important differences are moving into cache topology, batching, scheduling, schema handling, calibration, backpressure and the operational contract around the model. In production, that can matter more than whether the readout is a pointer head, an encoder marker or a diffusion slot.

For any of these projects I would separate two questions from day one:

1. Does the decision model improve quality / latency at the model level?
2. Can we serve it economically and predictably at our traffic shape?

A lot of otherwise promising model work fails at the second question.

---

16. What Jev-like models are good at

The use case I would start with is bounded semantic decisions: the input state can be messy, but the action space is known. In an enterprise product that includes:

• intent classification;
• tool routing;
• model routing;
• RAG/no-RAG decisions;
• response reranking;
• escalation;
• handoff;
• policy checks;
• semantic caching decisions;
• content moderation;
• ticket triage;
• dialogue-act classification;
• structured game actions;
• verification / critic layers.

They are particularly compelling when:

• the action space is known;
• the same state feeds several independent decisions;
• latency is important;
• decisions happen frequently;
• downstream policy needs confidence thresholds.

One thing I would make explicit here is that decision-model quality is not a property of the model alone. It depends heavily on how well the model, task and production pipeline fit each other. Language matters: a model that is strong in English may degrade materially on another language, and the problem gets harder again with ASR noise or code-switching. Domain applicability matters as well: a model that transfers cleanly across support, compliance and routing tasks on a benchmark may still fail on a narrow operational taxonomy it has never seen. Then there is the input contract itself: what goes into state, what is omitted, whether irrelevant context is allowed to accumulate, whether the decision is expressed as the right response type (Choice, Noul, Score), and whether the option set actually contains the right alternatives at the right level of granularity. Even after inference, the result still has to be interpreted correctly: a probability distribution, a derived confidence score and an operational threshold are not the same thing. And finally, none of the published calibration numbers should be treated as portable by default. If a probability is going to trigger an action, escalation or fallback, I would validate that calibration on the actual language, domain, state-construction logic, option-generation process and end-to-end production pipeline in which the model will run.

---

17. Where a generative LLM is still the right tool

I do not see these models as replacements for generative LLMs. They are the wrong primitive when the system has to create a novel sequence rather than choose among known alternatives, for example:

• writing a user-facing answer;
• synthesizing tool arguments that are not enumerable;
• open-ended planning;
• code generation;
• long-form explanation;
• multi-step reasoning where extra test-time compute is useful;
• tasks where the correct candidate may not exist in the supplied set.

Single-pass decision models also tend to struggle more with:

• compositional multi-hop reasoning;
• arithmetic/date logic;
• long unseen policy combinations;
• strong domain shift;
• missing-option detection without explicit training.

TypeSafe's own Jev 1.13 documentation is now unusually explicit about these edges. It lists literal interpretation, arithmetic and numeric precision, date/time comparison, multi-hop indirection, large irrelevant state, adversarial content, contradictory instructions/criteria and cross-formulation structural invariants as known failure modes.

Several of these have clear architecture implications. Keep arithmetic and date comparisons in code; reduce indirection; retrieve/filter before putting a large body of text into state; and do not assume Jev will ignore hostile instructions embedded inside the state. TypeSafe explicitly says that state is treated as data, not as hostile input by default, and adversarial text can move the answer.

That makes Jev useful as one discriminative guardrail, but I would not make it the only prompt-injection or security boundary around an agent.

What I expect instead is heterogeneous inference. That is already how I think about a real-time voice stack: endpointing, retrieval routing, policy checks, tool choice, argument generation and the final user-facing response have different latency budgets, error costs and output spaces. I see little reason to force them through the same model primitive just because one API can technically do all of them.

---

18. The enterprise architecture I would actually evaluate

A lot of agent stacks still put most of the control logic into one model:

user state
   ↓
big generative LLM
   ↓
JSON action
   ↓
tool

Then the same model, or another copy of it, is asked whether to retrieve, which tool to call, whether to escalate, whether the response is safe, and whether a stronger model is needed.

It is an efficient way to get a prototype working, but it becomes expensive and hard to reason about as the number of branches grows. I would evaluate something closer to:

                     ┌→ intent
                     ├→ RAG / no-RAG
shared state ────────┼→ tool class
                     ├→ guardrails
                     ├→ escalation
                     └→ model tier
                              ↓
                    deterministic execution
                              ↓
                 generative LLM where needed

The generative model is still central; I just want it doing the work where generation or open-ended reasoning actually creates value.

For the bounded branches, a decision layer gives the system three things a generated JSON answer does not naturally give you:

• a cheaper/faster inference path;
• a typed action space;
• probabilities that can potentially drive explicit policy.

And I would add a fourth requirement immediately: every decision needs a trace. Which state/version was used, which candidate set was available, what probability came back, which threshold fired, and where the request went next.

I would treat that trace as part of the product contract, not optional research telemetry.

I would also make the error cost part of the contract. Archestra's experiment is a useful example. Sonnet had the best overall accuracy at 98% (with constant baseline of 79%), but caught only 4 of 9 calls that should have required a trusted session. Jev caught 7 of 9. Laya caught all nine only because it over-blocked almost everything. Those are three very different products despite superficially similar classification metrics.

For an execution boundary, a false alarm may stall the agent; a false negative may leak data. The threshold and fallback policy should therefore be optimized against the actual cost of those two errors, not against macro accuracy.

---

19. Jev is not a silver bullet: you still need an ML harness around it

One potential misunderstanding of the whole “general decision model” idea is that it removes the ML lifecycle. It does not. What it can remove is the need to train, deploy and maintain a separate classifier for every small decision. The evaluation and operational layer around the model is still necessary.

Before putting a Jev-like model on a production branch, I would want a representative evaluation set from the actual workflow, not only public benchmarks. That means real states, real option sets, the languages the product actually sees, ASR errors if this is Voice AI, ambiguous cases, missing evidence, adversarial inputs and the rare cases that are expensive when the model gets them wrong. The Archestra experiment is a good example of why this matters: part of what initially looked like model error turned out to be ambiguity in the task definition itself.

The same applies after deployment. Every decision should leave a trace: model/version, state construction version, question, response type, options, returned distribution, derived confidence if one exists, threshold, final action and — when it becomes available — the actual outcome. Without that trace it is very difficult to distinguish a model regression from a change in retrieval, a new option taxonomy, a prompt/state-construction change or simply a shift in production traffic.

I would also continuously look at the production distributions, not only aggregate accuracy. The share of each class can move. The length and composition of state can move. Languages can move. The number and wording of options can move. Confidence and entropy distributions can move even while top-1 accuracy initially looks stable. A threshold fitted when 5% of traffic required escalation may behave very differently after the product, customer mix or upstream pipeline changes.

Calibration therefore has to be treated as part of the deployed workflow rather than as a property you inherit from the model. If p > 0.95 executes an action, I want to know the empirical error rate above that threshold on my traffic, by language, domain, important class and preferably by major customer/process slice. I would also measure risk/coverage curves rather than choosing one threshold and forgetting about it.

The minimum harness I would expect around a production decision model includes offline regression sets, shadow evaluation before rollout, trace collection, per-slice metrics, calibration and risk/coverage analysis, option-order and formulation tests where relevant, drift monitoring, model/version pinning, threshold/version control, and a fallback path for low-confidence or out-of-distribution cases.

This is an important distinction in the economics of Jev-like systems. They can reduce the marginal cost of adding another semantic decision, because you no longer need another full training project every time. But they do not make evaluation, observability and ML operations disappear. If anything, once one general decision model starts controlling dozens of branches, a good shared evaluation and monitoring harness becomes one of the most important pieces of the platform.

I would make this one of the central production sections. It strengthens your main argument rather than weakening it: Jev can eliminate a lot of per-task model engineering, but not the discipline of ML engineering.

20. How I would make the build-vs-buy decision

I would not frame the build-vs-buy decision as “Jev vs GPT.” The more useful question is which computational primitive should own this branch of the product? My default split today is:

I do not read the Jev launch as an argument to stop training classifiers. For a high-volume or high-risk path I still want a specialized model when it materially improves the product; we already make that trade-off in real-time voice.

The opportunity is the much larger set of decisions between deterministic rules and a frontier LLM. That middle layer is expensive today mostly for organizational reasons:

• someone has to create and version the dataset;
• someone has to train and ship the model;
• someone has to monitor drift;
• someone has to recalibrate it;
• someone has to repeat the process per language / market;
• someone has to keep the serving path alive.

The inference bill for an individual classifier may be tiny while the engineering bill around it is not. More importantly, the opportunity is not limited to replacing LLM calls that already exist. Companies leave many decisions unautomated because no single branch justifies its own dataset, classifier, deployment pipeline, calibration, monitoring and maintenance. If the marginal cost of adding another semantic decision falls enough, the number of decisions worth automating expands dramatically.

That is where a general decision model becomes commercially interesting to me: it can amortize that lifecycle across many decision points. The upside is not just cheaper inference on the architecture we already have; it is making a much larger set of small business-process improvements economical to build at all.

If I were buying or building this today, I would ask seven questions before looking at a leaderboard:

1. What decision volume moves off the frontier LLM?
2. What decisions become economical to automate that are currently rule-based, manually handled, or not automated at all?
3. What happens to p95 latency at our concurrency?
4. What is the fallback rate at the error budget we can tolerate?
5. How much per-customer fine-tuning or calibration is required?
6. Can we audit and replay every decision?
7. What do we own operationally: custom runtime, GPUs, data pipeline, retraining, or vendor dependency?

If those answers are bad, the benchmark score is largely irrelevant.

I would add one more operational rule for hosted decision models: pin the model version once you tune production thresholds. TypeSafe currently maps both jev-latest and jev-preview to jev-1.13.0, but the aliases can move when a new release ships. TypeSafe explicitly recommends pinning the versioned ID if confidence or probability thresholds have been calibrated against one model version.

For a chit-chat generative assistant, a silent quality improvement behind an alias may be welcome. For software that executes an action at p > threshold, changing the model behind the alias is effectively changing part of the business logic.

---

21. The biggest unresolved question: calibration under shift

The open-source work already reproduces a surprising amount of the inference pattern. The part I would trust least without my own evaluation is portable calibration.

A model can have:

• good in-domain ECE;
• good temperature scaling;
• strong top-1 accuracy;

and still become confidently wrong when:

• domain changes;
• language changes;
• option wording changes;
• candidate order changes;
• policy structure changes;
• production data drifts.

That means production evaluation should include at least:

• in-domain calibration;
• OOD calibration;
• option permutation tests;
• paraphrase tests;
• missing-candidate tests;
• per-language calibration;
• long-context tests;
• adversarial inputs;
• risk/coverage curves.

For me, a decision model becomes a real product primitive only when I can build policy around its uncertainty. Otherwise it is a faster classifier with a precise-looking number attached.

There is now first-party evidence for another limitation I would explicitly test: semantic equivalence does not imply numerical invariance. TypeSafe's Jev 1.13 documentation shows that the same underlying question asked as a Noul and as a yes/no Choice can produce materially different numbers. It also gives an example where a question and its logical negation, asked separately as two Nouls, return 0.72 and 0.47 — a sum of 1.19 rather than the arithmetic identity one might expect.

For me the production implication is straightforward: do not assume that a threshold calibrated for one question formulation, one primitive or one option set transfers to another logically equivalent formulation. If an invariant matters to the business logic, enforce it in code or calibrate the exact formulation you deploy.

Archestra's real-agent experiment adds a different kind of stress test. They repeated the same 100-call Jev evaluation several times and also reversed or cyclically shifted the Choice option order. On identical runs, 394–398 of 400 labels stayed the same, but only 35–39% of the probabilities were bit-identical. Median probability drift was small, around 0.01, while the largest run-to-run swing reached 0.17.

Changing option order caused more movement. Reversing the criteria produced 389/400 identical labels and cyclic shifting 392/400. After subtracting the baseline repeat noise, Archestra estimates that roughly 4 decisions per 100 changed because of option order alone, reducing overall accuracy by about 1.5–2 percentage points. Most flips were near ties, but not all: in one example a confident internal = 0.83 became public = 0.48 after reordering.

This is much better robustness than the small answer-token models that simply collapse onto option A or the last token, but it is not permutation invariance. If a production threshold sits close to the decision boundary, I would include repeated-call and option-permutation tests in the release harness.

There is now also an explicit first-party language caveat. TypeSafe says English is Jev's primary training language and currently its strongest language. Other languages, including CJK scripts, are supported but not equally strong. For me that means multilingual deployment is not just a translation problem: accuracy, calibration and the automation threshold need to be re-evaluated per language and, for voice systems, on the actual ASR error distribution as well.

---

22. What I think the Jev launch actually changes

After going through these implementations, I would not say that open source has already reproduced Jev. What it has done is more useful for engineering: independent teams have validated a large part of the design space around bounded decisions.

LLM prefill → logits
shared context → isolated decision branches
runtime candidates → NLI / reranking
whole candidate → sequence likelihood
frozen representation → tiny head
encoder → dynamic labels
proper scoring → calibrated probabilities
conformal prediction → abstain / fallback
diffusion → parallel decision slots
large teacher → tiny production policy

For the last few years, the easiest architecture decision in an AI startup has often been:

semantic problem
→ call the LLM
→ get text / JSON

That choice was rational while models were improving quickly, integration was trivial and engineering velocity mattered more than another 100 ms or a few cents. The economics look different once the product has real traffic, SLAs and a team watching gross margin, p95 latency and reliability. At that point “one large model for every branch” becomes less attractive, and the stack starts specializing again:

semantic problem
→ choose the right inference primitive

The foundation model does not disappear; it remains the source of representations, world knowledge, open-ended reasoning and language generation. I simply do not think enum, boolean, score, rank and probability need to go through text generation by default. That is the part of this release I expect to outlive the launch cycle.

I also hope this direction brings a little more ML back into AI. Over the last couple of years it became too easy to turn almost every semantic problem into “call a bigger LLM and parse the answer,” sometimes with much less attention paid to the decision boundary, hard negatives, calibration, error analysis and the evaluation harness around the call. The Jev ecosystem is a good reminder that task formulation, data, loss, inference topology and serving still matter.

The commercial test is less glamorous: does the model actually remove cost and latency from the workflow, or do we give the savings back in calibration work, custom serving and operational complexity?

My bet is stronger than “decision models become another component in the enterprise AI stack.” If we get models that are genuinely cheap, fast, well-calibrated and transferable across domains without a separate training cycle for every workflow, they make a large class of business-process improvements economical for the first time. I am not talking only about cheaper versions of things we already automate; I mean decisions that today are simply not worth building.

I also doubt that the final answer will be exactly Qwen-with-a-head, ModernBERT or today's diffusion implementation. These projects look to me like a search space from which a purpose-built decision architecture will emerge. If I were building one from scratch, that is where I would spend my time.

My forecast is intentionally aggressive: I would not be surprised if the eventual architectural impact were comparable in magnitude to the shift ChatGPT created — not because Jev itself is “another ChatGPT,” but because a whole class of semantic decisions may become cheap enough to automate properly.

There is also a practical side to this. This is very close to the class of problems we work on at AgentUnicorn.AI: production conversational and Voice AI systems where the hard part is not only the model call, but the combination of retrieval, decisioning, tools, guardrails, latency, evaluation and observability around it. If you are implementing conversational or Voice AI in a real business process — or already have an agent that needs to become faster, more reliable and more controllable in production — I am interested in those conversations. The same applies if you are working on the architecture and evaluation side of agentic or Voice AI and want to compare approaches.

And separately, I am increasingly interested in the model-layer question itself. I do not think the current JEV-like implementations are the end state, and I have a fairly concrete view of what I would try if I were building a foundation model specifically for this class of decisions. If you are an investor interested in exploring that thesis, that is also a conversation I would be happy to have.

Even if the larger forecast turns out to be too optimistic, Jev has already done something useful by forcing people to reopen a part of the ML design space that the “LLM for everything” phase made very easy to ignore.

---

FAQ: Jev and Decision Models

What is a decision model?

A decision model maps a state and a bounded question into a typed result such as a boolean, enum, score or probability distribution, rather than generating arbitrary text. Jev is one implementation of this broader idea.

How is a decision model different from a classifier?

A traditional classifier usually has a fixed label space and is trained for a specific task. Jev-like systems accept the decision definition, criteria and sometimes the candidate options at runtime, which makes them closer to general-purpose semantic decision engines.

How is Jev different from constrained decoding or structured JSON output?

Constrained decoding still runs a generative model and restricts which tokens it may emit. The more interesting Jev-like architectures change the computation itself: they reuse shared state, evaluate bounded alternatives directly and often avoid autoregressive generation altogether.

Can I reproduce Jev with an ordinary LLM and first-token logprobs?

Partially. Several open-source projects show that a causal LLM plus prefix caching and answer-token logits can implement a useful approximation. But that does not automatically reproduce Jev's calibration, candidate invariance, question isolation or serving behavior.

What are the main open-source alternatives to Jev?

There is no single open-source clone. Current approaches include causal-LLM logit readout, Kev-style shared-state branching, SemIf, whole-option likelihood, NLI, ModernBERT-based systems such as Laya, diffusion approaches such as djev/OpenJev, decider-style trained decision models, and serving-oriented systems such as openjev-sglang.

Do decision models need fine-tuning for every customer or workflow?

Not necessarily. Jev explicitly uses the same weights across customers, and several open implementations support runtime task definitions. But whether a model transfers well enough to a particular language, domain and decision boundary still has to be measured. Some open models benefit substantially from task-specific fine-tuning.

Are Jev probabilities calibrated?

TypeSafe trains Jev with calibration as an explicit objective, but calibration should not be treated as universally portable. The empirical relationship between a reported probability and actual error rate should still be validated on the deployed language, domain, state construction and workflow. TypeSafe's separate confidence field is also not the same thing as probability of correctness.

Can decision models replace generative LLMs?

Only for bounded semantic decisions. They are a poor replacement when the task requires explanation, synthesis, planning, open-ended reasoning or generation. A likely production architecture uses both: generative models where generation is required, and cheaper decision primitives for routing, policy, eligibility, tool selection and workflow control.

When should I use rules instead of a decision model?

Use deterministic code when the decision can be expressed reliably as arithmetic, exact logic, database predicates, date comparison or another explicit rule. Decision models are most useful in the middle ground where the decision is semantic but the output space is bounded.

What infrastructure is required to use a decision model safely in production?

You still need an ML harness: representative eval sets, decision traces, replay, calibration checks, per-slice metrics, distribution-shift monitoring, threshold versioning, model version pinning, fallback policies and regression testing. A general decision model reduces per-task model engineering; it does not eliminate ML operations.

How should decision models be evaluated?

Overall accuracy is rarely enough. Measure per-class precision and recall, majority-class baselines, calibration, risk/coverage curves, error cost, latency under realistic concurrency, fallback rate and robustness to changes in state, wording and option order.

Why are decision models particularly relevant to Voice AI and agents?

Agentic and Voice AI systems contain many small decisions on the latency-critical path: tool routing, RAG/no-RAG, interruption handling, escalation, policy checks, next-state selection and model routing. Removing unnecessary generative calls can reduce both latency and cost, provided the decision quality is validated on the actual pipeline.

Can decision models automate workflows that are currently rule-based or manual?

Potentially, and this may be more important than replacing existing LLM calls. If semantic decisions become cheap enough to deploy without building a dedicated classifier for every workflow, many low-volume or long-tail decisions that are currently handled manually, with brittle rules, or not automated at all become economically viable.

Is Jev a replacement for an ML team or ML platform?

No. It may remove a substantial amount of per-task training and model-serving work, but task formulation, evaluation, data quality, calibration, observability and production monitoring remain necessary.

---

Primary sources and repositories

The repos in this space are moving daily. Implementation-specific numbers above are tied to the linked project state and the article's September 23, 2026 update date.

Official TypeSafe / Jev resources

• TypeSafe AI
• TypeSafe — Introducing System One Models & Jev
• TypeSafe model reference
• TypeSafe confidence semantics
• TypeSafe Jev 1.13 jaggedness

Repositories

• JevBench
• mini-Jev
• reflex
• Kev
• SemIf
• daseinlabs/open-jev
• Bespoke Nimble
• Laya
• GLiClass
• Verdict
• Verdict 2.0
• decider
• poorjev
• DiffusionGemma/OpenJev
• openjev-sglang
• system-one-open
• Winnow-12B

Research and analysis

• Archer Hume — Jev's Architecture Unmasked
• Archestra real-agent benchmark

---