Hugging Face将TypeSafe的Jev推理接口合并进transformers

In my LLM inference roadmap, I split every model call into two phases. Prefill reads the prompt in parallel and is compute-bound. Decode emits output tokens one at a time, reading the entire weight matrix from HBM on every step, and is memory-bandwidth bound.

Most software decisions do not want text generation at all. Routing a support ticket, gating a risky tool call for human review, scoring an incident rubric from 1 to 5, or producing the P(bullish) signal I used in Laya Trader all ask for a probability distribution over a known set of outcomes. Calling /v1/chat/completions for those tasks pays the decode tax for dozens of tokens, hopes the JSON parser survives, and still leaves you without a trustworthy probability.

TypeSafe’s Jev (POST /v1/systemone) replaced that loop with a typed contract. You send one state (a document, JSON record, or ticket) and a map of named questions about that state:

The model runs prefill once, generates zero output tokens, and returns a probability for every option.

Over the last two weeks, this interface moved from a proprietary API to an open pattern you run and train yourself. Hugging Face merged native /v1/systemone support into transformers serve, turning 250+ open LLM and VLM architectures into logit-scoring decision models out of the box. And on JevBench, small open-weight decision models like SemIf (Qwen3.5-4B, scoring 73.1) and Winnow-12B Q8 (71.2) now sit right behind hosted Jev 1.13.0 (74.4) at roughly half the cost per thousand decisions.

This post walks through how to build and serve a decision model from the ground up: starting with Hugging Face’s zero-shot /v1/systemone server, then fixing its ceilings with reserved option tokens, calibrating probabilities with temperature and soft targets, packing multi-question requests with tree batching, and examining how Jev itself is wired under the hood.

Before training a single weight, serve the /v1/systemone contract locally on top of an existing open checkpoint. Hugging Face’s transformers serve CLI (added in PR #49322) formats your state and questions through the model’s chat template with thinking disabled, runs one batched forward pass (output_tokens: 0), and slices the next-token logits at the assistant prefix:

pip install "transformers[serving]" transformers serve

curl http://localhost:8000/v1/systemone
-H "Content-Type: application/json"
-d '{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "state": "Our integration has been failing for 3 days and we are losing sales!", "questions": { "team": { "type": "choice", "instructions": "Which team handles this?", "criteria": { "billing": null, "technical": "Bugs or integration problems" } }, "urgency": { "type": "score", "instructions": "How urgent is this ticket?", "criteria": ["low", "medium", "high"] }, "upset": { "type": "noul", "instructions": "Is the customer upset?" } } }'

The response maps each question key back to its probability distribution and a probability-weighted score:

{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "answers": { "team": { "type": "choice", "choice": "technical", "probabilities": {"billing": 0.000007, "technical": 0.999993}, "confidence": 0.999986 }, "urgency": { "type": "score", "score": 1.6739, "legend": {"0": "low", "1": "medium", "2": "high"}, "probabilities": {"0": 0.0031, "1": 0.3198, "2": 0.6770}, "confidence": 0.5109 }, "upset": { "type": "noul", "noul": 0.8520 } }, "usage": {"input_tokens": 213, "output_tokens": 0} }

Because transformers serve runs on top of the standard transformers processor pipeline, it also supports vision-language models: passing an images array with up to four data:image/...;base64,... strings attaches those images to the shared state for every question in the call.

How does transformers serve score an off-the-shelf chat model without custom training? By default, it maps choice and score options to single-token letters A through Z, maps noul options to Yes and No, and takes the softmax over those token logits at temperature = 1.0.

When you move from a zero-shot chat model to a fine-tuned decision checkpoint, you pass --decision-config decision.json at startup:

transformers serve openjev/openjev --decision-config decision.json

{ "labels": { "noul": ["A", "B"] }, "temperature": { "choice": 0.85, "score": 0.85, "noul": 1.5547129 } }

Those three fields in decision.json (labels, temperature, and chat_template) point straight at the three limits of zero-shot letter scoring. To understand why dedicated decision models beat zero-shot wrappers by 10 to 25 points on JevBench, we need to look at what happens at the token and logit layer.

Scoring the logits of plain letters A, B, C at the end of a prompt breaks in three ways:

A dedicated decision model solves this by adding reserved option tokens to the vocabulary (<opt_0>, <opt_1>, …, <opt_254>) along with a dedicated <readout> token at the end of the sequence. Most tokenizers already include spare slots that never fired during pre-training (such as Gemma’s <unused0>, <unused1>, …). Otherwise, append new rows to the input embedding matrix and the output head matrix.

Three architectural details decide whether those reserved tokens train cleanly:

The same one-token readout handles all three question primitives:

Why read from the model’s own vocabulary output head instead of freezing the transformer and fitting a small linear classifier probe on the last hidden state? Because the existing output projection matrix already shares the geometry of the model’s 200k+ word vectors. Unfreezing the backbone (via full fine-tuning or all-layer LoRA) and reading the option rows from the native LM head consistently outperforms a frozen probe.

When application code uses a decision model’s output as a number (if p_fraud > 0.92: block_payout()), argmax accuracy is only half the contract. A reported 0.90 probability needs to be right 90% of the time.

Fine-tuned transformers almost always drift toward steady overconfidence or underconfidence. The first correction is temperature scaling: divide the kept option logits by a positive scalar T before the softmax (softmax(z / T)). Values of T > 1 spread probability mass out, values of T < 1 sharpen it, and the top-ranked option never changes. Because binary, multi-way, and ordinal heads drift at different rates, fit a separate T for choice, noul, and score on a held-out validation set (mirroring how the --decision-config example earlier set 0.85 for choice and 1.5547 for noul).

Four calibration traps catch teams building their own pipeline:

Eliminating the decode loop explains why one question answers in ~150 to 250 ms. It does not explain how a decision server answers 32 or 64 questions about the same 8,000-token or 30,000-token document with almost zero extra latency.

If you wire a multi-question /v1/systemone endpoint on top of vLLM the naive way, you expand N questions into N separate sequences (state + q_1, state + q_2, …, state + q_N) and rely on prefix caching so the GPU computes the state’s KV blocks once. Latency still climbs steeply with N for two systems reasons:

Tree batching removes both bottlenecks by packing the entire request into one sequence of length S + N × L:

TypeSafe’s documented context limits for Jev show this exact tree structure in the numbers. Jev enforces 32k tokens per question and 64k tokens per request (with an opt-in header raising both to 262k). In a packed tree, 32k caps a single root-to-leaf path (S + L, the maximum context any one question attends to), while 64k caps the packed buffer (S + N × L, counting the state once). If the server duplicated the state per question, two questions on a 30k document would already blow past a 64k request ceiling. Archer Hume’s black-box latency study confirmed the same single-state, batched-branch behavior empirically.

TypeSafe describes Jev as a non-autoregressive transformer (The Batch) and keeps its base model and training recipe closed. Because a decision model must assign a probability distribution across whatever options you pass it, it cannot refuse a prompt the way a chat model does. Combined with the exact usage.input_tokens count returned on every API call, that property lets you fingerprint a black-box decision model across five independent tests.

If Jev sits on a 35B-A3B backbone and answers in a single prefill pass with zero reasoning tokens, how does it beat non-thinking Qwen3.5-35B-A3B on hard reasoning tasks?

You get that jump in one of two ways: distilling multi-step answers from a strong thinking model (Qwen or OpenAI’s gpt-oss-120b) into one-pass readout labels, or generating synthetic tasks where code computes and verifies the exact answer. On the Latent Space podcast, TypeSafe founder Diogo Almeida described that exact synthetic loop: all their training data is synthetic with “programs in the loop”, and on online RL rollouts he noted “zero is the optimal amount for our shape”.

That also demystifies RLCD (Reinforcement Learning for Calibrated Decisions), the label TypeSafe gives its training stage. When a program generates the task and knows the exact answer or probability distribution, you do not need PPO rollouts. Standard cross-entropy against program-verified soft labels trains a calibrated one-pass readout directly.

Sending a prompt with and without a candidate substring and diffing usage.input_tokens tells you whether that substring is a single token in the server’s vocabulary. Reconstructing Jev’s tokenizer token by token uncovers a 145,072-token vocabulary spliced from Qwen 3.5 (248,044 tokens) and GPT-4o / gpt-oss (199,998 tokens):

That third bullet is the unmistakable signature of Fast Vocabulary Transfer (also implemented in mergekit’s tokensurgeon) applied to an already-trained Qwen 3.5 checkpoint. When you add a composite token "embley" to a trained model, you initialize its input embedding row and output head row as the average of its two constituent rows ("em" and "bley"), then fine-tune so the spliced rows settle in. Adding a token with no existing pieces in the base model would require learning its vector from random noise.

Pruning 113,436 tokens and stripping BPE merge rules shrinks the memory footprint of the input and output tables on English traffic (1.0x token count relative to Qwen 3.5), at the cost of a steep token tax on other scripts: ~1.7x more tokens on Chinese, ~2.0x on Russian, Greek, Arabic, and Japanese, and ~4.0x on Thai. Because the API bills per input token, non-English states pay that multiplier on every call.

If you want to replace slow, uncalibrated chat completions with typed probability readouts today, you have four concrete options:

Sources and related reading: Hugging Face, Transformers Serve CLI: /v1/systemone, TypeSafe, Jev Model Documentation, JevBench v1.3 Leaderboard, Archer Hume, Jev’s Architecture, Unmasked, Just Ask Jev, Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration, Fast Vocabulary Transfer, Diogo Almeida on the Latent Space podcast, and my earlier posts on LLM Inference: a learning roadmap, Frink Inference, Laya vs Dijkstra, and Laya Trader.