From a Jev video to a local decision API

I watched a Two Minute Papers video about Jev and had the uncomfortable feeling that the interesting part was smaller than the hype and more useful than the hype at the same time. The claim around Jev is easy to oversell if you turn it into another story about a magic model. The useful part is quieter: a lot of model work is not writing. It is deciding between explicit answers.

Two short questions A support ticket does not always need a paragraph. It may need a route. A deployment note may not need a summary. It may need a yes or no on whether the rollout succeeded. An agent about to call a tool may not need a second chat model composing advice. It may need a guardrail check with three possible outcomes. That is the Jev-shaped idea that stayed with me after the video: make the output type small, fixed, and useful.

The first local clue The first thing I found that felt close was Kev, Jared Palmer's small Jev-like decision model family. Kev exposes the same kind of questions: noul for yes or no, choice for a declared option set, and score for an ordered rating. I tested it locally and it behaved like a different species of model call. It was not trying to be a conversationalist. It was turning evidence plus a question into probabilities.

Kev is important because it is not just a prompt wrapper. It is built on Qwen3.5 bases with trained adapters and a decision readout for this style of request. That makes it a more faithful open reconstruction of the Jev pattern than a plain general model. It also made the comparison useful: if Kev is the trained route, could there also be a tiny zero-shot route that gives people the same interface quickly?

A practical gap The gap was packaging. Kev is a real local route, but I wanted a tiny installable project that made the interface easy to try on an ordinary Linux machine, with model download handled by one command and a server that felt familiar from the first request. I also found SemIf, an open project that explores the same semantic-if pattern with GGUF models. That made the path obvious enough to try: keep the interface, use llama.cpp, read the answer-letter logits, return probabilities.

The result is local-jev The result is local-jev. It is not official TypeSafe Jev, does not ship TypeSafe weights, and has no TypeSafe affiliation. It is a local Jev-style package: local-jev setup creates a small runtime and downloads an open GGUF model, then local-jev serve starts a localhost API at /v1/systemone. The default model is Qwen3 0.6B quantized to Q4_K_M, about 484 MB. There is also a larger Qwen3.5 4B preset for people who want to trade disk and load time for better behavior.

The shape matters The server accepts the same basic shape every time. You send state, then a map of questions. Each question says whether it is noul, choice, or score, and for choice or score it declares the allowed answers. The model never gets to invent a JSON schema. It sees a prompt with evidence, criterion, and labelled answer slots from A to P. local-jev reads the logits for those letters, applies a softmax, and returns the distribution.

What a logit is Before a language model writes text, it scores every possible next token. Those raw scores are logits. Normal generation repeatedly turns those scores into text: score the next token, pick or sample a token, append it, then do it again. A normal structured-output wrapper often asks the model to write something like {"choice":"billing"} and then tries to parse that text.

local-jev stops before that writing step. It builds a prompt whose answer must be one of the option letters, runs the local GGUF model once, reads only the logits for the allowed letters, and normalizes those scores into probabilities. The JSON response is created by the server code, not by the model. That is the central difference: it uses a normal local LLM underneath, but it uses it as a decision scorer rather than as a prose generator.

The benefit is control. The schema is fixed, there is no malformed JSON to repair, the result includes an option distribution, and the model cannot add extra text. The non-benefit is just as real: the scores are not automatically calibrated, wording and option order can matter, and a small general model has not learned the decision task the way Kev has. This is why I call it Jev-style. It approximates the typed-decision readout, but it does not claim to be Jev.

curl -s http://127.0.0.1:8010/v1/systemone \
  -H 'content-type: application/json' \
  -d '{"state":"reset email never arrived",
       "questions":{"route":{"type":"choice",
       "instructions":"Which queue should handle this?",
       "criteria":{"account":"login and access",
                   "billing":"payment or invoice",
                   "sales":"buying or evaluation"}}}}'

On the machine On one local CPU machine the first one-shot CLI call took about 21 seconds because it had to load the model. Once the server had the model in memory, the useful path was much faster: a three-question support-style request took about 486 ms, and a slightly longer classification-style request took about 847 ms. The package itself is tiny, around 15 KB as a .deb, because the model is downloaded during setup instead of being bundled.

A small benchmark I then ran the package against 100 generic support-routing messages: 25 account, 25 billing, 25 shipping, and 25 technical. This used the default Qwen3 0.6B Q4_K_M preset on a warm local server with four fixed answer options. It is a smoke test, not a scientific benchmark.

By label, the run got account 18/25, billing 9/25, shipping 2/25, and technical 13/25. That is not good enough to hide. It shows exactly where this zero-shot approach is weakest: the tiny default model had a strong first-option bias, especially against shipping. Confidence helped only at the very top. Most calls sat below 0.75 confidence, and the middle confidence bands were noisy. So the score is a gate signal to validate, not a truth meter.

What it is good for The good use cases are boring in the best way: ticket routing, policy checks, triage, relevance checks, deployment success checks, confidence bands before automation, and agent-side gates before a tool call. The pattern is strongest when the decision can be written as a small set of options and when you care about probabilities more than prose.

What it is not The limits are just as important. The default model is small and can be overconfident. These probabilities are option scores, not measured truth. A larger model can help, but validation still matters. I would treat local-jev as a cheap local decision primitive, not as an oracle. That boundary makes it more useful, because you can design around it instead of pretending it is a general brain.

A small useful thing What I like about the project is that it turns the Jev conversation into something testable. Install it, run a request, watch the probabilities, and decide whether the interface fits your own workflow. Maybe the future has specialized hosted decision models. Maybe a lot of small local tools only need this much: a fixed set of answers, a local model, and a fast enough probability readout.