Testing Jev in a Lifting App: Where It Won and Lost

Almost everything written about Jev right now explains what it is. There is a Wikipedia page, a Forbes piece, a LangChain write-up, a Reddit thread with a few hundred comments. What there is very little of is someone reporting what the model actually did when it was pointed at real work, in a shipping product, and measured against a job that product already does.

We are a small hypertrophy logbook for iPhone and Apple Watch, not an AI company. But we do have one feature where a model reads what you did and proposes what to do next: Coach, which reads your logged sets and suggests the loads for your next session with the reasoning shown. So when Jev landed, we ran it against three separate parts of that turn and against a model we already ship, spent about a dollar and a half doing it, and wrote down what happened. This is that log. Some of it is a win for Jev. One of the tests it lost outright.

What Jev does differently, in one paragraph

Most language models answer in prose, and you parse the prose. Jev is a typed decision model: you hand it a block of text as the state plus named questions, each with a declared answer type (a yes/no probability, a choice among labelled options, or a position on an ordered scale), and it answers all of them in a single pass with calibrated probabilities and no prose at all. The vendor's own docs describe it as a decision engine rather than a chat model, and the harness authors who built early integrations, including LangChain, use it as a guard around a bigger model rather than as the thing that writes the reply. That shape is exactly why it looked useful to us: a lot of the small decisions inside a Coach turn are classifications, not compositions.

The three jobs we tested it on

A Coach turn is not one model call. Before the reply is written, the turn has to decide whether the message is even in scope, whether it is simple enough to answer with less effort, and whether any of the surrounding context (imported notes, exercise names, older messages) is trying to smuggle in an instruction. After a draft reply exists, something has to check that every claim in it is actually supported by your logged data before you see it.

Those are four small decisions, and we tested Jev on all of them:

• A scope gate. Is this message about training and recovery, or is it someone asking the model to write a poem?
• A complexity signal. Is this a trivial "what's next" or a real programming question?
• An injection screen. Does the untrusted context contain instructions aimed at the model?
• A reply reviewer. Does the drafted answer's every claim follow from the records it cites?

We also ran Jev against a completely separate model we already ship, to see whether it should replace it. That is where the one clear loss came from, and it is the most useful result in the whole exercise.

The 20-language scope test

The gate we cared about most was scope, because getting it wrong is expensive in both directions: refuse a real training question and you have broken the feature; wave through an off-topic request and you have spent a full, slow model turn on a poem. And because the app ships in twenty languages, a gate that only works in English is a gate that quietly fails most of the world.

So the first trial was 320 labelled messages: sixteen cases translated into all twenty shipped locales, sent with the exact questions the live code asks. Every one of them got an answer; none failed. The result was not clean on the first pass, and the way it was cleaned up is the part worth reporting.

With the original question wording, per-locale accuracy sat between 88% and 100%, with no single language standing out as the weak one. The misses were not random: they clustered on the same two cases in every language. App questions ("can it export to Hevy?") averaged around 0.45, and one locale's bad-sleep question came in at 0.22, because the question the model was being asked only mentioned "strength training" and said nothing about recovery or the app's own features. The gate was reading those messages as borderline off-topic because, strictly, they were not about lifting a barbell.

Rewording the question to name recovery and the app's features, then re-running all twenty locales, moved app questions up to 0.96 to 0.98 and lifted the weakest in-scope message to 0.53, while off-topic messages stayed at or below 0.11. A full 320-message re-run on the committed wording confirmed it:

The lesson there is not really about Jev. It is that a classifier is only as good as the question you ask it, and the model was right to be uncertain about a sleep question that mentioned nothing it had been told to care about. The only disagreements left after the fix were a marathon-planning case (whether running counts as in-scope is a product decision, not the gate's) and a protein question that landed at exactly 0.50 in two of the lower-confidence translations.

Two things made those thresholds trustworthy. First, the model is deterministic: five identical requests moved the answer by a median of 0.00 and a maximum of 0.07, so a threshold measured on Monday still holds on Tuesday. Second, the extremes are calibrated but the middle is not. Messages scoring in the bottom fifth were in scope about 1% of the time; messages in the top fifth, 99%. The middle buckets were close to coin flips. That is why every threshold we chose sits at an extreme, and why the gate refuses only strictly below a low bar rather than trying to read a confident answer out of the middle.

Where Jev lost: our own classifier

The scope, complexity and injection trials asked whether Jev is good enough at a job nothing in the app does today. The sharpest test was different: on a classification job we already ship a model for, is Jev better?

That job is Describe your workout, where you type what you want in plain language and a small on-device model turns it into a set of muscle regions, with the empty set meaning "could not tell" so the request falls through to the routine picker. It runs on the phone, offline, with no network call. We scored Jev on every one of the 14,670 held-out rows the shipped model is evaluated on, using the same metrics, at each model's own best operating point.

The shipped model wins. On the 4,600 rows where exactly one of the two is right, it took 2,449 to Jev's 2,151, a sign test at p below 0.0001. Two points on exact match and eight on the F1, and the F1 gap tells you what is going on: Jev is far more precise per region (89 to 98%) at far lower recall (51 to 64%). It names fewer muscle groups and is usually right about the ones it names. That wins on short, unambiguous requests and loses badly on the long ones, where our own convention expands a single phrase into three, four or five regions.

There was a methodology trap here worth passing on, because it nearly produced the opposite conclusion. The first run used an 800-row sample balanced by language and preset, and on that draw the two models tied exactly and Jev looked ahead on the East Asian locales by 13 points. A second balanced draw with a different seed put our model 9 points ahead and erased the gap. Our model scored the same on both draws; it was Jev's score that moved, because its errors concentrate on exactly the long multi-region requests that a balanced sample under-weights relative to the real mix of what people type. One sample was not enough to see that. Two contradicting samples were not enough to settle it. Asking every row was, and for about a dollar and a half in model calls, that should have been the first move, not the last.

Jev was genuinely ahead on the four East Asian locales, by 4 to 7 points rather than the 13 the first sample suggested, and that held up even after controlling for the fact that our on-device model is fed a normalised, lowercased form while Jev saw the original sentence. That is a real and useful signal about our own model, and we are tracking it as its own problem. But the decision was not close: Jev is behind on the metric the task is actually scored on, it would turn an offline, on-device inference into a synchronous network dependency inside the build path, and it would send the typed request to a processor for a feature that works fine without one. Nothing changed in the app.

The injection screen and the reviewer

Two of the tests were straightforward wins.

The injection screen takes the whole block of untrusted context (imported notes, exercise names, pasted text) as one state and asks a single question: is anything in here an instruction aimed at the model? Across 22 strings in four languages, half of them carrying planted instructions, clean strings scored at most 0.08 and injected ones at least 0.88. Every single case separated cleanly. There is no ambiguous middle to worry about here, which is unusual and welcome.

The reviewer is the job Jev is best suited to, and the one place it actually runs today. After Coach drafts a reply, the reviewer reads the reply against the specific records each sentence cites and asks, per segment, whether every claim is entailed by its sources, whether it claims an action that has no receipt, and whether it strays into clinical territory. On 27 labelled cases (false claims, invented schedules, reversed trends, wrong numbers written out in words, injections, plus grounded controls, in four languages) it agreed with all 27 labels: supported segments scored 0.81 and up, unsupported ones 0.22 and down, and it evaluated sixty questions in a single 0.67-second call.

What makes the reviewer worth shipping is not that it replaces the slower, more careful review round. It is that it does not try to. Segments in the confident zones are delivered or pruned immediately; only the genuinely ambiguous middle is handed up to the more expensive review. On real drafts, every candidate whose weakest segment sat in the 0.45 to 0.65 band went on to pass that fuller review, and the ones that scored 0.30 or below failed it. Jev is acting as a fast triage in front of the real judge, not as the judge.

The latency reality

One number is worth stating plainly, because it is easy to take a vendor's figure and repeat it. TypeSafe's docs put end-to-end latency at 70 to 500 milliseconds. From our dev machine, at the payload sizes a Coach turn actually sends, five warm calls came back at 0.64, 0.80, 0.77, 0.70 and 0.64 seconds, with about 0.19 seconds of that just the connection handshake.

That is our measurement, on our path, and it is not a verdict on the model. A network hop from a laptop is not the same as one from the server that runs the feature, and we have not yet measured the server-side path. But it did change a decision: an 800-millisecond timeout that had been chosen before anyone measured would have dropped roughly one call in five, and it would have dropped them from the slower, longer messages that a review most needs to see. We raised the ceiling once the real numbers were in.

What actually runs today, and what does not

It would be easy to read all of the above and conclude the app now leans on Jev. It does not, and the distinction matters.

In the shipped configuration, two of these uses are on: the reply reviewer, and a tool-planning step that decides which of your records a turn needs to read before the first model round. Those are the two that make a turn faster without changing what it is allowed to say, and the fuller review round is kept for the answers the fast pass is not sure about.

The scope gate, the complexity router and the injection screen are built and tested, but switched off. "Built but not switched on" is the honest description, not "live." There is a real reason for the gap: turning any of them on means a Coach message reaches a second processor, and the privacy page has to say so before that happens, not after. Since late September it does, naming TypeSafe alongside Google's Vertex AI and stating exactly what each receives.

If you are choosing a lifting app and you care about this kind of thing, that sequence is the point. The interesting decision was not "should we use the hot new model." It was "measure it against real work in every language we ship, in front of a job we already do well, and only keep the parts that earned it." Jev earned the reviewer and the tool plan. It did not earn a job our own on-device model does better, and it does not touch your data at all until the page that tells you so says it will.

That is also, roughly, how the rest of the app is built. Coach does not guess at your progress; it reads the sets you actually logged and shows the reasoning behind every load it proposes. The whole reason the target lives on your wrist and the rating happens at the moment of the set, rather than being reconstructed from memory later, is that a suggestion is only as good as the record under it. A model on top of bad data is just a faster way to be wrong.

So we spent the dollar and a half to find out which parts of a new one were worth trusting, and kept only those. If you want the argument for why the record belongs on your wrist rather than your phone, or a look at how logging apps compare, both are worth your time. Either way, log the work from your wrist and the model reads what is really there.