This appears the first decision AI that actually handles audio, video, and PDF, and I'm surprised this isn't highlighted everywhere.
Marketing seems to have dropped the ball on that. I doubt anyone's picking it for the +2% higher points, but being able to answer based on a PDF or call is groundbreaking.
Jev is still faster and cheaper for many of the common text uses.
I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
Wouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels.
Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution
Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
The thing I had to wrap my head around is: For a "classic" classifier, the classes are fixed and the data distribution is p(x,y). The calibration is checked over this single distribution. Jev makes the tasks part of the input, so the data distribution is p(task,x,y) and calibration is averaged over that joint distribution. What follows is that while for classic classifiers, calibration brakes down if your x was out-of-distribution, it can now also break if your task is out of distribution. The big question now is: Can subtle differences in phrasing your task already lead to this? And the only way to verify calibration is having ground truth data to validate on.
Also: Calibration over p(task, x, y) means calibrated on average across tasks. Even an in-distribution task can be over- or underconfident while the average looks fine.
> And the only way to verify calibration is having ground truth data to validate on
I’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
A model can be consistently over- or under-confident for similar inputs. The only way to check if it's actually calibrated is to compare against some known ground truth. I don't see any way around that.
In my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.
> It makes an unbounded number of future predictions
nit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.
From what I’ve gathered, it doesn’t work at all like a normal classifier. As in, for a classifier you’d train it for a specific task to reduce the cross entropy. So, in a hand-wavy way “the model knows the truth”. For Jev-type models it’s more like an LLM, in the sense that the “truth” is in the context you provide. But instead of producing a completion it fills in the schema. But it’s always - “given the context, the schema needs to be filled in like this”
Unless I am misunderstanding you then I don't think this is correct. Chain of thought was first proposed in 2022. Assuming you mean reasoning models then the correct comparison is GPT-o1, which came out in September 2024, 2 months prior to R1 lite preview. In the next 2 months, prior to the release of the full R1 model, both qwen and gemini released reasoning models. Dario explicitly notes in early 2025 that most major labs had been working on reasoning models in 2024, and explicitly credits OAI for pulling things forward with their release.
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
I wasn't suggesting they invented it. I was suggesting that they popularized it, made a huge amount of noise about it, and it seemed like after they did, there was a lot more attention on it.
I think if you were paying attention at the time, CoT and ReAct loops were already "well known". Lots of GPT wrappers at the time were doing it, I remember doing CoT to help measure the calories for a recipe. The problem is tokens then were quite expensive and doing experimentation on it was pretty expensive for an individual. I once accidentally spent $20 trying to categorize a Who's Hiring post from here lol.
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
Dont under estimate the power of branding/positioning/marketing. For more than a year I have been using a RL'd Qwen fine tune optimised for just taking a decision and outputting the decision number, initially I didnt have much training data so after a few manual examples I asked deepseek to make them for me and had codex train it.
I compared it to jev when it first came out and it was mostly neck and neck for my use case.
I can't have been the only one in fact, I'm sure there must have been other people who probably even open sourced, but I couldn't find anything. I think the question going forward will be how does one keep abreast of interesting stuff happening, now that its de-coupled from direct human labour so de-bottlenecked on production capacity - just bound by how residual time/attention one has (Terence Tao was talking about this exact predicament in the latest tyler cowen podcast).
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
That's how the original Transformer architecture (LaMDA) should have been released back in the day, but Google thought it would cannibalize their search revenue.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
That also makes me wonder: what other things are achieved by now and are up for grabs, and that will seem obvious in retrospect, but we don't notice it yet?
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
> Which makes me wonder: how do these models calculate probabilities?
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for most uses of decision models.
In some sense everything an LLM does is by unprincipled methods. But it works! We judge it empirically post hoc by its performance rather than relying on tuning.
Hell, human thinking is unprincipled in basically the exact same way.
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Not reliably - look at microsoft's guidance project that did some of this.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
OpenAI was already working on a Decisions API, and Jev launched theirs quickly enough to have 1 week in the spotlight before OpenAI put theirs in beta (which works great btw).
When trying it out, OpenAI's Decisions was ~2x slower, worse accuracy and more expensive. Unless used with image-related tasks, Jev is clearly superior.
I've found that they work great as gates ("does this require a human review?") or filters ("is this suspicious?") and occasionally as judges (based on this context, is the expected pain score for this user above 7.5/10?).
Interesting how do you know if it does a good job at these tasks? It seems that thinking in LLM's has been one of the things that has been so important to making sure it's a good response.
It doesn't. Decision models are not more accurate than small models with no thinking. If your classification task is accuracy sensitive, you still need to stick with one of the big frontier models.
When the use case allows room for an acceptable amount of incorrect decisions because there's a manual flow those cases can be routed to still anyways.
It can be used a fast/cheap router, if a response is above a specified probability then send it to LLM based workflow branch, otherwise nothing more to do
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
I'm using them to label training data for model development. You can do something like label a bunch of records with Jev, then train the model, then check results with opus and a hold-out set.
Task classification - given a set of classes, return the most likely class for each task
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
I work in manufacturing and would like to use LLMs for scenarios where there's obviously still rules for what should happen next, but there's just so many variables that I can't just code it into a decision tree easily.
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
2a) decision-tree for *who* to send it to for reworking things
3) scrap
As I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
Yes, literally built my own version of /goal in Claude/codex today for pi using jev. My agent was stopping so jev is called each turn to see if it’s done or not - similar to how it works in Claude, according to Claude…
Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
I'm trying to organize about 400k photos. What kind of questions are you asking it and what are you doing with that info? Applying tags/labels? Sorting into albums?
Basically just tags, but the questions branch based on the answers. Photo vs Graphic, Portrait vs Landscape, Person/People/No one, etc.
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
We're about at the point where the AI is now pretty insanely cheap. Passing 100K tokens of context then asking hundreds of questions of 100 tokens each would cost about as much as a single Google search API call.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
When I reviewed our internal AI workflows i was shocked at how often a decision API was a good fit or could streamline the process.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma4 26A4. That makes me optimistic for this release, despite the high cost.
I wouldn't even trust this is the high level of their pricing with the way they just casually doubled Gemini Omni pricing in 1.2 while trying to spin that it was somehow "cheaper"
You could instruct them to behave like one but they’ll still eat a lot of tokens, probably more, even though you’ll save a small amount on the output. It’s a separate thing because of significant token savings.
Audio, Video and File input? That is very appealing and I can imagine a few very unique ways to leverage this if it works well across all modalities.
For what it's worth, ran my quick and dirty evals with it and got in many usage scenarios a 4x lower brier over Jev (2x - 6x depending on task), which is an immense improvement and the first time a model so significantly has outperformed Jev in my experience. p50 was also over 100ms lower though all this cost twice as much across the entire eval end-to-end.
Dare I say, this is a very impressive release by the Gemini team. While testing GPT-6 Luna lead to an honestly very pleasant interaction with OpenAI staff on here, it did perform very uncompetitively so seeing a great first effort in basic text classification performance with broad multimodality, that is great to see. Course, wouldn't be surprised if Google had such a more broadly trained classifier for years as part of internal usage for platform features which they just now commercialised, but hey, I'll take it.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
That's always been the enterprise tech way. Using the product that comes with the platform the company is already signed up to pay for is easier than requesting a payment to a brand new service.
They don't need to be the best, they just need to not force a customer to go elsewhere.
I suppose the key question is come 5 years time is enterprise going to just become happy to use openrouter, or will they lose competitiveness/ market share to those who do.
Fyi artificialanalysis currently puts drex highest on price vs quality
What got me into Claude Code started with Google Antigravity. Somewhere along the way Antigravity took a dive. The rollout was great, then it started feeling like other G-products, the dreaded tail of lackluster developer energy after a peak.
We have Antigravity at work. It's become a lot better, especially since the Gemini 3.8 Flash and 4 Argon versions. The harnesses themselves (GUI and CLI) are updated quite regularly.
I don't really miss Claude when I'm using it for coding tasks. Also, it doesn't speak Claudish, which is a plus.
Opaque classifier as a service, for when you want to discriminate people by race and social status assumed from name, but want some other entity to be responsible for plausible deniability of the process.
Hey Jev, what mortgage rate multiplier should a person named James Smith get?
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
Startup / Founder focus vs Large Organizations with no vision.
With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
I still have no idea why this is useful. Models are atrocious without reasoning, for classification and otherwise, and this doesn't even just eschew reasoning afaik, but parts of the models as well for lower latency and cost. I genuinely do not see the point.
What differentiates this from training a classifier model? That it comes pretrained with general knowledge?
I think that is exactly the value prop, you don't need to train a classifier. You get something that works well enough to validate your product/feature/whatever before investing in training.
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
143 comments:
This appears the first decision AI that actually handles audio, video, and PDF, and I'm surprised this isn't highlighted everywhere.
Marketing seems to have dropped the ball on that. I doubt anyone's picking it for the +2% higher points, but being able to answer based on a PDF or call is groundbreaking.
Jev is still faster and cheaper for many of the common text uses.
>Marketing seems to have dropped the ball on that
Google's marketing consistently drops the ball on every opportunity so that's not surprising at all
I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
Wouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels. Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution
Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
The thing I had to wrap my head around is: For a "classic" classifier, the classes are fixed and the data distribution is p(x,y). The calibration is checked over this single distribution. Jev makes the tasks part of the input, so the data distribution is p(task,x,y) and calibration is averaged over that joint distribution. What follows is that while for classic classifiers, calibration brakes down if your x was out-of-distribution, it can now also break if your task is out of distribution. The big question now is: Can subtle differences in phrasing your task already lead to this? And the only way to verify calibration is having ground truth data to validate on. Also: Calibration over p(task, x, y) means calibrated on average across tasks. Even an in-distribution task can be over- or underconfident while the average looks fine.
> And the only way to verify calibration is having ground truth data to validate on
I’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
A model can be consistently over- or under-confident for similar inputs. The only way to check if it's actually calibrated is to compare against some known ground truth. I don't see any way around that.
In my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.
> It makes an unbounded number of future predictions
nit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.
From what I’ve gathered, it doesn’t work at all like a normal classifier. As in, for a classifier you’d train it for a specific task to reduce the cross entropy. So, in a hand-wavy way “the model knows the truth”. For Jev-type models it’s more like an LLM, in the sense that the “truth” is in the context you provide. But instead of producing a completion it fills in the schema. But it’s always - “given the context, the schema needs to be filled in like this”
as s.o. who registered their chatgpt account on the third day after they released, I can tell you chatgpt actually became mainstream overnight.
3.5, not 3
> A lot of AI stuff is like that, actually.
Chain of thought was that way. DeepSeek came out, and not long thereafter everyone had adopted chain of thought.
Unless I am misunderstanding you then I don't think this is correct. Chain of thought was first proposed in 2022. Assuming you mean reasoning models then the correct comparison is GPT-o1, which came out in September 2024, 2 months prior to R1 lite preview. In the next 2 months, prior to the release of the full R1 model, both qwen and gemini released reasoning models. Dario explicitly notes in early 2025 that most major labs had been working on reasoning models in 2024, and explicitly credits OAI for pulling things forward with their release.
https://darioamodei.com/post/on-deepseek-and-export-controls
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
CoT is far older than DeepSeek.
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
I wasn't suggesting they invented it. I was suggesting that they popularized it, made a huge amount of noise about it, and it seemed like after they did, there was a lot more attention on it.
I think if you were paying attention at the time, CoT and ReAct loops were already "well known". Lots of GPT wrappers at the time were doing it, I remember doing CoT to help measure the calories for a recipe. The problem is tokens then were quite expensive and doing experimentation on it was pretty expensive for an individual. I once accidentally spent $20 trying to categorize a Who's Hiring post from here lol.
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
Is this true? I don't remember this and looking back it appears that Google came up with it and openai was the first to release a commercial model.
Dont under estimate the power of branding/positioning/marketing. For more than a year I have been using a RL'd Qwen fine tune optimised for just taking a decision and outputting the decision number, initially I didnt have much training data so after a few manual examples I asked deepseek to make them for me and had codex train it. I compared it to jev when it first came out and it was mostly neck and neck for my use case. I can't have been the only one in fact, I'm sure there must have been other people who probably even open sourced, but I couldn't find anything. I think the question going forward will be how does one keep abreast of interesting stuff happening, now that its de-coupled from direct human labour so de-bottlenecked on production capacity - just bound by how residual time/attention one has (Terence Tao was talking about this exact predicament in the latest tyler cowen podcast).
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
> Dont under estimate the power of branding/positioning/marketing.
You're leaving out the UX (DX, developer experience) in this.
And the better zero-shot performance when you don't have a dataset yet (could argue this falls under DX).
Not to mention the faster time-to-decision and order-of-magnitude cheaper costs.
you should write a blog post about it
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
That's how the original Transformer architecture (LaMDA) should have been released back in the day, but Google thought it would cannibalize their search revenue.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
That also makes me wonder: what other things are achieved by now and are up for grabs, and that will seem obvious in retrospect, but we don't notice it yet?
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
> Which makes me wonder: how do these models calculate probabilities?
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for most uses of decision models.
In some sense everything an LLM does is by unprincipled methods. But it works! We judge it empirically post hoc by its performance rather than relying on tuning.
Hell, human thinking is unprincipled in basically the exact same way.
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
Zero-shot encoders even pre-date LLMs. People who worked in NLP in the transformers "BERT-era" may be familiar with natural language inference (NLI).
Decision models are really not much different than that, but (usually) scaled up to bigger causal models.
If you are looking to do Jev-like things with smaller models, check into NLI.
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Right, that's always been true for LLMs + constrained generation. It ensures syntactic correctness, but semantics are "best effort".
Not reliably - look at microsoft's guidance project that did some of this.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
I'm pretty sure these models also went through RL post-training, the question you ask still needs to be interpreted as an instruction.
> I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
Because it's especially parsimonious with token use.
Why introduce a feature that would cut token use by 1000x?
OpenAI was already working on a Decisions API, and Jev launched theirs quickly enough to have 1 week in the spotlight before OpenAI put theirs in beta (which works great btw).
When trying it out, OpenAI's Decisions was ~2x slower, worse accuracy and more expensive. Unless used with image-related tasks, Jev is clearly superior.
There were classifier models on hugging face for years though…
A couple of weeks back, someone had posted this on HN - an interesting note about the problems with asking LLMs for confidence: https://rumple.github.io/blog/2026/10/20/asking-llm-for-conf...
Thanks for mentioning this. I am the author for that post. There was a follow up on that regarding Jev: https://rumple.github.io/blog/2026/11/04/the-narrow-case-for...
I have still not used a decision model, How are people using them at the moment? is any one using them in a coding harness like Pi?
I've found that they work great as gates ("does this require a human review?") or filters ("is this suspicious?") and occasionally as judges (based on this context, is the expected pain score for this user above 7.5/10?).
Interesting how do you know if it does a good job at these tasks? It seems that thinking in LLM's has been one of the things that has been so important to making sure it's a good response.
It doesn't. Decision models are not more accurate than small models with no thinking. If your classification task is accuracy sensitive, you still need to stick with one of the big frontier models.
If your classification task is accuracy sensitive, you still need to stick with humans
> If your classification task is accuracy sensitive
Which classification tasks aren't?
Support ticket routing to a team. If the model gets it wrong, the team can put it back in the queue for manual routing.
And if the model gets it wrong in the other direction, well, you didn't really need that customer anyway.
When the use case allows room for an acceptable amount of incorrect decisions because there's a manual flow those cases can be routed to still anyways.
My problem is no other tokens means I can’t get reasoning or a justification.
It can be used a fast/cheap router, if a response is above a specified probability then send it to LLM based workflow branch, otherwise nothing more to do
Yep, I use pi-warden (https://github.com/DevMortimer/pi-warden) as part of my workflow.
I used it to classify 10k ish repos for https://hnrepos.com/
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
I adopted it as a super-fast credibility scan on the text (both the online version and the offline decision api inside chrome.
https://chromestatus.com/feature/5155080092385280 https://evergreen-labs.org/products/credibility-analyzer
I'm using them to label training data for model development. You can do something like label a bunch of records with Jev, then train the model, then check results with opus and a hold-out set.
I actually flipped it. Let the LLM label the records and do exploration. But the judge was Jev based. (Did the labeling meet criteria)
That seems an unusual order. I guess with jev you have a confidence number and you can adjust at what confidence you decide it was good or bad.
Selecting between user-provided tags to provide quick suggestions, semantic search, that sort of stuff.
Task classification - given a set of classes, return the most likely class for each task
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
> How are people using them at the moment?
We use it for our critique looping within our own harness.
But decision models can do things like write SQL queries, if you're willing to put a little bit of trickery upfront.
My team's writing a paper about it for ICDE 2027 as we speak (as a follow-up to the DICE demo/paper from VLDB 2014).
But my Linkedin post[1] has a quick summary of the idea, without explaining everything we do there.
[1] - https://lnkd.in/p/gAsWcrCd
Does anyone use it to classify LLM harness tool requests? Between allow, ask, deny.
Here is a recent case study on this: https://news.ycombinator.com/item?id=50436640
Tool routing. For example which MCP or Skill should i use for a given question.
Evals. is this hitting the criteria or not? (Test case passes or fails)
Game modding. I wanted to create a decent opponent AI for w40k on tabletop simulator.
What's the latency?
TTS is turned-based. I didn't notice so much.
I work in manufacturing and would like to use LLMs for scenarios where there's obviously still rules for what should happen next, but there's just so many variables that I can't just code it into a decision tree easily.
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
3) scrapAs I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
Email filtering, ticket triage/categorization/etc.
It would be awesome to be able to create mail filters using natural-ish language instead of grammar
Yes, literally built my own version of /goal in Claude/codex today for pi using jev. My agent was stopping so jev is called each turn to see if it’s done or not - similar to how it works in Claude, according to Claude…
Its bit more expensive than other other Decision Models in the same lane.
However, it can accept Audio and Video as input as well and for those specific cases it can be extremely useful.
Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
I'm trying to organize about 400k photos. What kind of questions are you asking it and what are you doing with that info? Applying tags/labels? Sorting into albums?
Basically just tags, but the questions branch based on the answers. Photo vs Graphic, Portrait vs Landscape, Person/People/No one, etc.
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
We're about at the point where the AI is now pretty insanely cheap. Passing 100K tokens of context then asking hundreds of questions of 100 tokens each would cost about as much as a single Google search API call.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
When I reviewed our internal AI workflows i was shocked at how often a decision API was a good fit or could streamline the process.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
I got the same that gotcha moment
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma4 26A4. That makes me optimistic for this release, despite the high cost.
At 0.10/M this feels very overpriced vs Liquid's 0.02/M, which on my own tests (text input) is very capable.
Wish they provided more comprehensive inputs to the model's capabilities and comparisons.
I wouldn't even trust this is the high level of their pricing with the way they just casually doubled Gemini Omni pricing in 1.2 while trying to spin that it was somehow "cheaper"
Can't LLMs just "degrade" into decision APIs whenever possible/necessary?
Why does this need to be a separate thing?
They generally use smaller, more specialised models which are substantially cheaper for similar performance e.g. https://strandsagents.com/blog/amazon-saves-millions-replaci...
You could instruct them to behave like one but they’ll still eat a lot of tokens, probably more, even though you’ll save a small amount on the output. It’s a separate thing because of significant token savings.
For anyone wanting something local, private, faster, which can also run and train on CPU, checkout https://http://jeffyclassify.com/
It comes with a bunch of pre trained classifiers, and you can use the tooling to train your own
There are also integrations with workflow tools like n8n
There’s also this guide by google to use embedding gemma 2 as a decision maker. Wonder how these compare.
https://developers.google.com/edge/mediapipe/solutions/decis...
Right. I also wonder how it compares to Strands Decisions with E4B or whatever.
Audio, Video and File input? That is very appealing and I can imagine a few very unique ways to leverage this if it works well across all modalities.
For what it's worth, ran my quick and dirty evals with it and got in many usage scenarios a 4x lower brier over Jev (2x - 6x depending on task), which is an immense improvement and the first time a model so significantly has outperformed Jev in my experience. p50 was also over 100ms lower though all this cost twice as much across the entire eval end-to-end.
Dare I say, this is a very impressive release by the Gemini team. While testing GPT-6 Luna lead to an honestly very pleasant interaction with OpenAI staff on here, it did perform very uncompetitively so seeing a great first effort in basic text classification performance with broad multimodality, that is great to see. Course, wouldn't be surprised if Google had such a more broadly trained classifier for years as part of internal usage for platform features which they just now commercialised, but hey, I'll take it.
API docs: https://aistudio.google.com/docs/decisions-api
Anyone hear anything around when it might land in Vertex? As far as I can tell it's just in AI Studio for now.
Surprised how close all these models are on the benchmarks
That would be just a sign that we need better benchmarks
Or that everyone is just copying each other
It's practically the same as Jev, but by Google
Presuming this started when Jev was released... If so, thats pretty quick time to release for Google.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
Standard practice by the big G.
That's always been the enterprise tech way. Using the product that comes with the platform the company is already signed up to pay for is easier than requesting a payment to a brand new service.
They don't need to be the best, they just need to not force a customer to go elsewhere.
Enterprise gonna enterprise.
I suppose the key question is come 5 years time is enterprise going to just become happy to use openrouter, or will they lose competitiveness/ market share to those who do.
Fyi artificialanalysis currently puts drex highest on price vs quality
https://artificialanalysis.ai/evaluations/artificial-analysi...
What got me into Claude Code started with Google Antigravity. Somewhere along the way Antigravity took a dive. The rollout was great, then it started feeling like other G-products, the dreaded tail of lackluster developer energy after a peak.
We have Antigravity at work. It's become a lot better, especially since the Gemini 3.8 Flash and 4 Argon versions. The harnesses themselves (GUI and CLI) are updated quite regularly.
I don't really miss Claude when I'm using it for coding tasks. Also, it doesn't speak Claudish, which is a plus.
Opaque classifier as a service, for when you want to discriminate people by race and social status assumed from name, but want some other entity to be responsible for plausible deniability of the process.
Hey Jev, what mortgage rate multiplier should a person named James Smith get?
This is literally the closest thing for an antidote to that.
Google shipped a "Decision Maker" API a while ago, see https://developers.google.com/edge/mediapipe/solutions/decis...
Apparently the two teams aren't talking to each other.
That is part of media pipe which does local inference. Not that it would be the first time Google shipped products that competed with each other
"should this person get citizen status, gemini give me a quick yes no" man i hate the future
"should we bomb this school? gemini give me a quick yes"
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Because Jules was explicitly an experiment, not guaranteed to "graduate".
Google is maintaining the AI models and APIs for much longer, even if many people don't know about them unless they use them to build things
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
Antigravity is far superior to Jules.
I say plant a thousand flowers and cull the ones that don't bloom.
Barring Search, Gmail, and YouTube I am really wary of using anything from Google now.
The difference is this is not a consumer-focused product.
Google kills consumer products because most consumers don't pay for them directly, so they often end up not making enough money.
With enterprise products, they have actual paying customers, so the situation is different.
So is it over for Jev now that the big boys are in the game
They just got a huge round, almost a billion in financing - that was after OAI Decisions API was released. I really wonder what their next move is.
Take the money and go home.
> I really wonder what their next move is.
bugattis
I guess it depends on how many good ideas Typesafe AI have. They never said that they were only about decision models.
It’s been a little while since I’ve looked, but when the first round of jev competitors came out, jev was still significantly faster.
Getting a response in 250ms opens up a lot possibilities over 750ms.
Startup / Founder focus vs Large Organizations with no vision. With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
"Input blocked: the prompt contains sensitive words that violate Google’s Generative AI Prohibited Use policy"
No I think Jev will do just fine. Not because Google can't even ship this day 1 on their enterprise platform (https://docs.google.com/forms/d/e/1FAIpQLSeupHxao3ajUvl9Dt7l...) but has time for moralizing a decision API.
It's not the spirit of a competitive company.
Looks like Google announced their own Jev inspired API and model
It was a hot topic 2 months ago.
Why’s the probability type called “noul”?
Because they model this on Jev's API instead of OpenAI's version.
It's an abbreviation for a "Bernoulli" random variable: https://en.wikipedia.org/wiki/Bernoulli_distribution
Was my question too. Apparently it is derived from Bernoulli (Can't find the source now).
Jev's founder mentioned the background on the latent space podcast.
https://youtu.be/cFx9Z3ZXca0?t=3340
I still don't really understand where and why these decisions or Jev is useful or interesting? Is there a clear demonstration or tldr anywhere?
Routing, classification, replacement for LLM-as-judge
I still have no idea why this is useful. Models are atrocious without reasoning, for classification and otherwise, and this doesn't even just eschew reasoning afaik, but parts of the models as well for lower latency and cost. I genuinely do not see the point.
What differentiates this from training a classifier model? That it comes pretrained with general knowledge?
I think that is exactly the value prop, you don't need to train a classifier. You get something that works well enough to validate your product/feature/whatever before investing in training.
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
Looking at their benchmarks, it's actually a better model than OpenAI and Jev (which in the same evals is worst than Jev?).
Looking at anyone's benchmarks, they're all better than each other.
Why is that important to you? Seems a bit twitchy.
Mostly curiosity whether openAIs relative speed is more innovative and can pump out new products faster, or if they are playing fast and loose