· AI
Notion Says Its AI Router Cuts Costs 90 Percent. Its Own Help Page Says 87.
A vendor's internal benchmark is not evidence until someone outside the vendor can check it, and right now nobody outside Notion can check this one.
Notion announced on October 19 that it rebuilt Auto, the setting that picks which AI model handles a given task, and that the new version delivers "similar results to Opus 5.5 and Astra at about 90 percent lower average cost." Its own help page, describing the same comparison, puts the number at 87 percent. Both pages are live right now, describing the same router against the same baseline model, and they still do not agree with each other. That gap is small. What it tells you is not.
My position: a model router that sends easy work to a cheap model and hard work to an expensive one is a genuinely good idea, and I would not fault Notion for building one. What I will not do is treat a percentage that a company generated by having its own AI judge grade its own router against tasks it selected itself as a number a customer can budget against. That is not evidence. It is a press release with a decimal point.
Here is the steelman, because the engineering case is real. Notion's help page says it tested Auto by replaying thousands of real tasks inside a simulated version of its own product, scoring each model's output against a rubric, converting those scores into Elo ratings, and then training a judge model to approximate that scoring at scale, since grading every comparison by hand would take too long. That is a more careful process than most vendors bother to publish, and Notion has its own reason to want the router to actually work: every token Auto saves a customer is also a token Notion is not paying an upstream model provider to generate. The incentive runs the same direction for both sides of the sale, which is more than you can say for most SaaS cost claims.
But none of that makes the number checkable. No raw task data, no list of what the 20-plus common use cases in the blog post actually were, no breakdown of how the simulated environment compares to how your team actually uses Notion. A model grading a model, with the company that built both the router and the judge choosing which runs to publish, is a closed loop. One independent Notion consultant, writing about the credit system before this update, told readers the opposite of what Notion's marketing tells you now: leave Auto on while the usage is free, but once you are paying, set the model explicitly on every agent instead of trusting Auto to pick well. He also disclosed he sells Notion consulting, so weigh that too, but it is worth noting his practical advice cuts against the vendor's own pitch rather than echoing it.
The part that actually changes your bill is the new infrastructure underneath Auto, not the percentage. Business and Enterprise seats now get 1,500 Standard credits and 500 Select credits a month, and agents that used to time out after five minutes can now run for more than 24 hours on a single task. A five-minute cap put a hard ceiling on how much any one runaway task could cost you. A 24-hour cap, times however many credits your plan allocates, does not. Notion itself says most users are unlikely to exhaust their monthly allotment, which may well be true for average use and is exactly the kind of assurance that precedes the unusual case showing up on an invoice.
None of this is an argument for a regulator to force vendors to publish raw benchmark data, and I would be skeptical of my own case if it turned into one. Disclosure mandates mostly produce compliance paperwork the largest vendors can staff their way through, not numbers a small business can actually use. The fix that works faster is boring and already available to you: run one full billing cycle on the new system before you build next quarter's budget around a vendor's press release. Track what Auto actually routed your work to, what it cost, and whether the output held up, the same way you would check any other vendor's claim before signing a renewal.
If you want a second set of eyes on what your AI vendors are actually charging you for, against what they claim, that is a fair chunk of what a fractional AI consulting engagement does. Get in touch before your next renewal, not after.
Sources
References used in this article. Links also appear alongside the relevant claims.
Let's make it happen.
You don't need a polished brief. A couple of lines about where you want your company to go is plenty, and we'll come back with what we'd tackle first.
We scope the work around your goals and timeline before quoting anything, so you know exactly what you're getting.