· AI
Nvidia's New Open Model Benchmarks Badly. That Is Not the Point.
A weak benchmark score is sometimes the most honest thing a model can show you.
Nvidia released two new open-weight models on October 15: Nemotron 3.5 Super VL, a 120 billion parameter vision-language model with 12 billion active per token, and Nemotron 3.5 Bolt, a 74 billion parameter model with 9 billion active, compressed to run on a single 24 gigabyte GPU (LetsDataScience). Artificial Analysis scored Super VL at 26.5 on its Intelligence Index. Bolt scored 21.4, the weaker of the two by a wide margin, and well behind open flagships like GLM-5.3 and Kimi K3, which scored 60 apiece in an August evaluation (Unite.AI).
My take: the low score is the honest part, not the embarrassing part. Bolt was never built to win a leaderboard. It was built to run an agent on hardware a business already owns, and for that job the number that matters is not the index score. It is whether the thing fits in memory at all.
Do the math on what "fits" means here. A 74 billion parameter model trained at the usual 16-bit precision needs roughly 148 gigabytes just to hold the weights, the kind of footprint that puts it in a data center, not on a desk. Nvidia quantized Bolt natively to about 2 bits per parameter, which brings the weight footprint down to roughly 18 to 19 gigabytes, close enough to the claimed 24 gigabyte card that the rest goes to context and overhead (LetsDataScience; Interesting Engineering). That is an eightfold compression, and Nvidia reports it still ranks first among open models in its size class on PinchBench and second on BrowseComp, two benchmarks built around actual agent tasks rather than general knowledge (Interesting Engineering). Both models ship with open weights, open training data, and open training recipes, and Bolt already runs through Ollama, LM Studio, and llama.cpp.
Now the steelman, because the skeptical read deserves a real hearing. Alexander Harrowell, an analyst at Omdia, told AIBusiness that Nvidia's open-source generosity "is not entirely pure." His argument: Nvidia sells GPUs, so a steady stream of models tuned to run well on Nvidia hardware is a business strategy, not charity. He also points out that open-weight models get replaced quickly, and that the real bottleneck for most companies is not model availability but whether their own infrastructure is ready to run packaged software locally at all (AIBusiness). He is right on both counts. Nvidia's self-interest here is not hidden, and a model that looks obsolete in six months is a real cost if you built a product around it.
But self-interest and a genuine deal to a customer are not opposites. That is the whole case for competition: Nvidia does not need to love its customers to hand them something useful, it needs a reason to compete for them, and open-sourcing a model that runs on hardware the buyer already owns is a sharper competitive move than keeping it locked behind an API meter. The same dynamic that cut inference prices across the industry this year, cheap Chinese open-weight models forcing everyone else to answer, is now showing up as compression engineering instead of just price cuts. Whether Bolt specifically survives six months is a real risk, and it is the buyer's risk to weigh, not Nvidia's promise to keep. Early adopters are already testing that bet: Scale AI built credit and due-diligence agents on the Nemotron 3.5 family for Skydance and EY, and Mercor post-trained it for coding benchmarks, so this is not a model running only in demo videos (Interesting Engineering).
The practical takeaway for a business owner: stop ranking open models by a single leaderboard number and start asking what job the model has to do. If the job is frontier reasoning on open problems, Bolt's 21.4 is a real weakness and GLM-5.3 or Kimi K3 are the better pick, cloud API and all. If the job is a bounded agent, coding help, document triage, research summarization, running on a machine you control, a model that trades leaderboard rank for a 24 gigabyte footprint and zero marginal cost per token is doing exactly what it is supposed to do. Matching the model to the job instead of the job to the biggest model you can afford is most of what a fractional AI advisor gets hired to sort out. If you are staring at a vendor quote and wondering whether you are buying more model than the job needs, that is a conversation worth having before you sign it. Get in touch.
Sources
References used in this article. Links also appear alongside the relevant claims.
Let's make it happen.
You don't need a polished brief. A couple of lines about where you want your company to go is plenty, and we'll come back with what we'd tackle first.
We scope the work around your goals and timeline before quoting anything, so you know exactly what you're getting.