NVIDIA's Nemotron 3 Ultra tops open-source models at 1/10 the cost of closed rivals

NVIDIA quietly dropped a benchmark result on Tuesday that reframes the open-source AI conversation. Its Nemotron 3 Ultra model — already known as a strong contender in the Llama-weight-class ecosystem — now holds the highest accuracy score among all open-source models on LangChain’s Deep Agents benchmark, and it does the job at roughly a tenth of what the top closed-source models cost per run.

The Deep Agents benchmark, built by LangChain, isn’t your typical multiple-choice test for LLMs. It evaluates what the industry calls “agentic” capabilities — multi-step reasoning, tool calling, memory access, and executing complex workflows within defined permissions. Think of an AI that doesn’t just answer a question but books a flight, cross-references a calendar, checks inventory, and files a report. That’s Deep Agents territory, and it maps directly to how enterprises actually deploy AI: automating business processes, retrieving information across systems, and coordinating cross-platform tasks.

NVIDIA’s team took Nemotron 3 Ultra and tuned the “system around the model” — the memory layer, tool-use configuration, evaluation pipeline, and model behavior — rather than retraining the model itself. The result: top open-source accuracy on the benchmark, higher throughput, and dramatically lower cost. The company says each inference run on the optimized model costs one-tenth what a leading closed-source model would charge for the same task.

That cost gap matters. In production AI deployments — where a single enterprise agent might run thousands of inferences per day — a 10x cost difference is the line between a prototype and a product. It also validates a thesis that LangChain CEO Harrison Chase has been pushing: that the model itself matters less than the system wrapped around it.

“Building better agents is about continuously improving the system around the model,” Chase said in a statement. “When teams can tune memory, tool use, evaluation, and model behavior together, these factors have a compounding effect.”

The timing is notable. Open-source model quality has been closing the gap with proprietary leaders for over a year, but the conversation has mostly centered on raw benchmark scores — MMLU, HumanEval, GSM8K. Deep Agents tests something closer to real-world utility: can the model actually do things across a business workflow, not just answer correctly? Nemotron 3 Ultra’s result suggests open-weight models are no longer sacrificing capability for cost.

“Our collaboration with NVIDIA shows that enterprises can achieve powerful performance using an open technology stack while maintaining control over the agent systems they’re building,” Chase added.

For companies weighing whether to build on open-source versus paying for API access to closed models, the math just shifted. Nemotron 3 Ultra’s accuracy is competitive with the best closed models — and at single-digit percentages of the operating cost, the “good enough and far cheaper” argument has never been stronger.