Defeating LLM Nondeterminism A Bite‑Size Guide for Non‑Technical Business Professionals (2025)

Date: 9.15.25

Introduction

Currently, large language models can give various answers to the same question, even with randomness “turned off.” That’s not just a quirk, it makes it hard to debug problems, test improvements, or prove compliance. If you can’t reproduce a result, you can’t reliably fix it, measure it, or audit it.

What Thinking Machines Labs (TML) showed is that this “randomness” isn’t mysterious GPU chaos, but the way servers bundle (batch) requests together under load. By fixing how those batches are handled and making the math path the same every time, TML proved you can get deterministic inference: identical outputs for identical inputs.

Why that’s huge:

• Researchers can run fair A/B tests and know differences are real, not random drift.

• Enterprises can get audit-ready outputs, critical for regulated industries.

• Trainers can finally do true on-policy reinforcement learning, which was nearly impossible before.

• Safety teams can reproduce hallucinations and trace them back to causes.

  
  

In short: it turns LLMs from “flaky but powerful” into something more like a reliable scientific instrument. And that shift unlocks the next stage of serious enterprise adoption and trustworthy AI research.

Part 1 coming soon.

Get Updates Straight to Your Inbox: Contact us 

Citation

He, Horace, and Thinking Machines Lab. “Defeating Nondeterminism in LLM Inference.” Thinking Machines Lab: Connectionism, September 2025, https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ .

Disclaimer:

Content provided by Generative AI Affiliates®, including materials shared via this website, email communications, and all official social media platforms, is for informational purposes only. Some elements, including but not limited to images, may be generated using artificial intelligence. This content does not constitute legal, financial, or professional advice.

Generative AI Affiliates® is platform-agnostic and provides services utilizing both open-source and proprietary AI technologies, tailored to meet the specific needs and preferences of each client.

No client relationship is formed without a formally executed agreement for paid services. Use of this website, affiliated content, or associated platforms implies acceptance of this disclaimer and agreement to our [Policy and TOS].