The Same Question, Two Different Answers, and Nobody Touched a Setting

Someone on the team reports that the AI feature is broken. You ask them to send the exact input. You paste it in yourself. It works perfectly.

So you do the thing everyone does. You ask them to try again. It works for them too, now. Nobody changed anything. Nobody deployed anything. The setting that is supposed to make the model boring and repeatable was on the whole time, and it was on for both of you.

You close the ticket as "could not reproduce" and feel slightly dishonest about it, because you know something happened and you have no idea what.

Here is what happened. Somewhere in a data center, a few strangers you will never meet sent their own requests at the same moment as yours. That was it. That was the entire difference. Your question got handled in a slightly more crowded room, and a crowded room does the arithmetic in a different order than an empty one.

The knob that promises no surprises

Every AI text tool has a dial that controls how adventurous it is. Crank it up and the model gets loose and inventive. Turn it all the way down to zero and you are telling it: no creativity, no dice rolls, at every step just take the option you rated highest.

Turned to zero, there is no randomness left in the decision. The model scores every possible next word, picks the top one, and repeats. Same question in, same scores, same top pick, same answer out. That is the promise, and it is the whole reason the setting exists. People build on top of it. Automated tests that check the model still says the right thing. Caches that assume the same input gives the same output. Bug reports that assume "here's my input" is enough to reproduce anything.

The promise does not hold. Ask the same question a thousand times, with that dial pinned at zero, and you do not get a thousand copies of one answer. You get a pile of answers that mostly agree and quietly refuse to be identical.

And the reason has nothing to do with creativity. The model is not making a choice. It is being handed slightly different numbers to choose from.

Computers round, and rounding does not wait its turn

Time for the boring truth underneath all of this, because everything else follows from it.

Computers cannot store most numbers exactly. There is a fixed amount of room for each one, so anything that does not fit gets rounded off, every single time, in every single step. Think of a till that only tracks whole cents. Not a bug, just the size of the box.

Now watch what rounding does when you add things up in a different order.

You are adding one enormous number and a long tail of tiny fractions of a cent. Start with the enormous number and add the fractions to it one at a time, and each fraction is too small to move a total that big. It gets rounded away. It contributes nothing. Do it the other way, add all the little fractions to each other first, and they pile up into a whole cent that is big enough to survive, and then you add that to the enormous number and it lands one cent higher.

Same numbers. Same addition. Different order. Different total.

This is not a rare edge case that shows up once in a million. It is the normal behavior of every computer doing decimal arithmetic, and it is the reason a spreadsheet occasionally disagrees with itself by a rounding error at the bottom of a column. Usually the gap is so small nobody notices or cares.

An AI answering your question does something in the range of trillions of these additions. The gap stops being invisible.

The explanation everybody gave, and why it was wrong

For years the standard answer to "why isn't this repeatable" was a confident shrug wearing a lab coat. Computers round, lots of things run at once on the graphics chips these days, so the additions land in an unpredictable order and you get a bit of harmless drift. Sorry, physics.

It sounds right. It was mostly wrong, and in September 2025 a team at Thinking Machines Lab published a piece called "Defeating Nondeterminism in LLM Inference" that took the folk explanation apart.

They went and checked the thing nobody had bothered to check. They took the individual chunks of math the model runs, fed them the exact same inputs, and ran them over and over on the same hardware. If the shrug were true, those chunks should have wobbled. They did not. Run the same piece of math twice on the same machine with the same inputs and you get the same result, down to the last digit, every time. The pieces were repeatable all along.

So the randomness was not coming from inside the math. It was coming from something outside it, something the math never sees and you definitely cannot see.

It is not you. It is the queue.

Your request does not get a computer to itself. That would be a catastrophic way to run a service. Instead the server collects the requests arriving right now, from you and from whoever else happens to be awake, and processes the whole armful together in one pass. Sharing the ride is what makes it affordable.

Here is the part that ruins everything. The number of people in that armful changes constantly, second to second, depending on who is using the thing. And the machine adapts to it. Handed four requests, it splits the work into a certain number of chunks and adds them up in a certain arrangement. Handed a hundred and thirty, it chooses a different arrangement, because a different arrangement is faster at that size. Perfectly sensible engineering, invisible from the outside, and it changes the order the additions happen in.

You already know what a different order does. It rounds differently. Fractions of a cent that got swallowed in the quiet arrangement survive in the busy one.

Which means the answer you get depends on how many strangers were in the room with you.

Worth saying plainly, because it sounds worse than it is: nobody's data leaks into your answer. The other requests do not touch your text and the model never sees them as words. Their only effect is on how the machine grouped the arithmetic, the way a bill split three ways rounds differently than the same bill split seven ways. Your numbers, their crowd, a different rounding.

You cannot control this. There is no header you can send. From your side, the only signal that anything is different is that you asked at 2 in the afternoon instead of 4 in the morning.

One word slips, and the sentence follows it somewhere else

Fine, you might think. A rounding error at the twelfth decimal place. Who cares.

Here is why you care.

The model writes one word at a time, and each word is chosen by scoring every option and taking the winner. Most of the time the winner wins by a mile and a microscopic wobble in the scores changes nothing. But somewhere in a long answer there is a moment where two options are almost exactly tied. "The system will" versus "The system can." A hair apart. The rounding difference is bigger than the gap.

The tie flips. And now the model is not writing a slightly different word, it is writing the next word after a different word. Everything downstream is a response to that. One flipped coin at position 103 and the two answers walk away from each other and never come back, not because the model changed its mind but because it is now finishing a different sentence.

The Thinking Machines team measured exactly this. A thousand runs of the same prompt, dial at zero, on a large open model. They got 80 different completions. The most popular version showed up 78 times out of a thousand. And the answers were word-for-word identical for the first stretch, byte for byte, until the first near-tie landed on the wrong side of a rounding error and they scattered.

That is the shape of the whole problem. Not noise sprinkled evenly through the output. A single point of divergence, then two completely different paragraphs.

It is fixable, and it costs you

The good news is that this is engineering, not physics, and the same team fixed it.

The fix is unglamorous and exactly what you would guess once you see the cause. Make the math take the same route regardless of how crowded the room is. Force the machine to split the work into a fixed arrangement instead of picking a clever one based on the current load. Same order every time, same rounding every time, same answer every time. They rewrote the handful of operations that mattered and released the code. A thousand runs afterward gave a thousand identical answers.

The bill arrives immediately. Locking the arrangement means giving up the fast path the machine picks when it has a big crowd to work with. In their test, a batch that took 26 seconds normally took 55 seconds with the deterministic version bolted on, and around 42 after they tuned the slowest piece. Somewhere between "noticeably slower" and "twice the cost," which is a real number on a real invoice when you are serving millions of requests.

So it is a choice, not a law, and it is not your choice. It belongs to whoever runs the servers. Most providers, most of the time, will take the speed. Repeatability is a feature you have to be offered.

What to actually do with this

Stop debugging the wrong thing. If your AI feature is intermittently producing a different answer to an identical input, the first suspect is not your prompt, not your temperature setting, and not a haunted cache. It is the shared queue, and no amount of staring at your code will show it to you.

That reframes a few things you have probably been fighting. An automated test that compares the model's answer to one exact expected string is not a flaky test, it is a test built on a promise the platform never actually made. Check for the properties that matter instead: is it valid JSON, does it contain the field, does it pick the right category. A bug report that says "same input, different output" is not a user misremembering. And "works on my machine" now has a genuinely new meaning, because the machine is the same one, and what changed was the strangers standing next to you on it.

The uncomfortable version, if you build on top of these things: the same input can give a different output, forever, unless someone upstream decides to pay for it not to. You can design around it. You cannot wish it away.

The model was never being creative when it did this. It was being crowded.