How the Model Thinks, and Why It Sometimes Confidently Lies

The previous chapter showed where and why the model slips. But one hidden piece was left out — exactly the piece that fools you the most: if the model genuinely doesn't know something, why does it sound so confident?

That confident tone has nothing to do with the model having fact-checked anything. There's no place inside it that first checks “is this true?” and then sets its tone accordingly. The fluency of a sentence and the correctness of a sentence both come out of the same process: guessing the most probable continuation. And the confident tone itself is just one of those probable things. The texts the model was built from are full of the confident writing of confident people — from papers and reference books to expert answers — so the model learned that a good answer “looks like this”: assertive, fluent, unhesitating. The model learned the sound of expertise, not expertise itself.

Here, one of our mental habits breaks down. When a colleague says something with confidence, that confidence usually means something; someone who isn't sure hedges, qualifies, says “I think.” For the model, that link is severed at the root. The model can speak with maximum assertiveness about something that doesn't exist. Its tone tells you nothing about whether what it's saying is correct.

And this is exactly why “lying” was the wrong word too. Lying requires two things the model has neither of: an intent to deceive, and a truth it knows and deliberately contradicts. What the model does is closer to that neurological patient who fills the gap in their memory with a coherent, plausible narrative — without intending to lie, or even knowing there was a gap at all. The name for this phenomenon is confabulation, and it describes what the model does far more precisely than “lying”: filling a void with something plausible, unaware that there's a void there at all.

Now, the practical consequence — more important than the discussion itself. Because tone has been severed from correctness, the model is most dangerous exactly where it sounds most confident. Ask the model two things. One inside your everyday domain of work, like a simple restorative protocol; if it gets it wrong, you'll notice immediately, because you yourself are competent there. Now the same model, with the exact same flawless tone, answers about a drug interaction or a figure that you would normally need to check against a source. The tone doesn't waver — it's identical to before. The only thing that's different is that this time, there's nothing at hand for you to check it against immediately. So the model's assertiveness should worry you precisely here, because this time you have no domain knowledge and no ability to tell right from wrong.

One open hook from the previous chapter still remains. There, we said there are tools that force the model to use only the sources you yourself give it (NotebookLM being the example — we'll come back to it properly when we get to content generation), and we asked why even these tools aren't completely immune to error. Now the answer is simple. These tools solve half the problem: when the model is forced to draw from a real document, fabricated references become far less likely, and fabrication is blocked. But reading and summarizing that very document is still the job of the same machine that doesn't understand and only reproduces. So it can quietly present a finding that only held under one specific condition, or only in the short term, as if it were settled and unconditional. The document is real, the link works, and the claim is still wrong.

The root of it is this: understanding an article doesn't come down to just citing the right source. It also takes a human grasp of that work's nuances — the things hidden between the lines that need to be told apart — and that is exactly what the model lacks. In a sense, this is more insidious than a fabricated reference, because you could catch that one with a simple search, while this one slips right past that same trap. Constraining the model to a source blocks fabrication; it does not block misunderstanding.

Two chapters were spent on one job: understanding what artificial intelligence is, and where and why it slips. But knowing where a tool slips isn't the same as working well with it. From here on, the question changes. We're no longer asking “what is this”; we're asking “so how do I talk to it so it slips less, and how do I work so that when it does slip, I notice.” This is where the book moves on from description and into the work itself.