When the Model Builds Something That Doesn't Exist

In the previous part we said that a language model can build something that has no external existence at all, and that this error has a name of its own. Now let's go and see it, face to face.

Imagine you are preparing a review article for a journal club and you ask the model to give you a few credible sources on the success rate of regenerative pulp treatment in immature teeth. The model answers without a pause. A clean list: the authors' names, the article title, the journal name, the year, the volume number, even the page numbers. Everything is exactly the shape a real reference should have. So tidy and professional that you have no reason to doubt it.

But when you go to find those articles, one or two of them don't exist at all. Not that the link is broken or unavailable; that article was never written. The authors may be real, the journal is real too, but this specific article with this title is entirely fabricated. The model made it up, with the same confidence it had given the other, real references.

This phenomenon is called hallucination (Hallucination), or in Persian, the AI's illusion. The name is a little misleading, because in humans “hallucination” means seeing something that isn't there, as if it were an abnormal, broken state. But for the model this is not a broken state at all; it is its everyday work. When the model builds that fake reference, it has not suffered a temporary glitch. It is doing exactly what it always does, only this time the result doesn't line up with reality.

And its most dangerous feature is right here: there is no difference in the appearance of the work. When the model gives a real reference and when it builds a fake one, both carry the same tone, the same confidence, the same shape. The model itself does not flag “I'm sure of this one and I made that one up.” For the model the two are no different. It is your job to tell them apart.

And hallucination isn't only references. It can be a drug dose that differs slightly from reality, a statistic that has no source at all, a clinical protocol that sounds reasonable but is recommended nowhere, or attributing a claim to a guideline that the guideline never made. Anywhere the model can place word beside word and build something that has the right shape, hallucination is possible.

Now you might ask: well, why? Why should a tool that seems this intelligent make something up instead of simply saying “I don't know”? That is exactly the question whose answer illuminates the whole logic of how the model works, and we will get to it — but not right now. That is the subject of the next chapter. For now it is enough to know that this happens, and what it looks like.

One thing, though, is clear from right now: if this tool is sometimes going to build, with complete confidence, something that doesn't exist, then you can't lean on it blindly. You have to work with it in a way that catches this error. And that brings us to the final point of this chapter.