The Method I Use Myself

The last part ended on a dead end. We tied the model to our own sources and saw that this blocks fabrication, but it doesn’t solve two things: whether the model actually saw everything those sources said, and whether what it saw was understood correctly.

The obvious answer is to go read the paper yourself from the start and check whether the model got it right. But look closer and this isn’t correction at all — you haven’t fixed the model’s work, you’ve redone the whole thing from scratch. And if you were always going to read the paper in full anyway, why did you turn to the model in the first place?

So if we actually want the model to buy us time and make our work easier, the checking process has to be lighter than the work itself. Otherwise we’ve gained nothing.

What follows is a method I built for exactly this problem myself, and have been using for a while. It isn’t a global standard, and I’m not claiming it’s the best way. It’s a method that has worked for my own use, and its logic can be explained — and that’s exactly what I want to open up here, so you can adapt it for yourself if you like it.

Starting Point: Two Different Questions That Always Get Mixed Up

In Chapter 5 we said you can’t judge whether “this statement is true,” but you can very easily check whether “this sentence exists in that paper.” There, we said it as a short piece of advice in passing. Now I want to dwell on it, because this entire method is built on exactly that one distinction.

The first question — is it true — is open and specialized. Its answer needs experience and clinical judgment, and if you already had that judgment you wouldn’t have turned to the model in the first place.

The second question — is this claim present in this text — is closed. Its answer has to be demonstrated in the text itself, and more importantly, it’s work a machine can actually do. Comparing a claim against a text isn’t creative work; it’s tedious, mechanical work, and it’s exactly the kind of task it’s sensible to hand to a model.

So the idea is: I keep the first question — the one that can’t be delegated — for myself. But I shrink its volume with the second question. Instead of asking the model whether a piece of text is true, I put its text in front of the source paper itself.

The Work Itself, Step by Step

Say you’ve read a review on antibiotic prophylaxis before implant surgery and asked the model to summarize it for you. You get back text that’s fluent, well organized, and carries a few specific numbers and recommendations. Now you don’t know which piece actually came from that paper and which piece the model pulled in from somewhere else.

Step one: I go to a second model. Preferably from a different company, and preferably the same source-bound kind we talked about in the last part.

Step two: I put two things in front of it together. First, the original paper itself, and second, the text the first model wrote. Both, not just one.

Step three: I don’t ask it whether the text is good. I say: check this text sentence by sentence against that paper, and for every claim, determine whether it’s in the paper, isn’t in the paper, or is in the paper but not faithfully quoted. And for every one that is in the paper, say exactly where.

And I add one more sentence that I think is the single most important sentence in this entire method: don’t just look for word similarity — check whether the patient population, the intervention, the outcome, the direction of the effect, and the numbers also match the paper.

Without that sentence, the second model goes looking for a similar-sounding sentence — and finds one. You’ll see the example of this in the next section.

The difference between this and “check this” is smaller than it looks, but the result is very different. I’ll say why further down.

What Comes Out

The output usually falls into three categories.

The first: sentences that are in the paper and cited right where they belong. Your work with these is done.

The second: sentences that weren’t in the paper, added by the first model from its own general knowledge. These aren’t necessarily wrong, but they’re no longer documented against that paper, and you need to know which ones they are. If this text is going to be published somewhere or become the basis of a decision, that difference matters.

The third is more dangerous than both, and its shape is this: the claim is in the paper, but it wasn’t faithfully carried over.

The simplest version of this is a dropped qualifier or condition. The number is correct, but it’s not clear which subgroup it belonged to. The recommendation is correct, but the paper said it applies to high-risk patients, and the text in front of you has dropped that qualifier.

The worse version is when the claim itself has changed without the words looking very different. Say the first model’s text reads “antibiotic prophylaxis reduces implant failure,” while the paper actually talked about a reduction in post-surgical infection. The outcome has changed. If the second model is only looking for a similar-sounding sentence, it finds one about antibiotics and reduction and says yes, it’s in the paper. That’s exactly why, in step three, I explicitly said to match the outcome and the direction of the effect separately.

This third category is exactly the one an ordinary read never catches, because the text doesn’t look wrong. It looks perfectly correct. It just isn’t what the paper actually said.

What This Is Actually Doing

Now that you’ve seen what the output looks like, let me say something important that’s easy to miss.

When I compare the model’s text against that paper, I’ve already accepted the paper beforehand. Meaning: before this stage, I’ve already judged that this paper is worth citing. That paper now plays the role of a reference for me — what’s called, in technical terms, a source of truth: the text that is meant to be the standard for this specific task.

That means when I make sure a claim is genuinely in the paper, and its condition and shape match what the paper said, I’m borrowing from the paper’s own credibility. Neither I nor the model built that credibility; it was already there, and I’ve simply attached the claim to it.

And now it’s clear why I added that second qualifier. The third category you saw above is exactly the point where this borrowing fails: the claim is in the paper, but what actually reached you isn’t that same claim. The link is connected, but the credibility hasn’t transferred.

Why I Call It the Sieve

Because that’s exactly what it does: it rejects and lets go whatever claim has no backing, and holds onto the coarser grains on top for you to look at.

You go in with a two-page text where every sentence could potentially have a problem, and you usually come out with a handful of specific spots worth checking by hand. The work of judgment is still yours, and nobody has taken it off your shoulders. It’s just no longer spread across the whole text — it’s concentrated on a few specific points.

And let me be clear about one thing right here, so the name doesn’t mislead you: the sieve doesn’t clean the text. What passes through the sieve isn’t necessarily true; it’s only a claim that has backing in the source. Whether that source itself is any good is a separate discussion, which I’ll get to below.

That’s the entire claim of this method, and I’m deliberately keeping it small. If you ever read that two models can get you to a certain answer, that claim belongs to exactly the kind of false guarantees we’ll talk about in the next chapter.

Why It Works At All

It’s a fair question. If the second model doesn’t understand any better, how is it supposed to catch the first model’s mistake?

Two reasons.

First, what we’ve handed the second model is a different kind of task. Writing a summary is open-ended work; the model has to decide for itself what to include and what to leave out, and wherever something is missing, fill the gap with the most likely guess. But matching a specific claim against a specific text is closed work. There’s much less room to drift, because it has to point to one specific place in the text.

Second, and to my mind more important: the second model wasn’t present in that first conversation, so it isn’t carrying the first model’s guesses and its chain of reasoning along with it. This is exactly what we said about ourselves in Chapter 5: once you’ve gone down a path, it’s hard to turn around and see that same path’s flaw. The same is true for a model — except the fix, for us, is simpler. We just call in someone else.

And that’s exactly why, back in Part 1, I said two free models from two different companies serve you better than one expensive subscription. That statement cashes out right here.

Where It Fails

Three places. And you need to know all three, or this method itself turns into the very thing we were trying to escape.

First, the second model may also have missed something. It uses the same retrieval mechanism we talked about in the last part. It might fail to find the exact passage it needed to compare against a claim and say “not in the paper” when it actually is. So a “not in there” from the model isn’t a verdict, it’s a question mark. Instead of throwing the claim out, go check that one spot in the paper yourself.

Second, models tend toward agreement. If you ask “is this text correct?” the odds it says yes are high. That’s exactly why the question has to be closed and neutral. The difference between “check this” and “for every claim, tell me which section of the paper it came from” is exactly this: the second leaves no room for courtesy. It has to point to a specific place, or admit it couldn’t find one.

Third, and more important than either of the above: this method only measures fidelity to the source, not the correctness of the source itself. If that paper is weak, the sieve will calmly hand you a weak but perfectly documented piece of text. That’s exactly why, in the last part, I talked about judging a paper’s credibility first. That step comes before this one, not after it.

There’s a fourth point too, one that comes up a lot in specialized literature and is worth knowing: when two models agree, that agreement doesn’t necessarily mean confirmation. If both were trained in similar ways, they might both make the same mistake together. Their agreement then is a shared error, not independent evidence. This is another reason I pick the second model from a different company.

How This Relates to What We Said in Chapter 4

In Chapter 4 we had a move where you’d tell the model to just check the situation first and report back, without offering a suggestion yet; then you’d read the report yourself; and then you’d say, now make a recommendation based on this exact report.

That one was manual quality control inside a single conversation, and you were the inspector. This one is cross-verification between two models, against an external source, and it catches something that could have slipped past both of you.

They aren’t rivals, and I use both.

And Next

Up to this point in the chapter, we’ve been talking about what to work with and how to keep errors out. That’s still protecting work that’s already been done.

The next part takes one step further back, to before the work even starts. Because a large share of the bad outputs people get don’t come from the model’s error at all. They come from the fact that the person themselves didn’t quite know what they wanted, and the model started building on its very first guess.