The Systematic Review and Meta-Analysis

So far we have looked at the types of primary study one by one, that is, studies that collect data from patients themselves: the RCT, the experimental study without a true control group, the cohort, the case-control study, the cross-sectional study, and the case report and case series. In this part we turn to a kind of study that does not collect data from patients. It finds the published studies on one question, appraises them and brings them together.

Why one study is not enough

Suppose an RCT has shown that a treatment is effective. Making a decision on the basis of that one study alone is risky, especially if we are going to change the way we treat because of it. One of the reasons is the number of patients. If two treatments differ greatly from each other, the difference can be seen even with a small number of patients. But if their difference is small or moderate, a very large number of patients must enter the study to see it, and a single study usually does not have that many patients. Very large trials (mega trials) with several thousand patients solve this problem, but they have rarely been carried out in dentistry.

The other way is to look at all the studies that have addressed one question together. That is exactly what a systematic review does.

What a systematic review is

A systematic review is a study that finds all the studies relating to a specific question using a rigorous method and brings them together. «Systematic» refers to exactly this rigorous method. The review must be comprehensive, meaning that no relevant study is left out. It must also be reproducible, meaning that if someone else follows the same steps, they arrive at the same studies.

A good systematic review first defines its question precisely. Then, before the search begins, it sets its eligibility criteria. An eligibility criterion is a condition a study must meet in order to be examined in the review. For example, a review might accept only RCTs carried out on adults (a hypothetical example). When these conditions are known in advance, the researcher cannot later set a study aside merely because they do not like its result.

It then runs a complete search based on these same criteria so that all relevant studies are found. In the next stage it appraises each study and assesses two things in it. The first is the risk of bias. Bias, as we saw in the RCT part, is any factor that pulls a study’s result away from the truth. The second is how directly that study’s result relates to the review’s question. For example, if the question is about adults and a study was carried out on children, its result bears on the question only indirectly (a hypothetical example).

Interestingly, the systematic review is itself an observational and retrospective study, because it takes no data from patients and works on studies that have already been carried out. Even so, this rigorous method is what keeps bias out and makes the systematic review a powerful tool for summarising data.

Meta-analysis

Every study ultimately reports its result as a number. Suppose three RCTs have measured the effect of a mouthwash on gingival inflammation. The first saw a large reduction, the second a small reduction, and the third something between the two (a hypothetical example). Now we want to know how much effect this mouthwash has overall.

If these studies have reported their results in a similar way, they can be combined using statistical methods to arrive at a single number. This is called meta-analysis, or a quantitative systematic review. The number obtained from this combination is called the pooled effect estimate, that is, an estimate of the treatment’s effect obtained by putting all the studies together.

When combining results, a meta-analysis pays attention to two things. The first is precision, that is, how far the final number can be relied on. A study with few patients may by chance have obtained a result far higher or lower than the true effect, and the greater the precision, the smaller the chance of such an error. The second is consistency, that is, whether the results of the studies are close to one another or not. In the mouthwash example, if one study saw a large effect and another saw no effect at all, the meta-analysis must take this disagreement into account as well.

So meta-analysis is a statistical step taken after the systematic review, and not every systematic review reaches this step. If combining the results is not appropriate, the systematic review ends without a meta-analysis.

Why the systematic review is valuable

A systematic review that has been well designed and carried out has gathered all the good-quality evidence on one question in one place. Instead of finding, reading and appraising dozens of articles ourselves, this work has been done once with a rigorous method. For this reason, if a systematic review exists for our question, using it is worthwhile. In the hierarchy of evidence, too, the systematic review is regarded as the highest level.

Summary

A systematic review finds, appraises and brings together all the studies on one question using a comprehensive and reproducible method. If the results of these studies can be combined, a meta-analysis turns them into a single number. With this part we have seen every type of study. In the next part we place them side by side on a ladder to see where each one sits in the hierarchy of evidence. And in the final parts of the series we will see that the systematic review’s place at the top of this ladder is not unconditional.