Meta analysis in research: meaning, steps and example

Meta analysisnoun

The statistical combination of results from two or more separate studies. This is the definition used in the Cochrane Handbook, the standard guide for health reviews.

The word comes from the statistician Gene Glass. In his presidential address to the American Educational Research Association, published in Educational Researcher in 1976, he described meta analysis as "the analysis of analyses". Each study analyses its own patients; a meta analysis analyses the results of the studies.

The Cochrane Handbook gives three reasons to do one:

• to improve precision, because many small studies together tell you more than any one of them
• to answer questions the single studies did not ask, such as whether an effect holds across different groups of people
• to settle controversies when studies seem to disagree, or to suggest new hypotheses

Shreya's nine trials were a classic case for the first and third reasons. Each was too small to be sure on its own, and together they seemed to disagree.

Students often use the two words as if they meant the same thing. They do not.

The Cochrane Handbook is clear that a systematic review need not contain any meta analysis. The reverse is the problem: a meta analysis built on papers you happened to find is only as good as that set of papers. That is why most guides say a meta analysis should sit inside a systematic review, with a search planned and recorded in advance. Our blog on how to do a systematic review walks through those steps, and types of literature review shows where both fit among other kinds of review.

A meta analysis can only combine numbers that measure the same thing. The Cochrane Handbook says to do one only if the participants, interventions, comparisons and outcomes are judged similar enough, and warns that meta analyses of very diverse studies can be misleading. Combining apples and oranges, it says, can give a meaningless result.

Shreya checked each of her nine trials against four questions:

1. Were the patients similar? All adults with type 2 diabetes.
2. Was the intervention similar? Walking programmes of at least eight weeks.
3. Was the comparison similar? Usual care, with no exercise programme.
4. Was the outcome similar? HbA1c, reported as a percentage.

All nine passed. A tenth study, which mixed walking with a new diet, did not, because nobody could tell which part did the work. She described it in words instead.

Pooling also needs numbers. Interviews and other qualitative studies cannot go into a meta analysis of this kind, though they have their own ways of being synthesised. If you are unsure which kind of evidence you have, our blog on qualitative vs quantitative research explains the difference.

Shreya's meta analysis followed the same steps most guides describe.

1. Fix the question and the protocol.Write down the population, intervention, comparison and outcome, and decide in advance how you will pool the results and which subgroups, if any, you will compare.
2. Search and screen systematically.Search several databases, record every search, and screen with two people working independently. Our guide to searching academic databases shows how to build the search.
3. Extract the numbers from each study.For an outcome measured on a scale, like HbA1c, the Cochrane Handbook asks for the mean, the standard deviation and the number of participants in each group. For a yes or no outcome, such as whether a patient was admitted to hospital, you need the number in each group who did and did not have the event.
4. Choose an effect measure.This is the number that compares the two groups in each study (see the table below).
5. Pool the results and draw a forest plot.Software does the arithmetic. Cochrane's own review tool, RevMan, is widely used for health reviews, and free R packages such as meta and metafor offer more methods.
6. Check heterogeneity, bias and robustness.Ask how much the studies disagree, whether small negative studies may be missing, and whether your answer changes if you make a different reasonable choice.

The result of a meta analysis is usually shown as a forest plot. Once you know its four parts, you can read one in any journal.

• The square for each study sits at that study's result. Its area shows the weight the study gets in the meta analysis.
• The horizontal line through each square is the study's confidence interval: the range of effects that fits its result.
• The vertical line marks no effect, where the two groups did equally well.
• The diamond at the bottom is the pooled result. Its centre is the estimate and its width is the confidence interval.

Why do some squares look bigger? In the common inverse variance method, each study's weight is one divided by the variance of its result. Larger studies have more precise results, so they get more weight. Here are three of Shreya's trials, with the pooled result for just these three. The numbers are made up for this blog.

Trial C on its own seems to say walking does nothing, and Trial A seems to say it does a lot. The pooled diamond sits close to Trial B, the biggest and most precise study, and its whole width lies on the side that favours walking. That is the point of pooling: one confident answer from three uncertain ones.

Studies rarely agree exactly. Variation in the effects found by different studies is called statistical heterogeneity. The statistic most papers report is I², which the Cochrane Handbook describes as the percentage of the variability in effect estimates that is due to heterogeneity rather than chance. It gives this rough guide:

The ranges overlap on purpose. The Handbook warns that such thresholds can mislead, and that I² is very uncertain when there are only a few studies.

Heterogeneity also shapes the choice of model.

When there is no heterogeneity, the two give identical results. Shreya's nine trials gave an I² of 38%, and since the programmes differed in length and intensity, Dr. Verma asked her to report a random effects model, as her protocol had said.

A meta analysis can only add up the studies that reached the world. If small studies with disappointing results were never published, the pooled answer will look better than the truth. Cochrane now calls this non-reporting bias: when decisions about how, when or where results are reported depend on what those results were.

The usual check is a funnel plot, a scatter of each study's result against its size or precision. Big studies sit near the top, close together; small ones spread out at the bottom. Ideally it looks like an upside down funnel. A gap in one bottom corner suggests small studies are missing. Two cautions from the Handbook: tests for funnel plot asymmetry should be used only when there are at least 10 studies, and asymmetry can have causes other than missing studies, so it is not proof of bias. With nine trials, Shreya drew the plot but did not run a test, and said why.

Next comes a sensitivity analysis: repeating the meta analysis with a different reasonable choice to see if the answer holds. Shreya re-ran hers without the two trials with a high risk of bias, and again with a fixed effect model. HbA1c still came down in every version, a little less in one.

Finally, report everything with PRISMA 2020, the Preferred Reporting Items for Systematic reviews and Meta-Analyses, published in the BMJ in 2021. It has a 27 item checklist and a flow diagram that accounts for every record from search to final set. Whether a meta analysis can count as an MD thesis, or a chapter of a PhD thesis, depends on your university, so check with your guide early.

Back at the Thursday clinic, Shreya now has an honest answer: regular walking probably brings HbA1c down a little, and the bigger trials agree more than the small ones suggested. For the chapter that will surround her numbers, she is using our guide on how to write a literature review.