It is important to consider the consistency of results in a meta-analysis. If the trials are producing wildly different answers, then the pooled result may depend more on the number of apples compared to the number of oranges included in the mix than on any single treatment effect of interest. Heterogeneity would usually be assessed by the chi-squared (c2) test for heterogeneity and the I2 statistic which is based on it.
A common response to heterogeneity in a meta-analysis is to analyse it using the random effects model. This model assumes that heterogeneity comes from real world sources and that the treatment truly does vary in effectiveness in different trials. The heterogeneity is incorporated into the pooled analysis, resulting in slightly different weights (smaller trials being given more weight) and wider confidence intervals, reflecting the additional uncertainty in results.
The following plot is taken from Sandercock, Parmar, Torri & Qian. Firstline treatment for advanced ovarian cancer: paclitaxel, platinum and the evidence. British Journal of Cancer, 2002, 87:815-24
In this meta-analysis there is a very clear disagreement between the trials. Two of the trials suggest a large and clinically important benefit to paclitaxel combined with platinum, whilst the other two suggest that it may be no better than a standard platinum-based treatment. The authors of the meta-analysis proposed that it was not sensible to pool these results. Instead they tested various hypotheses which might explain the disagreement: that there was too much crossover to paclitaxel in some of the trials; that different sorts of patients were included in the trials; that the paclitaxel regimens tested were different; that the control treatments used were different. They concluded that only one of these explanations was consistent with the data, and that this hypothesis had a strong basis in evidence from other trials and meta-analyses. They concluded that the heterogeneity was probably due to the use of a less effective control treatment in two of the trials.
A common technique to explore questions such as this is meta-regression. This uses the random effects model described above in a regression analysis in an attempt to ascertain which features of a trial are influencing the results, and to what extent.
Click the "up" link below to return to the index of the Interpreting meta-analyses section, or click here to move on to the Self-test quiz...