A systematic review requires the identification of the relevant studies, the determination how each study was conducted and under what precise circumstances, the collection of the associated data, the evaluation of the study quality, etc. Unless the information is unbiased and the statistical methods are sound, the numerical summaries from a meta-analysis may be misleading and no benefits may accrue from combining the studies.
The single most important reason for failures in meta-analyses is bias. Publication, selection and other study-related biases are rampant. Sequences of studies are usually done with differing standards, different definitions and methods, different populations, and so on. Does it make sense to estimate an overall effect via a meta-analysis? Sometimes, covariates on the study-level (meta-regression) may help in explaining the differences at least partially. In other instances it might be better to refrain from attempting a combination. Randomized controlled trials are considered the gold standard in medicine and health sciences, and restriction of meta-analyses to synthesis of their findings is strongly supported by the influential Cochrane Collaboration. The main concern in meta-analysis of observational studies is the existence of various biases due to non-random allocation.
Difficulties in publishing papers with insignificant findings, for example, will result in publication bias, which exaggerates the true effect. The history of science provides many examples, in which statistically significant effects were identified, published and even celebrated, only to be refuted later on. It seems all too easy to arrive at such wrong conclusions, particularly if the effect is believed to be real by the scientific community.
The recent adoption of the Clinical Trials Regulation by the European Parliament will help in reducing publication and selection biases in future meta-analyses, since all clinical trials must be pre-registered and a summary of their results published no matter the outcome. The analogous law was passed by the US congress in 1997 and the website clinicaltrials.gov since 2008 also includes results of the trials. Similar initiatives exist in some European countries, but up to now without the legal underpinnings.
As meta-analysis is more widely used, statistical issues become more visible. Statisticians have contributed to the methodology, but much more needs to be done. Many estimators, for example, are biased when applied to finite samples. Combining even a very large number of finite sample estimates does not make this bias disappear. On the contrary, its relative importance grows, because the variance decreases towards zero by combining studies.
The second order effects, such as variances or lengths of confidence intervals, are of lesser importance, but nevertheless deserve attention. One should make sure that all sources of variation are accounted for when computing the variance of an estimated effect. The resulting confidence intervals will be wider, but also more realistic. It clearly would also be desirable to include potential biases when calculating confidence intervals. But this requires a rethink of our current theories of testing and inference.
In view of the unmet needs in the area of bias and variance, it seems almost unnecessary to talk about the higher order effects due to deviations from Gaussianity. Clearly, however, more accurate models will lead to more accurate inferences and it presumably would be useful for meta-analysis to use more sophisticated models.
Currently, multivariate meta-analysis and meta-regression, sequential methods and network meta-analysis are hot topics. The most important problem when confronted with multivariate effects is the lack of information about correlations of the components. This demands the development of methods based on robust estimators or simplified correlation structures. Sequential meta-analysis has a degree of similarity with group-sequential trials, but the overall design is missing. The extension to random effects is currently being developed. Network meta-analysis attempts to compare treatments, which may never have been compared directly in any trial. This appears to be a brave and complex undertaking, requiring the existence of an accurate model. A detailed review of statistical problems in meta-analysis can be found in our paper published in International Statistical Reviews.
Meta-analysis can also be seen as a methodology for combining information from heterogeneous sources, which present the data at differing levels of aggregation and can vary greatly in quality. As such it is an appropriate tool to tackle problems posed by the analysis of big data. This is another reason for the continued interest in meta-analysis.