Grade Update of Papers
GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence

https://doi.org/10.1016/j.jclinepi.2017.05.005Get rights and content

Abstract

Objective

To provide GRADE guidance for assessing risk of bias across an entire body of evidence consequent on missing data for systematic reviews of both binary and continuous outcomes.

Study Design and Setting

Systematic survey of published methodological research, iterative discussions, testing in systematic reviews, and feedback from the GRADE Working Group.

Results

Approaches begin with a primary meta-analysis using a complete case analysis followed by sensitivity meta-analyses imputing, in each study, data for those with missing data, and then pooling across studies. For binary outcomes, we suggest use of “plausible worst case” in which review authors assume that those with missing data in treatment arms have proportionally higher event rates than those followed successfully. For continuous outcomes, imputed mean values come from other studies within the systematic review and the standard deviation (SD) from the median SDs of the control arms of all studies.

Conclusions

If the results of the primary meta-analysis are robust to the most extreme assumptions viewed as plausible, one does not rate down certainty in the evidence for risk of bias due to missing participant outcome data. If the results prove not robust to plausible assumptions, one would rate down certainty in the evidence for risk of bias.

Introduction

The extent to which risk of bias associated with missing participant outcome data (hereafter, missing data) reduce confidence in results represents a key issue for all systematic reviews [1], [2]. Currently, the Cochrane Collaboration Handbook [3] focuses on determining whether individual studies are at low or high risk of bias with respect to missing data. When considering whether to rate down for risk of bias across an entire body of evidence, this approach suffers limitations. Assume, for instance, that one sets a threshold of 10% missing data for high risk of bias, and of six studies in a meta-analysis, three have no missing data and three have 12% missing data. How is one to decide whether, across the entire body of evidence, one should—or should not—rate down for risk of bias due to missing participant data?

Sensitivity meta-analyses based on different assumptions can address these issues, particularly if such analyses consider issues beyond simply the frequency of missing data, such as the event rate in the intervention and control groups, the distribution of missing data in intervention and control groups, and the reasons for missingness. The Cochrane Handbook encourages such analyses but, with respect to missing data, does not provide specific guidance regarding how to proceed.

Three prior publications have filled this gap by presenting approaches for systematic reviews of randomized trials to address missing data for binary [4] and continuous outcomes [5], [6]. With some modifications, the GRADE Working Group has endorsed these approaches as GRADE guidance to assess the risk of bias associated with missing data in systematic reviews. In this article, we summarize our modified approaches, providing sufficient detail for their application, and provide several illustrative examples.

We present approaches for three situations: binary outcomes; continuous outcomes in which all studies have used the same instruments; and continuous outcomes in which studies have used different instruments to measure the same construct. In each case, the goal is to make inferences for the entire body of evidence for a particular outcome with respect to risk of bias. Within the GRADE framework, the issue is whether reviewers should rate down certainty in the evidence (quality of evidence, or confidence in evidence) for risk of bias due to missing data.

Section snippets

Development of methods

In developing our approaches, we formed a group consisting of clinical epidemiologists, methodologists, and biostatisticians, all with extensive experience in systematic reviews. We conducted a systematic survey of the literature addressing possible approaches to handling missing data when conducting a meta-analysis [7], [8], [9]. Iterative discussions among the investigators and testing our approaches in a number of systematic reviews completed the process.

The GRADE Working Group reviewed the

Scope and definitions

This guide is for meta-analyses of trial-level data and does not address methods for meta-analyses of individual participant data that may be available to investigators. We deal only with missing data and not other elements of risk of bias in a body of evidence (e.g., allocation concealment, blinding) that systematic review authors must address.

We define participant outcome data as “missing” if they are unavailable to the reviewers; that is, unavailable to investigators of the primary studies,

Common elements of the approaches

We recommend, as do other authors who have written about the issue of missing data in the context of meta-analyses, that systematic review authors' primary analysis include only those for whom data are available (complete case analysis) [7]. An alternative is to use imputation approaches for the primary analysis, an option that is particularly attractive if investigators have strong hypotheses regarding the direction and magnitude of bias associated with missing data. Generating these

Traditional imputations

There are many possible ways to impute missing data in individual primary studies. One might assume that all participants with missing data in either group had events, that no participants with missing data had events, or a worst-case scenario in which all participants with missing data in the intervention group suffered adverse events but none of the participants in the control group suffered such events. That worst-case scenario calculation assumes that the results of the primary analysis are

Binary outcomes—choosing the stringency of the imputations

Investigators using our approaches will need to decide on which extreme a RIMPD/FU they are willing to consider plausible. The choice will be based on factors such as the clinical scenario (e.g., higher value of RIMPD/FU in a trial of cardiac transplant in which participants are more likely to have suffered a bad outcome if lost to follow-up). Another consideration will be the frequency of the event of interest. If it is infrequent (say, 5%), it may be reasonable to assume a maximum RIMPD/FU of

Continuous outcomes—all studies using the same measure

Addressing risk of bias consequent on missing data in systematic reviews addressing continuous outcomes provides additional challenges, including the necessity of imputing both means and standard deviations (SDs). Once again, we suggest the primary meta-analysis used only participants with available outcome data (complete case). When pooled estimates are statistically significant, we suggest sensitivity meta-analyses imputing outcome data that are missing, to challenge the robustness of these

Continuous outcomes—studies using different measures

For certain continuous outcomes and in particular participant-important outcomes focusing on issues such as health-related quality of life (HRQL), clinical trial investigators often choose alternative measures of the same underlying construct. For example, there are at least five instruments available to measure HRQL in participants with chronic obstructive pulmonary disease (COPD) (Chronic Respiratory Questionnaire, Clinical COPD Questionnaire, Pulmonary Functional Status and Dyspnea

Alternative threshold for rating down for risk of bias: the context of health care guidelines

In the discussion thus far, we have suggested an approach to rating down using only one threshold: the 95% confidence interval includes a relative effect of 1.0, or an absolute difference of 0. This threshold corresponds to the P-value including the traditional boundary of 0.05.

This is not the only threshold one might use. Instead, one might choose the smallest effect that patients are likely to consider important and apply the approach to that threshold.

For instance, consider the outcome of

Dealing with limitations in reporting

Systematic review authors will find challenges when authors of primary studies fail to adequately report missing data. [10] For example, trial authors may not clearly report whether they imputed outcomes for participants with missing data. Consequently, a sensitivity analysis making imputations for participants with missing data risks double counting. Elsewhere, we have described in detail the solutions for a number of these challenges [10]. For trials in which authors do not report the

Discussion

We have developed structured and transparent approaches to determine the extent to which missing data across an entire of evidence introduce risk of bias and thus threaten the certainty in the evidence in systematic reviews. Our approaches to binary outcomes, and to continuous data when all studies use the same outcome measure, do not require a high level of statistical sophistication, and can be carried out relatively easily in many statistical programs including RevMan which is the software

Acknowledgments

Authors' contributions: G.H.G., S.E., P.A.-C., B.C.J., R.A.M., S.D.W., and E.A.A. contributed to the conception and design of the study. G.H.G., S.E., P.A.-C., B.C.J., A.G.M., M.B., R.A.M., X.S., S.D.W., D.H.-A., I.N., L.A.K., A.I., J.M., H.J.S., and E.A.A. contributed to the analysis and interpretation of the data. S.D.W. and D.H.-A. contributed to the statistical expertise. G.H.G. contributed to drafting of the article. G.H.G., S.E., P.A.-C., B.C.J., A.G.M., M.B., R.A.M., X.S., S.D.W.,

References (37)

  • E.A. Akl et al.

    Handling trial participants with missing outcome data when conducting a meta-analysis: a systematic survey of proposed approaches

    Syst Rev

    (2015)
  • E.A. Akl et al.

    Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey

    BMJ Open

    (2015)
  • E.A. Akl et al.

    Reporting missing participant data in randomised trials: systematic survey of the methodological literature and a proposed guide

    BMJ Open

    (2015)
  • I. Abraha et al.

    Modified intention to treat reporting in randomised controlled trials: systematic review

    BMJ

    (2010)
  • I. Abraha et al.

    Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study

    BMJ

    (2015)
  • K.F. Schulz

    Assessing allocation concealment and blinding in randomised controlled trials: why bother?

    Evid Based Nurs

    (2001)
  • M. Alshurafa et al.

    Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature

    PLoS One

    (2012)
  • J.P. Higgins et al.

    Imputation methods for missing outcome data in meta-analysis of clinical trials

    Clin Trials

    (2008)
  • Cited by (121)

    • The GRADE Method

      2022, Rheumatic Disease Clinics of North America
      Citation Excerpt :

      If the potential for bias is severe, raising substantial doubt in the results, confidence in evidence is downgraded by 2 levels. The assessment of bias is one of the most heavily proscribed aspects of GRADE, including advocating for the use of formal bias assessment tools and weighting of outcomes by likelihood of bias.8–12 However, the approach adopted by the ACR during guideline development tends to rely on more qualitative assessments and a less regimented approach to the handling of bias.

    • Evening versus morning dosing regimen drug therapy for hypertension

      2024, Cochrane Database of Systematic Reviews
    View all citing articles on Scopus

    Conflict of interest: All the authors have completed the ICMJE uniform disclosure form and declare no support from any organization for the submitted work and no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years. They declare being involved in previous publications making recommendations on the topic missing participant outcome data.

    Funding: This study is part of a project on addressing missing trial participant data in systematic reviews funded by the Cochrane Collaboration. P.A.-C. was funded by a Miguel Servet research contract from the Instituto de Salud Carlos III (CP16/00137). A.G.M. was funded by a Fellowship in Guidelines Methodology by European Respiratory Society (MTF 2015-01). The funders were not involved in study design and the collection, analysis, and interpretation of data and the writing of the article and the decision to submit it for publication. The researchers are independent from funders and had full access to all the data.

    View full text