Grade Update of PapersGRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence
Introduction
The extent to which risk of bias associated with missing participant outcome data (hereafter, missing data) reduce confidence in results represents a key issue for all systematic reviews [1], [2]. Currently, the Cochrane Collaboration Handbook [3] focuses on determining whether individual studies are at low or high risk of bias with respect to missing data. When considering whether to rate down for risk of bias across an entire body of evidence, this approach suffers limitations. Assume, for instance, that one sets a threshold of 10% missing data for high risk of bias, and of six studies in a meta-analysis, three have no missing data and three have 12% missing data. How is one to decide whether, across the entire body of evidence, one should—or should not—rate down for risk of bias due to missing participant data?
Sensitivity meta-analyses based on different assumptions can address these issues, particularly if such analyses consider issues beyond simply the frequency of missing data, such as the event rate in the intervention and control groups, the distribution of missing data in intervention and control groups, and the reasons for missingness. The Cochrane Handbook encourages such analyses but, with respect to missing data, does not provide specific guidance regarding how to proceed.
Three prior publications have filled this gap by presenting approaches for systematic reviews of randomized trials to address missing data for binary [4] and continuous outcomes [5], [6]. With some modifications, the GRADE Working Group has endorsed these approaches as GRADE guidance to assess the risk of bias associated with missing data in systematic reviews. In this article, we summarize our modified approaches, providing sufficient detail for their application, and provide several illustrative examples.
We present approaches for three situations: binary outcomes; continuous outcomes in which all studies have used the same instruments; and continuous outcomes in which studies have used different instruments to measure the same construct. In each case, the goal is to make inferences for the entire body of evidence for a particular outcome with respect to risk of bias. Within the GRADE framework, the issue is whether reviewers should rate down certainty in the evidence (quality of evidence, or confidence in evidence) for risk of bias due to missing data.
Section snippets
Development of methods
In developing our approaches, we formed a group consisting of clinical epidemiologists, methodologists, and biostatisticians, all with extensive experience in systematic reviews. We conducted a systematic survey of the literature addressing possible approaches to handling missing data when conducting a meta-analysis [7], [8], [9]. Iterative discussions among the investigators and testing our approaches in a number of systematic reviews completed the process.
The GRADE Working Group reviewed the
Scope and definitions
This guide is for meta-analyses of trial-level data and does not address methods for meta-analyses of individual participant data that may be available to investigators. We deal only with missing data and not other elements of risk of bias in a body of evidence (e.g., allocation concealment, blinding) that systematic review authors must address.
We define participant outcome data as “missing” if they are unavailable to the reviewers; that is, unavailable to investigators of the primary studies,
Common elements of the approaches
We recommend, as do other authors who have written about the issue of missing data in the context of meta-analyses, that systematic review authors' primary analysis include only those for whom data are available (complete case analysis) [7]. An alternative is to use imputation approaches for the primary analysis, an option that is particularly attractive if investigators have strong hypotheses regarding the direction and magnitude of bias associated with missing data. Generating these
Traditional imputations
There are many possible ways to impute missing data in individual primary studies. One might assume that all participants with missing data in either group had events, that no participants with missing data had events, or a worst-case scenario in which all participants with missing data in the intervention group suffered adverse events but none of the participants in the control group suffered such events. That worst-case scenario calculation assumes that the results of the primary analysis are
Binary outcomes—choosing the stringency of the imputations
Investigators using our approaches will need to decide on which extreme a RIMPD/FU they are willing to consider plausible. The choice will be based on factors such as the clinical scenario (e.g., higher value of RIMPD/FU in a trial of cardiac transplant in which participants are more likely to have suffered a bad outcome if lost to follow-up). Another consideration will be the frequency of the event of interest. If it is infrequent (say, 5%), it may be reasonable to assume a maximum RIMPD/FU of
Continuous outcomes—all studies using the same measure
Addressing risk of bias consequent on missing data in systematic reviews addressing continuous outcomes provides additional challenges, including the necessity of imputing both means and standard deviations (SDs). Once again, we suggest the primary meta-analysis used only participants with available outcome data (complete case). When pooled estimates are statistically significant, we suggest sensitivity meta-analyses imputing outcome data that are missing, to challenge the robustness of these
Continuous outcomes—studies using different measures
For certain continuous outcomes and in particular participant-important outcomes focusing on issues such as health-related quality of life (HRQL), clinical trial investigators often choose alternative measures of the same underlying construct. For example, there are at least five instruments available to measure HRQL in participants with chronic obstructive pulmonary disease (COPD) (Chronic Respiratory Questionnaire, Clinical COPD Questionnaire, Pulmonary Functional Status and Dyspnea
Alternative threshold for rating down for risk of bias: the context of health care guidelines
In the discussion thus far, we have suggested an approach to rating down using only one threshold: the 95% confidence interval includes a relative effect of 1.0, or an absolute difference of 0. This threshold corresponds to the P-value including the traditional boundary of 0.05.
This is not the only threshold one might use. Instead, one might choose the smallest effect that patients are likely to consider important and apply the approach to that threshold.
For instance, consider the outcome of
Dealing with limitations in reporting
Systematic review authors will find challenges when authors of primary studies fail to adequately report missing data. [10] For example, trial authors may not clearly report whether they imputed outcomes for participants with missing data. Consequently, a sensitivity analysis making imputations for participants with missing data risks double counting. Elsewhere, we have described in detail the solutions for a number of these challenges [10]. For trials in which authors do not report the
Discussion
We have developed structured and transparent approaches to determine the extent to which missing data across an entire of evidence introduce risk of bias and thus threaten the certainty in the evidence in systematic reviews. Our approaches to binary outcomes, and to continuous data when all studies use the same outcome measure, do not require a high level of statistical sophistication, and can be carried out relatively easily in many statistical programs including RevMan which is the software
Acknowledgments
Authors' contributions: G.H.G., S.E., P.A.-C., B.C.J., R.A.M., S.D.W., and E.A.A. contributed to the conception and design of the study. G.H.G., S.E., P.A.-C., B.C.J., A.G.M., M.B., R.A.M., X.S., S.D.W., D.H.-A., I.N., L.A.K., A.I., J.M., H.J.S., and E.A.A. contributed to the analysis and interpretation of the data. S.D.W. and D.H.-A. contributed to the statistical expertise. G.H.G. contributed to drafting of the article. G.H.G., S.E., P.A.-C., B.C.J., A.G.M., M.B., R.A.M., X.S., S.D.W.,
References (37)
- et al.
GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias)
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 3. Rating the quality of evidence
J Clin Epidemiol
(2011) - et al.
Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers
J Clin Epidemiol
(2013) - et al.
Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers
J Clin Epidemiol
(2014) - et al.
Three challenges described for identifying participants with missing data in trials reports, and potential solutions suggested to systematic reviewers
J Clin Epidemiol
(2016) - et al.
Probiotics and synbiotics for the prevention of postoperative infections following abdominal surgery: a systematic review and meta-analysis of randomized controlled trials
J Hosp Infect
(2016) - et al.
Culprit vessel versus immediate complete revascularization in patients with ST-segment myocardial infarction-a systematic review
Am Heart J
(2015) - et al.
GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes
J Clin Epidemiol
(2013) - et al.
Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers
PLoS One
(2013)
Handling trial participants with missing outcome data when conducting a meta-analysis: a systematic survey of proposed approaches
Syst Rev
Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey
BMJ Open
Reporting missing participant data in randomised trials: systematic survey of the methodological literature and a proposed guide
BMJ Open
Modified intention to treat reporting in randomised controlled trials: systematic review
BMJ
Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study
BMJ
Assessing allocation concealment and blinding in randomised controlled trials: why bother?
Evid Based Nurs
Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature
PLoS One
Imputation methods for missing outcome data in meta-analysis of clinical trials
Clin Trials
Cited by (121)
Allergen immunotherapy for atopic dermatitis: Systematic review and meta-analysis of benefits and harms
2023, Journal of Allergy and Clinical ImmunologyEffects of therapies for Ebola virus disease: a systematic review and network meta-analysis
2022, The Lancet MicrobeThe GRADE Method
2022, Rheumatic Disease Clinics of North AmericaCitation Excerpt :If the potential for bias is severe, raising substantial doubt in the results, confidence in evidence is downgraded by 2 levels. The assessment of bias is one of the most heavily proscribed aspects of GRADE, including advocating for the use of formal bias assessment tools and weighting of outcomes by likelihood of bias.8–12 However, the approach adopted by the ACR during guideline development tends to rely on more qualitative assessments and a less regimented approach to the handling of bias.
Evening versus morning dosing regimen drug therapy for hypertension
2024, Cochrane Database of Systematic Reviews
Conflict of interest: All the authors have completed the ICMJE uniform disclosure form and declare no support from any organization for the submitted work and no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years. They declare being involved in previous publications making recommendations on the topic missing participant outcome data.
Funding: This study is part of a project on addressing missing trial participant data in systematic reviews funded by the Cochrane Collaboration. P.A.-C. was funded by a Miguel Servet research contract from the Instituto de Salud Carlos III (CP16/00137). A.G.M. was funded by a Fellowship in Guidelines Methodology by European Respiratory Society (MTF 2015-01). The funders were not involved in study design and the collection, analysis, and interpretation of data and the writing of the article and the decision to submit it for publication. The researchers are independent from funders and had full access to all the data.