Abstract

Objectives: We sought to empirically evaluate whether the width of confidence interval (CI) of the relative risk (RR) and odds ratio (OR) can obviate the need for calculating the optimal information size (OIS) when making Grading of Recommendations Assessment, Development and Evaluation imprecision judgments. Study Design and Setting: We analyzed a convenience sample of meta-analyses extracted from the Cochrane Database of Systematic Reviews. From each meta-analysis, we calculated OIS based on relative risk reductions (RRRs) of 15%-50% and evaluated the ratio upper to lower 95% CI boundaries of RR (RR CI ratio) and OR (OR CI ratio). We calculated the positive predictive value (PPV) as the probability that a small CI ratio would correctly identify an OIS that was met, and the negative predictive value as the probability that a large CI ratio would correctly identify an OIS that was not met. Results: We analyzed 39,569 meta-analyses (266,877 studies). Analyses across various RRR's demonstrated high NPV's and low PPV's, suggesting that using the CI approach is helpful in identifying when OIS would not be met, but not as helpful in 'ruling in' that OIS would be met. Cutoffs previously proposed for RR CI ratio (>2.5) and OR CI ratio (>3) had very high NPV's (>90%) except when the RRR used to calculate OIS was high (30%-50%) and when the baseline risk or number of included studies increased. Conclusion: This empirical analysis supports using CI ratios of RR and OR to 'rule out' meeting OIS (ie, to rate down the certainty evidence due to imprecision). The CI approach is not helpful to 'rule in' meeting OIS and is not helpful when the plausible effect size > 30% RRR. (c) 2024 Elsevier Inc. All rights are reserved, including those for text and data mining, AI training, and similar technologies.