Abstract
Background: Current evidence appraisal concepts, such as the Assessment, Development and Evaluation (GRADE) approach and Cochrane’s Risk of Bias (RoB) tool, rely on assumptions related to the classic problem of inductive reasoning and may suffer from insufficient inter-rater reliability.
Discussion: The Composite Quality Score (CQS) has emerged as a possible trial appraisal tool that does not rely on inductive assumptions and has been shown to be of potentially very high inter-rater reliability.
Conclusion: Although the current CQS concept is still under development, its current evidence is encouraging and justifies further study. This article presents the rationale and currently available research concerning the CQS and shows where further research is required.
Graphical Abstract
[1]
Anttila S, Persson J, Vareman N, Sahlin NE. Conclusiveness resolves the conflict between quality of evidence and imprecision in GRADE. J Clin Epidemiol 2016; 75: 1-5.
[http://dx.doi.org/10.1016/j.jclinepi.2016.03.019] [PMID: 27063204]
[http://dx.doi.org/10.1016/j.jclinepi.2016.03.019] [PMID: 27063204]
[2]
Langendam M, Carrasco-Labra A, Santesso N, et al. Improving GRADE evidence tables part 2: A systematic survey of explanatory notes shows more guidance is needed. J Clin Epidemiol 2016; 74: 19-27.
[http://dx.doi.org/10.1016/j.jclinepi.2015.12.008] [PMID: 26791431]
[http://dx.doi.org/10.1016/j.jclinepi.2015.12.008] [PMID: 26791431]
[3]
Sterne JAC. Savović J, Page MJ, et al. RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ 2019; 366: l4898.
[http://dx.doi.org/10.1136/bmj.l4898] [PMID: 31462531]
[http://dx.doi.org/10.1136/bmj.l4898] [PMID: 31462531]
[4]
Guyatt GH, Oxman AD, Vist GE, et al. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336(7650): 924-6.
[http://dx.doi.org/10.1136/bmj.39489.470347.AD] [PMID: 18436948]
[http://dx.doi.org/10.1136/bmj.39489.470347.AD] [PMID: 18436948]
[5]
Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011; 64(4): 383-94.
[http://dx.doi.org/10.1016/j.jclinepi.2010.04.026] [PMID: 21195583]
[http://dx.doi.org/10.1016/j.jclinepi.2010.04.026] [PMID: 21195583]
[6]
Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011; 64(4): 401-6.
[http://dx.doi.org/10.1016/j.jclinepi.2010.07.015] [PMID: 21208779]
[http://dx.doi.org/10.1016/j.jclinepi.2010.07.015] [PMID: 21208779]
[7]
Popper K. The two fundamental problems of the theory of knowledge. Taylor & Francis eBooks 2012.
[8]
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1): 159-74.
[http://dx.doi.org/10.2307/2529310] [PMID: 843571]
[http://dx.doi.org/10.2307/2529310] [PMID: 843571]
[9]
Mustafa RA, Santesso N, Brozek J, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol 2013; 66(7): 736-742.e5.
[http://dx.doi.org/10.1016/j.jclinepi.2013.02.004] [PMID: 23623694]
[http://dx.doi.org/10.1016/j.jclinepi.2013.02.004] [PMID: 23623694]
[10]
Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH. Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma. PLoS One 2011; 6(2): e17242.
[http://dx.doi.org/10.1371/journal.pone.0017242] [PMID: 21390219]
[http://dx.doi.org/10.1371/journal.pone.0017242] [PMID: 21390219]
[11]
Hartling L, Hamm MP, Milne A, et al. Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 2013; 66(9): 973-81.
[http://dx.doi.org/10.1016/j.jclinepi.2012.07.005] [PMID: 22981249]
[http://dx.doi.org/10.1016/j.jclinepi.2012.07.005] [PMID: 22981249]
[12]
Armijo-Olivo S, Ospina M, da Costa BR, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One 2014; 9(5): e96920.
[http://dx.doi.org/10.1371/journal.pone.0096920] [PMID: 24824199]
[http://dx.doi.org/10.1371/journal.pone.0096920] [PMID: 24824199]
[13]
Mickenautsch S. Miletić I, Rupf S, Renteria J, Göstemeyer G. The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time. Clin Oral Investig 2021; 25(10): 6015-23.
[http://dx.doi.org/10.1007/s00784-021-04099-w] [PMID: 34379191]
[http://dx.doi.org/10.1007/s00784-021-04099-w] [PMID: 34379191]
[14]
Berger VW. Selection bias and covariate imbalances in randomised clinical trials. Wiley 2005; pp. 1-128.
[http://dx.doi.org/10.1002/0470863641]
[http://dx.doi.org/10.1002/0470863641]
[15]
Mickenautsch S, Fu B, Gudehithlu S, Berger VW. Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: A simulation-based investigation. BMC Med Res Methodol 2014; 14(1): 114.
[http://dx.doi.org/10.1186/1471-2288-14-114] [PMID: 25283963]
[http://dx.doi.org/10.1186/1471-2288-14-114] [PMID: 25283963]
[16]
Odgaard-Jensen J, Vist GE, Timmer A, et al. Randomisation to protect against selection bias in healthcare trials. Cochrane Libr 2011; 2015(4): MR000012.
[http://dx.doi.org/10.1002/14651858.MR000012.pub3] [PMID: 21491415]
[http://dx.doi.org/10.1002/14651858.MR000012.pub3] [PMID: 21491415]
[17]
Papageorgiou SN, Antonoglou GN, Tsiranidou E, Jepsen S, Jäger A. Bias and small-study effects influence treatment effect estimates: A meta-epidemiological study in oral medicine. J Clin Epidemiol 2014; 67(9): 984-92.
[http://dx.doi.org/10.1016/j.jclinepi.2014.04.002] [PMID: 24855929]
[http://dx.doi.org/10.1016/j.jclinepi.2014.04.002] [PMID: 24855929]
[18]
Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: Meta-epidemiological study. BMJ 2013; 346(14): f2304.
[http://dx.doi.org/10.1136/bmj.f2304] [PMID: 23616031]
[http://dx.doi.org/10.1136/bmj.f2304] [PMID: 23616031]
[19]
Mickenautsch S. Are most of the published clinical trial results in restorative dentistry invalid? An empirical investigation. Rev Recent Clin Trials 2020; 15(2): 122-30.
[http://dx.doi.org/10.2174/1574887115666200421110732] [PMID: 32316900]
[http://dx.doi.org/10.2174/1574887115666200421110732] [PMID: 32316900]
[20]
Göstemeyer G, Blunck U, Paris S, Schwendicke F. Design and validity of randomized controlled dental restorative trials. Materials (Basel) 2016; 9(5): 372.
[http://dx.doi.org/10.3390/ma9050372] [PMID: 28773493]
[http://dx.doi.org/10.3390/ma9050372] [PMID: 28773493]
Related Journals
Applied Drug Research, Clinical Trials and Regulatory Affairs
