http://orcid.org/0000-0002-4904-1394
Figures
Abstract
Background
The Consolidated Standards of Reporting Trials (CONSORT) statement aims to improve clarity and consistency of transparency of reporting in Randomized Controlled Trials (RCTs). The Cochrane Risk of Bias (RoB) tool for RCTs helps authors to judge the RoB. as ‘‘low”, “high” or “unclear”.
Objective
In this study we aimed to assess whether the implementation and updates of the CONSORT statement influenced the trend of “unclear” RoB scores of RCTs included in Cochrane systematic reviews.
Methods
All Cochrane reviews published in December to October 2016 were retrieved. The publication year of RCTS included in the reviews were sorted into time frames (≤1995, 1996–2000, 2001–2009 and ≥2010) based on the release- and updates of the CONSORT statement (1996, 2001 and 2010). The association between “unclear” RoB versus “low or high” RoB and the year of publication in different time frames were calculated using a binary logistic regression.
Results
Data was extracted from 64 Cochrane reviews, with 989 RCTS (6471 items). The logistic regression showed that the odds of RCTs published ≥2010, compared to ≤1995 were more likely not to report an “unclear” RoB for the total data (Odds Ratio (OR) 0.69 (95% Confidence interval: 0.59–0.80)), random sequence generation (OR 0.32 (0.22–0.47), allocation concealment (0.64 (0.43–0.95)) and incomplete outcome data (OR 0.60 (0.39–0.91)).
Citation: Rademaker MM, Ramakers GGJ, Smit AL, Hooft L, Stegeman I (2020) The effect of the CONSORT statement on the amount of “unclear” Risk of Bias reporting in Cochrane Systematic Reviews. PLoS ONE 15(7): e0235535. https://doi.org/10.1371/journal.pone.0235535
Editor: Vance Berger, National Cancer Institute, UNITED STATES
Received: November 12, 2019; Accepted: June 18, 2020; Published: July 10, 2020
Copyright: © 2020 Rademaker et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
There is increasing concern about the accuracy of the outcomes of biomedical studies and the translatability in clinical care [1]. Inadequate reporting of results leads to imperfect healthcare decisions, that may lead to a waste of health care funds and potential harm to patients [1,2]. The absence of relevant information about the methods of the study, or information about the intervention, and the study results makes replication of studies and implementation in clinical care difficult [3]. In 2014 the Lancet published a series of five articles called ‘Increasing value, reducing waste’ [4–8]. This series identified five stages of avoidable waste or inefficiency in biomedical research. One of them is the writing of unbiased and usable research reports [8].
In an effort to improve the quality of research reports, reporting guidelines have been developed. The Consolidated Standards of Reporting Trials (CONSORT) statement (www.consort-statement.org) was developed in 1996 for randomized controlled trials and was updated in 2001 and 2010 [9–12]. Nowadays, many biomedical journals endorse the CONSORT statement [13]. However, Turner et al. showed in 2012 that there are still many flaws in the adherence [14]. For example, where CONSORT provides a flowchart for RCTs, only 263 out of 469 (56%) investigated randomized controlled trials (RCTs) included such a diagram in 2009 [15]. Even the five highest impact general medicine journals show variable and incomplete adherence to the CONSORT statement for abstracts [16].
So far it is unclear whether the publication and updates of the CONSORT statement affected the overall quality and completeness of reporting of RCTs. Standardized quality assessments have been developed to create transparency in the reporting of bias. For randomized controlled trials the most well know is the Cochrane Risk of Bias (RoB) tool introduced in 2008 and updated in 2011 [17]. By this tool, selection bias, performance bias, detection bias, attrition bias and reporting bias are assessed by the judgement of seven items (random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting and other bias) as “low”, “high” or “unclear” risk of bias [17].
In this study we aim to investigate the effect of the publication and updates of the CONSORT statement on the amount of “unclear” RoB scores of RCTs included in Cochrane systematic reviews. We hypothesize that the CONSORT statement improved completeness of reporting by lowering the amount of “unclear” RoB reporting in systematic reviews.
Methods
Selection of Cochrane reviews
We retrieved all therapeutic Cochrane reviews published in December, November and October 2016. We did not make any limitations for topic.
Data extraction
Data about the risk of bias assessment of all RCTs of included Cochrane systematic reviews were extracted by G.R. The following information from the Cochrane reviews was extracted; the names and year of publication of the original RCT; the RoB assessments (“low”, “high” or “unclear”) as judged by the Cochrane review authors. 10% of included reviews was randomly selected by Research Randomizer and accuracy of extracted data by the first screener out of these reviews was randomly checked by M.R [18]. To adhere to the most recent Cochrane handbook (version 5.1) the judgements on the seven key items were extracted (sequence generation, allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting and other bias). All included Cochrane systematic reviews used the Cochrane RoB tool. However, some reviews did not include all seven key items or used sub-categories for certain types of bias. Data on sub categories was not assessed.
Data analysis
The frequency of low, unclear and high RoB rating was calculated per item. The year of publication of the individual RCT was related to the publication (1996) and updates (2001, 2010) of the CONSORT statement whereby four time frames were constructed: ≤1995, 1996–2000, 2001–2009 and ≥2010. The proportion of RoB-items scored as unclear, low or high were calculated per time frame, in total and per item. A binary logistic regression analysis was performed. The outcome was RoB assessment “unclear” versus “low and high” RoB scores, per RoB item and in total. The timeframe ≤1995 was the reference timeframe. An odds ratio was deducted (OR, 95% confidence interval (CI)). A binary logistic regression analysis was also performed with the same outcome (RoB “unclear) with time as a continuous variable.
The statistical analyses were performed using SPSS v25.
Results
Selection and general characteristics
The search retrieved 75 systematic reviews from the Cochrane Library. 11 were so called ‘‘empty reviews” and were excluded. The remaining 64 therapeutic Cochrane systematic reviews included a total of 1008 RCTs. Of these, 19 RCTs were excluded because of several reasons; protocols (n = 5) and no clear publication year (n = 14). Therefore, 989 RCTs (1 double, 988 unique RCTs remaining) were included for further analysis, including a mean of 15.45(range 1–64) RCTs per review. Fig 1 shows the inclusion process. 10% of the reviews were randomly checked for accuracy of data extraction (n = 7) which encompassed 145 RCTs and 1015 RoB assessments. 19 of 1015 RoB assessments were missings.
Of the 989 included RCTs, 153 RCTs were published in or before 1995 (16%), 94 RCTs in 1996–2000 (10%), 361 RCTs from 2000 to 2009 (37%), and 381 RCTs from 2010 till 2016 (39%). The median publication year was 2008 (interquartile range 12, with a minimum 1967 and maximum of 2016).
RoB scores of included RCTs
Of the 989 RCTs, random sequence generation was assessed 989 times (100%), allocation concealment 989 times (100%), blinding of participants and personnel 957 times (97%), blinding of outcome assessment 798 times (81%), incomplete outcome data 989 times (100%), selective reporting 955 times (97%) and other bias 794 times (80%) by the authors of the Cochrane systematic reviews. In total 6471 items were assessed, the median number of items per RCT was 7 (IQR 1, minimum 4 –maximum 7).
Assessment of “unclear” RoB reporting
A total of 39% of all items were reported as “unclear”, with a maximum of 61% for allocation concealment. The proportion was lower for blinding of participants and personnel (24%) and incomplete outcome data (25%). (Table 1) The reporting of “unclear” RoB over time is illustrated in Table 1 and Fig 2.
RS = random sequence generation, AC = allocation concealment, BPP = blinding of participants and personnel, BOA = blinding of outcome assessment, IOD = incomplete outcome data, SR = selective reporting, OB = other bias.
Time
Binary logistic regression shows no statistical significant differences in the total percentage of unclear RoB scores of RCTs between ≤1995, and both 1996–2000 and 2001–2009. However, RCTs in the time frame ≥ 2010, the odds were significantly more likely not to score an “unclear” RoB (OR 0.69 (95%CI 0.59–0.80)) (Table 2)
The odds were significantly more likely not to score an “unclear” RoB in RCTs that were published in the two latest time frames, for the item random sequence generation (2001–2009: OR 0.41 (95%CI 0.28–0.61) and ≥ 2010: OR 0.32 (95%CI 0.22–0.47)). For RCTs published in time frame ≥ 2010, the odds were significantly more likely not to score an “unclear” Rob for allocation concealment (OR 0.64 (95%CI 0.43–0.95)) and incomplete outcome data (OR 0.60 (95%CI 0.39–0.91)).
With time as a continuous variable, the odds were significantly more likely not to score an unclear risk of bias for all RoB items together (total) (OR 0.987 (0.0981–0.992). (Table 3) The same is applicable for random sequence generation (OR: 0.959 (95% CI 0.945–0.973), allocation concealment (OR 0.985 (95% CI 0.985 (0.970–0.999) and incomplete outcome data (0.982 (95% CI 0.967–0.997).
Discussion
In this study we provided an overview of the “unclear” RoB reporting of RCTs published between 1967 and 2016 and included in a therapeutic Cochrane systematic review published in October till December 2016. By creating time frames based on the release and updates of the CONSORT statement (1996, 2001, and 2010) the development of “unclear” RoB reporting was calculated over time. In general, a slight decrease of “unclear” RoB reporting over time was found. Over all items the odds were significantly more likely to not score an “unclear” RoB ≥ 2010, compared to ≤ 1995. For the individual item random sequence generation, a key factor in creating RCTs, the odds were significantly more likely to not score an “unclear” RoB in two time frames (2001–2009, ≥2010). The same applies in the last time frame (≥ 2010) for allocation concealment and incomplete outcome data.
Interestingly, the three items that show significant lower odds of “unclear” RoB ≥2010 compared to <1995 (random sequence generation, allocation concealment and selective reporting), also show the highest three percentages of unclear reporting over all time frames. Allocation concealment, blinding and method of randomisation might modify effect estimates in RCT’s [19].
The outcomes of our study are in line with earlier studies in other research domains. By our own research group the adherence to the CONSORT statement in otorhinolaryngological literature was studied in 2015, demonstrating that the quality of RCTs published in ENT journals was suboptimal [20]. Similar outcomes have been found for publications in diabetes journals, restorative dentistry and anaesthesiology journals [21–23].
Considering the data on time evolution, similarities can be found with Dechartres et al., Reveiz et al. and Peters et al.; a decrease have been demonstrated in “unclear” risk of bias over time, especially in random sequence generation and allocation concealment, and an overall improvement in the reporting quality was found over time [9,24,25]. No comparisons were made with the CONSORT statement. It appears from our study that the first steps have been taken to increase value and reduce waste, especially in reducing waste from incomplete or unusable reports of biomedical research.
There is uncertainty in the term and meaning of ‘‘unclear” RoB. One could argue that unclearness of the existence of a risk, implicates a risk. The missing of information in clinical trials could raise a dilemma in the form of two questions to the risk assessors: 1) are we to give them the benefit of the doubt or 2) were the authors strategic in their decision on what items not to describe? [26] Still, it turns out that a large amount of studies with missing information about blinding had in fact performed the blinding [26]. One could argue the suitability to be more conservative with judging missing information [26,27].
There are concerns about the completeness and correctness of the CONSORT statement regarding trial reporting. It is debated that CONSORT does not include all items to properly assess trial quality, such as the lack of a randomization log, the lack of precision in the intention to treat (ITT) description and the lack of adequate handling of missing data [28–31]. In this paper we have used the CONSORT statement because it is a commonly used tool, that was adopted by many journals [13,26]. A lower quality of reporting, (e.g. higher amount of unclear risk of bias), does not necessarily mean that the outcomes of the reviews were incorrect. In an era of evidence based medicine, however, the lack of possibilities to replicate and reaffirm does introduce questions about the methodology, outcomes and the use of the findings in clinical practice [28].
A number of methodological issues need to be addressed. A large sample size was used of 6471 RoB- items out of 989 therapeutic RCTs out of different medical fields, published in October, November and December 2016. We believe this provides us with an extensive and representative sample. We relied on the judgment of the authors of the included Cochrane systematic reviews for the RoB assessment of the 7 key-items. Variety might exist; one could argue the consistency of the RoB judgements. We assumed all authors worked according to the Cochrane risk of bias Handbook, introduced in 2011 [17]. However there are limitations to the reliability of the Cochrane RoB tool since low agreement rates have been found between Cochrane- and external reviewers [32,33]. Second, one could imagine that review authors might have excluded RCTs because of their inadequate designs, which can affect our outcome. Third, in this paper we use the RoB tool as a measure to assess quality of reporting. This implicates that no statements can be made about adherence to (potentially missing) individual items of the CONSORT statement. Fourth, in this paper only the seven items from the Cochrane RoB tool were included. If authors had adjusted the tool (e.g. added an item), these adjustments and outcomes were not taken in account. This could have led to missing information.
In our study we found that the odds of RCTs published after 2010 were significantly more likely not to score an “unclear” RoB in total. The same applies for several individual items, such as random sequence generation, allocation concealment and incomplete outcome data. These improvements are encouraging. However, the use of the CONSORT statement, and the level of adherence varies greatly amongst RCTs [20,34]. In order to ‘Increase value and decrease waste’ in biomedical research a further encouragement to adhere to the CONSORT statement must be established to improve the quality of reporting of RCTs and consequently better patient care.
References
- 1. Chalmers Iai, Glasziou P. Avoidable waste in the production and reporting of evidence. Lancet [Internet]. 2009;374(9692):786. Available from: pmid:19733780
- 2. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. J Am Med Assoc. 2004;291(20):2457–65.
- 3. van Heteren JAA, van Beurden I, Peters JPM, Smit AL, Stegeman I. Trial registration, publication rate and characteristics in the research field of otology: A cross-sectional study. PLoS One. 2019;14(7):e0219458. pmid:31291350
- 4. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. pmid:24411644
- 5. Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet [Internet]. 2014;383(9912):166–75. Available from: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62227-8/fulltext pmid:24411645
- 6. Chalmers I, Salman RA-S, Macleod M, Beller E, Savulescu J, Hemminki E, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet [Internet]. 2014;383(9912):176–85. Available from: pmid:24411646
- 7. Chan A-W, Dickersin K, Jefferson T, Ghersi D, Gøtzsche PC, Song F, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66. pmid:24411650
- 8. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet [Internet]. 2014;383(9913):267–76. Available from: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62228-X/fulltext pmid:24411647
- 9. Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, et al. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: Research on research study. BMJ. 2017;357:1–14.
- 10. Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement. Obstet Gynecol. 2010;115(5):1063–70. pmid:20410783
- 11. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. Int J Surg. 2012;10(1):28–55. pmid:22036893
- 12. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. J Am Med Assoc. 1996;276(8):637–9.
- 13.
Organizations E. The CONSORT Website. CONSORT endorsers- journals.
- 14. Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev [Internet]. 2012;1(60):1. Available from: http://www.embase.com/search/results?subaction = viewrecord&from = export&id = L368727196%5Cnhttp://www.systematicreviewsjournal.com/content/1/1/60%5Cnhttp://dx.doi.org/10.1186/2046-4053-1-60
- 15. Hopewell S, Hirst A, Collins GS, Mallett S, Yu LM, Altman DG. Reporting of participant flow diagrams in published reports of randomized trials. Trials. 2011;12.
- 16. Hays M, Andrews M, Douglas K, Callender D, O’Malley PG, Wilson R. Reporting quality of randomised controlled trial abstracts among high-impact general medical journals: a review and analysis. BMJ Open. 2016;6(7):e011082. pmid:27470506
- 17. Higgins J, Green S. Cochrane Handbook for Systematic Review of Interventions. Version 5.1.0 [Internet]. The Cochrane Collaboration. 2011. p. 1–639. Available from: www.cochrane-handbook.org.
- 18.
Urbaniak, G. C., & Plous S. Research Randomizer (Version 4.0) [Internet]. 2013 [cited 2020 Apr 3]. Available from: http://www.randomizer.org/
- 19. Savović J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: Combined analysis of meta-epidemiological studies. Health Technol Assess (Rockv) [Internet]. 2012;16(35):1–81. Available from: http://annals.org/data/Journals/AIM/24890/0000605-201209180-00007.pdf%5Cnhttp://ovidsp.ovid.com/ovidweb.cgi?T = JS&PAGE = reference&D = emed11&NEWS = N&AN = 2012647635
- 20. Peters JPM, Hooft L, Grolman W, Stegeman I. Assessment of the quality of reporting of randomised controlled trials in otorhinolaryngologic literature—Adherence to the CONSORT statement. PLoS One. 2015;10(3):1–12.
- 21. Zhai X, Wang Y, Mu Q, Chen X, Huang Q, Wang Q, et al. Methodological reporting quality of randomized controlled trials in 3 leading diabetes journals from 2011 to 2013 following CONSORT statement. Med (United States) [Internet]. 2015;94(27):e1083. Available from: http://www.embase.com/search/results?subaction = viewrecord&from = export&id = L605949083%5Cnhttp://dx.doi.org/10.1097/MD.0000000000001083
- 22. Sarkis-Onofre R, Poletto-Neto V, Cenci MS, Pereira-Cenci T, Moher D. Impact of the CONSORT Statement endorsement in the completeness of reporting of randomized clinical trials in restorative dentistry. J Dent [Internet]. 2017;58:54–9. Available from: pmid:28159507
- 23. Janackovic K, Puljak L. Reporting quality of randomized controlled trial abstracts in the seven highest-ranking anesthesiology journals. Trials. 2018;19(1):1–6.
- 24. Reveiz L, Chapman E, Asial S, Munoz S, Bonfill X, Alonso-Coello P. Risk of bias of randomized trials over time. J Clin Epidemiol [Internet]. 2015;68(9):1036–45. Available from: pmid:26227423
- 25. Peters JPM, Stegeman I, Grolman W, Hooft L. The risk of bias in randomized controlled trials in otorhinolaryngology: Hardly any improvement since 1950. BMC Ear, Nose Throat Disord. 2017;17(1):1–8.
- 26. Berger VW. Conservative handling of missing information. J Clin Epidemiol. 2012;65(11):1237–8. pmid:23017643
- 27. Berger VW. Internal validity and the risk of bias: A case for a comprehensive review. J Anesth. 2012;26(5):802–3. pmid:22653407
- 28. Palys KE, Berger VW. Re: Quality of reporting of modern randomized controlled trials in medical oncology: A systematic review. J Natl Cancer Inst. 2013;105(3):244.
- 29. Berger VW. Intent-to-treat: Further insight. Res Nurs Heal. 2011;34(1):4.
- 30. Berger VW. Conservative handling of missing data. Contemp Clin Trials. 2012;33(3):460. pmid:22366777
- 31. Lachin JM. Worst-rank score analysis with informatively missing observations in clinical trials. Control Clin Trials. 1999;20(5):408–22. pmid:10503801
- 32. Armijo-Olivo S, Ospina M, Da Costa BR, Egger M, Saltaji H, Fuentes J, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in Physical Therapy trials. PLoS One. 2014;9(5).
- 33. Berger VW, Mickenautsch S. On the need for objective measures of risk of bias. Contemp Clin Trials. 2015;41:202–3. pmid:25666855
- 34. Jin Y, Sanger N, Shams I, Luo C, Shahid H, Li G, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years?–A systematic review of reviews: An update. J Multidiscip Healthc. 2018;11:495–510. pmid:30310289