Reporting Overviews of Reviews: PRIORitizing a Reporting Guideline

Lisa Hartling; David Moher

doi:10.56986/pim.2024.06.001

Articles

Page Path: HOME > Perspect Integr Med > Volume 3(2); 2024 > Article

Editorial Reporting Overviews of Reviews: PRIORitizing a Reporting Guideline: Lisa Hartling^1,*, David Moher²; Perspectives on Integrative Medicine 2024;3(2):65-68.
DOI: https://doi.org/10.56986/pim.2024.06.001
Published online: June 30, 2024

¹Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Canada

²Clinical Epidemiology Program, Ottawa Hospital Research Institute, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada

*Corresponding author: Lisa Hartling, Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Alberta, Canada, Email: hartling@ualberta.ca

• Received: April 18, 2024 • Accepted: May 9, 2024

This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

759 Views
33 Download

Full Article

Download PDF

Visual abstract
Article information
Data Availability
References

Visual abstract

Two recent publications in Perspectives on Integrative Medicine have employed an emerging form of evidence synthesis, however, both papers refer to different designs in their titles–overview of systematic reviews and umbrella review, respectively [1,2]. What are these? Do they use the same design and methodological processes? Can the results be trusted? These are some of the questions a reader might ask regarding these and similar publications. In this editorial, we aim to define this newer form of evidence synthesis; provide some history regarding its evolution and methodological approach; discuss some caveats around the methods and potential flaws; and present a reporting guideline [preferred reporting items for overview of reviews (PRIOR)] that can guide authors, peer-reviewers, editors, and readers in understanding these and similar evidence syntheses, their results, and implications for end-users (e.g., clinicians, policymakers, patients, and their caregivers).

Overviews of reviews (also referred to herein as overviews) have emerged as a form of evidence synthesis over the past 20–25 years with one of the first overviews published in 2000 [3]. They are similar to systematic reviews in that they aim to provide a comprehensive and unbiased synthesis of the literature on a topic by following rigorous, systematic, and predefined methods. However, the unit of analysis is systematic reviews rather than primary studies. The methodology has evolved, and the numbers have increased concurrent with the increase and advances in systematic reviews. Moreover, with the steady increase in the number of systematic reviews, with estimates of more than 80 being published per day [4], overviews of reviews can provide a valuable resource for end-users by bringing together evidence across a broad topic. This has great relevance to complementary therapies and integrative medicine where often a given intervention may be used across different health conditions, or multiple interventions may be considered as options for the same condition, particularly those that have a range of presenting symptoms and/or varying patient responsiveness across intervention options.

Within the Cochrane Collaboration, an international organization dedicated to producing high quality evidence syntheses to inform health decisions, a working group was established in 2004 to guide methods for overviews. Cochrane considered these products to provide a “friendly front-end” to reviews, by bringing together evidence from individual systematic reviews on related topics. To this end, Cochrane Child Health produced over 30 overviews of reviews (published in Evidence-based Child Health: A Cochrane Review Journal) to facilitate the uptake of evidence in practice and policy [5] (e.g., non-pharmacological interventions to treat attention deficit hyperactivity disorder [6], interventions to treat major depression in children and youth [7], and interventions to prevent eczema [8]). Two of the most common approaches to topics (or review questions) for overviews are the examination of the effectiveness and/or safety of multiple interventions for the same condition (e.g., treatments for women with pain or subfertility associated with endometriosis [9]) or a single intervention across multiple conditions (e.g., aromatherapy for management of health [1]). Also, of interest to end-users, overviews of reviews may provide evidence for subgroups or subpopulations of interest (e.g., children, elderly) by focusing on data related to those groups from the relevant systematic reviews (e.g., the reference [10]).

Alongside the production of overviews, much work has been performed to advance methods. Cochrane first published methodological guidance for overviews in 2008 [11]. This guidance was updated in 2020 [12]. New guidance for overview methods continues to emerge (e.g., the references [13–19]), with some guidance specific to the field of traditional and complementary medicine [20]. The Cochrane guidance focuses on overviews of intervention effectiveness, recognizing that subsequent guidance will be needed to address reviews with different types of questions such as diagnostic test accuracy, prognosis, and risk factors/associations [21]. JBI (originally known as The Joanna Briggs Institute), an international organization that supports evidence-informed decision-making to improve health and health service delivery, has also developed guidance for what they refer to as umbrella reviews [22]. The JBI concept of umbrella reviews differs somewhat from Cochrane’s overviews: while Cochrane’s overviews focus on systematic reviews of effectiveness, JBI umbrella reviews may incorporate different forms of systematic reviews [e.g., effectiveness (quantitative); meta-aggregative, integrative (qualitative)] to address “a broad scope of issues related to a topic of interest,” e.g., effectiveness as well as patient experiences with an intervention [22]. Terminology regarding the evidence synthesis product is critical for readers’ expectations and appraisal, to differentiate methodological approaches and rigor, and particularly, to distinguish an evidence synthesis from a nonsystematic summary of select literature (e.g., a more traditional literature review) [23].

Overviews and umbrella reviews are considered rigorous scientific endeavors, that take careful planning, time, and relevant research expertise (ideally experience with systematic reviews). Organizations may commission an overview, or researchers may propose such an approach, for time-dependent decision-making (e.g., the references [24,25]), without understanding the potential complexities. Overviews need to begin with clear objectives, well-formulated research questions (with specification of population(s), intervention(s), comparison(s), outcome(s) and other relevant variables), and a priori defined methods that aim to avoid bias. As with all research projects, preparing a protocol before conducting the overview and making it publicly available through a journal publication or repository (e.g., Open Science Framework) is a hallmark of methodological rigor and enhances transparency and openness for end-users [26]. The key difference between systematic reviews and overviews, is the unit of analysis (primary studies vs. systematic reviews), which can lead to specific challenges in conducting an overview and interpreting results. Three important challenges include the potential for overlapping studies across included systematic reviews, reporting (or non-reporting) biases, and inappropriate indirect comparisons.

Firstly, overviews may include systematic reviews that contain the same primary studies. This may occur when including multiple systematic reviews on the same topic. It may also occur when primary studies have multiple arms examining different active interventions. In this case, the same primary study may appear in individual systematic reviews examining specific active interventions. This can lead to overstating the sample size and number of events (i.e., outcomes), and can exaggerate the degree of precision in an analysis and subsequent estimate [12]. Care needs to be taken by the overview researchers so that the data from the same primary studies is only included once in any syntheses or estimates of effects. Methods are available for identifying and managing such overlap in overviews [12,17–19]. Overview authors should describe how they have addressed potential overlap (PRIOR Checklist items 9b and 17) [27].

Secondly, overviews are susceptible to at least two forms of reporting bias: publication bias and selective outcome reporting bias (PRIOR Checklist items 13 and 20) [27]. Overviews rely on the methods employed by the included systematic reviews, including methods to ensure comprehensive searching, identification, and inclusion of studies regardless of study results. Methodological research has shown that studies with negative (or statistically non-significant) results are less likely to be published and less likely to be published in English [28]. This may be particularly relevant to complementary therapies and integrative medicine; they are often practiced in countries whose first language is not English [29]. Overview researchers need to address the potential for publication bias as systematic reviews that have failed to comprehensively identify all relevant studies may present biased results that either over-estimate or under-estimate the true effects. Furthermore, overview researchers need to prespecify outcomes of interest (ideally in a protocol) and report on findings specific to these outcomes. It is unsatisfactory (and potentially biased) to simply describe what the systematic reviews have observed or what the primary studies have reported because of the likelihood to report on outcomes that show significant results [28].

Thirdly, caution with respect to interpreting results of overviews is the use of informal indirect comparisons between interventions. For example, some overviews will present results of systematic reviews examining different interventions compared with usual care, placebo, no control, or another active intervention. It can be tempting to assume that an intervention (A) that performs better than a given comparator (C) in one set of studies (or systematic review) may be preferred to a different intervention (B) that does not perform better than the same comparator (C) in a separate set of studies (or separate systematic review), i.e., making informal indirect comparisons between interventions A and B. Cochrane methods strongly discourage indirect comparisons unless formal statistical analyses (i.e., network meta-analysis) have been conducted to integrate and appropriately analyze data from direct and indirect comparisons [12]. This is rarely possible within the context of an overview because of the need for intimate knowledge of the individual studies to ensure a key assumption (transitivity) is met.

The ability of readers to assess these and other potential sources of bias, and ultimately the validity and applicability of results, will depend on transparent and detailed reporting. Reporting guidelines might be considered an intervention to ensure that reports of primary studies and evidence syntheses accurately and sufficiently reflect the methods and results and allow for replication. The Enhancing the QUAlity and Transparency of Health Research (EQUATOR) Network [30] is an international initiative that promotes transparent and accurate reporting of research. The EQUATOR Network library houses over 500 reporting guidelines for a wide range of study designs including primary studies across research pillars (basic, preclinical, clinical, health services, and population health), and evidence syntheses. The EQUATOR Network also endorses specific methods for the rigorous development of reporting guidelines [31,32].

Several key reporting guidelines have been developed for evidence syntheses, including the Preferred Reporting Items for Systematic Review and Meta-Analyses (abbreviated to PRISMA) [33,34], and more recently the Preferred Reporting Items for Overviews of Reviews (PRIOR) [27]. The development of PRIOR followed established methods which are detailed elsewhere [27,31,32]. Of note, rigorous reporting guidelines are developed based on evidence, and where evidence is lacking or unclear, consensus among a range of stakeholders. PRIOR was established based on scoping reviews of the literature for methodological guidance, a Delphi study involving 100 stakeholders from multiple end-user groups (i.e., authors, journal editors, funders, relevant international organizations, patients), and consensus meetings with stakeholder representatives. The final PRIOR guideline involved a collaboration among 19 authors from 8 countries and 5 continents. Reporting guidelines, including PRIOR, typically include a checklist of items for authors to report, a flow diagram that describes the search and selection process, and an Explanation and Elaboration document with rationale and examples for each checklist item. The PRIOR reporting guideline should be used by authors, editors, and peer-reviewers to ensure that key aspects of an overview of reviews are reported such that readers can assess its validity and applicability to their context.

Drummond Rennie (former Deputy Editor of the Journal of the American Medical Association) wrote that the “whole of medicine depends on the transparent reporting of trials” [35]. This is also essential with evidence syntheses because of the potential for errors and biases to be carried forward and amplified. Authors of systematic reviews and overviews need to employ methods and rigorously appraise their included studies (or reviews) such that accurate, reliable, and valid results are presented with appropriate and relevant conclusions. Reporting guidelines are not only relevant to authors when reporting the results of their studies; they should guide protocol development. Authors should look at reporting guidelines when preparing their protocols and methods to ensure that they can adhere to the reporting requirements once their evidence synthesis is complete. Furthermore, reporting guidelines are very relevant to editors, peer-reviewers, funding bodies of research, and users of research, and should be encouraged (if not mandated) to ensure that end users have the information they need, contextualized within potential limitations, to inform healthcare practices and policies. We encourage researchers (and publishers) within the fields of complementary therapies and integrative medicine to elevate the quality of research through consideration of, and adherence to, established reporting guidelines.

Article information

Author Contributions

Conceptualization and writing–original draft: LH. Writing–review & editing: DM.

Conflicts of Interest

The authors have no conflicts of interest to declare.

Funding

Dr. Hartling is supported by a Canada Research Chair in Knowledge Synthesis and Translation.

Ethical Statement

This research did not involve any human or animal experiments.

Data Availability

All relevant data are included in this manuscript.

References

[1] Kil KJ, Lee MS. A protocol for the overview of systematic reviews of aromatherapy for management of health. Perspect Integr Med 2023;2(1):56−8.Article PDF
[2] Kim D, Baek GG, Shin BC. An umbrella review of systematic reviews for Chuna (or Tuina) manual therapy on musculoskeletal disorders. Perspect Integr Med 2023;2(3):142−54.Article PDF
[3] Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000–2011. PLoS One 2012;7(11):e49667. Article PubMed PMC
[4] Hoffmann F, Allers K, Rombey T, Helbach J, Hoffmann A, Mathes T, et al. Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000–2019. J Clin Epidemiol 2021;138:1−11.Article PubMed
[5] Thomson D, Foisy M, Oleszczuk M, Wingert A, Chisholm A, Hartling L. Overview of reviews in child health: evidence synthesis and the knowledge base for a specific population. Evid Based Child Health 2013;8(1):3−10.Article PubMed
[6] Foisy M, Williams K. The Cochrane Library and non-pharmacological treatments for attention deficit hyperactivity disorder in children and adolescents: an overview of reviews. Evid Based Child Health 2011;6(2):283−97.Article
[7] Zinck S, Bagnell A, Bond K, Newton AS. The Cochrane library and the treatment of major depression in children and youth: an overview of reviews. Evid Based Child Health 2009;4(4):1336−50.Article
[8] Foisy M, Boyle RJ, Chalmers JR, Simpson EL, Williams HC. The prevention of eczema in infants and children: an overview of Cochrane and non-Cochrane reviews. Evid Based Child Health 2011;6(5):1322−39.Article PubMed PMC
[9] Brown J, Farquhar C. Endometriosis: an overview of Cochrane reviews. Cochrane Database Syst Rev 2014;2014(3):CD009590. Article PubMed PMC
[10] Pollock M, Sinha IP, Hartling L, Rowe BH, Schreiber S, Fernandes RM. Inhaled short-acting bronchodilators for managing emergency childhood asthma: an overview of reviews. Allergy 2017;72(2):183−200.Article PubMed PDF
[11] Becker LA, Oxman AD. Chapter 22: Overviews of reviews. Edited by Higgins JPT, Green S: Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011.
[12] Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L, Chapter V. Overviews of reviews. Edited by Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA: Cochrane handbook for systematic reviews of interventions version 6.4 (updated August 2023). Cochrane, 2023.
[13] Hunt H, Pollock A, Campbell P, Estcourt L, Brunton G. An introduction to overviews of reviews: planning a relevant research question and objective for an overview. Syst Rev 2018;7(1):39. Article PubMed PMC PDF
[14] Bougioukas KI, Bouras E, Apostolidou-Kiouti F, Kokkali S, Arvanitidou M, Haidich AB. Reporting guidelines on how to write a complete and transparent abstract for overviews of systematic reviews of health care interventions. J Clin Epidemiol 2019;106:70−9.Article PubMed
[15] Bougioukas KI, Liakos A, Tsapas A, Ntzani E, Haidich AB. Preferred reporting items for overviews of systematic reviews including harms checklist: a pilot tool to be used for balanced reporting of benefits and harms. J Clin Epidemiol 2018;93:9−24.Article PubMed
[16] Fusar-Poli P, Radua J. Ten simple rules for conducting umbrella reviews. Evid Based Ment Health 2018;21(3):95−100.Article PubMed PMC
[17] Lunny C, Pieper D, Thabet P, Kanji S. Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews. BMC Med Res Methodol 2021;21(1):140. Article PubMed PMC PDF
[18] Bougioukas KI, Vounzoulaki E, Mantsiou CD, Savvides ED, Karakosta C, Diakonidis T, et al. Methods for depicting overlap in overviews of systematic reviews: An introduction to static tabular and graphical displays. J Clin Epidemiol 2021;132:34−45.Article PubMed
[19] Bougioukas KI, Diakonidis T, Mavromanoli AC, Haidich A-B. ccaR: A package for assessing primary study overlap across systematic reviews in overviews. Res Syn Meth 2023;14(3):443−54.Article
[20] Kim T-H, Tian Z-Y, Liao X, Robinson N. Essential elements required for conducting and structuring an overview of systematic reviews in the field of traditional and complementary medicine. Eur J Integr Med 2022;49:101605. Article
[21] Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviews in the medical and health sciences. BMC Med Res Methodol 2018;18(1):5. PubMed PMC
[22] In: Aromataris E, Lockwood C, Porritt K, Pilla B, Jordan Z. [Internet]. In: JBI manual for evidence synthesis: JBI: 2024 Available from: https://synthesismanual.jbi.global
[23] Sutton A, Clowes M, Preston L, Booth A. Meeting the review family: exploring review types and associated information retrieval requirements. Health Info Libr J 2019;36(3):202−22.Article PubMed PDF
[24] Khalili M, Chegeni M, Javadi S, Farokhnia M, Sharifi H, Karamouzian M. Therapeutic interventions for COVID-19: a living overview of reviews. Ther Adv Respir Dis 2020;14:1753466620976021. Article PubMed PMC PDF
[25] Rios P, Radhakrishnan A, Thomas SM, Darvesh N, Straus SE, Tricco AC. Preventing the transmission of COVID-19 and other coronaviruses in older adults aged 60 years and above living in long-term care: a rapid review. Syst Rev 2020;9:218. Article PubMed PMC PDF
[26] Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008. Article PubMed PMC
[27] Gates M, Gates A, Pieper D, Fernandes RM, Tricco AC, Moher D, et al. Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. BMJ 2022;378:e070849. Article PubMed PMC
[28] Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. Edited by Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA: Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023.
[29] Wardle J. The Australian government review of natural therapies for private health insurance rebates: what does it say and what does it mean? Adv Integr Med 2016;3(1):3−10.Article
[30] EQUATOR Network [Internet]. Enhancing the QUAlity and transparency of health research: Available from: https://www.equator-network.org
[31] Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med 2010;7(2):e1000217. Article PubMed PMC
[32] EQUATOR Network [Internet]. How to develop a reporting guideline: [cited 2024 Mar 28]. Available from: https://www.equator-network.org/toolkits/developing-a-reporting-guideline/
[33] Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. Article
[34] Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021;372:n160. Article
[35] Rennie D. CONSORT revised–improving the reporting of randomized trials. JAMA 2001;285(15):2006−7.Article PubMed

Figure & Data

References

Citations

Citations to this article as recorded by

PubReader
ePub Link

Cite

Download Citation

Download a citation file in RIS format that can be imported by all major citation management software, including EndNote, ProCite, RefWorks, and Reference Manager.

Format: