Skip Navigation
Skip to contents

Perspect Integr Med : Perspectives on Integrative Medicine

OPEN ACCESS
SEARCH
Search

Articles

Page Path
HOME > Perspect Integr Med > Volume 3(3); 2024 > Article
Letter
Research Methodology Matters: A Comment on the Newly Published Article in the Journal of Evidence-Based Medicine
Tae-Hun Kim*orcid
Perspectives on Integrative Medicine 2024;3(3):184-185.
DOI: https://doi.org/10.56986/pim.2024.10.009
Published online: October 31, 2024

Korean Medicine Clinical Trial Center, Korean Medicine Hospital, Kyung Hee University, Seoul, Republic of Korea

*Corresponding author: Tae-Hun Kim, Korean Medicine Clinical Trial Center, Kyung Hee University, Korean Medicine Hospital, 23, Kyungheedae-ro, Dongdaemun-gu, Seoul 02447, Republic of Korea, Email: rockandmineral@gmail.com
• Received: April 23, 2024   • Revised: July 2, 2024   • Accepted: July 9, 2024

©2024 Jaseng Medical Foundation

This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

prev
  • 74 Views
  • 5 Download
Dear Editor,
Blinding is accepted to be important in a clinical trial to reduce bias and improve the validity of the evaluation on the specific effects of the test intervention. However, in the absence of an appropriate placebo control intervention, as in trials employing acupuncture, blinding is a challenging goal to achieve. In the article recently published by Long et al titled, “The impact of blinding on estimated treatment effects in randomized clinical trials on acupuncture: A meta-epidemiological study” [1], the authors concluded that, “It (blinding of participants) underscores the practical difficulties of blinding in acupuncture randomized controlled trials (RCTs) and the necessity to distinguish between trials with and without successful blinding to understand treatment.” There are questions regarding the methodology of this study and its reliability to determine, “Blinding of participants and outcome assessors does not significantly influence acupuncture treatment efficacy” [1].
Firstly, there may be errors in the search process and literature selection of this meta-epidemiological study. Searching only Chinese databases, PubMed, and Embase for acupuncture studies could result in publication bias [2]. For example, the failure to search the Cochrane library is a factor that should be carefully considered. It is difficult to verify whether transparency in the literature selection process was maintained. It would be beneficial to detail the reasons which determined the exclusion of studies so the process would be more transparent [3], particularly when 568 articles were excluded due to risk of bias and 595 reviews were excluded due to incomplete information. In addition, only 1 meta-analysis was included in the analysis, and the rationale for selecting this study was based solely on its having the highest number of included RCTs. To demonstrate the validity of the study selection process, it would be appropriate for the authors to state whether other meta-analysis studies were considered and whether sensitivity analyses were performed to establish robustness before the decision to include only 1 meta-analysis in the study was made. Furthermore, it is challenging to accept the feasibility of identifying trends across studies which employ diverse acupuncture interventions and control measures (in terms of tools, frequency, severity, and duration) for different conditions/diseases and evaluate various outcomes. Such clinical heterogeneity needs to be considered seriously in meta-analyses. Given the existing evidence that the effect size can vary depending on the interventions used in the control group [4], and that the magnitude of the placebo effect differs among conditions/diseases [5], heterogeneity raises serious concerns about the appropriateness of the analytical methods employed in this study.
Secondly, the criteria used by the authors to judge the success of blinding in the included RCTs of the meta-analysis appears to be inappropriate [1]. Statistical methods exist for evaluating blinding in RCTs [6], and blinding successfulness of the individual studies should have been evaluated based on these, comparing studies that were judged as “well-blinded” against those that were not. However, in this meta-epidemiology study, the criteria for judging blinding were based on whether blinding was described in the literature, and whether a sham device was used. Particularly, the statement, “If single blinding is reported and no form of placebo acupuncture is used in the control group, it is considered a low risk of blinding of outcome assessors” raises doubts about whether the studies analyzed were appropriately differentiated. While it is understandable that many studies might not be included due to the nature of the research, if the judgment on appropriate blinding is uncertain, it may not yield the appropriate results.
Thirdly, questions arise as to whether the analysis methods used were appropriate. In this study, an odds ratio of greater than 1 was considered a beneficial intervention effect for binary data, and a standardized mean difference of less than 0 was classified as a beneficial effect for continuous outcomes [1]. However, not all individual meta-analyses consistently showed outcomes in the same direction for benefit or harm. It is uncertain as to whether the direction of the outcomes presented in individual studies was appropriately adjusted, and the supplementary file does not provide information on which meta-analyses were included or what effect sizes were used for their outcomes, which further undermines confidence in the reported outcomes of this research.
The success of blinding has been recognized as a significant factor influencing the effect size in clinical trials, and empirical evidence supports this hypothesis [7]. If the findings of this study are valid, then acupuncture clinical trials should no longer consider blinding issues important, nor discuss interventions to distinguish between nonspecific effects thus negating the use of sham acupuncture. However, if the design of this study indicates the authors’ conclusions are unreliable, it raises concerns around future acupuncture research and how it should be implemented.

Conflicts of Interest

There is no conflict of interest to declare.

Funding

None.

Ethical Statement

This article does not include any personal information. General research ethics guidelines were followed.

There was no usable data in this article.
  • [1] Long Y, Zhang N, Wang X, Tang R, Guo Q, Huang J, et al. The impact of blinding on estimated treatment effects in randomized clinical trials on acupuncture: a meta-epidemiological study. J Evid Based Med 2024;17(1):54−64.ArticlePubMed
  • [2] Kim TH, Kang JW, Lee MS. When conducting a systematic review, can one trade search efficiency for potential publication bias? Res Synth Methods 2022;13(6):662−3.ArticlePubMedPDF
  • [3] Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008. ArticlePubMedPMC
  • [4] Lee B, Kwon C-Y, Lee HW, Nielsen A, Wieland LS, Kim T-H, et al. The effect of sham acupuncture can differ depending on the points needled in knee osteoarthritis: a systematic review and network meta-analysis. Heliyon 2024;10(4):e25650. ArticlePubMedPMC
  • [5] Cao B, Liu YS, Selvitella A, Librenza-Garcia D, Passos IC, Sawalha J, et al. Differential power of placebo across major psychiatric disorders: a preliminary meta-analysis and machine learning study. Sci Rep 2021;11:21301. ArticlePubMedPMCPDF
  • [6] Bang H, Flaherty SP, Kolahi J, Park J. Blinding assessment in clinical trials: a review of statistical methods and a proposal of blinding assessment protocol. Clin Res Regul Aff 2010;27(2):42−51.Article
  • [7] Page MJ, Higgins JP, Clayton G, Sterne JA, Hróbjartsson A, Savović J. Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies. PLoS One 2016;11(7):e0159267. ArticlePubMedPMC

Figure & Data

References

    Citations

    Citations to this article as recorded by  

      • PubReader PubReader
      • ePub LinkePub Link
      • Cite
        Download Citation
        Download a citation file in RIS format that can be imported by all major citation management software, including EndNote, ProCite, RefWorks, and Reference Manager.

        Format:
        • RIS — For EndNote, ProCite, RefWorks, and most other reference management software
        • BibTeX — For JabRef, BibDesk, and other BibTeX-specific software
        Include:
        • Citation for the content below
        Research Methodology Matters: A Comment on the Newly Published Article in the Journal of Evidence-Based Medicine
        Perspect Integr Med. 2024;3(3):184-185.   Published online October 23, 2024
        Close
      • XML DownloadXML Download
      Research Methodology Matters: A Comment on the Newly Published Article in the Journal of Evidence-Based Medicine
      Research Methodology Matters: A Comment on the Newly Published Article in the Journal of Evidence-Based Medicine

      Perspect Integr Med : Perspectives on Integrative Medicine
      TOP