You have been rigorously trying to use the literature to help you make difficult clinical decisions for one of two reasons: 1) You have been trained in the methods of evidence-based practice, or 2) You are determined to attain the skills required to become an evidence-based practitioner. You are well aware that the best information often can be found in those papers highest on the evidence hierarchy, such as systematic reviews and meta-analyses. In fact, you know that a meta-analysis is a kind of systematic review that attempts to aggregate the data from a group of studies with similar populations, outcomes and statistical methods. You know that the purpose of this kind of review is to enlarge the sample size to provide a more realistic depiction of a true effect size for a question like this: “Is manipulation effective for the treatment of low-back pain?”
You know it’s possible to look at individual trials to try to answer the question. But you are aware that each trial, taken in isolation, has a relatively small number of participants. A meta-analysis combines all the studies and gives you a better treatment effect. Lately, however, you have been hearing about something called “publication bias.” You initially understand this to mean that it is more likely for studies that report positive results to be published, compared with those that show negative results (i.e., to be technical, those that support the null hypothesis that there will be no difference between groups). Intuitively, this makes sense to you: Negative results are less appealing. But then you begin to wonder if there is more to it than this. You decide to look for some information on publication bias.
An Evidence-Based Consideration
You decide to conduct a literature review on this topic. Initially, you use Google for basic information. You are given a number of links to papers on this topic, as well as a few links to YouTube and TED clips, most of them involving the medical writer and provocateur Ben Goldacre. Here is one that really strikes you: www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe.
You also know that there are good scientific data available from PubMed, so you do a simple search using the term “publication bias,” looking for clinical trials and systematic reviews. This paper is found: “Publication bias in clinical trials due to statistical significance or direction of trial results.”
Your paper is a systematic review from the Cochrane Collaboration. You know that this group, which conducts systematic reviews for all kinds of interventions, will have scoured existing literature on this topic to locate papers for analysis. In their paper, the authors looked at whether a publication is influenced by the statistical significance, perceived importance or direction of results of a given paper.1 Indeed, the authors found that trials with positive findings were more likely to be published than trials with negative results or null results (i.e., no difference between groups at the end of the study). The authors turned up another interesting finding: Published papers with positive results were published nearly two years faster than those with negative results.
There are those who believe that the failure to publish results is research misconduct. Why would that be the case? Let us put this in the context of papers looking at the effectiveness of a drug. We have already stated that a meta-analysis is the best way to determine the true effect size of an intervention. If journal editors tend to accept only positive results, and researchers who later conduct a meta-analysis can locate only the papers with those positive findings, the researchers’ analysis will not reflect the true reality of the effectiveness of that drug. The negative papers never were published. To the researchers, they do not exist!
Ben Goldacre gave a telling example. He began the TED talk I linked above by showing the audience a paper on precognition; that is, the ability among college students to predict the future. This paper was indeed truly published, and it had what appeared to be solid methodology. When a second research group set out to replicate the study, it had negative results. The researchers still submitted the paper to the same journal, only to be told that the journal was not interested in replication studies. The odd and unusual finding was published; the more banal and perhaps expected finding was not. And yet we say that science is built upon the ability to replicate the work of others.
Now, the failure to publish negative results happens for a number of reasons. One is that there is a human desire to see the unusual published. A second reason, especially in drug trials, is that negative studies can be quashed so that only the positive ones are published. This has been the case in several highly newsworthy events wherein drugs have been taken off the market. An example is Vioxx.
Our problem is this: How do we even know that a study was done but not published? Various answers have been proposed, such as trial registries and other requirements. But it has been shown that even where required, studies may not be registered. This is a true problem with biomedical studies.
We live in an evidence-based world. Our decisions have to be based on the best possible evidence. Some of that evidence, however, may never see the light of day. Therefore, our view of reality can be skewed without our knowing it. We have no knowledge at all about possible publication bias in chiropractic research. Are there studies that have been conducted but not published? How can we find them?
It turns out that there are tools that may help. One is a simple device called a funnel plot.2 This is a graph that allows you to plot the results of a set of studies, whether they show positive or negative results. Typically, with a given set of studies, there should be dispersion around the line of no effect, with an approximately equal number of positive vs. negative studies. If the plot shows many on one side and few on the other, it may indicate the existence of publication bias.
We need to interpret even our best and most rigorous studies with some caution. There is no evidence at present of publication bias in chiropractic research. This does not mean it does not occur, nor does it mean it does. It suggests we need to look at the question. We should examine trial registries to see if all trials in chiropractic are indeed published. We should query journal editors about their willingness to accept negative studies, and survey those whose papers have been rejected for publication. This will be a challenge. But given the importance of the information, it is a necessary challenge. In addition, from a human research perspective, we should never enter a human being into a clinical trial if in the end the results are not published. If that happens, it means that the patient was exposed to risk and no one anywhere will ever benefit. This is ethically specious.
1. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results (Review). Cochrane Library, 2009, www.thecochranelibrary.com/userfiles/ccoch/file/INternational%20Clinical%20Trials%20Day/MR000006.pdf, accessed July 11, 2014.
2. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Br Med J 1997;315:629-634.
Dana Lawrence, DC, MMedEd, MA, is senior director for the Center for Teaching and Learning and interim senior director for Continuing Education and Events at Palmer College of Chiropractic, Davenport, Iowa.