In This Case, the Journal Science is Holding Back the Progress of Science

Sometimes, scientific journals hold back the progress of science.

***Please note that while political beliefs are mentioned in this post, it is NOT a political post and political commentary will NOT be approved.***

Science is supposed to be self-correcting. The history of science is full of mistakes, but over time, those mistakes are usually found, and the findings are communicated to the scientific community so that the mistakes no longer influence scientific thinking. Unfortunately, one of the main ways that findings are communicated is through scientific journals, and there are times when scientific journals are not interesting in correcting mistakes, especially when those mistakes reflect badly on the journal’s reputation. I recently ran across a story that illustrates this point.

Back in 2008, the most prestigious scientific journal in the United States, the journal Science, published a study that attempted to understand the root causes of political beliefs. They exposed several participants to images and sounds designed to evoke fear and correlated the participants’ response to their political beliefs. Based on their results, the authors concluded:

…individuals with measurably lower physical sensitivities to sudden noises and threatening visual images were more likely to support foreign aid, liberal immigration policies, pacifism, and gun control, whereas individuals displaying measurably higher physiological reactions to those same stimuli were more likely to favor defense spending, capital punishment, patriotism, and the Iraq War. Thus, the degree to which individuals are physiologically responsive to threat appears to indicate the degree to which they advocate policies that protect the existing social structure from both external (outgroup) and internal (norm-violator) threats.

In other words, if you are prone to fear, you are more likely to be a conservative. If not, you are more likely to be a liberal.

The study was ground-breaking, and it has strongly influenced scientific research in the field. Indeed, at the time of this posting, the study has been referenced in 257 subsequent studies. There’s only one problem. It probably isn’t correct. How do we know? Because some researchers who were initially interested in expanding on the results of the study began doing some experiments, but the experiments didn’t seem to support the conclusions of the 2008 study. In an attempt to see what they were doing wrong, the researchers contacted the authors of the 2008 study so that they could replicate their methodology. They weren’t trying to demonstrate that the 2008 study was wrong. In fact, they were trying to use its methodology to “calibrate” their study so that they could get consistent results.

What they found is that they could not replicate the results of the previous study. What is supposed to happen when something like this occurs? It is supposed to be communicated, right? After all, researchers need to know that there was something wrong with a study that is still influencing the field. Where should this be communicated? In the journal that published the original study, of course. That’s what these authors tried to do. They submitted the above-linked paper to the journal Science, and the editors refused to even consider publishing the paper.

When a scientific journal considers publishing a paper, it sends the paper out for peer review. That means other scientists in the field evaluate it to see if they can find any errors in the study. If the reviewers think that small errors exist, the journal might still publish the paper, as long as the authors correct those small errors. If the reviewers think there are serious errors in the paper, the journal will not publish the paper. The problem is that in this case, Science refused to even send the paper out for peer review. They just summarily rejected it, without any evaluation as to its quality.

Why? The editors thought that the field had “moved on” from the initial study (it was 11 years ago, after all), so there was no reason to publish the fact that the initial study was probably wrong. However, we know that is simply false. Of the 257 papers that cite the study, nine of them come from this year, 25 of them from last year, and 41 from the year before. Clearly, then, the initial study is still influencing research in the field, and the journal Science refuses to let scientists in the field know that the study is probably wrong!

I think the authors of the new study say it best:

Science requires us to have the courage to let our beautiful theories die public deaths at the hands of ugly facts…Our takeaway is not that the original study’s researchers did anything wrong. To the contrary, members of the original author team — Kevin Smith, John Hibbing, John Alford and Matthew Hibbing — were very supportive of the entire process, a reflection of the understanding that science requires us to go where the facts lead us. If only journals like Science were willing to lead the way.

4 Comments

  1. John D. says:

    It seems pretty unscientific to peg something as complicated as policy / party preference on something as simple as fear. I’m not surprised the findings couldn’t be duplicated. The entire premise seems to reek of partisanship from the get go.

    Your link to the 257 studies in which the study’s been cited requires a password so I wasn’t able to see how the findings were used – but I’d be willing to guess that they’ve been used to bolster prejudices already held by political scientists (i.e. policies stemming from certain party platforms basically stem from phobias – such as xenophobia)

    Seems overall like yet another attempt at fatalistic determinism. Behaviorist scientists looking for the good ole hard wired motives.

    Do you think that they won’t correct it is simply because it’s embarrassing… or maybe because it would be better that such a seemingly powerful study continue to be cited (similar to textbooks hanging on to false evolutionary studies like vestigial organs)

    1. Jay Wile says:

      I really don’t know the motivation behind the decision to not even send the article out for peer review. All I can say for sure is that the stated reason is false. The field has definitely not moved on from the 2008 study!

      1. Spencer Brown says:

        This reminds me of something I saw recently, where the current funding models and measures of success for scientific research discourages replications studies and corrections – because Journals refuse to publish them because they aren’t “novel” enough.

        Veritasium – “Is Most Published Research Wrong?”
        https://youtu.be/42QuXLucH3Q

        1. Jay Wile says:

          I love that YouTube channel, but I hadn’t watched that video. What he says near the end is great:

          What gets me is the thought that even trying our best to figure out what’s true, using our most sophisticated and rigorous mathematical tools, peer review, and standards of practice, we still get it wrong so often.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: