Posted by jlwile on January 19, 2010
I am currently doing an Alaskan tour of six cities in seven days, working with educators in a state-wide, publicly-funded charter school system. Even though it is cold, it is a lot of fun. Alaska is beautiful, and the charter school system is excellent. It is great to see quality education occurring in such a novel way.
Because airplanes are the main way one gets from city to city in Alaska, I have been spending a lot of time sitting in airports, on tarmacs, and occasionally on an airplane that is actually flying. As a result, I have been doing a lot of reading. I came across an interesting article in
Science Science News today1, and I think it is a great illustration of something I stress in most of my science books.
For several years now, scientists have been using functional magnetic resonance imaging (fMRI) to observe functioning brains. The main technique involves using a magnetic resonance imaging machine to look for small changes that occur within the blood vessels of a person’s brain. The very reasonable argument proposed is that the more active a neuron is, the more blood it needs. Thus, if the fMRI sees an increase in blood flow to a particular region of the brain, the neurons in that region must be more active. So…a subject is stimulated in some way, and the fMRI looks for increases in blood flow. Any region of the brain that “lights up” must be the region that is responsible for either processing whatever stimulus was provided or producing a response to it.
Several hundred papers have been published discussing the results of all manner of fMRI experiments, and they have made all sorts of definitive conclusions regarding what regions of the brain are responsible for processing various stimuli or producing various responses to those stimuli. Well, Craig Bennett wanted to see how reliable fMRI experiments are, so he decided to do a very simple baseline test. He used fMRI to study the way a dead fish’s brain responds to stimuli.
The results were quite interesting.
It turns out that when the dead fish was shown pictures of scenes that depicted emotion, a part of the dead fish’s brain lit up under the fMRI. If the fish had not been dead, the definitive conclusion would have been that the regions of the brain that were lit up were the regions responsible for how the fish processes the images it was seeing. Obviously, however, Bennett knew that his study said nothing about how a dead fish’s brain works. Instead, it says something about the reliability of fMRI studies.
In order to see what was going on, he tested the statistical methods used to take the volumes of raw data the fMRI collects and translate them into the result the experimenter actually reads – the region of the brain with increased bloodflow. Not surprisingly, he found that the way the raw data were processed (which no one ever really checks or questions) can take random noise and make it look like increased blood flow. What he as an experimenter read, therefore, was just nothing but random noise. The problem is, had he not been using a dead fish, he would never have questioned the result.
The paper goes on to list several other problems associated with fMRI experiments. For example, because of the nature of fMRI, the researcher cannot measure activity over the entire brain at once. He or she has to focus on a specific area. That, in itself, produces a bias in the results. Also, because of the nature of these kinds of experiments, they are rarely repeated. In some of the few cases where experiments have been repeated, the two trials did not produce consistent results. The paper even says that recent research casts doubt on the fundamental (and very reasonable) assumption that increased blood flow means increased neuron activity.
In short, then, an experimental technique that has been used for 20 years to make definitive statements about brain function is, to some extent, inherently flawed. Indeed, according to the article, a review of all papers using fMRI techniques published in Nature in the year 2008 found that almost half of them contain significant methodological errors.
So what’s the point? Is science flawed? Of course it is, but that’s not news to any scientist. Because it is flawed, is it useless? Of course not. Science makes lots of mistakes, but over time, it produces a lot of reliable, useful information. The point is simple: Scientific conclusions are based on assumptions. Sometimes those assumptions are testable, sometimes they are not. Sometimes, they are testable, but no one really thoroughly tests them. As any good scientist knows (but few in the general public understand), this makes scientific conclusions very tentative.
The fact that a whole lot of expert scientists believe something, therefore, means rather little. A whole lot of scientists who are experts in brain research believed the conclusions of fMRI studies, and now we see that a significant number of the studies that led to those conclusions are flawed. As a result, a significant number of the conclusions are probably incorrect. Because of such events, a good scientist continually checks assumptions against the data, regardless of how many scientists believe the assumption. As a frequent commenter on this blog (Norwegian Shooter) recently found out, for example, simply listening to the majority of scientists can often make you dead wrong!
Of course, being told by a bunch of scientists what to believe is much easier than actually thinking for yourself. Unfortunately, the easy way is rarely the best way.
1. Laura Sanders, “Trawling the Brain,”
Science Science News 176:16-20, 2009.
Return to Text