Print Reading versus Digital Reading: Which Produces Better Comprehension?

I saw this Science Alert article come across my Facebook feed a few days ago, and I read it with interest. Written by two researchers from the University of Maryland, it makes some pretty strong statements about the effectiveness of reading a digital article compared to a print article. Essentially, the researchers say that for specific kinds of articles, students’ comprehension is better if the article is read in print form as opposed to digital form. They make this statement based on a review of the studies that already exist as well a study they published two years ago. While I think they are probably correct in their assessment, I am struck by how small the difference really is.

For the purpose of this article, I will concentrate on their new study. In their Science Alert article, they refer to it as three studies, but it is published as a single paper. In the study, they had 90 undergraduate students who were enrolled in human development and educational psychology courses read a total of four articles: two digital and two in print. Two of them were newspaper articles and two were excerpts from books. They were all roughly the same length (about 450 words). They dealt with childhood Autsim, ADHD, Asthma, and Allergies. Presumably, all of those topics would be of interest to the students, given the classes in which they were enrolled.

Before they did any reading, the students were asked to assess themselves on their knowledge of the four topics about which they would be reading. They were also asked which medium they preferred to read: digital or print. They were also asked about how frequently they used each medium. They were then asked to read the articles, but the order in which the articles were read changed from student to student. Some would switch between digital and print, while others would read the first two in one medium and then the second two in the other medium. That way, any effect from switching between the media would not be very strong.

After each reading, students were asked to identify three things: the main idea of the article, its key points, and any other relevant information that they remembered from the article. The researchers had asked the authors of the articles these same questions as well as two independent readers. Those were considered the correct answers. Two trained graders independently compared the students’ answers to the correct answers, and the grades they assigned were in agreement 98.5% of the time. For the 1.5% of the time they didn’t agree, they then discussed the grading and came to a mutual agreement.

After all four readings and tests, the students were then asked in which medium they think they performed best. As you will see, that’s probably the most interesting aspect of the study.

What were the results? Students, on average, preferred the digital medium, but the amount of preference changed depending on the situation. Students heavily favored the digital platform for newspaper and academic reading. They also preferred the digital platform for “fun” reading (like reading on vacation), but not as strongly. What about the students’ comprehension of what they were reading? Here is the table the researchers present in their article:

These are the combined results, averaging all four readings done by each student. The “Mean Score” is the average, while “SD” is the standard deviation, which is a measure of how much variation exists among the subjects. When the standard deviation is low, the subjects scored very similarly. When the standard deviation is high, there are big differences among the subjects’ scores. The “Maximum Score” is what a student would get if he or she answered every question perfectly.

Now here’s the thing you have to understand about comparing people: since each person is different, you expect differences to exist naturally. As a result, if you do a study like this, some of the differences you measure between print and digital will come from the inherent differences among the students, not the differences between the media themselves. Those inherent differences produce what is referred to as statistical error. A nice rule of thumb is that the statistical error depends on the square root of the number of data sets you have. Take the square root of your number of subjects, divide by your number of subjects, and multiply by 100. That will give you a rough estimate of the statistical error in your study.

So, for this study, a rough estimate of the statistical error is the square root of 90, divided by 90, times 100. That’s about 11%. So, if the differences between the scores for digital and print are within 11% of one another, you can’t say they are different. They could very well be the same, and the differences you see could be the result of statistical error. Using this rough estimate, then, you can’t say that any of the differences between digital and print are real. All of the differences in the mean scores could very well be the result of statistical error.

So does that mean that this study doesn’t really say anything about the difference between digital and print? Not exactly. Since all three digital scores are lower than all three print scores, there might be some difference. If the error were truly statistical, you would expect that the digital score might be higher in at least one of the three cases. It is not. So in the end, there probably was less comprehension among the students when they read the digital articles, but the effect was pretty small (less than the statistical error).

Here’s the fascinating part of the study: The students were asked which medium they think gave them the best scores. They weren’t shown their scores. They were just asked what they thought would happen when their digital scores were compared to their print scores. 69% said that they scored better using the digital medium, while only 18% said that they scored better using print. The others thought that the medium didn’t affect their scores. So while the majority of students thought they were comprehending more using the digital medium, they were actually comprehending more using print! That result, more than any other, is the one that stands out to me. Why would students not be able to judge the medium from which they learned best?

In the end, I think this is a good study, but I don’t think the results are definitive. A similar study with a lot more participants might help better pin down the differences between print and digital media. In addition, a similar study should be done using longer articles to see what difference that produces.