Bad Research: A Comparison of iPad, Kindle and Book Reading SpeedsWe love a usability study as much as the next person. But we love well-designed, elegant studies that rightfully point out their own limitations and are printed in peer-reviewed journals most of all. We have less love for studies that act as propaganda, or researchers who draw conclusions not supported by their own data.

Bad research can be found anywhere — even by consultants who make their living from doing research on usability. Well-meaning folks like Jakob Nielsen for one. Recently he released a study on his website on the usability of the Kindle, the iPad, a PC and a book for reading a piece of short fiction.

After noting there was no statistical difference between reading on the Kindle or iPad, and then noting that the data did not reach statistical significance for the iPad versus a book, the authors of this report still declare that “Books Faster Than Tablets.”

Well, yes, that’s true on the face of it. But the point of research and statistical analysis in the first place is to go beyond what seems to be true and see if the difference is meaningful or not. After all, data may look like they mean something, but if the statistics don’t back it up, then the appearance of meaning is just an illusion. One that shouldn’t be emphasized in one’s sub-titles, since it’s misleading.

In fact, the data from this particular study found that only reading on the Kindle was statistically different than reading a book. But that’s a far less sexy conclusion than the broader, “Books Faster Than Tablets.”

Alternative explanations for the results were not offered in this study, a glaring omission. The most obvious alternative explanation for the results found is simply that folks have a great familiarity and experience with reading the printed word in a book — often times decades’ worth! On the other hand, the time and experience spent reading from a newer device — whether it be a Kindle or iPad or even the PC — has been extremely limited. In any case, people have not spent decades reading books from one of these devices. So it could simply be a matter of familiarity — books are easy, familiar ways to ingest reading material. The Kindle and iPad are not (yet).

This alternative explanation also explains the difference between the Kindle and iPad data findings (the Kindle was statistically significantly slower than a book, while the iPad was not). The iPad uses a screen that most people are already familiar with, and Apple’s usability expertise in these kinds of hand-held devices is well-recognized. The Kindle, on the other hand, uses a screen technology few people have ever encountered. And while far more usable in everyday situations (such as reading outdoors or not worrying about having to recharge it everyday), the Kindle is simply a more unfamiliar device using unfamiliar technology.

Limitations of the study include its design and subject pool. While 24 subjects may give us sufficient power for statistical analysis, people’s reading styles are different enough that a larger experiment might give us more robust and conclusive results. Asking a person to read the same 17 minute story four times in a row also seems less-than-ideal. While the order of devices in which they read on was randomized, I can’t help but think how boring and monotonous it would be to read the same story over and over again, 4 times in a row.

Offering different types of reading material — both fiction and non-fiction — would also rule out any possible effects that the type of content being used as a measure is not inadvertently affecting reading speed too. Hemingway — the author used in the current research — is a great author, but I’d hardly say he’s to everyone’s liking or taste. Reading material you have little interest in may also impact reading times.

What this study demonstrated was that different table computing devices for reading a work of short fiction don’t significantly impact reading times for those with no specific familiarity with the devices (except on the Kindle, where reading time was significantly slower). Not surprisingly, the iPad, Kindle and book all had similar user satisfaction scores, demonstrating that the devices have no significant usability issues that detract from an end user’s satisfaction of using them.

Read the full study: iPad and Kindle Reading Speeds (Jakob Nielsen’s Alertbox)