I hate to give press to a “research firm” that doesn’t know the first thing about reporting statistics or basic methodology in their own “research” report. I guess that’s what happens when you get a bunch of people together who are mostly technologists, not statisticians or social scientists.
This past week, MyType, a Facebook personality application that takes your data and then sells aggregated reports based upon your answers to their quiz, released a report about the iPad. They suggested that iPad owners and those looking to buy one were “selfish elites” while those who were iPad critics were characterized as “independent geeks.”
You can already tell that this isn’t exactly going to be a scientific analysis, right?
First off is MyType’s reporting of its results. What isn’t included in the actual report are the actual statistical tests used, statistical significance (p values), and numbers of subjects in each group studied. You know, the basic kinds of data other researchers are looking for in which to evaluate the results presented. Without that basic information, this kind of “research” is no better than an undergraduate’s paper for their Psychology 101 class.
A person by the name of John Hall called them out in their comments section, at which point they started giving broad statements about p values (but again, no specifics, breaking each analysis down, which is what you’d expect in market or scientific research). Dr. F tried educating the MyType people about basic statistical theory, since they apparently don’t know much about how to design a good research study.
What’s also clear from the comments section is that someone is editing the comments to remove things they later regretted saying. John Hall quotes apparently Tim Koelkebeck saying that because he has a computer science degree (apparently a bachelor’s), that somehow qualifies him in research design and methodology as well as statistics. Odd.
But the biggest giveaway that this isn’t a serious research study is the adjectives chosen to describe the two groups of people. They took some arbitrary personality traits to describe each group. They could have just as easily said that non-iPad owners are neurotic, insecure, proud and careless. Is that a legitimate characterization of non-iPad owners, though? Of course not, that’s only what their tiny, biased dataset shows. Given that doesn’t even pass the smell test, it would’ve made most researchers pause to wonder if they were going down the wrong road.
Going by their own data, they could have characterized iPad critics to be intolerant, dislike charity and friendship, aggressive and cynical teenagers. Wow, no surprise there — you’d have to tease out whether gender or age were skewing these results (and offer an alternative explanation to the traits noted). The “researchers” could have also described iPad owners as family-oriented, smart, high-achieving adults who are both imaginative and sophisticated.
But they didn’t. They tried to summarize a bunch of disparate traits into catchy marketing phrases to make news headlines — phrases that were neither particularly accurate, nor particularly scientifically valid. That didn’t stop dozens of legitimate, mainstream news outlets like CNET, ZDNet, and even Wired from reporting on the results without raising an eyebrow about the lack of scientific rigor or method.
Read their “conclusions:” iPad Personality Clash: Elites vs. Geeks