This is a lesson in creating valid scientific instruments that actually measure something, rather than providing pretty numbers that purport to measure something. Infoworld is one of a number of weekly IT magazines that is distributed freely to IT professionals working in the IT world. Some cynics would argue that these types of periodicals are nothing than a set of repackaged press releases from the computer and software companies. However, Infoworld purports to be different in its Test Center reviews.

Background

Infoworld’s Test Center features a set of product reviews (of computer equipment and software) in nearly every weekly issue, many times pitting similar products from different manufacturers against one another. Surprisingly, there’s little information on the Infoworld website about the “Test Center,” its purpose, the objectivity of its reviewers, whether Infoworld has to purchase the items it tests (ala Consumer Reports or whether they are provided free of charge to the company), etc. We did, however, find this note in the About Us section of their site under “Editorial Philosophy” (do they teach a course on that in journalism school?):

InfoWorld provides in-depth technical analysis on key products, solutions, and technologies for sound buying decisions and business gain. InfoWorld.com is the place to turn for the latest breaking news and in-depth coverage of the issues, trends, and products that run your enterprise. InfoWorld.com also features interactive discussion forums, trusted industry columnists, and incisive product test results and reviews backed by the renowned InfoWorld Test Center.

According to the Test Center scoring list, products are ranked as follows:

• 8.7 – 10.0, Excellent
• 8.0 – 8.6, Very Good
• 7.0 – 7.9, Good
• 6.0 – 6.9, Fair
• 5.0 – 5.9, Poor
• 0 – 4.9, Unacceptable

The first clue that something may be a little unbalanced at the Test Center is that the score ranges for each category are unequal. If this scale was derived from some sort of statistical procedure, that’s understandable. But there’s no information how this scale was derived, and it could’ve just as easily been pulled from some editorial meeting where a bunch of writers came up with it. By definition, however, we would expect that any product that is commercially viable is not going to be “unacceptable” nor likely to be tested by the Test Center. So that makes half of the scale useless as any sort of meaningful rating, since no product is likely ever to obtain a score below 5.0.

Realistically, then, all products tested will fall into one of five categories, three of which are positive (excellent, very good, or good), one neutral (fair), and one decidedly negative (poor).

Methodology

We visited Infoworld.com and clicked on the “Test Center” tab. Then we chose the “Products Review Guide” sub-tab. Going back to February 25, 2005 (the earliest reviews we could pull from its website, although it’s not clear why older reviews are not available), we did a simple analysis of the results of Infoworld’s tests by choosing its “Product Reviews Guide.” We counted for all products listed, those that scored Excellent, Very Good, Good, Fair, Poor and Other (Beta or 0.0 scored products).

Results

We found 85 reviews listed. Of those reviews, 24 products were rated “Excellent,” 38 were rated “Very Good,” and 19 were rated “Good.” We could only find 2 products rated “Fair” and no products that were rated “Poor” (although two were rated 0.0 and one was marked “Beta”). We also found no products rated “Unacceptable.”

Discussion

From this data one can surmise that if you submit your product for review to Infoworld, you have a 73% chance of receiving either a “Very Good” or “Excellent” rating! Those are pretty good odds for any vendor. (Settle for just a “Good” rating, and your chances increase to 95%.)

Infoworld also publishes a weekly “Leaderboard” which appears to be a sampling from their Test Center reviews. In a recent issue, for instance, they listed “Hardware Servers.” All six servers listed received an “Excellent” with a range of only 6 points between all products tested. In the same issue, another category listed “Hardware Printers: Monochrome” and 14 products were listed. 9 were listed as “Very Good,” 1 was “Excellent,” and the remaining 4 were “Good.”

If we were looking at a normal bell curve distribution of this data, you would see the vast majority of products that Infoworld tests are “Very Good,” with a smaller minority of products either being “Excellent” or just “Good” on either side of the distribution.

Don’t get us wrong — the narrative test reviews of the products reviewed by the Test Center are usually excellent and well-written. But it’s pretty obvious their scale and scoring system needs an overhaul, as these distinctions between products have become meaningless. If the scoring system is supposed to be a short-hand for the quality and usefulness of the product (as determined by the narrative review), then it has lost its meaning. If every product you test is pretty much guaranteed to be at least “Good,” then most vendors can’t go wrong submitting their product for review.

We have to wonder though, if all this stuff is good, then why do people continue to have so much trouble with computer hardware and software? Is Infoworld only testing the best of the best??

But the most important question to ask of all of this is, What is the point of a rating system where it appears everyone’s getting an A or B? Where are all of the C, D and F products to balance these reviews out??

This only goes to show that even established publishers can simplify things to the point of the data losing meaning.

As an aside, it’s interesting to note how Infoworld blurs the line between editorial and advertising content on its website, even in the Test Center reviews table (which is an editorial table). Every entry includes a column for “Special offers,” which is simply advertising tied directly to that product. Makes you wonder…

Dr. John Grohol is the CEO and founder of Psych Central. He is an author, researcher and expert in mental health online, and has been writing about online behavior, mental health and psychology issues -- as well as the intersection of technology and human behavior -- since 1992. Dr. Grohol sits on the editorial board of the journal Cyberpsychology, Behavior and Social Networking and is a founding board member and treasurer of the Society for Participatory Medicine.

Like this author?
Catch up on other posts by John M. Grohol, PsyD (or subscribe to their feed).

Comments

This post currently has one comment. You can read the comments or leave your own thoughts on our new comments page.

Trackbacks

No trackbacks yet to this post.

    Last reviewed: By John M. Grohol, Psy.D. on 23 Jun 2005
    Published on PsychCentral.com. All rights reserved.