Most people gain value from reading reviews about products and videos and such online. We read a review of a book on Amazon.com and believe we can trust most of the reviews to provide a narrative that we either find helpful or not.
When it comes to product websites, and even some vendor sites like Circuit City, you get something less and you get something more. Usually the narratives are less detailed and informative, but you get a numerical rating which gleams respectability and validity. (To read about the truth of these numbers, however, read my concerns about the reliability and validity of online ratings in general.)
When we go to YouTube and see people rated a video 3 out of 5 stars, that tells us something instantly. Whether its reliable or not probably doesn’t matter so much, since a video isn’t likely to change any person’s life one way or another.
But when it comes to rating people on their livelihood, like a doctor (or CEO), should their entire life experiences and career be whittled down to a single number (or a bunch of numbers) that try and capture the wide range of variables in their career and professional lives?
Imagine someone who is an expert in the same field as you comes to your workplace, reviews your personnel file, and after a 30 minute interview, puts up a rating on an employee website about how well they think you do as an employee. Imagine employees rating their bosses, or managers rating their vice-presidents, or vice-presidents rating their CEOs. Or a company’s board of directors rating their CEO.
Yeah, we thought that might jog a few brain cells.
The Problem with Rating People on Their Jobs
The problem with rating individuals on their livelihood is that even if we were to spend hours with that individual, observing them do their daily routine, even for days at a time, we still wouldn’t be in a position to rate that individual on their work. Could you imagine someone boiling your work, your entire career, down to such a review? Very few people could. People have bad days. Heck, some people have bad years. We’re only human, after all.
We expect the law of averages to, well, average out the lows and highs in such online ratings, and that the “truth” will somehow be magically revealed by the sheer quantity of people rating a thing online. And for things, this may or may not be true, because there are literally millions of people who may own or use that thing. But there are not millions, or even thousands of people, who use or interact with a single person in their profession. Even a waitress only interacts with a few dozen people on any given day. Same with a doctor.
Now imagine that in hundreds of interactions, a single person decides to give that waitress or doctor an online rating or review at one rating website. One of dozens. Because the population of people who could possibly rate this individual are so much smaller, that one has a much bigger impact on that professional’s score. But do you know this, does anyone tell you this? No, instead rating doctors online is compared to rating Internet videos.
Yet the latest rage is to start rating people on their livelihoods. Students have been doing it with their teachers for years, and now companies want to start doing it with doctors. And not with discrete, detailed criteria, but with broad, sweeping generalizations and categories that group together things like “accurate diagnosis” with receiving “lifestyle changes” advice. Really? Are those two really in the same category? Or how about getting a response to one’s questions… Is that really in the same category as getting a timely appointment?
Rating doctors really shouldn’t be like rating an Internet video or whether someone is “hot or not.” A broad category of ratings shouldn’t place a doctor who provides accurate information but wasn’t very personable in the same place as a doctor who provides wildly inaccurate misinformation but was very agreeable and listened to all of my questions. The first is pretty much the norm for many doctors nowadays, while the latter is grounds for a medical liability lawsuit. Yet some websites would have you rate the two in the same category without noting any difference between these two very different issues.
But What’s Important to Me? It’s Supposed to Be About Me, Remember?
Worse yet, some of these online doctor rating sites don’t let people tell them what’s important to them in searching for a healthcare provider. For example, maybe it’s really important to me that a doctor returns my call in a timely manner and listens to my concerns. The site has this information, but it won’t let me search by this criteria. You’re left with the usual search by geographic region or by gross recommendations (e.g., the recommendation averages of all 1 or 2 users who’ve seen the same professional).
Should we really be showing ratings based upon a single data point or two? If this were a review of a book or Internet video, sure, why not? Who cares? But for someone’s livelihood? Should we be playing so fast and loose with the data??
What’s the outcome of these ratings efforts? Right now, I’d say they are pretty rough around the edges and showing us what Amazon.com, “Hot or not” or Circuit City showed us circa 2005.
But this is 2007 and people expect and want more. You can do doctor ratings well if you do them thoroughly and with respect to the profession. You can do doctor ratings badly if you treat it like rating Internet videos or rating a stove or have people who have little healthcare experience or background designing your ratings criteria. People don’t want unreliable ratings. They don’t want broad categories that don’t take into account their specific needs and wants. And they certainly don’t want any more unreliable data, given that most people have had enough of that in dealing with the health care system already. If an online rating system can’t do better than this, they should throw in the towel before the lawsuits begin.
Grohol, J. (2007). Hey, It’s 2007! Let’s Rate Doctors!. Psych Central. Retrieved on October 30, 2014, from http://psychcentral.com/lib/hey-its-2007-lets-rate-doctors/000905
Last reviewed: By John M. Grohol, Psy.D. on 30 Jan 2013
Published on PsychCentral.com. All rights reserved.