Psych Central’s Online Resources Review Guidelines
We’ve been indexing and reviewing online resources since 1992, beginning with Internet newsgroups in psychology and support topics (like support groups for depression). In 1995, we expanded my efforts to include websites and in 1996, we formalized internal guidelines for such reviews that our resource editors (at the time, when I was working for another company) used to evaluate and rate websites. Yes, this was long before HONCode and a year before the American Medical Association (AMA) released their own guidelines and criteria, because we recognized early on that with billions of pages of information out there, consumers (and professionals) would need some guideposts along the way. Psych Central has always acted as that guidepost.
Over the years, we’ve informally discussed many of these guidelines, but we’re now publishing them for others to make use of them. Perhaps it will better one’s own efforts at recognizing “good” sites from “bad” sites.
A Word of Caution
No matter what type of guidelines a person or organization may publish, it is perfectly easy (and becoming easier) to publish a website with a particular agenda that adhere to every single guideline we and others publish or recommend, and still be 100% wrong and a very unhelpful resource (while posing as otherwise).
We run across these sites everyday in our online travels. Perfectly legitimate-looking websites with perfectly legitimate-looking designs, navigation, articles and other content, while offering biased or misleading information. How is this possible? Because in health and mental health, biases are subtle and rarely stated. For instance, a researcher who has spent their entire lives pursuing one possible explanation for the cause of depression has a lot of their time, ego, and publications invested in such an explanation, regardless of whether it has enough data to back it up. But the researcher isn’t going to say, “Hey, make sure you comb through my data results carefully (and don’t just take my word for it in the discussion section of my peer-reviewed, published research study), because it’s easy to portray data in ways non-statisticians may not fully appreciate.”
Websites are even more challenging, because you may never even know who’s publishing the site or what their intentions are. I know, because I’ve spent the past 16 years in this industry and know this specific niche like the back of my hand. I have a pretty good understanding of every major mental health resource out there, and can delineate their positives and negatives and even take a stab at their biases (as others can easily do so for Psych Central). But trying to explain all of this is a 300-word summary of a site is nearly impossible (though try as we do). The big mental health portals are easy, because I either ran them myself at one time or did consulting work with them in one form or another. It’s when you start digging down into smaller websites, many of which are just run by ordinary people, that it gets more and more difficult.
The History of Online Ratings
We actually began the ratings of websites in 1995, but didn’t put it down into writing to follow until 1996 when we hired others to help us keep up with all of the online resources available. We did this because we knew we had industry-specific mental health knowledge that folks like Yahoo didn’t have and didn’t care about (at the time, the big online player in search via its directory, which is still available, but largely overlooked these days). We also recognized that health and mental health concerns weren’t just like other information online — bad or poor information could ostensibly hurt someone.
So when we moved to rating websites, we would rate them on content as well as their presentation. There were so many poorly designed websites in this timeframe that while they may have had really good information, it may have been difficult or distracting to actually access. We meticulously reviewed Web resources and gave them an appropriate rating based upon four main rating categories: Content, Presentation, Ease-of-Use, and Overall Experience.
In hindsight, of course, presentation and ease-of-use, while nice, are not really vital if what they are publishing is a load of bull. You can have the prettiest, most Web 2.0 website available, and it can still publish stuff that has little to no basis in research or facts. Combined with MFA websites (Made for Adsense, a Google advertising program), we now identify a lot of bad websites with these common attributes — a slick, clean “Web 2.0” design, Adsense ads, and content with no research basis. We most often find these sites linked from social networking websites like Digg.com, which is a social networking blog that allows its users to rate articles based upon popularity. In fact, we don’t think it would be difficult to find an inverse relationship between popularity (as measured by a site like Digg.com) and legitimacy of their health or mental health content.
Our General Guidelines
Our guidelines have been developed over the years with an emphasis on content and understanding attribution (who is publishing what you’re reading and why). We have developed more of a secondary emphasis on design and usability, since we find most modern websites don’t suffer from the serious usability issues of earlier sites that were commonplace in the late 1990s.
We break our system down into Primary Criteria, Secondary Criteria, and Overall Impressions. Primary Criteria are of primary importance when reviewing and rating websites, because if a site fails one of these criteria in a significant way, there may be serious problems with the site. Secondary Criteria are of less significance, as they have to do with a site’s look and feel, and usability. Overall Impressions take all of these criteria together to form an overall judgment about a site.
Content is the meat of the site. It is the breadth and depth of the material, no matter what the subject matter, provided through that site. Content may be reading material, information, contact information for real-world organizations, and links. Questions to ask when rating content:
- Does the site add new and unique material to the online world, or does it simply list or link to existing material well-covered by other sites already? (e.g., we’ve seen hundreds of websites that publish little more than NIMH or NIH content, content freely and readily available already from the NIMH or NIH)
- Does it offer new insights into a disorder, or offer information not found elsewhere? For instance, a patient’s story is more interesting to others than yet another “new breakthrough” suggesting vitamins or a health product will solve all of a person’s ills.
- Does it offer refreshing perspectives, and/or regularly updated new content in the form of news articles, opinions, or other forms of communication?
- Does it include interactive features that help provide more information to an individual that otherwise would be difficult (or impossible) to provide?
- Is it just an advertisement for an organization or company or individual?
- If the site has articles that discuss research, is that research cited appropriately and fairly reliably?
- Is the content written for a consumer audience or a professional audience? Does it meet minimal grammar, spelling and English standards?
Attribution, Transparency and Biases
Everyone has their own biases, from pharmaceutical companies and the NIH, to reporters and doctors who treat people with depression. The key isn’t to look for people or companies without biases, but to look for them that clearly talk about their biases and acknowledge them up-front. Editorial content must be separated from advertising, which must also be clearly labeled and defined as such (including “sponsored content”). If a site is a larger publisher of information, we look for an editorial policy and one reaffirming their editorial independence from financial influences. We also look for sites that keep their information updated and attributed. Sites that don’t put authors’ names or dates on their articles usually don’t want you to know.
- Is every article or entry on a website authored and properly attributed?
- Does every article or entry on a website have a date?
- If an article is modified or updated, is the “last modified” date noted?
- Is advertising and advertising units on each page clearly defined and delineated, and not mixed in with editorial content? (e.g., advertising should not be mixed inline with editorial content to try and confuse readers to click on the advertisement because they mistakenly think it’s a part of the site’s content)
- Does the site offer “Sponsored health centers” or similarly labeled content that is provided by advertisers, but published under the site’s logo and in the site’s templates to suggest it is editorial content?
Community and Social Networking
As social networks have become all the rage, lots of people flock and enter in private health data into such networks without sometimes clearly understanding the ramifications of such behavior. We look for sites that help people understand how such data may be indexed in a search engine for all time. Because health information can affect future employment and other major components of a person’s life, we believe health sites need to be more aware (and help their users become more aware) of the ramifications of such information being shared.
- Does the site offer a way for a user to delete their user profile and all of their contributions to the website if they so choose?
- Who owns a user’s contributions to the website? (e.g., does the individual retain ownership and copyright of their own contributions, or do they become a part of the site and can never be removed?)
- Are user contributions, a user’s profile and other possibly personally-identifiable information indexed in search engines?
- What types of user generated content is and is not allowed? Is such a list readily defined and available? Is such content appropriate for the type of health or mental health concern?
- What user protections are available to reduce or address cyberstalking and other user harassment concerns?
- Who runs the community aspects of the site? Are support staff available 24/7 or only during business hours?
- What is the site’s disaster and backup plan? Do they even have one, or share it with their users?
Presentation is, at the heart of it, how good does a site look. A very plain site with little or no graphics can score just as high as a site with loads of fancy graphics, if it is well-arranged and -designed. Usually, though, most sites which score higher on presentation do so because of well-designed graphics with limited or no use of animated graphics (which most users find annoying).
- Is the information laid out in a logical and well-organized manner?
- Are graphics designed appropriately for the site and do they load quickly?
- Do the graphics overwhelm the user and the content?
- Is the site advertiser-sponsored (and hence, adding to the “busy-ness” of each page which sports a banner advertisement)?
- Is it arranged so the best material the site has to offer is clearly delineated from other, less-important material?
Ease-of-use denotes how easy it is for you to move around the site and find specific information relatively easily and most of all, quickly. Large sites should have or use a search engine to help you find information in this manner. All sites should be logically arranged and organized. Often it helps to pick a specific piece of information you would hypothetically be searching for. For instance, on a site devoted to depression, perhaps we’d be interested in learning more about what makes someone depressed or what treatments are available for it. Contact information, in the form of at least an e-mail address or a link to something similar, should be provided on every page.
- Can you find our way to X content quickly using the navigation aids provided, and then get back to the home page as painlessly?
- Is a search engine or utility provided to help you find information more quickly (when appropriate)?
- Can you easily find contact information on the page to provide feedback about broken links?
This category takes into account the previous categories and allows you to evaluate your overall experience and feelings about the site. Although this is a relatively subjective rating, you should be able to make it less subjective by simply taking into account the 3 previous categories.
- What kind of feelings did you have when we came away from the site?
- Was it enjoyable to read through or was it painful?
- Were you expecting more and became disappointed by the content or presentation or difficulty in navigating through the site?
- Would you bookmark the site yourself or find yourself wanting to return on a regular basis?
- Would you recommend the site to a friend or family member?
Other Codes and Rating Systems
All rating systems have their pros and cons; therefore no single system will ever be adequate for a consumer to rely on. For instance, we’ve discovered troubling and inconsistent policing of the HONCode, finding it on many websites that don’t actually subscribe to all of the 8 principles. When complaints are filed against a site, action may or may not be taken. Eight months after filing a complaint for one popular site, we still note that most of their articles don’t have dates or author attributions, two important components of the HONCode. We have found the Health on the Net Foundation (HON) is largely unresponsive to inquiries.
One of the things we encourage consumers to do is to review multiple recommendations and guidelines (e.g., from us, HONCode, the AMA, and others), and develop their own set of criteria that works for them.
Disclaimers and Considerations
Even our rating system is far from perfect. Websites are constantly undergoing revision and a site we rated even a month ago can be completely changes and turned into some sort of spam site overnight (which we’ve seen happen more times than we can count). Other times, even when you make your best efforts to figure out what a site is about, you can still make mistakes because ultimately, we’re human and so are the judgments we’re making about other sites. As humans, we have and will continue to make mistakes.
But we’re committed to doing the best job possible, given all the information we can read and glean from any website we review.
One of the things to be aware of is that an article may follow all of these guidelines to a tee, and still not be accurate. Why? Because large publishers know about these guidelines and publish their articles to publicly adhere to them, without putting the mechanisms in place to actually follow them. For instance, I’ve consulted with large online health companies where they claimed every article on their website was reviewed by a physician or medical director, but few, if any, really were. Nobody can tell you that a company is actually doing what they’re claiming to do, so at the end of the day, you will always be left with imperfect information based upon company’s own self-reports (and perhaps an external verification or two).
We hope you find this information perhaps helpful in your own online research and travels.
Grohol, J. (2016). Psych Central’s Online Resources Review Guidelines. Psych Central. Retrieved on October 19, 2017, from https://psychcentral.com/lib/psych-centrals-online-resources-review-guidelines/