Quality on the Net:
Many Standards, No Conclusive Criteria

By John M. Grohol, Psy.D.
April 1999; Reviewed: February 2004, August 2007

Determining what is a high-quality health or mental health Web site online nowadays is a daunting task (Jadad & Gagliardi, 1998). If you've spent more than a few hours surfing the Web, you've already discovered the great variability between well-crafted, high-quality sites, and sites of lesser quality. This article attempts to cull some of the standards put forth by various professionals organizations for gauging somewhat easily whether a site is worth your time or not.

The Health Information Technology Institute developed a white paper on this very issue entitled, Criteria for Assessing the Quality of Health Information on the Internet (HITI, 1997). They've broken down the problem into the following categories:

2. Quality Criteria for Evaluation of Health Information
C1 Credibility
C1.1 Source
C1.2 Context
C1.3 Currency
C1.4 Relevance/Utility
C1.5 Editorial Review Process
C2 Content
C2.1 Accuracy
C2.2 Hierarchy of Evidence
C2.3 Original Source Stated
C2.4 Disclaimer
C2.5 Omissions Noted
C3 Disclosure
C3.1 Purpose of the Site
C3.2 Profiling
C4 Links
C4.1 Selection
C4.2 Architecture
C4.3 Content
C4.4 Back Linkages and Descriptions
C5 Design
C5.1 Accessibility
C5.2 Logical Organization (Navigability)
C5.3 Internal Search Engine
C6 Interactivity
C6.1 Mechanism for Feedback
C6.2 Chat Rooms
C6.3 Tailoring
C7 Caveats
C7.1 Alerts

Items like Accuracy are nearly impossible to define in the world of behavioral healthcare, where even professionals and academicians disagree about the most basic of issues (e.g., the most appropriate care for a given person, valid treatment options, etc.). Anything which doesn't fit within the neat confines of Western idealogies surrounding medical and health treatment also don't do well under this model. For example, Eastern philosophies and methods of treatment simply wouldn't hold up to the Western standards of measurement. Yet such Eastern methods of care are widely viewed by many international professionals as holding a great deal of treatment and curative value. There is value and limitations to any method, and the scientific method embraced by Western culture is no different. When dealing with as complex an organism as the human body, and especially the least understood organ, the human brain, it helps to keep an open mind. Under criteria like these, developed under a specific theoretical model of healthcare and treatment, many worthwhile information resources online may be incorrectly categorized.

Such detailed criteria as the HITI categories are interesting reading, but make for difficult, timely quantitative judgments. It would take an average person up to a half-hour to evaluate all of these criteria objectively and adequately. What a Web reader needs is some quick and reliable indicators for quality content.

The American Medical Association considered the problem and wrote an editorial about it back in April, 1997 entitled, Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet (Silberg, Lundberg, & Musacchio, 1997). They offer a core set of standards in which to evaluate medical and health information online:

Authorship: Authors and contributors, their affiliations, and relevant credentials should be provided.

Attribution: References and sources for all content should be listed clearly, and all relevant copyright information noted.

Disclosure: Web site "ownership" should be prominently and fully disclosed, as should any sponsorship, advertising, underwriting, commercial funding arrangements or support, or potential conflicts of interest. This includes arrangements in which links to other sites are posted as a result of financial considerations. Similar standards should hold in discussion forums.

Currency: Dates that content was posted and updated should be indicated. (p. 1245)

These criteria are much more succinct, and therefore easier to apply in your day-to-day surfing online. Unfortunately, they do not really address issues of quality. One could easily publish a Web site full of quackery, yet readily meet each of these standards.

There is little agreement between Web sites themselves what constitutes quality. Content makes up only one category out of six in the Six Senses review criteria, and one out of three in our own ratings criteria. Medical Matrix uses a form consisting of six categories as well, two of which appear to address content issues directly. Physicians' Choice utilizes five separate, subjective categories, most of which emphasize content-oriented aspects of the site. Other organizations, such as the Internet Healthcare Coalition and the International Society for Mental Health Online would appear to offer the hope of some greater standards or review criteria to add to the mix; however, nothing has been published by either of them on this subject. The Health On the Net's (HON) Code of Conduct, outlining 8 principles of conduct of medical or health-oriented Web sites, is one of the oldest such attempts at defining minimum standards, much like the AMA's editorial does. One would think the American Medical Informatics Association would be on top of this issue, but their own working group's Web site on the Internet hasn't been updated in over a year. Nor could we find any white papers or articles on this issue on their Web site. Not an encouraging sign. A MEDLINE search turned up other criteria which have also been suggested (e.g., Pealer & Dorman, 1997).

Nobody is trying to suggest that any one of these ratings criteria is complete or sufficient. Given the disagreement about how such sites should be rated at all amongst such organizations which do this kind of rating, it is not surprising that anyone simply surfing the Web on their own spare time may find the whole process burdensome and overwhelming.

The fact is, there is no one set of standards which will work in any given situation. You can, however, use the AMA's set of core standards to keep in mind the legitimacy of any given site you're reading. If a Web resource fails to meet these 4 core standards, then it is unlikely to meet more detailed criteria, such as the HITI criteria.

We believe the most important of those four standards to be attribution. A person or Web site should give references for his or her own work on the Web, when presenting material of an informational nature. If no such references are given, you can assume that the material comes from the author's own head. If authorship is properly disclosed, you can then make the determination as to whether that's sufficient for meeting your minimum needs of quality, or whether you would like further information.

You can tell a lot from the types of references quoted for a stated position. If you scroll on down to the References section (assuming the online document has one) and find that most of what is quoted is their own work online, or some strange-sounding journal names, it may be that that is enough information to make a qualitative decision. For example, a hypothetical article supporting the use of EMDR, a therapeutic technique, might quote items from a hypothetical organization called the EMDR Organization. We would not be surprised that such an organization would support the use of its own techniques (e.g., self-serving, selective references). This says nothing about the inherent quality of the article (perhaps it is just an overview of the EMDR technique and history). You have to use such information in the context of what is being presented. Given the vast amount and types of information available online, there will be a lot of room for subjective, individual judgment here. There is simply no getting around that at this time.

You can't expect everyone who publishes a Web site today to conduct the equivalent of a comprehensive, objective review of the research literature, however. It just isn't an important aspect of most people's Web publishing efforts. You would expect that people use information from well-known publishers and organizations, however, such as the NIMH, the APAs, NMHA, or other well-respected institutions. Contrast these references to a hypothetical reference to Dave's Wacky Guide to Schizoaffective Disorder Online!

Disclosure is important; a Web site which suggests that it is the singular best resource for a mental disorder like depression should offer a vast array of information about the pros and cons of treatments available. That might not be the case if that Web site was funded, let's say, by a pharmaceutical company. (This particular issue varies from pharmaceutical company to company... Some are very good about bringing in the wide range of objective materials and research, while others are blatantly using the site as a marketing vehicle for their own medications.) Currency is the least important, since information in the mental health and behavioral health field doesn't go stale very quickly. New treatments do become available, and should be so noted as appropriate on a Web site. Most treatments for most mental health problems, however, have been around for years. The most significant changes in mental health treatment usually surround the introduction of a new class of medications. So it should not be shocking or even surprising to come across a site which may have not updated certain materials every month.

Last, beware of any site which suggests it is the end-all, be-all for accurate and respected health or mental health information online. One such site simply doesn't exist. The richness of the Internet is defined by its vast and varied content. Explore, be skeptical, and keep these criteria in mind. They should act as a good starting point in evaluating the quality of health and mental health information online.

References

Health Information Technology Institute (HITI). (1997). Criteria for Assessing the Quality of Health Information on the Internet (Draft). Online at: http://hitiweb.mitretek.org.

Jadad A.R., & Gagliardi A. (1998). Rating health information on the Internet: navigating to knowledge or to Babel? The Journal of the Americal Medical Association, 279 (8), 611-614.

Pealer L.N. & Dorman S.M. (1997). Evaluating health-related Web sites. Journal of School Health, 1997 67(6) 232-235.

Silberg, W.M., Lundberg, G.D., & Musacchio, R.A. (1997). Assessing, Controlling, and Assuring the Quality of Medical Information onthe Internet: Caveant Lechtor et Viewor (Let the Reader and Viewer Beware). The Journal of the Americal Medical Association, 277 (15), 1244-1245.

Reference
Grohol, J.M. (Apr 1999). Quality on the Net: Many Standards, No Conclusive Criteria. [Online].

Last reviewed: By John M. Grohol, Psy.D. on 6 Sep 2007
    Published on PsychCentral.com. All rights reserved.

 

 

Understanding is the soil in which grow all the fruits of friendship.
-- Woodrow Wilson