YOUR TRUST
It's Being Violated in Subtle Ways

by John M. Grohol, Psy.D.
March 1, 1999

There are two things I'd like to talk about in this essay -- two Web sites which are trying to provide a helpful service, but which don't disclose enough about the limitations of the services they are providing. Names aren't important; what is important is that you be on the lookout for these types of violations of your trust and recognize them when you see them.

Biased Research Results
The first site provides individuals with direct searches of the MEDLINE biomedical research database for common mental disorders, such as depression or panic disorder. It has pre-defined terms embedded in one-click links that allows you to search MEDLINE quickly and easily. It's a powerful resource and tool. But it has one serious limitation which isn't mentioned anywhere on the site.

 MEDLINE is a biomedical public research database. It covers only a smattering of the hundreds of journals which focus on human behavior, psychological research, and behavioral healthcare treatment. These journals publish thousands of potentially useful research articles on topics directly relevant to those pre-defined terms mentioned above. But MEDLINE doesn't include them.

 It would be fine for any site to provide such helpful links to MEDLINE. But they should also include a disclaimer to the effect that these research results are not representative of the scientific field as a whole because of the limitations on the MEDLINE database. Psychologists and scientists within the behavioral research field don't generally even use MEDLINE to do literature searches and find information relevant to their work as a researcher or practitioner. They use a completely different database which focuses on the psychological and human behavior research field. This database, called PsycINFO(tm), is unfortunately a proprietary, commercial database which is not available online for free. You should have the whole picture, however, and any serious research done online needs to take this factor into account.

Tests, tests, tests
A popular site for online "psychological" tests which purportedly measure everything from your EQ to facets of your personality has a few problems as well. The first problem is that the tests on this site were simply created by a single individual and most of them have not undergone the traditional peer review process which verifies that the tests you are spending a lot of time on are indeed valid measures of what they say they are.

 I suspect one of the reasons the author hasn't vetted these tests in any traditional professional forum is because many of them simply wouldn't stand up to our field's scientific rigor. Some of them are significantly lacking the statistical significance which is needed to ensure that they can discriminate amongst the various kinds of variables they say they are measuring.

 The author, for some unknown reason, hides behind a pseudonym on the site. Her degrees are in the field of psychiatry, apparently lacking formal training in psychological test development which is standard for psychologists.

 The psychometric properties of her tests (published only on her site) show questionable attributes. For instance, her EQ tests shows that the test scores' correlations are low across real-life measures, such as personal, social, or professional success. What this means is that your score on the EQ test correlates next to nothing in your real life (accounting for little of the variance of these areas). The extroversion/introversion inventory shows virtually no differences between test scores for genders, contrary to the published literature. The psychometric report for this measure also doesn't seem to accurately reflect the actual test, which has additional unnumbered questions at the end. The self-esteem inventory is only slightly more convincing, but is still weak. Other tests all seem to suffer from flaws which draw into question the generalizability and usefulness of the results given.

 The psychometrics reported, while interesting, aren't as telling as the psychometrics not reported. No test-retest statistics are reported, so we don't know whether any test is measuring a stable trait or variable state of the person. Nearly all of the tests lack a measure of content validity and criterion validity -- statistics which ensure the items on the test actually measure what they purport to measure. Where were the test items derived from? An empirical pool of items, or just randomly created by the author? In taking a number of the tests, I found many responses very similar (and confusingly similarly-worded). In other questions, I found the responses available didn't capture the accurate response I wanted to give -- I had to give an inaccurate response. While poorly designed tests have this characteristic, well-designed instruments do not. I guess you get what you pay for (in this case, free)!

 What is just as disturbing as the variability in test quality across the measures offered is the author's marketing practices. Under the media kit section of the site, in the Ad Prices category, the site makes the following offers to advertisers:

Special Audience Level #1
Dynamically generated pages that are returned to visitors after submission of a test for scoring. These pages generally contain test scores with short interpretation. Banner is visible on the screen without or with minimal scrolling.

Very high clickthrough rate (4 to 10 %); excellent visibility; visitors at the peak of their attention curve

Especially suitable for marketing of one-on-one services (counseling and consulting), self-help material, health products and services

Custom-selected Audience
Dynamically generated pages that are returned to VISITORS WITH A SPECIFIC PSYCHOLOGICAL PROFILE after submission of a test for scoring. These pages generally contain test scores with short interpretation. Banner is visible on the screen without or with minimal scrolling.

Very high clickthrough rate (over 10 %); custom-made campaign according to the profile of your typical customer; visitors at the peak of their attention curve; surprisingly cost-effective

Especially suitable for marketing of one-on-one services (counseling and consulting), self-help material, health products and services

Without telling visitors to her site, the author is attempting to manipulate her users' behavior through these kinds of offers. The site doesn't tell test users that their scores will be used to market products or services to them. While this is great for advertisers, it appears to be taking advantage of users who generally are not privvy to this information.

 I find such kinds of marketing practices to be underhanded, deceptive, and unethical. Anybody who will try and basically sell your test scores to advertisers is somebody I'd rather not take a test for, even "just for fun."

 And that's the real problem here. Most people who come to a site like this are looking for additional information about themselves. I'm not sure whether they take their responses to the tests seriously or not, but they are nonetheless devoting a sometimes-significant portion of time (some of the tests on this site may take 20-30 minutes to fill out!) in doing so. If users are not told that (a) the tests may not be accurately measuring what they purport to measure and (b) their scores will be used to direct-market to them, then they are not being given the whole picture. Or the truth.

Summary
I'm all for these kinds of interesting and potentially-useful resources online. But the publication of these resources requires a certain level of responsibility and implicit trust between the Web developer and you, the user. This responsibility and trust is defined by letting you know about the proper uses and limitations of the resources they provide. Without this information, a user is left to assume, for better or for worse.

 In the first case illustrated above, the assumption is that the literature search of the research is complete, comprehensive, and up-to-date. In the second case, the assumption is that the tests are accurate and reliable, measure what they purport to measure, and that the user's privacy will be maintained. Clearly, in both cases, your trust is being broken. Being on the lookout for these kinds of violations of your trust, and giving the sites feedback when you find them, will ensure the integrity of future resources.

Last reviewed: By John M. Grohol, Psy.D. on 11 Mar 2009
    Published on PsychCentral.com. All rights reserved.

 

 

I do not think there is any other quality so essential to success of any kind as the quality of perseverance. It overcomes almost everything, even nature.
-- John D. Rockefeller