Your Trust: It’s Being Violated in Subtle Ways
There are two things I’d like to talk about in this essay — two Web sites which are trying to provide a helpful service, but which don’t disclose enough about the limitations of the services they are providing. Names aren’t important; what is important is that you be on the lookout for these types of violations of your trust and recognize them when you see them.
Biased Research Results
The first site provides individuals with direct searches of the MEDLINE biomedical research database for common mental disorders, such as depression or panic disorder. It has pre-defined terms embedded in one-click links that allows you to search MEDLINE quickly and easily. It’s a powerful resource and tool. But it has one serious limitation which isn’t mentioned anywhere on the site.
MEDLINE is a biomedical public research database. It covers only a smattering of the hundreds of journals which focus on human behavior, psychological research, and behavioral healthcare treatment. These journals publish thousands of potentially useful research articles on topics directly relevant to those pre-defined terms mentioned above. But MEDLINE doesn’t include them.
It would be fine for any site to provide such helpful links to MEDLINE. But they should also include a disclaimer to the effect that these research results are not representative of the scientific field as a whole because of the limitations on the MEDLINE database. Psychologists and scientists within the behavioral research field don’t generally even use MEDLINE to do literature searches and find information relevant to their work as a researcher or practitioner. They use a completely different database which focuses on the psychological and human behavior research field. This database, called PsycINFO(tm), is unfortunately a proprietary, commercial database which is not available online for free. You should have the whole picture, however, and any serious research done online needs to take this factor into account.
Tests, tests, tests
A popular site for online “psychological” tests which purportedly measure everything from your EQ to facets of your personality has a few problems as well. The first problem is that the tests on this site were simply created by a single individual and most of them have not undergone the traditional peer review process which verifies that the tests you are spending a lot of time on are indeed valid measures of what they say they are.
I suspect one of the reasons the author hasn’t vetted these tests in any traditional professional forum is because many of them simply wouldn’t stand up to our field’s scientific rigor. Some of them are significantly lacking the statistical significance which is needed to ensure that they can discriminate amongst the various kinds of variables they say they are measuring.
The author, for some unknown reason, hides behind a pseudonym on the site. Her degrees are in the field of psychiatry, apparently lacking formal training in psychological test development which is standard for psychologists.
The psychometric properties of her tests (published only on her site) show questionable attributes. For instance, her EQ tests shows that the test scores’ correlations are low across real-life measures, such as personal, social, or professional success. What this means is that your score on the EQ test correlates next to nothing in your real life (accounting for little of the variance of these areas). The extroversion/introversion inventory shows virtually no differences between test scores for genders, contrary to the published literature. The psychometric report for this measure also doesn’t seem to accurately reflect the actual test, which has additional unnumbered questions at the end. The self-esteem inventory is only slightly more convincing, but is still weak. Other tests all seem to suffer from flaws which draw into question the generalizability and usefulness of the results given.