Last Friday, the Wall Street Journal reopened one of the more contentious debates about psychological research. This debate is not about any particular results or theories, but rather centers on the validity of all results. Since the 1960s, college students, who are cheap and convenient, have been the primary subject pool for nearly all psychological research conducted at universities across the country. What at first must have seemed a perfect match has now become an issue of some concern as heavy reliance on students in laboratories has raised critical questions about the potential for sampling bias.
College students, especially undergrads, have many qualities that set them apart from the public at large. They have not only been selected for elite academic qualities, but they are also at a uniquely turbulent stage in their lives. A more representative survey would have less bias, but budget constraints and simple ease of use almost always win out. As the Journal points out, some are now wondering, if students are different from the general population, then how can their results be extrapolated to paint a picture of the rest of society?
Dr. David Sears first focused attention on this problem with his 1986 call-to-arms College Sophomores in the Laboratory. His paper meticulously documents the trend towards sole reliance on students and then outlines many of the possible biases of this method. Sears does not single out psychological research as inherently flawed, for he points out that other scientists rely on rats to make general statements about human health. However, he warns that the only way to rectify a flaw is to recognize that flaw and actively correct for the bias. Pharmacologists work to understand the way that rats differ from humans. Similarly, he points out, psychologists should study the way that college students differ from average people.
Dr. Sears pulls together a number of findings about college students to find examples of this difference. He suggests that much of the image of human nature painted by laboratory studies is biased because college students have weaker senses of self, more malleable attitudes, more compliance with authority, and stronger cognitive skills. If this is true, we may have to review all of the studies purporting to show that humans are easily influenced into changing their minds and following orders.
Dr. Robert Peterson from the University of Texas took up Sears's call and has since been trying to study the extent of this bias. In 2001, Peterson compiled a meta-analysis in a report published in the Journal of Consumer Research to show that results for students and non-students differed widely. Not only were the magnitudes of some results markedly different, but in some cases the results were opposite. Dr. Peterson told us that he was troubled to find that the opposite result was obtained from non-students in almost one-fifth of cases.
The Wall Street Journal reports that Dr. Peterson is working on a new paper demonstrating that students are not as homogenous as previously thought. In his 2001 study, Peterson stated that students seemed slightly more homogenous than non-students. However, this newer paper may knock out one of the last supports for the viability of student-only research. If students were all very similar in obvious ways, then it would be possible to study their qualities and correct any biases. However, if students are different from the general population and from undergraduates from other colleges in ways too complex to easily adjust for, psychologists may need to start offering more than free candy to starving student volunteers.
Dr. Peterson pointed to current trends to predict that in ten years almost all psychological research will be done on students if nothing is done to rectify the problem. Dr. Brian Nosek, a psychology professor at the University of Virginia, is not content to do nothing. He is trying to solve the student bias problem by making it easier to use alternative subject pools. Dr. Nosek spoke with us recently about the problem:
The mechanisms of reward for individual scientists, the means of actually getting samples to work with, are not structured to benefit the use of representative samples. And that is a big problem.
Dr. Nosek explained to us that scientists know they are using less-than-ideal subjects, but that they will not pursue more representative samples because of the incredible difficulty and expense. Instead of admonishing researchers, Dr. Nosek is trying to make it easier to collect diverse samples.
His first step towards this is Project Implicit, an online laboratory set up with the help of the National Institute for Health. Project Implicit can administer dozens of tests that are easy to take and accessible from anywhere in the world. Currently, around 20,000 tests are completed every week. Internet users can access this site at their leisure and participate in any number of ongoing tests. By filling in specific data, anonymous users provide researchers with hordes of information. Researchers have the ability to design studies in any number of ways, even inserting questions to help improve upon the accuracy and legitimacy of future research. Beyond feeling like they've contributed to a good cause, users are sometimes given immediate feedback in the form of personal test results that they can then compare to the study cohort at large. Serving only novel purposes really, it is still always fun to see how one compares to many.
Dr. Nosek has ambitious plans to expand his virtual laboratory. He shared with us a grant proposal that he just submitted to the NIH to get funding for the creation of a massive International Participant Pool. If it becomes easier to test diverse citizens from all over the world, then there will be no reason to rely exclusively upon college students. The entire field of psychological research will be enhanced as we become more confident in the generalizability of results.
|