Any company that uses reaction time tests to make personnel decisions is fooling itself. These game-like measures simply don’t measure much at all.
By Ryne Sherman, Ph.D.
You might see this article as an attack on your business practice. Maybe you’ll even find it an affront to your livelihood. I don’t mean to offend, just to tell the truth.
A few years ago, on the promise of revolutionizing the employment screening industry, two companies began using laboratory tasks borrowed from the field of cognitive psychology to accomplish two things: assess individual differences in personality and use the results to help companies make personnel decisions. It’s a tribute to their sales and marketing departments that these organizations are still in business.
Sign up for the monthly TalentQ Newsletter, an essential roundup of news and insights that will help you make critical talent decisions.
This article concerns three issues: First, it describes the reaction time-based cognitive psychological measures these companies use. Second, it shows how the businesses market their products in misleading ways. And finally, it explains why reaction time measures of personality can’t predict the kinds of outcomes employers care about, like job performance, turnover, and workplace safety.
What Are Reaction Time Measures of Personality?
According to the companies that use reaction time measures, the Go, No-Go task assesses impulsivity and attention span. Meanwhile, other similar tasks assess such personality traits as risk-taking, persistence, and motivation.
The easiest way to understand reaction time measures—the kinds these companies are using—is to complete one yourself. Take two minutes to try the “Go No-Go” task here. Your mission: Press the spacebar as fast as you can whenever you see a green dot, and don’t do anything at all if you see a red dot. That’s it. The computer records your response time and accuracy.
Although these companies are selling laboratory tasks used in cognitive psychology research, they call their tasks neuroscience. By adding game-like elements (points, color, and context), they also refer to these cognitive psychology tasks as games.
Although experts quibble about what is and isn’t neuroscience, they all agree that biological systems—primarily the brain—are the core of neuroscience. Unless you directly measure part of the nervous system via methods like EEG, MRI, or record neuron activity, you’re not doing neuroscience.
But you don’t need to take my word for it. Check out the table of contents for a book on neuroscience methods. Here’s another. You won’t find reaction time measures mentioned in either book. Why? Because they don’t measure brain activity, and thus, they aren’t neuroscience. They also aren’t measures of personality … but we’ll get to that.
If you completed the Go No-Go task linked above, you probably didn’t think it was a game. Would you spend your free time completing a Go No-Go task? I doubt anyone would. As Richard Landers explains in his definitive talk on gamification, dressing up tasks and adding arbitrary point systems doesn’t make something a game.
Look, I get it: Neuroscience sounds sexy. And game-based assessments sound like fun. Indeed, there are game-like cognitive ability assessmentson the market that actually might be fun. But characterizing these assessments as neuroscience-based games is a marketing ploy designed to sell otherwise boring psychological laboratory tasks.
Why Reaction Time Measures Don’t Work
Consider the differences between personality and cognitive psychology. Personality psychology is about differences between people; personality assessment is designed to capture those differences. Cognitive psychology, meanwhile, concerns finding general laws of thinking; cognitive psychological assessment is designed to focus on average trends and minimize individual differences.
So tasks designed to minimize individual differences, such as the Go No-Go, can’t be used for personnel selection because everyone gets basically the same score. The lack of individual differences in scores also doom such assessments because they produce low reliability and validity.
Reaction time measures of personality also don’t work because they’re unreliable. Reliability is the degree to which people get the same scores each time they take an assessment. This is important because it reflects the degree to which an assessment can predict itself. Simply put, an assessment that can’t even predict itself certainly can’t predict something else.
A recent large-scale analysis of test-retest reliability concluded that “while self-report measures generally have high test-retest reliability, behavioral task measures have substantially lower test-retest reliability, raising questions about their ability to serve as … measures of individual differences.” Bad news for reaction time measures.
Finally, these measures of personality don’t work because they have low validity. Validity concerns the degree to which an assessment predicts outcomes like job performance, turnover, and workplace safety. Valid assessments allow us to select the right candidate because they predict job performance. As I just mentioned, if a test can’t predict itself, how can it possibly predict anything else?
Two different sets of researchers have studied the ability of reaction time measures to predict meaningful outcomes. The conclusion isn’t surprising: Reaction time measures of personality don’t predict real-world outcomes. One study showed that standard personality measures predicted 20 out of 30 life outcomes (compulsive spending, exercise, life satisfaction, school grades, work quality), whereas the cognitive task measures didn’t predict a single life outcome.
Why They Don’t Want to Talk About Validity
If reaction time measures of personality don’t predict the outcomes businesses care about, like job performance, then I have to ask the crucial question: Why do we even use them?
The answer, of course, is marketing. Companies use the labels of neuroscience and gamification to fool their clients. The organizations that use these tasks can’t advertise validity because they don’t have any. Instead, they promote hiring benefits that are irrelevant to job performance: number of applicants, diversity of hiring class, and time to hire.
Do these outcomes sound appealing? Absolutely. But here’s the truth: A random number generator will produce the same results, and it’ll cost a whole lot less. If you don’t care about validity, it’s easy to increase your applicant pool and reduce time to hire. Just hire people at random!
In contrast, the beauty of well-validated personality assessments is that they give you both job performance and fairness; personality tests don’t discriminate against minority groups and women.
Modern psychological assessment is like the American Wild West: There are few rules and regulations. This is a stark contrast to something like the modern pharmaceutical industry. In the United States, if we want to sell a new drug, we must go through a rigorous scientific review process and submit exhaustive validity evidence for a drug to be approved by the Food and Drug Administration (FDA), the regulating body for medical drugs. Without FDA approval, we cannot sell any drug. In the psychological assessment industry, there is no regulating body.
Anyone—literally anyone—can create a psychological assessment and start selling it. Validity be damned.
Ryne Sherman, Ph.D., is the chief science officer of Hogan Assessments. Prior to joining Hogan, he was an associate professor in the department of psychology at Texas Tech University, where he researched the importance of individual differences, the psychological properties of situations, and developing tools for data analysis.
