As is my habit, I recently attended the annual conference for the Society for Industrial/Organizational Psychology (SIOP), which is the professional organization for people who apply psychology to understanding human behavior in the workplace. Trust me, we I/O psychologists actually do know how to party.1 Apparently some of us also know how to play video games, because I happened upon one panel discussion called “Serious Games and Virtual Worlds: The Next I/O Frontier.” The purpose of the panel was to explore how video games, virtual realities, and other immersive technologies can help us measure and predict worker behaviors.
One concept that particularly clicked with me was the idea of making games out of what are called “situational judgment tests,” or “SJTs.” These are tests used to screen job applicants and help decide if a person likely has abilities needed for success on the job. They typically present test-takers with a situation, give them relevant facts, and then ask them how they would react or solve a problem requiring some kind of application of rules and a judgment call. For example, a SJT for first line supervisors might describe (or show via video) an employee coming in to work late and giving an explanation of his tardiness. Using this scenario, a copy of the fictional company’s attendance policy, and access to the offending employee’s past time sheets, applicants are asked to indicate what they would say in response to the employee’s request to make up the time later. This can help prospective employers decide if an applicant should be able to appropriately handle similar situations on the job.
Think about that. If you’ve ever played a Western role-playing game, especially something by Bioware, you’ve taken a situational judgment test. I was recently playing The Witcher, for example, and during the tutorial you’re tasked with having to decide how to deal with an attack on your stronghold –do you want to chase down the powerful mage who’s trying to steal your stuff or do you want to deal with the monster rampaging through your courtyard? Your choice may say something about your preferences and your priorities, not to mention your tactical skills. Or take the conversation in Mass Effect 2 that has you guide Commander Sheppard through a conversation with Mordin about the alien doctor’s involvement with the bioengineered, near-destruction of an entire species. That’s a delicate exchange, and true to the character of a good role-playing game, Bioware lets you proceed through it in a number of different ways, each of which could reflect something different about you.
Ahhh, Mordin, I'm gonna need you to go ahead and come in on Saturday to wrap up that whole genophage thing, 'kay?
I think the idea of using these kinds of games to test job applicants on their ability to make various real-life judgments is a fascinating one. One could easily imagine a game where applicants are told to interact with an in-game character and solve problems as they would in real life –sort of a role-playing game where the point is not to role-play.
What’s more, we know other things about human psychology in video games that could improve our measures of work-related abilities. We know, for example, that people who are deeply immersed in a technology or engaged in psychological flow are more likely to forget about their surroundings and forget (or at least downplay) the fact that they’re playing a video game. One of the biggest problems in using tests or interviews to assess job applicants is that they may behave differently relative to a normal day on the job. But if casting an assessment in the mold of a game helps them forget that they’re taking a test or disregard it, then we’ve got a much more accurate measurement of their ability. Game designers could be brought in to make the assessment more like a game, and while you’re at it you could take the opportunity to convey important information about the company and its values in order to make the assessment informative to the applicants, too.
As I’m so fond of saying, somebody get on that. And remember: I’m available for consulting work.
I’m not sure your proposal would work. The demand characteristics (specifically ‘good participant role’) of someone engaged in an interview-related task would likely prevent deep immersion or flow.
I’m not sure. Games have demand characteristics as well, but you can still experience some degree of immersion. It’s not all or nothing; you can be more or less immersed.
Don’t forget the Ultima series of RPGs which, at least in some instalments, used situational judgement tests to roll your character, as opposed to you picking preferred stats yourself. Something similar happened in Fallout with the G.O.A.T.
Anyway, that’s just an aside…
“But if casting an assessment in the mold of a game helps them forget that they’re taking a test or disregard it, then we’ve got a much more accurate measurement of their ability.”
See, when I play a game, I automatically try to figure out how my answers will influence gameplay, and I’ll answer according to that. Trying to hide tests in this fashion would probably fail me.
The Ultima thing is a good example. one of the en vogue methods of delivering personality assessments is instead of having ratings of 1 to 5 on a “I strongly disagree with this” to “I strongly agree with this” scale, you give test-takers two statements and have them pick the one that more accurately reflects them. Over many items you can build a profile of their personality. Ultima’s gypsy lady is not unlike this –in form, at least.
Also, to your other point, I’m not talking about hiding the test in a game. People are doing this right now and it’s always clear that this is a test on which your employment prospects hinge. And you’re usually asked to respond in a way that reflects what YOU would do on the job, not some character that you’re role-playing.
“See, when I play a game, I automatically try to figure out how my answers will influence gameplay, and I’ll answer according to that”
I agree that you usually go for either “good guy” or “bad guy” (or similar) and follow through with this choice accordingly during your gameplay, but in some games and in personality tests such as Belbin, often there is no obvious “Good/bad guy” choice, but several middle ground. Forced to choose will reveal your typical personality characteristica. Implement those kinds of choices in a game situation where you are somewhat immersed compared to a normal interview situation, i think, could help better reveal your personality traits…
Tristan: Innovative companies are already using videogame-like tools for assessment. Although your point is well-taken: employers need to understand what exactly they’re measuring (e.g., typical or maximum performance).
I think the biggest hurdle (other than cost) will be that not all jobs–particularly the ones that organizations likely have the most problem screening for–lend themselves easily to an engaging interactive experience. But that being said, companies like Shaker Consulting Group seem to be doing just fine.
Thanks for the post! Good stuff.
Yep, it’s not a one size fits all solution for sure. There are probably jobs and test types for which it works better or worse.
what do you think of videogames to measure not only Sitautional Judgment but also aptitudes required for the workplace? Like playing a level of portal to help determine problem solving
… or maybe something lacking a passive agressive AI conducting the test but you get what I’m saying
I was just forwarded this by a colleague, as I was on that Serious Games panel. Thanks for the write-up!
There was another session that I was involved with at SIOP, which specifically looked at simulated/multimedia situational judgment tests. It’s early days, but the validity data is encouraging. (If you’d like a copy of that slide deck, let me know and I’ll email it to you.)
There’s a big difference between using *elements* of gaming (e.g. interactivity, animation, sound etc) for assessment and actually building a game that also assesses. In a game, the user has control and can make choices that the game responds to. But that’s at odds with our approach to in assessment where we want to ensure that each user goes through exactly the same experience so that we can directly compare the performance of different users.
That said, I think all of us who remember the movie “The Last Starfighter” have this vision of a fair, valid and reliable assessment is *so* engaging that people will play it just for fun. It’ll take a while, but we’ll get there…
Nice dialogue. These are the things we need to be thinking about for the future. These threads are really new ones and these are the first discussions I have seen that link the worlds of IO and gaming based on insight into how both function. Agreed we are very very early in this and the amount we do not know is staggering. But these dialogues are essential to progress. Carry On!
The question that needs to be asked is this – Who will be using the test and information gained from it to decide whether I get the job?
Professional psychologists who understand the test and how to interprest the results and any anomalies or middle managers with a check sheet that says If question X is a No and question Y is a Yes then interview failed.
It’s an interesting way to do a psychology test, but I wouldn’t want to bet my future on a Budget Manager’s reading of the results. And this is what we’re looking at in the real world
I hope there really is a priest and a Likert scale joke…
Pingback: New Emerging Field of Video Game Psychology : Knickerbocker Ledger
Pingback: Video Game Psychology: A New Emerging Field | ggvogue
Pingback: Leveraging the Power of Video Games