A Better Situational Judgment Test

Lisa Halter PSI Blog

A Better Situational Judgment TestWhen you’re developing new testing products, sometimes it helps to think about the history of tennis racquets.  Any time before the 1970s, tennis racquet heads had about half the surface area of today’s racquets.  Not surprisingly, this made playing tennis a lot harder.  In 1976, the appropriately named sports entrepreneur Howard Head introduced a racquet with a larger head.  This, obviously, made playing tennis a lot easier, so almost immediately every manufacturer followed.

Why then, for decade after decade, did all manufacturers make the hitting surface of their racquets so small?  It turns out there wasn’t a reason, not a good one anyway.  It was simply that, in their minds, this is how things were done.

There are many examples of this kind of locked mindset in almost every industry.  Whenever PSI develops a new product, we try to think like Howard Head, and PSI’s new True-to-Life™ situational judgment tests (SJTs) are a good example.  SJTs present job applicants with work-relevant situations, followed by a number of ways to respond.  Candidates indicate, in some way, how effective or ineffective those responses would be.   But it is innovations in the way these responses are translated into hiring recommendations that set PSI’s SJTs apart.

Let’s look at an example.  Imagine that two people, we’ll call them Jack and Jill, are being considered for a job that will require, among other things, that they go up a hill.   So to decide who to hire, we present these two young applicants with a list of different ways climb the hill, and ask them to rate how effective each of these methods would be.

When Jack or Jill are hired, we’d like them to perform like people who really know what they’re doing, so before we score their responses, we ask a group of job experts how effective each of the hill-climbing methods would be.   Now we have a standard:  The closer Jack or Jill’s responses are to the experts, the higher their score.

Situational Judgment Test pic 2So now let’s look at the figure showing the expert ratings and the answers provided by Jack and Jill.  For the first response, “Drive the Jeep”, Jack’s rating is one away from the experts, but Jill’s is two away, so Jack scores somewhat better on that one.  Adding up the differences like this across all three hill climbing options, Jack is a total of four spots away from the experts, and Jill is six spots away.  Conclusion, we hire Jack, because overall he’s closer to the experts.  This all seems very reasonable.   And, after all, for situational judgment tests, it’s how things are done.

But now let’s step back and think like Howard Head.  What are we actually trying to accomplish here?  The answer, for any employment test, is that we want to have some idea of what a job applicant will actually do on the job after he or she is hired.  So now let’s imagine that Jack, Jill, and the experts are all on the job together facing a range of hill-climbing situations.  In the first situation, they have all of the three options mentioned available to them.  Jill and the experts board the ski lift together, because they each rated that option as better than the other two, but Jack drives up alone in the Jeep, his highest-rated option.  If the ski lift is broken, everyone rides up together in the Jeep, but if the Jeep is broken, Jack again breaks from the group and walks up the hill, because he rated that option as better than taking the ski lift, the option everyone else uses.

You’ll notice something here — Jill is acting like the experts, but Jack often isn’t.  In fact, for these particular climbing approaches, no matter what happens in terms of the availability of the different methods, Jill and the experts will always do exactly the same thing on the job.  This is because the pattern or shape of their ratings, represented by the blue and green lines in the figure, is identical.  Conclusion, definitely hire Jill.

PSI’s True-to-Life™ SJTs are perhaps unique among currently published SJTs in that we will recommend hiring Jill, even as our competitors will try to sell you on Jack.  Methods that look at similarity in the shape of different patterns of responses have been around for a very long time, and are widely used in other settings.  Historically, however, not in scoring SJTs.  This locked mindset has created an opening for PSI to, like Howard Head, offer a better way.


Victor JockinVictor Jockin, Ph.D., is Senior Manager of Assessment Solutions at PSI.

Click here to receive more information about PSI’s assessment solutions.