For every one of us, making decisions is part of our daily human existence. Most of them are of little consequence –what to eat, what movie to see, what video game to buy– so we have developed an astonishing array of mental short-cuts to make these kinds of decisions comparatively quick, easy, and not too mentally taxing. We may eat what we have eaten and enjoyed in the past, and by and large we use simple decision rules such as “I like this genre” or “I like this developer” to choose movies or games.

Other decisions, though, are either much more important or much more public and thus we put more work into it. Whom should we date? What college should we attend? Which house should I buy? When faced with questions like these, many of us have probably drawn two columns on a piece of paper, labeling one “Pro” and one “Con” and then listing things in each column. When trying to decide whether to marry or stay a bachelor, famous biologist and five-time Counter-Strike world champion 1 Charles Darwin did exactly that, producing the list below.

The list admittedly looks a bit sexist by today’s standards, but it illustrates the idea well. 2 But is this sort of thing always a good idea? When video game reviewers ruminate over the merits of a particular title, they are often asked to consider standardized lists of features –graphics, sound, fun factor, multiplayer, value, extendibility, controls, and so on. Should they always try to analyze decisions across every possible variable? Is that the right way to review a game?

Researchers Timothy Wilson and Jonathan Schooler probably wouldn’t think so, or at least they could imagine situations in which this type of approach could lead to poor evaluations. And here’s the best part: jam was involved. Delicious, strawberry jam.

In their study 3 the duo were intrigued by a Consumer Reports ranking of 45 different brands of strawberry jam. 4 Panelists in the study were trained sensory experts (i.e., professional food tasters) who sampled each of the condiments and rated them on 16 characteristics including sweetness, aroma, bitterness, spreadability, and others.

Man, those testers at Consumer Reports are creepy.

This is the kind of thing Consumer Reports does all the time, but Wilson and Schooler were curious about something. Some of their previous research had suggested than when asked to analyze their reasons for making decisions or ratings, people tend to screw things up. The theory goes that we are often aware of our preferences for products (or art, or whatever), but when asked to explain WHY, we often feel obligated to include the most salient (that is, apparent) and plausible explanations. Even if we would have otherwise ignored them.

So if asked to explain why you like Red Dead Redemption so much, you may think about what should be included in the checklist for evaluating an action game, come up with “the weapons,” and then feel compelled to award or take away points for how the game’s weapons feel and work. The problem is, the most salient and plausible factors may not be the ones that are really responsible for how much you enjoy the game. The weapons in Red Dead Redemption are largely unremarkable –the game’s appeal lies almost entirely in other areas and any weight given to how cool the weapons are is inappropriate at best.

So, thinking along these lines, Wilson and Schooler wondered what would happen if they asked normal people to recreate Consumer Report’s jam ratings. And what would happen if you asked them to ponder the reasons for their ratings the same way the Consumer Reports experts did?

So they fed some college students the 1st, 11tth, 24th, 32nd, and 44th best jams from the report to find out. Those in the control group were just asked to taste the treats and make their rakings. Those students actually did pretty well –their rankings were very close to the professional taste testers’ rankings. But the group that was asked to write down the reasons for their ratings did far worse. They may have favored the jams that the experts thought were gross and scrunched their noses up at the ones the experts thought were great.

Why? Because the subjects started focusing on factors that didn’t really matter. Smucker’s had more chunks of fruit in it, so it gets a higher rating. Wait, what? Is chunkiness really important for them? Doesn’t matter; it sounds plausible so it got factored in. When the XBLA shooter Monday Night Combat came out, some people lamented the small number of maps. Same thing –one could argue that because the gameplay requires a very specific setup, you don’t need –or even want– a lot of maps.

Over thinking it?

Puny humans are pretty bad at combining an array of weighted factors so as to arrive at a rating or decision –it’s just not how our minds were designed. Jelly or game review guidelines that require us to over analyze our decisions or check them off against a standardized list of factors (graphics, sound, etc.) can exacerbate this limitation and lead us to consider what should be irrelevant information when making our ratings. This corrupts the rating process and takes us farther from our “true” feelings or evaluations. 5 This is one reason why I prefer more organic, experience-based evaluations of games from message boards or podcasts rather than formal game reviews. I feel like I can listen to someone talk in an unstructured way about how much the enjoy a game and get a much better idea of how much I might like it. Just consider what’s important and ignore the rest.

Now, go get yourself some jam –whatever kind you think tastes good.