Let’s say that you’ve hired a coach to help you improve your Slayer game in Halo 3. I’ve heard of stranger things. Let’s say this coach looks like Mr. Miyagi but he curses WAY more. He uses a variety of training and motivational techniques, ranging from grenade throwing drills to trigger finger sprints, doing everything he can to drive you towards perfection. You notice, though, that he eventually stops praising you whenever you rank at the top of a match. He did at first, but now when you earn more than your usual number of kills your coach stands stoically by, straight faced and not giving you a single word of praise for those outstanding rounds.
Eventually, you ask him why he never praises you when you do a really good job.
“Because,” he says, “I’ve noticed that praise doesn’t work. Every time I praised you for a really good round, your next round is always mediocre. And what’s more, when I yell at you for playing poorly, next round you always do better. Praising not only doesn’t make you better, it makes you worse.”
You pause for a second, then cry “You’re not my real dad!” and run out of the room, bawling like a child. Yes, you do. That emotional outburst aside, though, is your coach’s logic sound, given that you DO in fact perform worse every time he praises you for doing well and perform better whenever he rebukes you for doing poorly? Praise makes you do worse and berating makes you better, right?
Nope. Your performance following stellar rounds of Halo or Starcraft II any other game involving skill can be best explained not by the effects of praise or punishment, but by something called “regression to the mean.”
Let’s assume that if we looked at your performance over a bunch of matches and plotted them out with ending scores along the X axis and how often you end a match with that particular score on the Y axis. They’d probably form something close to a normal, bell-shaped distribution like this:
Figure 1: Your sick skilz, plotted
If we were to pick any single match at random, it’s more likely that your performance would be about in the middle somewhere –somewhere near the “mean,” which is basically another word for “average.” 1 In this example, that’s 10 frags. It’s rare that you’re at the very top (17 frags) or bottom (3 frags). In fact, if your performance follows a normal distribution like the one above, then the following will be true:
- 68% of your matches will end with scores between 8 and 12
- 95% will be between 6 and 14
- Only 0.6% will be under 4 or over 16 2
And even if your distribution is a little skewed because you do well more often than you do poorly, the numbers won’t change much until things get REALLY skewed. At which point no amount of coaching is going to change your game in either direction.
This is the reason that you seem to do worse after good matches and better after bad ones. The particularly good or bad matches are rare, and because they’re rare it’s improbable that you’d have two in a row no matter what your coach does.
So don’t get discouraged when you can’t consistently come out on top multiple times in a row in any game of skill. You may be able to move your distribution up the right-hand side of the scale and/or squish it together so that there’s less variation, but you’re always going to regress to the mean somewhat because every round can’t be your best (or worst) round.
Now go give your dad a hug.
Oh god! It’s like my college math/stats class all over again.
See, it WAS important!
This raises an interesting question – regression to the mean aside, what do we know about the actual impact of praise on performance in games of skill? Most games incorporate some form of “praise”, or at least positive feedback – Dragon Age gives you loot and XP points, Geometry Wars unlocks additional play modes, and Peggle treats players to a dose of the ole Ludwig Van, complete with fireworks, every time they complete a level. Do we have any way of assessing how far mechanisms like this push the bell curve to the right?
@Ethan Maron
hmmm,i don’t think the things you said have any effect at all,i mean i for one certainly wouldn’t like peggle any less if it didn’t show that sequence at the end of the level.What you are calling “praise” is actually gives a sense of progression in games.
Positive feedback, if timely and specific, certainly helps in learning and motivation. Same for negative feedback. The point of this article is more around the magnitude of those effects relative to the effect of regression towards the mean.
This is very tangential, but on the tv show Sports Science they compared a basketball player’s performance and heart rate after being “instructed” by either a coach that encouraged him, and one that yelled at him. I think that he scored fewer shots with the yelling, and his HR jumped up quite a bit. I tried to find a link to the episode but I cannot for the life of me. Anyway, based on what they found I think the Halo Coach’s shouting might actually make the person play worse…but that show doesn’t have the most rigorous scientific standards, so that should be kept in mind.
RPG’s with stat progression send players the message that if they’re not hitting the mean performance on their first try, that they should go elsewhere and grind for awhile, then come back stronger. Although I’ve played some rpg’s where even that couldn’t save me.
I have to be honest and say an explanation of the normal curve has to be the most underwhelming article yet.
Well, next week’s will probably be better. Because of regression to the mean.
Eh? See what I did there? Eh? Eh…
WHAT!? This argument is totally flawed! You’re implying that Regression to the mean is somehow a causative force in normalizing this player’s score. That’s the Law of Averages! (which is a false premise, of course)
In reality, You could go on a hot streak and get 17 kills several games in a row, or on a poor streak and get 2 kills several games in a row. This normal curve, however, will predict the distribution of your score over the LONG TERM, not from game to game.
Either way, this is my first time reading your blog, and I’m quite excited to see you applying statistics and psychological analysis to video games. Can’t wait to read more articles!
@Merzhost
Actually, I believe he was arguing that there is no causative relation between the coach and how the person plays, not how the person plays one game to the next. The argument is that the further away from your average game you get, the more likely the next will be closer to the mean…no causation, just probability.
@Merzhost
Actually, I believe he was arguing that there is no causative relation between the coach and how the person plays, not how the person plays one game to the next. The argument is that the further away from your average game you get, the more likely the next will be closer to the mean…not causation, just probability.
Yep, Alex pretty much has it. Regression to the mean isn’t causing the changes in performance from round to round, it’s just explaining them. The causes are your skills relative to the other players, which probably don’t vary THAT much around an average performance. Statistically, it’s very unlikely that you’d get several games in a row with 17 kills if your average is 10 with a average difference from that score (i.e., a standard deviation in statistics gobblety-gook) of 2.
You should totally provied someplace other then comments to discuss other topics we stumble upon.
Anyway,ever since i bought my 5850 ati card i have been having problems with it:GSOD hey call if(guess what that stands for).Basically this card either has problems running simple 2d stuff or high 3d stuff and randomly crashes.After months of awful wait,ATI claims the problem has been addressed in the latest driver/hotfix but this site clearly says other wise.
http://forums.amd.com/game/messageview.cfm?catid=260&threadid=124747&enterthread=y&STARTPAGE=1
Interesting thing is this:they seem bent on completely ignoring all their customers with this problem!Now this seem hardly a good choice business wise,i wonder what makes them act this way?an funny thing is,i have seen the same thing with saboteur game on win7+ati systems which havent been addressed:
http://forum.ea.com/eaforum/forums/show/3817.page
If you are interested in the effects of feedback on motivation (not performance)you can check out this article I wrote a while ago for Gamasutra:
http://www.gamasutra.com/view/feature/4171/staying_power_rethinking_feedback_.php?print=1
Pingback: Blood Street Associates » Blog Archive » More on Decision Making Errors