Why do surveys overestimate the number of people experiencing the Xbox 360’s dreaded Red Ring of Death?

Lord knows I’m familiar with the Xbox 360’s “red ring of death,” or “RROD” as it’s not so affectionately called. I’ve encountered it twice myself. The term refers to what you get on the front of Microsoft’s console when its notoriously high failure rate kicks in and the thing stops working. This tends to be a touchy subject for Xbox owners, who tend to light up the torches and grab the pitchforks whenever it’s brought up.

Earlier this year the magazine Game Informer made a lot of headlines by reporting that according to their research, a mind blowing 54.2% of Xboxen crapped out, which is a failure rate traditionally reserved for mundane things like marriages. 1 Immediately across the Internet people started screeching this number as fact.

I should know about the RROD. I've had 2 myself.

But how much stock can we put in the survey and the methodology used to conduct it and interpret its findings? Sure, I think it’s safe to say that the RROD rate is high (Microsoft’s Peter Moore admitted as much publicly), but is that 54.2% overblown? (SPOILER ALERT: Yes. Yes it is))

Game Informer surveyed 5,000 of its print subscribers to gather the data. I couldn’t find a copy of the actual survey, but let’s give them the benefit of the doubt and assume that there were no leading questions or other shenanagains that would influence subjects’ responses. The problem that persists, though, is related to what psychologists and others refer to as “sampling errror.” This is when the sample of data you collect is somehow biased, skewed, or otherwise not representative of the larger group of people that you’re really interested in (“the population” in inferential statistics parlance).

Presumably, what Game Informer wanted to do was infer that was was true of its sample (the readers it surveyed) was also true of the population of interest (all Xbox 360 owners). 2 What Game Informer did was send out a survey and then ask people to voluntarily respond. I can easily think of three reasons why this may have inflated their results:

1. People who had experienced a RROD and been justifiably pissed off about it would be more likely to respond to the survey (an example of what’s known as “self-selection bias”)
2. People who subscribe to enthusiast magazines probably play more games and thus put their machines through more wear and tear
3. People who subscribe to enthusiast magazines are more likely to be early adopters who bought initial runs of the console before Microsoft improved their process and reduced RRODs

Think of it this way: If you were interested in measuring the prevalence of drinking in your home town would it be wise to only survey patrons at bars? 3

So what should Game Informer have done? The best way to eliminate sampling error is to survey people from the population of interest randomly and not rely on self-selection to be in the sample. Sending surveys randomly to people who have registered an Xbox 360 would reduce (but not eliminate) sampling error. Same for randomly surveying people in a shopping mall or cold calling them. Sure, this is hard and expensive and not always practical, but the bottom line is that if your research has flaws like potential sampling error you should note it, and reporters –even in the gaming enthusiast press– should be savvy enough about these things to note them when reporting on them instead of screaming “54%! 54%!” because it makes for good headlines.

As an example, look at this online survey done by CNET UK on exactly the same question. The survey has most of the problems described above, but the authors are good enough to cop to it:

This was a self-selecting survey, so it doesn’t represent a random sample of console owners. It’s likely that people whose consoles have had problems are more motivated to fill out the survey, but the results are still interesting when you compare the Xbox 360 to its competitors.

…The survey did not distinguish between the Xbox 360 Arcade and Elite versions, which are very similar, or the PS3 and PS3 Slim, which has only just been introduced.

So good on them. The next time you see survey results cited anywhere, think about how credible they are by taking sampling error into consideration.