Peter Moore recounts $1.15bn Xbox 360 Red Ring of Death saga
"If we had tried to fudge over the problem the Xbox brand and Xbox One wouldn't exist today."
Video game executive Peter Moore has recounted the infamous Xbox 360 'Red Ring of Death' saga that cost Microsoft over $1bn.
Soon after the launch of the Xbox 360 in 2005, some customers found their console unusable. The console's way of telling people something very bad had happened was by displaying three flashing red lights, what was dubbed the "Red Ring of Death".
The problem got so bad that in 2007 Microsoft put aside $1.15bn to cover an unprecedented repair offer that meant affected customers would get a fixed Xbox 360 as quickly as possible while extending the console's warranty to three years.
Now, eight years later, Peter Moore, now of EA but then of Xbox, recalled the Red Ring of Death furore in an IGN podcast also starring current Xbox boss Phil Spencer and Seamus Blackley, one of the creators of the original Xbox.
"We were seeing failure rates and starting to get reports through customer service," Moore said.
"This was a thing where we couldn't actually figure out what was going on.
"We knew we had a problem. I remember going to Robbie Bach, my boss, and saying, I think we could have a billion dollar problem here. As we started to do the analysis of what was going on, we were getting the defectors in, it was a challenging problem for our engineers, and we couldn't quite figure out what it was. We knew it was heat related. There were all kinds of fixes. I remember people putting wet towels around the box."
Moore, alongside his financial team, set about working out how much it would cost to address the issue - an issue that was becoming a critical PR problem for the Xbox brand.
He worked out that it would cost $1.15bn to put things right - and then plucked up the courage to tell then boss of Microsoft Steve Ballmer.
"The moment I'll never forget, I said to Robbie, we've got a business review meeting in Building 34, with Steve Ballmer. I said, we've got to tell Steve, here's what we have to do: we need to FedEx an empty box to a customer who had a problem - they would call us up - with a FedEx return label to send your box, and then we would FedEx it back to them and fix it. Either keep your hard-drive or send it to us.
"I calculated with my finance team, Dennis Durkin (now the Chief Financial Officer of Activision), Doug Ralphs (then Senior Director of Finance at Microsoft's Interactive Entertainment Division)... $1.15bn, right out there. I always remember $240m of that was FedEx. Their stock must have gone through the roof for the next two weeks.
"And, I am trembling sat in front of Steve, who I love to death, but he can be an intimidating human being. Steve said, 'okay, talk me through this.' I said, 'if we don't do this, this brand is dead. This is a Tylenol moment.'"
Moore's reference to Tylenol relates to the Chicago Tylenol murders, a series of poisoning deaths resulting from drug tampering in 1982. The victims had all taken Tylenol-branded capsules that had been laced with cyanide. The company behind Tylenol, Johnson & Johnson, distributed warnings to hospitals and halted Tylenol production and advertising before issuing a nationwide recall of Tylenol products.
Moore continued: "Steve looked at me and said, 'what have we got to do?' I said, 'we've got to take them all back, and we've got to do this in a first class way,' because when you take a console away from a gamer, and you're going to spend three weeks fixing it... so we've got to FedEx this all the way. We've got to FedEx this all the way. We've got to overnight it back in two.
"He said, 'what's it going to cost?' I remember taking a deep breath, looking at Robbie, and saying, 'we think it's $1.15bn, Steve.' He said, 'do it.' There was no hesitation.
"I'm thinking, I'm about to crater Microsoft's stock. Actually, nothing moved."
Blackley said while speaking to the likes of Steve Ballmer and Bill Gates was an intimidating experience, they were willing to help solve problems.
"If you were full of shit you were dead," Blackley said. "If you were honest about the problem, it was, 'okay, how can we help you?' That's why that business was so great."
Moore credits Ballmer with green-lighting the huge amount of money required to repair the damage the RRoD furore had caused.
"I'll never forget that moment," he said. "If you're an Xbox gamer, you can thank Steve Ballmer for not even hesitating. Now, we were a wealthy company who could afford to do that, but not even hesitating because the brand was more important.
"If we hadn't made that decision there and then, and tried to fudge over this problem, then the Xbox brand and Xbox One wouldn't exist today."
It sounds like it was a difficult time for Moore and his colleagues at Xbox.
"It was sickening," Moore recalled. "I was doing a lot of interviews. It's not like today with social media, which would have been horrific. We couldn't figure it out."
Moore and Microsoft hardware engineer Todd Holmdahl were left scratching their heads as to why Xbox 360s were breaking.
"There was a theory," Moore remembered. "We had changed our solder, which is the way you put the GPU and the fans, to lead-free. Todd, who is the most sincere human being, was going crazy trying to figure it out.
"We knew it was heat related. We think it was somehow the heat coming off the GPU was drying out some of the solder, and it wasn't the normal stuff we'd used, because we had to meet European Standards and take the lead out."
Whatever the cause of RRoD, Moore now believes Ballmer's decision to fork out the money to address the PR disaster it sparked saved the Xbox brand.
"It was that moment of decision that Steve Ballmer made, that I will never forget. He didn't even think twice about spending $1.15bn to protect a brand that's probably worth three or four times that today.
"Xbox One wouldn't have happened."
The podcast is well-worth a listen. It recounts the early days of Xbox, its launch and marketing efforts, as well as Xbox One and its recent ups and downs.