Digital Foundry vs. ApocalyPS3
The chip culprit, and Sony's response.
The panic is over. Just as quickly as they were crippled, "fat" PlayStation 3 consoles worldwide started working again as their faulty internal clocks moved away from the fateful date of 1st March 2010. Sony can breathe a deep sigh of release: no firmware update patch will be needed (for a while at least), no hyper-expensive recalls will be required. Everything is as it was. Balance has been restored.
So, what actually happened? And will it happen again? Throughout the debacle, Sony referred to the issue erroneously as a "network connectivity" problem, with the blame pinned squarely on poor old PSN.
However, as gamers rallied and experiences were shared, it quickly became obvious that the issue was far more serious: consoles that were disconnected from the internet displayed the same problems as those that accessed PSN. This wasn't anything to do with the online service, but everything to do with a specific piece of hardware within the PlayStation 3.
The PS3's internal clock is completely invisible to the end user and is used to sync with PSN, as well as time-stamping trophies and downloadable content activation certificates. Yesterday, with this invisible master clock now set to a date that simply didn't exist due to a misunderstanding of leap years, most trophy-supported games wouldn't launch either online or offline, PSN couldn't be accessed, and the activation certificates for downloaded content became invalid. Those affected were left with a games console that wouldn't play games.
Early yesterday afternoon, PS3 users on the #Efnet IRC network claimed to have pinpointed the problem. A small ARM Syscon CPU dedicated to menial tasks within the PlayStation 3 was known to have an issue dealing with leap years. The same piece of silicon had a history of causing similar aggravation for mobile devices with the likes of Zune and some Blackberry devices afflicted. In those cases, MSFT and RIMM were on top of the issue. Sony, unfortunately, was not.
As it happened, the fix for this issue was remarkably straightforward. Similar to a PC motherboard exhibiting CMOS memory issues, the ApocalyPS3 could be resolved by opening up the PlayStation 3, removing the button-shaped battery and letting the power dissipate internally. If you re-inserted the battery and reconstructed your PS3 10 minutes later you were good to go. Great if you're confident dealing with electronics and happen to have a set of the special Torx security screwdrivers needed to disassemble the PS3. Not so great if you're just a regular gamer, nor if you fancied retaining your warranty (if you still have one for a "fat" PS3 anyway).
In the here and now, the problem has righted itself and a patch for the ARM controller can be incorporated at Sony's own pace into a future firmware update. Internally the company must surely be relieved about the bullet it has dodged. Unofficial community-compiled lists of "fat" PS3 hardware affected by the problem suggest that of 11 different SKUs, eight of them use the bugged controller chip, resulting in a colossal amount of affected users.
| PS3 SKU Type | Affected by ApocalyPS3 bug |
|---|---|
| CECHA01 | Yes |
| CECHA01 | Yes |
| CECHC04 | Yes |
| CECHE01 | Yes |
| CECHG01 | Yes |
| CECHG04 | Yes |
| CECHH01 | Yes |
| CECHK01 | Yes |
| CECHK04 | Yes |
| CECHL01 | No |
| CECHL04 | No |
| CECHP01 | No |
This isn't just thousands or tens of thousands of units, it's surely millions of them worldwide. While an internet-delivered mandatory firmware update could have solved the problem, it's difficult to imagine how Sony would have handled the multitudes of these units in homes with limited or non-existent internet connectivity.
Thankfully such desperate measures not required. Power up your PS3 and you're back in action. The raging, NSFW fat bloke on YouTube can return to "owning noobs" and that comment-writer on the EU PlayStation blog can probably revise down his estimate that ApocalyPS3 is equivalent to "9/11 x 1,000". However, one of the most disturbing things about the whole episode was the lack of accurate or worthwhile information from Sony itself.
While pointing out the slim owners were safe in its initial blog posting, there was never any acknowledgement that offline consoles didn't work either. Promised updates on Sony's Twitter page never happened in a timely manner, nor offered much in the way of useful information.
As the real cause of the issue became common knowledge within the community by mid-morning, Sony urged gamers not to "trust info regarding this issue unless from an official Sony source" but failed to produce any meaningful information until the late afternoon when the internal clock issue was finally confirmed to be the culprit. The lack of clear, unambiguous information on the root cause and how it would be fixed was disappointing to say the least.
Clearly there are lessons to be learned here and it will be interesting to see how Sony chooses to handle the aftermath.
You may also like...
-
Metal Gear Solid: The "Lost" HD Remasters
-
Face-Off: SoulCalibur 5
-
Eurogamer.net Podcast #98: Resident Evil and the Circle Pad Pro
-
Who Killed Rare?
-
The Darkness 2 Review
-
Valve makes Portal 2 Space Core mod for Skyrim
-
App of the Day: Armed!
-
Kingdoms of Amalur: Reckoning Review
-
Mobile Controller Group Test
-
Official Mass Effect 3 Xbox 360 and PS3 console mods revealed
-
Mutant Mudds Review
-
Killzone 1 for PlayStation 3 "delayed indefinitely"
-
Sony improves PlayStation 3 web browser with system update v4.10
-
Mass Effect 3 gets From Dust day-one DLC
-
Ubisoft apologises after online server switch snafu
-
Vodafone 3G Vita offers free WipEout and 4GB memory
-
Massive Square Enix sale hits Xbox Live
-
Skyrim gets high-res PC texture pack
-
Square Enix makes Sleeping Dogs official
-
The Simpsons Arcade on EU PlayStation Store today
-
Square Enix announces Sleeping Dogs at retailer event
-
PSP UMD discs not transferable to Vita
-
New LG TVs can play Unity games with Magic Motion Remote
-
Full SSX soundtrack track list revealed
-
Yakuza: Dead Souls release date announced









Comments (64) Latest comment 2 years ago
Comments threads automatically close after 30 days, but please feel free to continue chatting on the forum!
Comment below viewing threshold Show
"Has your gaming experience been affected by an injury in an internal time keeping chip? Then call claims direct on..."
Comment below viewing threshold Show
I love your conclusion. I can see why you felt the need to write an article about it. Eye-opening to say the least.
Comment below viewing threshold Show
Comment below viewing threshold Show
Can we have Richard on the eurogamer podcast sometime to discuss such matters in the group?
Comment below viewing threshold Show
If they really want to charge for PSN communication to the users must be improved....
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
Either way, Sony should have known about this before it even happened. If all consoles are affected then fine, this would have been a different matter. Unless the newer PS3 consoles use a different ARM CPU to handle the calendar they obvioulsy are running updated code. If that is the case then someone should pay a bit more attention to the changelog.
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
Personally, I think they didn't have a clue. As with all companies, you can't just advise on something if you're not 100% sure of what it is. It may have done more harm than good to say something and then turn out to be wrong.
Comment below viewing threshold Show
I think you are probably right. They did not know but should have known. They would have not deliberately "not fixed" the issue since it could have been rolled into a simple firmware update.
More likley a process/management failure. ARM would have told them but their organisation did not get that infomation to the right place to do anything about it. Just one of those problems with large organisations. Probably on some poor programmers' "TODO list" who had his priorities messed around with too much to get it sorted and it fell into obscurity
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
Genius bit of waggery.
Comment below viewing threshold Show
Sony screwed me with a launch PS2, just completely ijgnoring the well-known drive failures and using my broken unit as an excuse to suggest I just buy another (repairs are so costly and take so long, after all). They said the same thing then as now -- "don't listen to all those millions of people reporting the same thing on the Internet. You can't trust any of them. We'll totally tell you the truth. And the truth is, Sony is Da Greatest".
As has been pointed out, this is essentially the same problem that bricked 30Gig Zunes last year for one day . . . but at least MS was vocal about the cock-up and what to do ("Yes, we screwed up. No its not permanent. It'll fix itself automatically in 24 hours. We'll make sure it doesn't happen again. We are very, very sorry"
MS may have had its share of problems (to be fair, more than its share), but at least in the games and entertainment division they have worked hard to earn some trust. Sony has not.
From last genation to now . . . I just don't trust Sony anymore. And this is from a PS1, PS2, Bravia, Vaio, and Sony Audio system owner. Yes, I'm a Sony "hater" but its incidents like this that keep reminding me WHY I am.
Regardless, I'm glad the PS3rs out there can get back to gaming. A one day gaming break isn't so bad in the grand scheme of theings, but going thru it (especially if you don't know if the end is a few hours away or a few weeks away) is a painful thing. My sympathies for my fellow gamers, and here's hoping that the "lost save data" claims really ARE just Internet rumors.
Comment below viewing threshold Show
Comment below viewing threshold Show
Although a little harsh I understand your sentiments. So much has not gone to plan with the PS3 and surely lessons have to be learnt? Yes ground has been made up and the PS3 does give a good experience but ....
Comment below viewing threshold Show
So when faced with a potential recall of perhaps millions of faulty consoles, it's not surprising that Sony would say nothing, until they knew for certain that they either
a) had a problem
or
b) had a fix
Comment below viewing threshold Show
Microsoft communicated things better than Sony, but ultimately they also just let the Zune fail and waited until it fixed itself.
RIM was actually much worse than Sony. They recommended their users to do a full wipe of their phone, which obviously caused data loss, and which didn't actually fix the problem.
Comment below viewing threshold Show
Comment below viewing threshold Show
Lack of accurate or worthwhile information?? Lets think back: an early announcement to say they were aware of the problem and were investigating; a followup to say the problem had been verified and was being handled with urgency; a further note to warn people not to do any of the stupid things being suggested online; a subsequent description of the problem and advice on how to avoid the worst issues whilst a solution was found; a final announcement to confirm that the issue was over, people could return to normal practices, and suggestions for how to resolve lingering issues.
Really, what more do you want? Pathetic.
Comment below viewing threshold Show
Comment below viewing threshold Show
TRUE!
Finally, someone with some common sense
@onyxbox
If they had said that, and it wasn't truem then what? think about it
Comment below viewing threshold Show
"So much has not gone to plan with the PS3 and surely lessons have to be learnt?"
Many of them could have been learnt by observing their direct competition - PSN is still flaky today, and I'm getting signed out continually. Sony's response and explanations have been lacking, to say the least.
Liking the fact that the SDF are marking my previous comment down, BTW - so the PS3 has been perfect for you then? I doubt it
Comment below viewing threshold Show
The PSN has been fine for me until recently. I was getting signed out on a regular basis for a couple of days. Seemed to have righted itself though.
It's fair enough that you're annoyed with Sony but is it because of this problem where you had to wait until midnight for it to right itself? Is it because of the lack of communication, which Bigglesworth has pointed out wasn't actually lacking? Or was it just the straw that broke the camels back?
You're a Sony 'hater' and you've stated as much. Perhaps some of us just aren't as annoyed as you are.
Comment below viewing threshold Show
The sad thing is, I'm not a hater. I have owned every Sony console so far at UK launch (or before with the PSP), still get excited at new game/product announcements for the current machines and own several other pieces of Sony kit too. Sadly, their talent for shockingly bad customer relations and continual self-ownage (SIXAXIS/rumble, shitty BD drives, this calendar bug - just this gen!), coupled with the barely-competitive hardware at a higher price.... well, my patience has worn rather thin.
Comment below viewing threshold Show
Sorry. I wasn't trying to accuse you as being a Sony hater. I was suggesting that your post hinted as much because you had said it. Who knows... If I'd had the problems you have, I'd probably be just as annoyed.
I just personally think that one company is as bad as another and a lot of people are a bit eager to jump at the other company when the slightest* thing happens.
Edit: *Not that this was a 'slight' problem but I certainly don't think it's as detrimental as people are making out.
Comment below viewing threshold Show
Comment below viewing threshold Show
Shit. Ok big apology here. I was getting your post mixed up with Busboy's post. I take it all back.
I am crap
Comment below viewing threshold Show
Kinda like DF did with this article. A full article basically saying "Sony didn't see it coming despite the problem being known about in the electronics industry and lied about the causes and didn't update twitter like they said they would and blah blah blah"
It's good to see an impartial interpretation of the facts available, doesn't happen that often unfortunately as both EG and DF live off of the additional hits created by sparking rows between fanboys with articles such as this.
Comment below viewing threshold Show
/still can't get on PS Home though, never have been able to, and Vidzone is toss too.
Comment below viewing threshold Show
To be fair to Sony, no doubt their main worry would have to be collateral damage to system and network data resulting from unpredictable side effects of, and repercussions to the root problem. That's why their suggestion was for people just to wait it out.
Had they been upfront about the cause of the problem, no doubt many people would start poking around inside the case to reset the battery, and/or try all kinds of silly stuff to see what still worked what didn't, possibly causing more problems down the line by creating files with dodgy date-stamps.
The point is that had they gone on record with a precise diagnosis, should something else go wrong as an unforseen consequence of the calendar bug they'd look even worse and more incompetent.
More than anything else though, this whole episode showed how paranoid and badly informed large swathes of the internet are. By that I mean the idiots suggesting this was the harbinger of a total recall/mass bricking, when it was clearly apparent what the cause of the problem was, and how it was going to resolve itself pretty much from the start.
Comment below viewing threshold Show
Come on! You should know that it is "resulting in a colossal number of affected users".
Sort it out!!
Comment below viewing threshold Show
Comment below viewing threshold Show
They seemed to hope, that it would resolve itself after midnight, but damn...they weren't sure at all.
Else they would have said: "The problem will be resolved after midnight".
I hope that some heads are rolling in the tech department of Sony.
Comment below viewing threshold Show
Maybe so, but when only 3 people own a zune who cares how they handle it?
Comment below viewing threshold Show
Comment below viewing threshold Show
If people who were told "chill out till tomorrow, it should be fixed by then" could just bring themselves to DO exactly that, they would already have all the information they really need.
This reminds me of Duke Nukem Forever, where some people just needed the dev to say "we are still working on it, here are some screens" every bloody month or they would suffer some kind of seizure.
And the guy (child... obviously) who compared this to 9/11 is the world's biggest prick.
Comment below viewing threshold Show
I realize people love their videogames but the uproar across the internets was embarrassing to say the least. Videogamers are often stereotyped as being stunted man-children and this incident did nothing but reinforce that.
Comment below viewing threshold Show
I'm not part of your pityful war. The only thing I spring to the defense of is common bloody sense.
Comment below viewing threshold Show
That makes you part of the SDF apparently...
Comment below viewing threshold Show
edit: ah, but... would that even have worked with that bug? *scratches head* :/
Comment below viewing threshold Show
"That makes you part of the SDF apparently... "
In the eyes of an idiot, apparently so.
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
It was actually a stupid comment irrelevant to the subject at hand.
Comment below viewing threshold Show
Ok, if I wasn't one of the targets then its my bad. I was assuming that you were suggesting anyone "defending" Sony on this had fanboy motives for doing so, and as I was in a way defending them (against an onslaught of tabloid rage) I assumed I was in part being labelled SDF.
Sorry for calling you an idiot.
"Really? Sony's PR machine being in full swing in the threads is irrelevant. Hmmm. Yeah. Right."
But now you are doing it again. Enjoy your paycheck? Is it not possible that some of the people who disagree with you are doing so because you are wrong? I'm still sorry for calling you an idiot, but I don't agree with you.
Edit: I wrote this before you specifically referenced my idiot comment. I think the above explains it (if not excuses it).
Comment below viewing threshold Show
Comment below viewing threshold Show
These things happen, it will all be forgotten soon enough. Except by rival fanboys.
Comment below viewing threshold Show
= 818.181818181818
It's a pretty funny number alright....
Comment below viewing threshold Show
Comment below viewing threshold Show
PS: Some people are waaay too into gaming, complete meltdowns after just one day of no gaming.
Comment below viewing threshold Show
Comment below viewing threshold Show
In any case ,it was more or less some kind of PR thing...saying that it was a PSN problem served to calm some people as the PSN is Sonys responsability so it was up to them to solve the problem and they were at that.It also avoided headlines about 18 million faulty consoles breaking simultaneously as in some true console Armageddon and diverted it to a simple "temporal error in the service".
Thing is that it didnt let players play offline either ,and our downloaded PSN games didnt work either (bar some exceptions as Critter Crunch).So it was clear it wasnt only a PSN thing.
The problem was pinpointed early on,many users at Gaf 1 hour after the debacle already knew exactly what was happening and called for an autofix in 24 hours.
Probably Sony thought they could cape the temporal of 24 hours with 2-3 tweeter messages and assuring everybody that in 24 hours everything would be allright as they must have been sure was the case.
But things didnt play out like that ,yesterday there was a mass hysteria ,with hordes in the net blaming Sony ,exchanging experiences ,finding their own workarounds on the problem ,haters rejoicing ,imaginative people creating hilarious Gifs after Gifs .....it was really a great day as strange as it may sound!
Comment below viewing threshold Show
"Actually didn't MS just do exactly the same thing when the Zune buggered up? Keep schtumm and wait for the problem to right itself?"
MS was getting the word out that there was a widespread problem by 8am that morning:
[link url=http://zuneinsider.com/archive/2008/12/31/ 30-gb-zune-issue-update.aspx
]http://zu neinsider.com/archive/2008/12/3...[/link]
and was releasing press statements explaining what was going on by 2pm:
[link url=http://zuneinsider.com/archive/20 08/12/31/30gb-zune-issues-official-update.aspx
]http://zu neinsider.com/archive/2008/12/3...[/link]
Now whether that was enough or too little is a matter of fair debate. Personally, when I compare that with Sony's handling, I'm more impressed with MS than Sony, but I'm sure this won't convince anybody on either side if the issue either way.
@Sharx:
I’m lying about my opinion? Well . . . damn. You got me there. I implicitly trust Sony without reservation, I was just testing you. Good job spotting the subterfuge tiger.
Btw, you might want to wipe away some of the spittle and froth from your shirt. Makes you look a little like a retarded clown shoe, which I'm sure isn't the look a considered and reasonable gentleman like yourself is going for.
Comment below viewing threshold Show
Comment below viewing threshold Show
Comment below viewing threshold Show
First they wouldn't tell me anything. So I said 'some people think it's the internal clock' - 'ah yes, we think it might be that'. I pushed on: 'some people are opening up their machines and fixing the clock themselves' - 'yes that might work' (not a HINT that I could damage my machine if done incorrectly, or asking if I still had a warranty to explain that it'd be invalidated if I did so). My final question was if reports that trying to turn on the machine might lead to loss of trophies or other data were true 'oh you don't need to worry about that, absolutely no risk whatsoever'.
I suggest next time Sony HQ staff keep an eye on the forums. Most people already knew what was happening, how to fix it and what the risks were, and that it'd probably fix itself.
I've had some bad advice from call centre staff but Sony outdid themselves that day.
Comment below viewing threshold Show
Comment below viewing threshold Show
start latest heavy rain save, exit game, synch trophies.