Digital Foundry vs. PS3 Skyrim Lag

Updated: Obsidian's Joshua E Sawyer explains memory management issues affecting Fallout: New Vegas that sound very similar indeed to what is happening with PS3 Skyrim.

Original story: Given its scale, Skyrim was always going to be a game with rough edges. For some, however, its issues are cutting into the gameplay experience to an unacceptable degree. Rising above all the specific glitches and quirks on the other formats, the PS3's lag bug is currently considered the one in most dire need of addressing, with huge dips in performance being reported by those with save files of 5.5MB and above. Cue Bethesda with the release of a new patch (2.01) earlier this week which, according to the notes, "improves occasional performance issues resulting from long term play".

After updating, some players on the Bethesda forums have reported that it has helped alleviate the problem slightly, while others remain adamant that it hasn't changed a thing. So the big question is: has Skyrim update 2.01 actually improved performance on PS3 at all?

First, let's put things into perspective by seeing the bug in action on patch 2.00 - the day one update users upgraded to when the game was released. Here we have a comparison of performance between a completely fresh, brand new game, and a whopping 65 hour save, with each file taking up 3MB and 12MB respectively.

For the sake of this test, we find that a simple stroll through villages and pre-defined cobble pathways is all we need to show the extent to which the frame rate crashes down (no battling necessary - you'll see why). The issue isn't immediately evident on booting the game, but just 30 minutes of play on this 65 hour save is all we need to kick-start the stuttering, with the most effective method of reproducing the effect involving fast travel between a series of six major destinations on the map.

As you can see on the 65 hour side, the effect is impossible to miss and given its severity, the outcry that has followed on forums is very much justified. These prolonged bouts of stuttering render the game almost completely unplayable, even during non-intensive walks down pathways, with some freezes lasting long enough to drag the frame-rate down to zero in places - the first time we've seen this in years of performance analysis here at Digital Foundry.

Expectedly, it's business as usual for the new game save, which only suffers from a few hiccups when attempting to stream geometry or passing through areas with a high density of NPCs. These are a minor nuisance, and generally don't detract from the gameplay to anywhere near the same extent.

When the 65 hour game isn't stuttering, we notice that it tends to settle on 20FPS as its baseline, with the new save holding out at around the 30FPS mark. Considering that the same environments are being rendered in each case, this strongly suggests it isn't an issue that can be solved by direct graphical tweaks. Both runthroughs are seeing the same geometry, weather effects, shadows, and presumably have the same LOD settings in play.

Interestingly, walking past NPCs seems to be a more consistent instigator of these stutters when they do happen, particularly as you pass through heavily occupied places like the Whiterun market, which coincides with the larger freezes. There's a definite sense of cause and effect here which indicates that areas where a player has an extensive history of interaction, via quests events or dialogue trees, are prone to the greater pitfalls. This isn't a hard and fast rule though, as walking through incomplete dungeons or outdoors terrain can also bear down heavily on the console and make for some choppy motion - just not as badly.

To What Extent Does The Patch Help?

So now for the real test: in the video below, we take the same 65 hour save through the same locations on patch 2.01, and compare the results to 2.00. Again, we give each playthrough a similar 30 minute "burn-in" period to make sure the bug is in full force before proceeding.

So has the new patch improved performance? Yes, it has benefited, with the aforementioned 20FPS baseline being raised to something closer to 25 on average. However, the root cause of the stuttering - whatever it may be - is still there and the potential for zero frame refreshes per second is still a possibility, leading to some very unwieldy controller response and an extremely poor gameplay experience in general. To all intents and purposes, this updated version of Skyrim on PS3 is still unplayable for those with a huge time investment in the game - unless you're only willing to play Skyrim in half-hour bursts, saving and restarting the game. Hardly ideal.

So what might be the problem? Seeing as the 360 doesn't appear to manifest these issues under the same conditions, it raises the question as to why this glitch is proving to be such bugbear for Sony's platform alone. There are certain aspects which separate the two consoles, such as the split RAM setup of the PS3, and the mandatory 4.3GB install, but neither of these differences seem to hint at an absolute explanation for the problem.

The theory currently being proposed on the Bethesda forums is that there's a memory leak. The idea here is that the PS3's RAM is slowly being filled up with data during the course of play, which forces the game to stream more regularly from the HDD, causing these momentary freezes.

Having profiled the phenomenon, we have our theories about what's going on too. Skyrim is a game with an enormous amount of objects and characters in play. The gradually expanding size of the save game strongly suggests that it acts as a database that keeps track of how the active pool of objects, characters and other elements has altered since the game began. Perhaps after reaching a certain size, database management requires much more processing.

Another theory concerns where Bethesda store this database. The split-pool RAM architecture on PS3 is more constrictive than it is on 360, and this is clearly a RAM-intensive game. We wonder if the save game resides in the graphics RAM instead of the XDR, which Cell addresses much more quickly. Bandwidth between the PS3 CPU and VRAM is a mere 16MB/s, compared to the 22GB/s access the 360 has to its unified 512MB of GDDR3. Advanced database management on a much larger save would have clear performance implications bearing in mind this bottleneck.

An Unbounded Game Running on Space-Constricted Systems

However, the most likely explanation is that an advanced Skyrim gameplay state - like the 65 hour one we tested - proves to be too much for the garbage collector, or memory defragmention method, struggling to cope with the multitude of gameplay elements that have been altered from the default "virgin" state stored on the hard drive. To test the possibility of this, we tried loading our 65 hour save and playing with it for 30 minutes, and then directly loading our new character save from the menu - the result being that the stuttering does indeed carry straight over to our virgin save. It suggests that memory has been allocated that the garbage collector cannot reclaim, which isn't flushed when the player returns to the menu screen and starts the game anew.

The bottom line is that Skyrim is an unbounded game world running on a space-constricted system - and this applies regardless of the platform you play it on, hence reports of the PC version running out of address space and displaying solid colours instead of textures. The PlayStation 3 is unfortunate in that it's the platform with the most oppressive RAM issues (in addition to the split-pool set-up of the memory, the OS has a larger footprint than its 360 equivalent) so it makes sense that it has the most noticeable issues. But we do have to wonder how 360 owners with a 65 hour-plus save are coping, and whether the same issue manifests for them - just further along in the gameplay. If any 360 owner has a similarly mammoth save, please get in touch.

Overall it's clearly disappointing that these performance issues persist on PS3 even after patch 2.01. The way in which Bethesda addressed the 360's texture streaming issue gives us hope that the company is aggressively chasing problems but the fact that this issue hasn't been resolved with its first attempt clearly casts doubts over whether an absolute fix is possible.

It suggests that the firm knows what the problem is but fixing it completely is much more of a challenge than simply reworking some errant code. Certainly the patch note suggesting that the issue relates merely to "occasional performance issues resulting from long term play" is an extraordinary level of understatement based on the evidence on this page, not to mention community feedback on Bethesda's forums, plus the sterling work by Future's Tim Clark in highlighting the problem.

Of course, playing the PS3 version in short 30 minute bursts and rebooting is unacceptable as is expecting players to fork out for another version after investing so much time into their existing purchase. Ultimately, it appears the only way forward for now is to leave feedback for Bethesda, detailing where the problems crop up most in your game. It's a position paying customers shouldn't find themselves in and the implication that the game was released without proper testing is troubling, but the sooner the problem can be isolated, the better for everyone.