It's been a while since we've run a tech interview, but when Microsoft approached us with the opportunity to quiz The Coalition about the technical work behind the recently released Gears of War: Ultimate Edition, we seized the opportunity. The original Epic release was a landmark last-gen title - a game that thrust Unreal Engine 3 into the limelight, its technology and its artistic approach helping to define the aesthetic of a vast range of Xbox 360 and PlayStation 3 titles.
What's clear is that the Ultimate Edition is a full-on remake done right - fans and purists will have their own view on how the game should have been updated for the current-gen era, but there's no doubt that the vast majority of the game's visuals are updated beautifully, honouring the spirit of the original material while at the same time presenting a radically new look, based on today's cutting-edge rendering principles. And at the same time, what's equally clear is that the Ultimate Edition remains a Gears game through and through - the biggest surprise for us being that the original gameplay, faithfully replicated here, still manages to hold up nine years after the title's original Xbox 360 release.
In this interview, The Coalition talks us through the whole process of bringing the Xbox 360 game to Xbox One. You'll learn about the way the original game was ported across and built upon, how the team updated the rendering technology, as well as the approach taken in remaking all of the environments without impacting upon the way the original game played. On top of that, the team also clue is in on how the Windows 10 DirectX 12 version is progressing, the kind of visual enhancements we should expect to see there, and how the PC version scales across hardware platforms. There's also a small hint on how Xbox One's backwards compatibility works (native x64 executables?) - thus far, technical details there have been thin on the ground.
We had a lot of fun playing Gears of War: Ultimate Edition. Indeed, our resident Gears fan John Linneman put his heart and soul into producing our recent piece on the remake. But the bulk of Digital Foundry's work is based on the viewpoint of an outsider looking in. This interview allows us to tell the story behind the making of the game from an entirely different insider's perspective and we hope you enjoy it. Answering our questions, we have:
- Mike Rayner, Studio Technical Director
- Jaysen Huculak, Technical Lead
- Stu McKenna, Senior Software Engineer
- John White, Senior Software Engineer
- Bart Mazus, Online Services Lead
- Cam McRae, Technical Director - Windows 10
Digital Foundry: So you're given the green light to give Gears of War a full remake. Where do you begin with such an open-ended brief, and how did your design strategy crystallise into the product we have here today?
Jaysen Huculak: I still remember the day I found out. I was called into Mike Rayner's (Studio TD) office and after the surprise and excitement settled, we discussed the possible approaches which basically broke down into three options: start with the original, start with the latest UE3 engine or update to UE4.
We had a philosophy of keeping the game playable throughout development so we felt that to achieve our goals and ensure we were playing the game as often as possible, our best starting point was with the original codebase. From a content perspective this meant things would already be working on day one. Our engineering team has a long history working with Unreal technology but at that time we had fairly limited exposure to the gameplay code for Gears. We discussed the approach as a team, did a site visit to Epic and discussed it with them and outside of the challenges of working with 10-year-old tools, we agreed this was the best approach. That is how our journey with the original codebase began.
The key challenge in either of the other options was that all the level data would be tough to keep working through the transition and would result in a large amount of time when the game was unplayable. And ultimately we wouldn't be able to guarantee the same functionality as the original since the underlying systems had changed. The only other option would be to throw it all away and rebuild all the level scripting from scratch and we felt the chances of missing something in the gameplay experience would be too great.
With that decision made, our engine team began work immediately on getting the game up and running on Xbox One while our gameplay team went to 'Gears of War Gameplay University' to dissect and learn everything that makes Gears work. This combination allowed us to evaluate gameplay additions based on trying them in-game on Xbox One at 60fps earlier in development.
We are very happy with the decision to go this route but we did underestimate some of the challenges of working with such an old editor. Unreal Engine 3 went through numerous updates and improvements in the editor and we could have improved our efficiency by bringing more of these over earlier in production. The trade-off with staying on the original engine with all the original gameplay scripting was that we didn't need to rebuild everything from scratch to create something that just looked like Gears 1 gameplay, our version would be true to the original.
Digital Foundry: Based on earlier statements, it seems pretty clear that a shift to UE4 just wouldn't work. But there are clearly major rendering enhancements - so how did this work? Did you import the original source into a later version of UE3, or build upon the original codebase?
Jaysen Huculak: We started with the original codebase and brought changes and upgrades in based on recommendations from our rendering team. The Coalition and Microsoft are active contributors to the Unreal Engine and have a really good understanding of how different rendering features perform on the Xbox One. That knowledge allowed us to start with the original games features running on Xbox One and then we harvested later additions from UE3 as well as a few from UE4. Most of the editor-facing features were pulled from UE3. An example of this was Lightmass (the Unreal Engine lighting solution), which went through several months of work to get features up to roughly the 2011 time frame. Other features were drawn more from UE4 or our own work on UE4-like physically-based shading and many of our own Xbox One specific enhancements like ESRAM management. We also experimented with some asynchronous compute for optimisation that wasn't available in the UE3 time frame.
Digital Foundry: The jump from PowerPC to AMD's x86 Jaguar has proven to be less then trivial in some cases, particularly in light of some of the performance issues seen in the Master Chief Collection where there are (or rather, were) clear CPU bottlenecks. Does this in part explain the reason to lock to 30fps in the campaign?
Mike Rayner: We decided 30fps for campaign and 60fps for multiplayer very early on as a design decision and not a technical limitation. The original was a visual showcase for Xbox 360 and we wanted to continue that on Xbox One by significantly updating the campaign visuals at 1080p while also keeping a more consistent 30fps frame-rate than the original. Running the campaign at 60fps is doable, it just doesn't look nearly as good as the 30fps version.
Digital Foundry: There's little sense from our analysis that you're GPU-bound - how did you set your render budgets and what was your optimisation strategy?
Jaysen Huculak: As part of our 'always playable' methodology, most of our budgets are set and maintained by monitoring real world performance in the game. Through our development this allowed us to know where we stood from a performance standpoint with all the original content and monitor and update as each map was remade. We had both automated and manual daily playtests of each map in active development. Specifically for the GPU we targeted between 8-12ms on a multiplayer map with only your character in it to allow for fluctuation with gunfire, grenades etc.
There were many rendering challenges throughout development but we were more often CPU-bound feeding the GPU due to the number of objects added and we had to employ tools to merge geometry together as well as offloading additional rendering to more cores. A lot of hard work went into making our GPU performance solid and this involved rewriting nearly every post process effect in the game, maximising memory bandwidth usage with ESRAM and many more optimisations.
Digital Foundry: There's been a lot of discussion about the 32MB limit on Xbox One's ESRAM - what's your strategy for its utilisation in Gears Ultimate?
Jaysen Huculak: Fun question and you get a description right from the primary developer of our ESRAM management.
Stu McKenna: Initially we only put the colour/depth buffers into ESRAM as we are forward-rendered, so main buffers (colour/depth) fit nicely into ESRAM at 1080p - this was a huge performance boost. Later in the project we looked at our ESRAM usage again to get extra performance back. From looking at the analysis you can see that only certain passes and resources will give you any benefit inside ESRAM as we were not ROP-write bound in many cases (ie ALU or texture read bound).
We added a new ESRAM allocator that leverages virtual memory to map pages to ESRAM/DRAM per 64k. We added lifetime management for resources we want inside ESRAM so that once they are released the physical pages are returned for other resources to use. Depth and colour targets on Xbox One can be compressed and we noticed in some cases splitting that planes between DRAM/ESRAM can also be a win as it frees up ESRAM memory for other more important resources.
The new allocator was a good performance win for us, especially during shadow passes where we could do all our shadow depth writing and reading inside ESRAM. Using a naive approach of putting 'everything' into ESRAM you could certainly hit the 32MB limit, especially with a deferred renderer. In our case it wasn't an issue and we still had the option to use the DMA engines to move resources in/out if we needed more space.
Digital Foundry: Pre-launch, there was plenty of discussion about the SHAPE processor in Xbox One and the implications of its use in keeping CPU utilisation low for audio tasks. Post-launch, we've not heard much more about it. Does Gears Ultimate leverage this element of the hardware?
Jaysen Huculak: We do use the audio processing available on Xbox One and the good news here is that we never saw audio as a CPU bottleneck and things just worked. Given the increase to 7.1 this is a huge increase in the number of individual sounds we're processing and the hardware compression is very handy for helping to manage this.
Digital Foundry: Can you talk us through the key enhancements you made with regards the lighting? Presumably you're using Lightmass for global illumination but can you tell us more about the ambient occlusion and the physically-based rendering system? Any particular PBR model you're utilising?
Jaysen Huculak: For sure! We are using Lightmass for the lighting and started with the 2006 version. We went through two major integrations, one to bring it up to roughly 2009 and one to bring it to 2011. This work sometimes has to go in stages to make sure all the pieces are working to move to the next level. The first integrations got us global illumination, which in itself was a big step up visually. Later we brought in support for dominant directional lights as well as signed distance field shadows and pre-shadows. This improved specular lighting as well as allowing for dynamic objects to receive and cast shadows that correctly blend with the environment. Our physically-bases shading is based on the UE4 method.
John White: The lighting model was switched to use the UE4-based PBR. This is Lambert for diffuse with Schlick Fresnel and geometry terms with GGX normal distribution. Normal maps could be optionally processed with Toksvig mapping to help with Spec AA. Keeping the UE4 model was very intuitive for the artists who were very familiar authoring using the model.
All lighting is now in full linear HDR space. The post-processing stage was completely rewritten to do all processing in linear HDR space and performs dynamic exposure adjustment, physically correct bloom, lens dirt, bokeh DOF, motion blur, vignetting, filmic tone-mapping, gamma correction, colour grading via 3D LUT and FXAA. Multiplayer omitted motion blur and DOF and was 1.1ms for all other effects enabled. [For] Ambient occlusion, SSAO was depth based on volumetric obscurance and twin pair sampling with a separate dilate and blur pass.
Digital Foundry: If we recall correctly, the original version made use of the Xbox 360's 7E3 ("FP10") format for HDR lighting, are we up to FP16 this time around?
John White: We used the 32-bit RGB101111 format instead of FP16x4. Using FP16x4 gave us no noticeable quality increase so we kept the full rate 32-bit format, which also helped with ESRAM utilisation.
Digital Foundry: Multiplayer hints at elements hailing from the Gears 3 - weapon skins, enemy spotting, movement etc. How did you balance the need for authenticity to the original game with the temptation to include gameplay improvements from the sequels?
Jaysen Huculak: I'm going to answer this from a technical perspective in terms of what features we brought to enhance the experience. Peder, our design lead would have more feedback from a design perspective on the design choices. From a technical standpoint we wanted to provide dedicated servers, responsive controls (fast frame-rate, client side hit detection) and a consistent high frame-rate.
Bart Mazus: We also wanted to ensure that the matchmaking and multiplayer experience was as up to date and as solid as possible. This meant adding support for such things as squads, playlists and the ability to interact and be able to communicate with the community. Ultimate Edition also relies far more on external services then any of the previous Gears of War titles. From the matchmaking point of view our goal was to get you into a game as quickly as possible, with the best match possible. We added skill-based matchmaking, introduced the levelling system (similar to Gears 3), and in general worked closely with the platform folks to ensure a best-in-class multiplayer experience. One of the things that allowed us to do this was various playtests, both internal and external throughout the development of the project.
Digital Foundry: By extension, are there any behind-the-scenes revamps on improving the co-op code?
Jaysen Huculak: We did some updates to the guts of co-op to make replication and joining in progress work better in some areas but the most visible things would be going with split-screen pillar boxing for better FOV (field of view) visibility and all of the improvements for versus play that also apply to co-op.
Digital Foundry: Can you talk us through the process of remaking the environments? Did you build upon the existing levels, or did you simply start again, completely from scratch?
Jaysen Huculak: The short version is that we started with the original and made sure we replaced everything but the collision as a replacement phase and then enhanced it further with lighting, VFX and additional hero assets and work. Outside of collision we re-did the levels from scratch.
On the technical side we had to do a few things to support this. First we build out a big database of all of the assets in Gears and what levels they were referenced from. This helped the art team in planning asset creation. Our tools team further analysed the meshes and determined where objects were scaled and if that scaling was uniform so we had an idea of how many additional assets would have to be made to keep the texel density consistent throughout the game. The original development team was very creative and efficient in re-using assets but there are visual tradeoffs we didn't feel were acceptable in 2015.
Another thing we did early on was to lock down all of the collision etc so that things couldn't be modified by accident. It is sometimes easy to move something by mistake so we defaulted to locking collision to avoid small errors. The bulk of the technical work came from partnering with the art and technical art team through each phase of visual development to ensure that the features needed for levels were working and that the map performance was in where it needed to be. We were optimising and adding features through all of the 18 months of development, which sometimes made it difficult for the art team. We had a really strong team of technical artists both at The Coalition and Splash Damage that really helped balance these issues.
One thing that has to be said is that people cannot see the authoring challenges that go into making a game like the Ultimate Edition. For the majority of the project our content teams were working with a 10-year-old engine. Things you take for granted in the latest UDK or UE4 simply weren't there. So we updated what we could to make the workflow better and had a fair number of coding contributions from our technical art teams to help with issues.
Digital Foundry: How do the poly-counts compare for characters, weapons and environment assets versus the originals?
Jaysen Huculak: For environments, our initial budgets were for the replace stage and covered vertices and textures. For vertices, our budgets were fairly light on increases in polygons as our artists found they could rebuild the models and with only a 10 to 30 per cent increase in geometry and improve them substantially. With textures, we have twice as many and a resolution higher for 4-16x the memory usage in some cases. The larger additions came from replacing 'kit-bashed' assets and enhancement phases where additional assets beyond what was in the original are created and placed. Finally, optimisation passes would be done that would trade grouping objects together into stitched similar assets for faster rendering. Characters in some cases like the Corpser doubled the vertices and had similar texture increases.
Digital Foundry: Animation looks like a fairly close match for the original. Are there any changes here, or would it have compromised the authenticity of the original experience?
Jaysen Huculak: The raw animation data for characters was kept the same to ensure the gameplay experience was preserved. The character models and their rigging as well as secondary animation was overhauled. For example the characters armour vs skin was weighted more appropriately so that armour did not stretch as much and objects on character's belts animate in the campaign. Weighting was also extended to allow up to eight bones per vertex rather than the original four.
Digital Foundry: We rather like the changes you've made to the aesthetic, but purists seem to want the more monochromatic presentation. Did you consider including the various colour filters (lucid/soft/vivid) found in the original game?
Jaysen Huculak: In our concept phase we looked at what it would take to provide a full toggle similar to the Halo Combat Evolved Anniversary Edition. We didn't want to compromise how far we could go to accommodate a full Xbox 360's worth of memory as well as the impacts on load times so this feature was removed in pre-production. This led to a discussion about providing a colour filter or estimating the original look with a lower resolution. Ultimately however we felt it would be better if we focused all efforts on just making it look the best it can.
Digital Foundry: There's a nice moment in the UE where you're pushing a flaming car down a hill to protect yourself from the Kryll. In the original TGS 2005 presentation the flaming car was a full light source, but this was omitted in the final version of the game. This is restored for Xbox One. Was that a deliberate design? Any other examples of compromises made during the original development you could deploy in the Ultimate Edition?
Jaysen Huculak: The level designers and artists were empowered to reimagine the lighting as they saw fit so long as the levels stayed within the performance restrictions. This enabled enhancements like you mention as well as light functions throughout the levels that allow lighting textures with colours in areas or around some fires.
Digital Foundry: The existing Xbox 360 Gears titles come to Xbox One via backwards compatibility - is this essentially the exact same code, or is there any kind of conversion or optimisation process involved?
Mike Rayner: It is essentially the exact same code. The Xbox team converts the 360 game and 360 flash PPC executables into native x64 executables, packages those up with the 360 game assets, 360 flash and emulator as a regular Xbox One game, and publishes it.
Digital Foundry: Ultimate Edition is coming to DX12 and Windows 10 - can you talk us through how the engine takes advantage of the new API?
Cam McRae: We are still hard at work optimising the game. DirectX 12 allows us much better control over the CPU load with heavily reduced driver overhead. Some of the overhead has been moved to the game where we can have control over it. Our main effort is in parallelising the rendering system to take advantage of multiple CPU cores. Command list creation and D3D resource creation are the big focus here. We're also pulling in optimisations from UE4 where possible, such as pipeline state object caching. On the GPU side, we've converted SSAO to make use of async compute and are exploring the same for other features, like MSAA.
Digital Foundry: It's been mentioned before that the PC version supports higher resolutions and frame-rates. Will there also be other anti-aliasing modes supported (MSAA, SMAA etc)? Better ambient occlusion, particle effects, shadow-map quality etc?
Cam McRae: We have put significant effort into making the Windows 10 version a showcase at 4K, geo and textures were re-authored with 4K in mind so the visual fidelity will really scale up on higher end hardware. We plan to uncap the frame-rate and will ship with a built-in benchmark mode. For anti-aliasing we'll support MSAA and FXAA.
Digital Foundry: On the flip-side, how low will the PC version scale down?
Cam McRae: While we haven't locked how far the game will scale down, it is important to us to ensure good performance (at least 30fps) on a variety of hardware. We will support a broad range of resolutions as well as a variety of graphics options for gamers to tweak their setup.