During production of yesterday's Digital Foundry vs. Forza Horizon article, we reached out to Playground Games with a view to adding their input to our analysis on this exceptional game. We could only include some of their information in our piece, but the team was so lavish and exhaustive in its replies to our questions that we realised that we'd be doing our readers a disservice by not publishing the entire transcript verbatim.
So here it is - a remarkably insightful look into the making of Forza Horizon, covering the genesis of the game, Playground's take on the Turn 10 codebase, how the team grappled with the latency issues in dropping down to 30Hz, the challenges of the new open world, enhancements and changes made to physics, AI, and rendering - and so much more.
Talking us through the technical development of Forza Horizon is Playground Games' technical director Alan Roberts, lead rendering engineer Andy Sage and lead gameplay engineer Karl Hammarling.
Digital Foundry: It's remarkable for a first-time studio to be handed the keys to one of Microsoft's prestige franchises, let alone a studio thousands of miles apart from series founder Turn 10. How did this collaboration come about and what convinced Microsoft and Turn 10 you were the people for the job?
Alan Roberts: Yes, it's a remarkable story and we're really pleased to have been able to fulfil the ambitions we had when we set up the studio.
In the early days of Playground we concentrated on building a company that could scale up to meet the demands of a big budget game. We realised that the best way to sustain serious conversations with publishers was to have a way to prove to them that we could collaborate to make it work. We were a group of very experienced developers, but it was clear that we weren't a ready-made studio and we knew it would take someone with vision to get on board and give us the backing we were asking for.
We'd been talking with Microsoft about a number of things prior to meeting with the Turn 10 guys, but after we met them we knew that the opportunity to contribute to the Forza franchise aligned perfectly with our company goals.
Turn 10 spent lots of time on-site with us and it became clear that we had plenty in common, including our attention to detail and the way we approach game development. Ultimately I think the shared ambition and passion of both teams made it happen.
"Our first meeting with Microsoft was at E3 2010. We pitched the Forza Horizon concept some months after that. All in all it was roughly two years in the making."
Digital Foundry: Can you talk us through the collaboration process with Turn 10 as the project progressed? Factoring in that you're thousands of miles apart, how did that actually work?
Alan Roberts: At the start we spent lots of time on-site with them understanding how their teams and technology worked together. We were able to bring new thoughts and experience to the process, but were also very aware that we needed to stay closely aligned, so very quickly the teams developed a strong level of trust and respect.
The time-zone difference was very challenging to start with, as I'm sure anyone dealing with an eight-hour difference will know. They get to work just as we're going home for the day so we have to be careful when communication happens. They have an experienced team dedicated to supporting the relationship and this is something that has been a huge help. Both teams also regularly visit so that we get plenty of time face-to-face to iron out any kinks.
We were set a goal of working together without impacting each other's projects and I believe we have achieved that. We were able to close out Forza Horizon successfully and on time without the need for significant resource from Turn 10, which is a testament to the way the teams collaborated to solve the unique challenges of the project.
Digital Foundry: From gestation period to gold master, how long did Forza Horizon take to develop?
Alan Roberts: Our first meeting with Microsoft was at E3 2010. We pitched the Forza Horizon concept some months after that. All in all it was roughly two years in the making.
Digital Foundry: Did you take the open world concept to MS/Turn 10, or was it a core concept that they wanted you to develop?
Alan Roberts: Their original direction was that we should create a game that was part of the Forza franchise, but that brought something different to Forza Motorsport. They left it up to us to decide how that would work.
We pitched the concept of an open world Forza game centred on a summer music festival and they loved it. The final game is remarkably similar to the game we pitched, which is a credit both to our design team and the way in which Turn 10 and Microsoft supported us to achieve that.
They were always keen that we held the vision for the game so that we bought into what we were creating, and I would say that's one of the best things about working with them as a partner.
"It was clear from the start that the Motorsport codebase is great and has a lot of strengths. It gave us a head start so that we could focus on adding key functionality and new features."
Digital Foundry: Dan Greenawalt says that work began by handing over the Forza 4 engine to you in its entirety: graphics, AI, physics - the whole shooting match. We must be talking about an enormous codebase here - how do you begin to get to grips with it, let alone customise it?
Alan Roberts: That's a great question, and yes, it's an enormous codebase!
We have some of the best technical guys in the business at Playground, and so do Turn 10. When you get guys like these together working on something they love they're keen to roll up their sleeves and dive in, so it started taking shape remarkably quickly. Turn 10 were concentrating on closing out Forza Motorsport 4, so they mainly took an advisory role but were very supportive.
The first thing we did was get Forza Motorsport 4 up and running. Then we started analysing the major areas that would need changing in order to make Forza Horizon. We had a lot of questions in the early days!
It was clear from the start that the Motorsport codebase is great and has a lot of strengths. It gave us a head start so that we could focus on adding key functionality and new features. It was quite close in design to technology we had created in the past, so it was great to see that two independent groups had been thinking along similar lines without knowing it.
In the beginning, even with our small team of seven engineers and a handful of artists, we had the proof-of-concept of our open world streaming technology, complete with fully functional gameplay prototype, working within six weeks. I've never seen such a small group achieve so much in that sort of time frame. It gave us a really solid start to the project.
Digital Foundry: Clearly, Horizon is a very different beast to Forza 4. How easily did existing tech integrate with your own code? Was it possible to slot in modules from the existing engine or did you build the new game around the fundamentals of the Turn 10 engine?
Alan Roberts: We built the game around the Turn 10 engine, as we were really keen to retain the Forza DNA, such as the car physics engine. It was surprisingly quick to get started, but we were making some pretty fundamental design decisions that required serious divergence, so in some areas we have completely changed the architecture. Streaming cars whilst the game is running is a really good example of this.
Digital Foundry: You've spoken about the Forza physics model being employed in your game, updating with 360 samples per second. How did this system - and the other legacy code - square with being run within a 30FPS environment? Did any changes need to be made?
Andy Sage: It was important to us that we kept the physics running in the same way as Forza 4 even though the renderer was changing to 30FPS. Forza 4 already runs multiple physics updates per 60Hz frame, so when we halved the frame-rate we made sure the simulation was running twice as often per 30Hz frame to compensate. This means we keep the same lightning fast physics update. All of the other simulation systems, such as AI, were updated in the same way. We spent a lot of time profiling and testing that the behaviour remained consistent with Forza 4.
"We built the game around the Turn 10 engine, as we were really keen to retain the Forza DNA... but we were making some pretty fundamental design decisions that required serious divergence, so in some areas we have completely changed the architecture."
Digital Foundry: Latency is a major challenge to overcome in the move from 60FPS to 30FPS. There's a feeling of crispness to a 60FPS game that most 30FPS titles simply can't match. Others would say that 60FPS is a defining factor in the Forza interface. What was your approach here in ensuring the best possible experience?
Andy Sage: Early on in pre-production we experimented with introducing varying amounts of controller latency into Forza 4 whilst keeping the 60Hz update - to try and assess what kind of margins we had to work with.
We paid a lot of attention to ensuring the controller input and the player's car physics calculations are as late as possible in the update code - there are lots of other calculations, eg. the AI cars, streaming work, that can be done earlier in the frame - and it all helps reduce the latency between player input and what you seen on the screen. Another thing we do to help is ensure that the latency between the physics and the audio is as low as possible as this doesn't need to be impacted as much as a result of the switch to 30Hz rendering.
Digital Foundry: You point to the sheer consistency in the 30FPS update as the defining factor in giving a great feeling of movement to the game. Do you employ any techniques such as motion blur to mitigate the judder reducing frame-rate inevitable causes?
Andy Sage: Yes, we use post-processed motion blur based on per pixel velocity to improve the smoothness. We added several techniques onto this to allow it to run within the post-process frame budget and still look as smooth as possible, such as screen space dithering and velocity curves.
Digital Foundry: What's the secret in maintaining frame-rate bearing in mind the multitude of factors that can influence the time taken to render the next frame? Bizarre's Blur used an innovative LOD-balancing system, for example.
Alan Roberts: Solid performance is something that needs to be bred by a studio culture that underpins everything you do. This means artists, designers, audio guys and coders all pulling in the same direction to make sure budgets are hit. We're really in alignment with Turn 10 on this and you can see the results of employing that culture both in Motorsport and Horizon regardless of the target frame-rate.
Andy Sage: The approach we took was to ensure that our worst-case CPU and GPU usage for each system was always within a specified budget, which meant we didn't have to do any dynamic load balancing which can lead to visual popping. Achieving this requires constant attention to performance and optimisation of all systems during the development of the game, and ensuring that this information is presented clearly and can be acted on by the content creators.
"Early on in pre-production we experimented with introducing varying amounts of controller latency into Forza 4 whilst keeping the 60Hz update - to try and assess what kind of margins we had to work with."
The only area we did dynamic load balancing was with the post process anti-aliasing system, so screen quality increases when there is spare GPU time to do so.
Digital Foundry: The major game-changer this time around is the open world - the ability to move beyond circuit racing and take these amazing machines out onto the open road. From a technical perspective, what were the major challenges here?
Andy Sage: The streaming system had to be heavily updated to cope with being able to drive multiple routes and large open spaces. This also lead to some real challenges when considering how much data could be read from the DVD given the speed at which you can move through the world.
Coping with the number of objects in the world meant updating several systems so we could still fit the game in memory - in particular some content is procedurally generated to reduce its footprint.
The texture variety in the world also proved a particular challenge. We had to invest a lot of time in ensuring that everything would fit into the memory budget available for the environment, given the diversity and the fact that you can see from one side of the map to the other.
Alan Roberts: Forza Motorsport is really well optimised to gain the most out of the platform for the particular challenges it faces. However, with open world we clearly had some very different challenges that needed specific attention and optimisation.
Aside from the changes Andy already talked about we had to stream many more additional data types, such as physics and AI, to give us more memory to spend on graphics.
We also have very different requirements when it comes to bringing the world to life, as we're not constrained to racing circuits. We had to employ a lot more animation and particle effects to make the world feel lived in. The festival area itself was a particular challenge as there's a lot going on there. All of this hits streaming bandwidth, performance budgets and is heavy on memory so we had to write bespoke systems to deal with it.
We even had to make specific changes to cater for cars being at large distances from the world origin, such as precision updates for rendering systems to avoid z-fighting and a different way of packing synchronisation data for the cars when playing online game modes.
"Forza Motorsport is really well optimised to gain the most out of the platform... However, with open world we clearly had some very different challenges that needed specific attention and optimisation."
Digital Foundry: You have some spectacular vistas here, a scale far beyond the existing Forza titles - what kind of maximum draw distance is there? What rendering techniques did you employ to emphasis the sense of scale?
Alan Roberts: Colorado definitely provided challenges in that respect. When you're at the top of Red Rock you can see the mountains on the other side of the map. It's all modelled and we couldn't get away with photos of the horizon like you would in some other types of game. It really is the case that if you can see it you can drive to it. The artists have done an incredible job dealing with this sort of scale.
Andy Sage: The maximum draw distance is around 20km. To achieve this we had to ensure that the environment could LOD seamlessly across this distance. We developed custom technology to deal with this that we call 'Uber-LODing'.
Uber-LODs allow us to hierarchically combine objects within the scene when far from the camera. This reduces the overheads of drawing the meshes and means that less data needs to be streamed when driving through the world. More detail can then be focused on the near-distance without impacting draw distance.
We also concentrated on getting the atmospheric effects to accurately model the visual impact of the large distances involved, including investigating a large amount of research material and writing a new atmospheric scattering model and a detailed dynamic sky simulation.
Digital Foundry: Another fundamental difference is the lighting implementation. Dan Greenawalt has spoken about Forza 4's HDR model and image-based lighting forming the basis of your initial experimentation here, but from what we've seen we're looking at a much more advanced model. Do you use deferred shading?
Andy Sage: From the start of the project we realized that we had a great starting point with the high quality and feature set of the lighting model used by the Forza 4 engine, including a fully gamma correct lighting pipeline and image-based lighting model. We wanted to leverage this system as much as possible, and build on it for our requirements for Horizon.
We decided early on that we wouldn't be able to transition to deferred shading due to its inherent drawbacks, in particular the material variety limitations that would have impacted the vehicle rendering. Instead we used a hybrid forward rendered approach that allowed for multiple dynamic light sources along with static lightmaps for environment lights. We also added a range of other lighting effects, concentrating on the car light sources in particular, and also ensuring that everything was fully controllable by the artists with real-time visual feedback.
Another important aspect of the changes required involved dealing with a fully dynamic 24 hour cycle, for which we added over 200 time dependent parameters to control all aspects of the day to night transition. In addition this also meant we had to develop a high quality real-time shadow system utilising multiple cached and non-cached cascaded shadow maps.
Digital Foundry: Turn 10 experimented a fair bit with anti-aliasing in Forza 4 - 2x MSAA on racing, 4x MSAA on the 30FPS replays and on time trials. Elsewhere, there's been a gradual shift to post-process AA generally within games development. What route did you choose for Horizon and why?
Andy Sage: Our approach is based on 4x MSAA combined with FXAA. During the prototyping phase we tried several approaches and found that this gave us the highest visual quality (albeit at the expense of extra post process GPU usage).
We found FXAA alone didn't cope with the amount of high frequency detail that can be present in some images, and starting from only one sample per pixel doesn't present enough information to avoid high frequency noise and aliasing. In addition, 4x MSAA allows us to use the order independent transparency advantages of alpha-to-coverage. Where post process anti-aliasing really gives us a benefit is on our intermediate render targets that don't utilise MSAA, for instance we use a screen space mask for dynamic lights to which we then apply FXAA, which allows us to reduce aliasing.
"The [car] assets had to undergo a significant number of changes to support new features, such as headlights and indicators, emissive interior dials for night time, dynamic lighting, and dynamic shadowing."
Digital Foundry: Are vehicle assets interchangeable between Forza 4 and Horizon? Or are they independently tailored to suit different performance targets? What sort of polycounts and texture sizes are there on a typical vehicle?
Andy Sage: The cars are a stand-out part of the Forza franchise, so we were keen to make sure they remained as such, but we couldn't keep them interchangeable due to the differing demands of the games. The assets had to undergo a significant number of changes to support new features, such as headlights and indicators, emissive interior dials for night time, dynamic lighting, and dynamic shadowing. They were also optimised to allow them to stream faster and take up less memory. Despite this, we made sure we retained the huge poly counts and high resolution textures you see in Forza 4.
Digital Foundry: Aside from the actual modelling, what is your process in bringing a car to life within the game? Is it a matter of plugging in real-life variables into the simulation and then tweaking it until it feels right? Or is there a more scientific process?
Alan Roberts: A massive amount of detailed research goes into the cars. The car guys even go to the lengths of dismantling cars to weigh and measure components. It's a hugely impressive process.
All the data they build up on each individual part gets plugged into the 'Automagic' tool, which is an offline simulator that does some very intensive mathematical modelling. The result of this, combined with the real-time physics simulation, is the real secret behind the Forza physics engine.
The sheer number of combinations of cars and possible setups mean that the simulation has to be spot-on without tweaking by hand or making subjective calls on how a car 'should' drive.
Digital Foundry: Being an Xbox 360 exclusive, you have the opportunity to develop the game around the specific strengths of the hardware - to what extent do you feel you've pushed the 360's bespoke features? Do features like memexport, VMX or even the almost forgotten tessellator get a look in?
Alan Roberts: A lot of the senior developers come from multi-platform development houses so we're used to having to compromise on quality to ensure things work in multiple hardware and OS configurations. It has given us a great new lease of life to focus on a single-platform game; we can be a lot more aggressive and the freedom is refreshing. We found we can extract a lot more performance from the Xbox 360 and still feel like we could do more.
Andy Sage: We pushed the game very hard from an optimisation point of view as we only had a single platform to work with, and we could code to its strengths.
This mainly involved carefully balancing the performance trade-offs inherent with a console architecture to achieve the best look possible rather than using bespoke features. For example, we spent a long time tuning performance factors such as vertex/pixel load, texture/ALU ratio, and bandwidth. With a single platform this allowed us to avoid the 'lowest common denominator' syndrome, and to really push the content creation towards the areas that the hardware excels at.
In addition we also spent plenty of time on detailed performance analysis and scheduling of the CPU workload to ensure a high utilisation across all six cores available on the 360 during each frame.
"It has given us a great new lease of life to focus on a single-platform game... We found we can extract a lot more performance from the Xbox 360 and still feel like we could do more."
Digital Foundry: What was your approach to AI within Horizon? Was this a component that could be dropped in from Forza 4 or did it require significant customisation? Did the open world aspect require a new approach?
Karl Hammarling: We wanted to make the world feel like it is inhabited by other drivers who are exploring and competing with one another at the same time as you, so we added more layers of dynamic behaviour to allow them to be much more reactive to situations. Thus you will see, for instance, groups of festival drivers hooking-up and racing together through the world.
We re-used a significant portion of the Forza's 'Drivatar' AI system as it's very good at finding its way along a single track ribbon - the challenge was constructing these ribbons dynamically for any given route through the world, which can be decidedly complex, especially when passing through several junctions in quick succession. As a bonus, this allows us to project the raw, festival-goer attitude of the racers, as we can build-up variable, character-driven racing lines, as opposed to the single groove racing line traditionally seen at real-world circuits.
The vehicle avoidance was the most significant update to the existing Forza AI, to cope with vehicles approaching from arbitrary directions. We had to deal with much higher closing speeds due to stationary vehicles or head-ons, which required a lot of testing and tuning. The AI has to be able to cope with navigating these conditions whilst racing at full speed, so the margins for error are very small.
Finally, the local traffic was introduced as an entirely new AI system on top, interfacing with the Forza AI mainly for avoidance purposes.
Digital Foundry: With the game complete, what are your final thoughts on your creation? Have you begun the post-mortem process yet?
Alan Roberts: We're really proud of what we've achieved, especially given that we had to build a studio environment, development team and game at the same time. Even so, we have post-mortems on the go at the moment as there are always things that could be done better. It's going to be great to see what we can do in the future with the talented team and studio we have built up. Watch this space!