Digital Foundry vs. Project Natal

Microsoft's announcement of Project Natal at this year's E3 was, for many, the event of the show; an exciting piece of brand new technology promising to revolutionise the way we play games. No joypads, no sticks, no buttons... no controller! Combining a traditional RGB camera with an infra-red sensor, along with a multi-array mic and Microsoft's own voice-recognition technology, Project Natal is all about bringing people into the game, literally. Capable of tracking and interpreting motion data for up to four people simultaneously, the human body becomes the controller. In effect, it's a pretty sophisticated motion-capture set-up designed for the home, and it's exclusive to Xbox 360. It's also pretty much the main reason I've come to gamescom - I have so many questions about the device, and none of the answers have been forthcoming in anything I've read about it so far.

The ever-engaging Kudo Tsunoda is creative director of Project Natal, and is our pilot through the presentation, which is essentially the same as the E3 demo albeit without the inclusion of Peter Molyneux's Milo & Kate. "Xbox is becoming the entertainment hub of the living room and we're getting a lot of new content - like downloading movies, you've probably seen the announcement for Facebook coming to Xbox, Twitter - those sorts of things," he begins. "We have a lot of different content out there that appeals to a lot of different people, and so it becomes very important that we develop a new control system where anybody can use our controls. Right now our current controller is very nice, I like playing games with it, it's awesome. But it's got a lot of buttons and a lot of analogue sticks and it's very hard for some people to use, and it's a barrier to some people for enjoying our console."

Tsunoda reckons, then, that the current joypad is intimidating for some people and puts them off console gameplay. Not only that, but with the system also doubling up as a media player, he talks about how his family would call him over to navigate through the interface using the 360 controller just to fire up their movies. "It became very important for us to create a new control scheme where anybody - no matter what your age or gaming ability - can just get in there can play with Xbox. No instructions, just very simple and easy to use," he continues. "But at the same time we wanted to give extra fidelity for core gamers. So, simple and approachable, extra fidelity - it seems like opposite things, but those are both things we can do with Project Natal."

Onto the first demonstration then: a 3D Breakout game. Powered by Unreal Engine 3, it places you in front of a wall with a barrage of red balls being fired in your direction. The objective is simple: use whatever body part you have available to bash the balls back to the bricks, which breaks them. A suitably energetic demo from Tsunoda ensues. "To play the game all you need to do is move your body," he says. "If you move your body, you'll know how to play the game. It's simple and approachable. I don't care how many buttons or how many analogue sticks you put on your controller: you're never going to get simultaneous movement of your entire Avatar as you can do with Natal."

He's right. What is particularly noteworthy here is that the on-screen Avatar in this demo is closely (if not identically) based on the entire 48-point vector skeleton Natal is processing internally. The way in which the Avatar matches even the slightest of your movements is quite remarkable. It's not just limited to limb movement - your entire body is being accurately mapped. Tsunoda mentions that the dashboard Avatars will also perform in a similar manner, immediately making them more individual, human-like and more closely related to you. I've never been enthralled by the Avatar concept, but this will work in really individualising them and making them seem more like mini-CG replicas of yourself. The applications within Avatar-driven games (and of course via Xbox Live) are potentially spectacular.

"Plus the great thing is that you don't need to stand in just one place and move," Tsunoda continues. "You can totally walk around the room, and your Avatar walks with you. I can walk backwards, I can walk forwards, left and right... it all just works, right?" Tsunoda proceeds to play the demo, jumping about like madman. "I've lost over eight kilograms since I started working on Project Natal," he says. I can believe him. [He really is completely bonkers to watch. The only point he reins himself in is to avoid kicking Rich in the face. - Ed]

A more lethargic performance from myself follows. Curiously, Natal doesn't immediately lock onto me. Tsunoda's assistant runs his hand in front of the sensor, which seems to reset it, and all is well. I'm going to guess that this is actually a good thing: Natal is presumably still attempting to track Tsunoda, but he's no longer 'on the grid'. "You can see also that because the sensor is seeing everything in the room in 3D, the Avatar changes," Tsunoda says, referring to the translucent game character on-screen. "It was an Avatar that looked more like me but has changed to an Avatar that more closely fits how he [he being me] looks." It's simple, good fun, and I'm particularly struck at how well Natal judges the velocity of moving limbs.

It takes some getting used to, but the gameplay has the feel of authenticity about it that you would want. Give the ball a solid whack, or a good kick, and the reaction on-screen is exactly how you feel it should be. Basic, fun, concept-driven stuff like this is exactly where Nintendo excels, and it's good to see Microsoft following a similar line of thought. Just from playing this small demo, it becomes obvious that Microsoft could make a hell of a good Wii Sports-style game with this, and the body-mapping looks so precise that any fitness software it would want to produce has the potential to be astonishingly good. In fact, you can transpose just about any of the motion-driven Wii games and imagine just how much more involving it will feel with Natal. And authentic: no more Wiimote-shaking to cheat the running sections in Wii Fit, for example.

The Breakout-style demo is also interesting in that you would think it will give us some idea of just how much latency there is in processing the full-body skeleton. Trying to get a feel for it during gameplay is difficult simply because there is already 'latency' in the human body itself when moving a major limb. What we do know from previous presentations is that Natal produces its 48-point skeletons at 30 frames per second. Burnout Paradise operates at 60FPS and aims for a 50-millisecond latency, so we can assume that 30FPS scanning gives us a baseline of 100ms. During the Breakout demonstration, I sneak in a move or two specifically aimed at giving us some idea of just how fast Natal updates.

Even running in real-time, the video of me waving my right arm up and down gives some indication of the lag, but it's important to emphasise the "some indication" bit in the strongest-possible terms. The video recording was made at 1080i - 60 fields per second - and this can be interpolated out with reasonable accuracy to a progressive 60FPS video. Frame-counting from moving my arm down to the on-screen Avatar following suit appears to be close to 200ms (12 frames, a fifth of a second). Before we go ballistic on this, it's got to be put into context in a number of respects:

• We don't know the latency of the display, which could be anything between 1-5 frames (it's a Samsung, so it'll likely be towards the lower end).
• We don't know the latency caused by the game code - it varies dramatically, even between 30FPS games.
• Of course, Natal isn't finished yet, and best guestimates say it'll be 14 months until it's out in the shops. There's a very strong possibility that the final production unit will be different.

With that in mind, an educated guess on the optimistic side would be that this Breakout-style demo operates between 133ms to 166ms - effectively the same range of response as Halo 3 working with the joypad. Assuming it's an ultra-quality, low-latency display, that delta shifts to 166ms-200ms - similar to Killzone 2/GTAIV controller lag. Certainly in this test, the latency is noticeable, and I think it's clear to see when the video is running in real-time, but until we get the tech into controlled conditions, preferably in a straight head-to-head with the conventional joypad performing like-for-like tasks (dash navigation, for example), a precise latency figure isn't possible. As I've said, what we see here is an indication of likely performance.

Now, onto the second demonstration: the legendary Burnout Paradise demo. There's nothing mad or crazy going on here coding-wise, it's exactly the same game that emanated from Criterion's Guildford development lair. "The reason we did this demo is because we wanted to show that you have just as much responsiveness in control with Natal as you do with a regular controller," Tsunoda explains. "The reason we chose Burnout Paradise is that with any type of racing game, it's very fast-paced action, you're moving very quickly, and if there's any lag or delay in the controls, you're going to notice it very quickly in a big fashion."

In essence, Microsoft is scanning Natal inputs and converting them into the equivalent Xbox joypad commands. In this case, the hands represent the steering wheel, while acceleration and deceleration work by moving your foot forwards and backwards. It works, it's entertaining and it will make racing games far more appealing to the non-traditional gaming audience than they have ever been before, but I'll be honest - it will be a cold day in hell before I hang up my joypad and play an ultra-fast 'twitch' action game like Burnout Paradise with any kind of motion controller when the traditional pad must surely get the same job done a lot more quickly.

Instructions from Microsoft say that my fists should be placed close to each other for the steering motion to work properly, but it's not especially realistic, so I adopt a more traditional steering posture with my hands further apart. Tsunoda is triumphant. "You can see that even though he's trying to test the controls by moving his hands closer together or farther apart it's still very responsive and works in all cases."

Once again, he's right, but it's very difficult to tell just whether it is as responsive he claims. Before playing Natal, I spent some time on Gran Turismo 5 at the Sony stand on the show floor. The force-feedback wheel was great, but it took ages to adjust to, having been used to the pad. I feel similarly detached here and, while Burnout Paradise is clearly playable, I can't say I feel the 50ms precision Criterion strived to implement in the basic console version. Is it me? Is it Natal? Is it just the case that moving hands and feet physically takes longer than pressing buttons on a joypad and is therefore inherently less responsive? What about the process of drifting: dabbing the brake, hitting the gas and turning the wheel? A doddle on the controller, but with the Natal implementation here, it's very difficult to tell just what level of control is possible.

Kudo Tsunoda's clearly a brilliant Natal evangelist, but as much as he might tell us it's got the "extra fidelity", without having an extended period of time to get used to this very different system of control, it's impossible to tell. What we can do is give you another video to look at. Compare the left/right steering wheel movement to the corresponding movement on-screen, and see what you think. Once again it's difficult to ascertain quite what's going on, as you can't have immediate response on the display: presumably the wheels on the car need to start turning before the screen shifts.

Whether Natal controls will work seamlessly with other enthusiast favourites also remains to be seen. Shooters, for example. Can you point and shoot with Natal? Does it track the direction of your finger, and thus mirror the infra-red beam element of the Nintendo Wii controller? Bearing in mind how popular the machine is with FPS fans, it's not surprising that this is the number one question I am consistently asked about Microsoft's new tech and, until now, all I could do was shrug pathetically and speculate.

For the shooting game example, Tsunoda reckons that pointing won't work, but that you could position your arms as they would be when carrying, say, a shotgun. Which is all well and good, but assuming that Natal can't pinpoint the movement of your fingers, it's never going to know when you're pulling the virtual trigger. Some other mechanism would be required, and perhaps the intuitive nature Microsoft wants would be lost in some alternative kind of implementation.

Make no mistake though, Natal is very, very smart. Natal is going to do amazing things for the Xbox 360, and I daresay that there will be a very cunning way in which it is integrated into Halo: Reach. But it does have its limitations. In respect of direct pointing at the display, both the Nintendo Wii remote and, I suspect, the PlayStation motion controller will have the better of it. Both are built around the concept of being handheld wands, similar to TV remotes, and thus pointing is the most natural thing to do with them. Natal will need to find its own way. Outside of gaming the 'pointing requirement' isn't quite so stringent - there's nothing to stop an on-screen hand being mapped to your own appendage, and it's here that interface navigation will feel much more authentic and easier to get to grips with in comparison with the usual controller.

In terms of the presentation, is there anything held back that we aren't allowed to show you or talk about? Surprisingly, not much. Only two limitations are put on our video work. Firstly, we're not allowed to film the actual camera/sensor itself (I doubt there's anything sinister here over and above the final product looking different). Secondly, a Natal technical read-out screen is demonstrated that is hugely informative in terms of showing you how the system sees the world. It's a massive shame that we can't show it, because for the geeks amongst us, it's probably the highlight of the presentation. But I'll do my best to describe it to you.

It comprises several windows - the main one being a representation of the area being scanned, with a vector graphic display of the 48-point player "skeleton" combined with a rough approximation of Natal's infra-red display, which is overlaid on top. Next to that is conventional RGB image (similar to what you'd get from a normal webcam), and, below that, the pure infra-red camera feed, which really is quite cool. Move closer to the sensor, and you literally appear "warmer" to Natal. It's how it perceives depth, and it's why its performance will not be impeded in less-than-ideal lighting conditions.

During this demonstration, Tsunoda shows how Natal's brain works. The sensor's in-built processor constantly maintains your skeleton's points and pieces them together to form the vector image. Should one or more of those points disappear, Natal uses basic common sense on how the human body moves, and how it is able to move, and interpolates the connections with unerring accuracy. Up to four players can be tracked in this way, and even when swapping positions, or obscuring parts of the other player's body, the "brain" sorts it out. Very cool. Also impressive is how the skeleton-tracking is maintained even as the player moves towards the camera, with more and more of the skeletal points moving out of shot. Even with just the hips and a small portion of the legs in shot, Natal is still able to accurately determine leg movement. And yes, for the less energetically inclined, Natal works with the player sitting down as well. Presumably because of the properties of the infra-red sensor, background items are automatically factored out of the picture.

One thing I think is worth pointing out is that Natal only seems to scan up to the hand, which is just one of the 48 tracked points. I was curious as to whether the tech could lock onto a player's hand and map individual fingers. The implications for Guitar Hero-style games, for example, could be significant. "It could see my fingers pretty well," Tsunoda replies. "But a little kid's fingers... it's a bit harder at the maximum distance."

So Activision and EA's sales of plastic instruments are seemingly quite safe for the time being, and the tech demo itself is quite revealing. Assuming that what we're seeing on the tech screen is native resolution, the size of the RGB and IR windows suggests that even with an adult, finger-tracking could be asking a bit too much: resolution from the camera certainly isn't HD, and I didn't expect it to be, bearing in mind that Natal interfaces with the Xbox 360 via USB. This introduces a bandwidth bottleneck that at its theoretical maximum is 60MB per second. A 720p uncompressed stream running at 30FPS in 24-bit RGB easily outstrips that. While Tsunoda won't be drawn on Natal's actual resolution, intriguingly he does mention that the normal camera's resolution can be scaled down and that the IR feed can be scaled up. That being the case, I'm guessing that developers will be able to allocate bandwidth accordingly.

As the demo plays out, Tsunoda explains how the data is used to provide services such as an automatic sign-in to Xbox Live. The Natal sensor literally recognises you when you walk in front of it. "We use the infra-red to get a 3D scan of who you are," he says. "And then also using the voice combined with the infra-red to determine who it is." Natal's voice recognition sounds similar to that found in the new iPhone 3GS, so in theory you can negotiate the interface with aural commands such as "play movie". The tech can also be deployed in a vast range of other non-gaming situations. "You can do a video party chat, no headsets required," Tsunoda enthuses. "You can have a living room of people video-chatting with another living room of people using Live, the video camera and the multi-array mic together."

All of the key image-processing is done by Natal's in-built silicon, leaving the Xbox 360 free to power the game itself, and you would think that processing up to four different skeletons simultaneously could impose some kind of performance hit. "If you're going to be tracking all 48 points, it's going to be a little more on the processor than just one person," reckons Tsunoda. "And you can do up to four, but it's not going to be hugely processor-intensive either way." Not a hugely revealing answer - and it's something that can't be tested during the demo, as we only see two skeletons being processed simultaneously - but if the doubling of the load is going to impact Natal's refresh rate, we don't see anything on the tech screens to indicate this. It looks absolutely solid.

Overall, you can easily see why Microsoft is excited about this. Nintendo took the GameCube, added a new interface and a few tech tweaks and turned a poorly performing product into a market leader. Xbox 360 is already a commercial and critical success, it has a clear HD advantage, and its motion-sensing system is like nothing else. In many ways, Natal is the natural evolution of the Wii remote - one of the first impressions I have when using it is that the human body doubles up for almost all of the functions of the Nintendo controller. You are the Wiimote, and its potential to expand the appeal and the reach of the Xbox 360 is astonishing.

The applications of the technology are immense, and will be limited only by the imagination of the developers - and this is where Microsoft's first parties in particular have got their work cut out for them. Wii works because the innovative hardware is backed by some of the most gifted developers in the world, who have successfully managed to capture the imagination of a new type of gamer. Compare and contrast this with the company that has brought us commercial flops like Lips and You're In The Movies. In short, the raw tech is there to completely and utterly outquaff the Wii controller in every single way, but Microsoft's biggest challenge is going to about upping its game-making skills to match and exceed the best that Nintendo has to offer.

As for the core gamer, the jury's still out. The Burnout Paradise demo proves that Natal can be interfaced nicely with the fast action arcade game, and yes, it's playable, but in many ways the concept is held back by the inefficiencies of the human body. Pulling your foot back to engage the brake simply takes longer for a human being to manage as opposed to pulling a trigger or pressing a button - even if Natal operated with zero latency, which certainly in the Breakout demo it doesn't, it would not be the optimal way to play it.

In FPS parlance, it would be similar to comparing a joypad to the keyboard/mouse combo: both playable, but with one being much more precise. I can see alternate control systems being factored in, and I can also see clever functionality tweaks, additions and shortcuts being added to the control systems and interfaces of core games using the Natal tech, but still with the pad as the primary interface. Technologically, there's nothing to stop developers using both simultaneously. For example, the archetypal Xbox 360 shooter could still use joypad commands, but melee combat would work far more nicely if you were literally smacking your opponent in the face via motion control. Similarly, lobbing a grenade might be another application that would be better suited to Natal.

One thing I do find intriguing is the concept of Natal perhaps being able to track head movements, and thus turning the player's display from a flat representation of the world into more of a window into it - a sort of revisiting of the old VR concept. Head-tracking using hacked Wii controls produces quite outstanding results, as seen on YouTube. The author of that video, Johnny Lee, is now NDA-bound and apparently working for... Microsoft. Similar to the way game-makers push console technology to the limits and beyond, I can see the same thing happening for Natal, and that is hugely exciting.

The bottom line is that is that the core principles of the standard controller are decades old now, and with good reason. Natal will get more casual gamers interested in what are traditionally enthusiasts' games, but Microsoft will really have a tough job convincing core gamers to move across full-time once the novelty factor has worn off. As it is, in the form I play it at gamescom, I can't see Natal changing the old style of games and the way we play them that much. Similar to what Nintendo achieved with the Wii, we will simply see new types of software more suited to the unique properties that Natal brings to the table - and while the playable demos I get to sample are interesting, the true greatness and the real potential of Natal has yet to be seen.