Tech Analysis: Kinect

By Richard Leadbetter Published 15 June, 2010

Digital Foundry on latency, CPU overheads and how it all works.

Page 1 of 2

It's the day after the night before, and a chance to reflect on our hands-on playtest of the new Microsoft Kinect for Xbox 360 platform, contact our sources and attempt to put together some semblance of the technical picture behind the device formerly known as Project Natal.

It's difficult to dislike what Microsoft has done, despite the fact that none of the games on offer were designed to appeal to the core audience that has loyally stuck by the platform over the last five years. Behind the Avatar-driven, cutesy, cartoon-style games is a technological masterpiece that is simply a breathtaking achievement: full-motion capture of multiple players simultaneously combined with excellent quality voice recognition, all in a consumer-level package.

So what are the crucial components within Kinect for Xbox 360, and how have we seen them implemented in the titles we got to play on Monday night?

Kinect has a traditional RGB camera in it, as found in a multitude of webcams and mobile phones, and it's capable of a standard 640x480 resolution, operating at 30 frames per second. Alongside this are the depth sensors. These bathe the area in an infra-red wash, colour-coding the scene based on how far away the objects are. This is the key to Kinect's unique capabilities. Not only does it allow games to know where everyone and everything is in 3D space, but it also means that even without the RGB data it can operate just fine in any lighting conditions - even pitch black.

The depth map is the most crucial weapon in Kinect's arsenal, and it can also be integrated with the traditional RGB webcam image in a process known as registration, although the integration of the two planes together does incur a small additional CPU load. However, even without registration, we can see that developers are making use of it in the launch titles, visualising it directly into the game.

Perhaps the most dramatic example of this is in Ubisoft's Your Shape: Fitness Evolved. Here your on-screen persona is effectively a post-processed rendition of the depth map, with the main figure (i.e. the player) cut out, with additional particles effects overlaid to create a much smoother look.

We also see the depth map in effect in Harmonix's Dance Central. Occasionally the on-screen dancers fade out, to be replaced with another heavily post-processed rendition of the depth map, complete with a range of psychedelic effects. Dance Central is actually an interesting case in point because unlike Your Shape, the map isn't quite so clearly filtered: background items and players will "leak" into the image.

'Tech Analysis: Kinect' Screenshot debug1
'Tech Analysis: Kinect' Screenshot debug2

Kinect Joy Ride (left) just wasn't playing ball with EGTV chief Johnny Minkley, leading the game's handler to boot up the debug tool, pictured to the right. There you get some idea of how the depth map looks and how the skeleton is created and transposed onto the Avatars in the top-right of the image.

The question is, fancy technical trickery aside, does it actually work? Up in the massive penthouse suite Microsoft had reserved for the event, things were already getting busy when we arrived. While the gameplay areas around each pod were taped off, there was still plenty of potential interference from people wandering about into the camera's field of view and also from flash photography potentially upsetting the IR beams from the depth cams.

However, in all but one instance Kinect worked beautifully, with only a single pod - running cartoon racing title Joy Ride - posing any sort of issue. Even this turned out to be a blessing in disguise. Looking to debug the problem, the game's caretaker returned to the development dash and loaded up the "NUI" debug tool. I managed to get a sneaky photo of this tool in action - essentially it shows the people in view picked up on the depth camera, and then assigns skeletal movement points to them.

So, having established that the system actually works, it was then time to revisit our thoughts on the lag. If you recall, the latency inherent within the new control scheme was one of our biggest reservations about Kinect when we saw it last year in its pre-production Project Natal guise. To give some idea of comparison, we chose to run our patented wavy-arm test on the very same game, albeit an updated rendition.

A fresh analysis of the latest Kinect Adventures demo reveals that any performance changes since the tech demos last year are fairly minimal.

So, not much has changed in terms of the performance level compared to what we played a year ago. You still need to think ahead and react in advance somewhat to make sure that you hit all those balls, with lag in the 200ms range (including the latency from the display, of course). It's the sort of shift that you're likely to make naturally as you get to grips with the way the system works.

Page 1 of 2

Comments (51) Latest comment 2 years ago

Comments threads automatically close after 30 days, but please feel free to continue chatting on the forum!

  • RodHull #1 2 years ago

    Is there anything stopping developers from using Kinect along with a traditional controller? For instance using the sticks to move yourself around and fire in an FPS, and gestures such as a punch or whatnot to instigate a melee attack?

  • Centrifugal #2 2 years ago

    I'm surprised Microsoft didn't demonstrate how Kinetc could interface with hardcore gaming, by utilising a standard controller with Kinect.

  • Britesparc Verified Creative, ITV #3 2 years ago

    I think Fable III uses Kinect, so maybe Molyneux will talk about that today?

    I hope so, i'm very excited by Kinect but in all honesty most of the games they've shown so far will be party game material.

    /was rooting for Black and White Kinect with strokable creature

  • Yossarian #4 2 years ago

    RodHull, in theory, no, there is nothing stopping them.

  • Raya #5 2 years ago

    Maybe it doesnt work with the controller... Scratch you nose whilst playing and you could knock yourself out in game. Much has been said about things like driving along in a car and using Kinect to look out of the window etc. All drivel of course as if turned you head then you wouldn't see the tv!

    It has limitations. Expect waggle fitness and cuteness to swamp over the 360. just think of all the wii ports you can play!

  • cen4pgb #6 2 years ago

    @raya

    Only if you had head turning in the game mapped 1 to 1 muppet.

    I expect a whole raft of kinitec augmented control scemes for the more hardcore titles, a few of the non casual racing games might include full kinetics.

  • Peew971 #7 2 years ago

    "But where does this leave us, the hardcore gamers?"

    We just want to play with a controller! The only Kinect hardcore games I'd be happy with would be Heavy Rain like experiences.

  • jiveguy #8 2 years ago

    All drivel of course as if turned you head then you wouldn't see the tv!

    I've seen this done pretty well elsewhere. They just amplify your head movement so a little turn to the side effectively makes your man look all the way around: http://www .youtube.com/watch?v=_AO0F5sLdVM
    Edited by 2 at 15/06/10 @ 09:24

  • rotmm #9 2 years ago

    Love that first photo with Minkley playing Joyride, where the camera has captured the infrared waves emitting from the Kinect unit.

  • hiddenranbir #10 2 years ago

    Hard core adventure games could conquer consoles with this, I think! :x

  • Arwin #11 2 years ago

    Yeah, I think 30% of one of the three cores, is 10% of all three. So these figures make sense.

  • Dolly #12 2 years ago

    I wonder what the limit of players it can read at the same time will be? There's 3 in that picture, but i wonder what the max is?

  • HornsDino #13 2 years ago

    Hmm. 200 ms Latency as something you easily adjust to, CPU overhead that "can't be too bad", total failure to map a person a blessing in disguise as you get to see the debug interface. This is a real soft soap of a tech analysis.

  • Dizzy #14 2 years ago

    The hardcore gamers like us will probably only get UI extras or mini-games out of this or we can *gasp* play a casual game from time to time?

    I am not bothered... being a gamer since the 80s I will play anything if it is fun. Hardcore, softcore or casual...

  • Peew971 #15 2 years ago

    @ notmyrealname, I'm not saying it will succeed but to answer your points:

    1. By now we all knew before yesterday it wasn't for the core market and if Microsoft gets a slice from Nintendo's pie they won't care too much about the critics from core players.

    2. That's what bundles are for and they know how to make these successful.

    3. That's actually a good thing, that's how we'll get innovation. If anything the regular control schemes are the ones holding devs back. If the iPhone had a joystick the touch screen games wouldn't be as good as they are now.

    4. Initial sales will dictate developers investment in Kinect but I wouldn't say what we've seen so far costs as much to create as your typical nextgen game.

    5. See 2 and 4

    6. See 1 and casual gamers don't care about latency.

  • KayTannee #16 2 years ago

    "/was rooting for Black and White Kinect with strokable creature "

    Couldn't agree more, well not about the stroking bit. Was playing B&W2 the other day and thought would be perfect for Natal. Only issue is being able to grap, if got no finger control, all you can do is move your hand around.

    Microsoft really need to release a controller for each hand. Split a controller in half, so got one for each hand and make them cheap. And onto a winner.

  • abigsmurf #17 2 years ago

    "There's little doubt that the action on-stage wasn't scripted"

    Double negatives that don't use absolutes make my brain hurt.

  • Omroth #18 2 years ago

    This was way too soft on the latency issue - it's a HUGE problem. I'm excited for Kinect, but lag is just a massive issue for a lot of the exciting applications.

  • IronGiant #19 2 years ago

    So it's laggy and that won't bother your gran or a 5 year old. Both of who will prefer to just get the Wii out of the cupboard every so often.

  • Arwin #20 2 years ago

    @notmyrealname: it takes just one killer app, one application that is convincing on the scale of Wii Sports, to get it accepted. This could even just be controllerless XNE control, though something like the Dance or Fitness game could do it as well, or something completely new altogether.

    As for lag, you're misjudging Move here - Move has no significant lag overhead versus the sixaxis! Because of this, Move has definitely strong potential to work in hardcore games.If you want to see for yourself, just look carefully at this tech demo here: http://te chland.com/2010/06/14/exclusive...

  • Calgon #21 2 years ago

    Arwin no the figure was just 10 - 15% full stop(they'd never give a figure thats split "across 3 cores";) I think early on it may have been higher but they refined it, plus with it being software driven there's perhaps the potential to improve it even further.

    Also the latency really ISN'T a problem people... shut up untill you've played it, 150ms is pretty darn impressive considering what its doing(theres already input lag for core games, some of you seem to be expecting 0ms which is just utterly batshit crazy thinking), if they can get it to 100ms then I think even the pickest people will need to STFU. Honestly I hate how people make a problem where there isn't one and its from the people who haven't even got near one these units yet is the irony, telling everyone they know better or claiming they are overlooking it when they clearly addressed it already.
    Edited by 4 at 15/06/10 @ 15:21

  • makeamazing #22 2 years ago

    it takes just one killer app, one application that is convincing on the scale of Wii Sports, to get it accepted. This could even just be controllerless XNE control, though something like the Dance or Fitness game could do it as well

    I agree to some of that... BUT ;) ...

    1. XNE is not going to make this sell Wii numbers.
    2. Dance of Fitness, totally agreed IF the wii hadnt already done this first.

    Thats the problem, if MS had released this at the same time as the wii, i have no doubt this would have the possibility of wiping the floor with the wii. A number of years on and Wii has squeezed out fitness, casual and dance that I am not convinced that those people who have a wii will upgrade just to get more of the same. For Natal to succeed (sorry prefer natal to kinect)... they need to do something that hasnt already been seen, and i am not sure there is anything else left in this space to do that.

    IMHO they would have been better off holding this off for the next gen console... I think it would have had more impact (and would have given them more time to perfect it).

  • Zoink #23 2 years ago

    200ms may be ok for a party game, but I can't believe racing games would work too well with that. Also, I'm still waiting to hear how you're supposed to brake/accelerate. It looked like in the press conf Forza was handling that automatically (as in the kart game). That doesn't seem like much fun.

  • Spydy #24 2 years ago

    From a technical point of view, to me at least, MS have totally dropped the ball. None of the titles I saw yesterday were anywhere near the quality they promised last year (in fact Sony's Move demo from last E3 was far more technically accomplished than Kinetic this year).

    The depth perception of Kinetic is there but hasn't really been capitalised on. The Forza demo for example (Not to mention how exhausting it will be to use 2 hands without a controller to drive a car), I would expect a 3D hand to move in an out of 3D space and turn on the ignition and yet it looked like a mouse pointer on a 2D plane. The Kintanimals demo held my interest until I released it was completely staged and fake. (As was the Forza demo apparently).

    Perhaps the API's will grow and mature over time but it looks to me like the depth mapping is way off target and isn't anywhere near what it should be.

    I honestly believe results like what MS showed yesterday could be achieved using the EyeToy.

  • Dizzy #25 2 years ago

    " For Natal to succeed (sorry prefer natal to kinect)... they need to do something that hasnt already been seen, and i am not sure there is anything else left in this space to do that. "

    Give them so time. Natal (yeah I know) technology allows for some extra things that the Wii cannot do. The Yoga game for example is already way beyond the Wii because it can track your skeleton. The callory burn API is also something the Wii can't really do. Think of this as the beginning of something interesting for the next 10 years. Maybe (and a big maybe) this tech will finally enhance hardcore gaming in the next gen as well. Natal will be upgraded, tweaked and APIs added. More computing power of next gen systems will also mean more things to compute. Most serious "reviews" of the technology has been rather positive (DF and Engadget), only the emo neo-gaffers seem to think it is a waste of CO2.

    Compare starting Live to what it is now and I think you will see Natal tech having the same evolution.
    Edited by 1 at 15/06/10 @ 13:07

  • Douche #26 2 years ago

    Some of the comments for yesterdays hands on were ridiculous. Nobody had even tried the technology and it was utterly slated. I find it hard not be impressed by the technology. I thought the ubisoft part of the microsoft presentation at E3 was pretty cool and, although a lot of the other stuff was pretty cringeworthy, it demonstrated pretty well what the tecnology can do. I certainly won't be using it for Forza, but i'm sure the tecnoloogy and games will improve.

    It cleary isn't perfect but it is definately a step forward and i hope it is a success.

  • makeamazing #27 2 years ago

    @Dizzy...Yoga game... already done in Wii sports, EA active on the wii.

  • Feanor #28 2 years ago

    "Coming back from the hands-on event, it's clear to see that Microsoft - depending on the price of the final product - has something of a winner on its hands"

    Clear as mud.

  • Dizzy #29 2 years ago

    >@Dizzy...Yoga game... already done in Wii sports, EA active on the wii

    Nope.. that is not a Yoga game. I just measures balance. I wasn't saying it hasn't been done, I was just saying that Natal tech can do a lot "more" (or can be quite different). So expect games to differentiate themselves. One side will be Wii/Move and the other will be Kinect (still don't like the name)

  • Shikasama #30 2 years ago

    Anyone who refers to himself as a 'hardcore gamer' can go and sit on their own thumb quite frankly

  • MENTAL1ST Verified Senior Software Engineer, Picsel UK Ltd. #31 2 years ago

    Kinect definitely gives us something that could not be achieved with Eyetoy or any of the other regular webcams used in gaming.

    But it does look very much like it is just "Eyetoy done properly". So we'll have fitness, dance and novelty minigames that work far more like the equivalent eyetoy stuff was supposed to, but little more than that. I think the processing overhead has rules it out for most 'core' games, except as a novelty add on for menu screens and Forza's intimate car porn mode.

    It's almost like Xbox-with-Kinect is another console entirely. And as such, we'll have to wait and see if it's successful with Jo Public first, then if so, whether core games can be adapted to it later.

    Edit: I was about to change 'Jo' to 'Joe', then I realised it was probably more appropriate that way.
    Edited by 1 at 15/06/10 @ 14:00

  • Sunyavadin #32 2 years ago

    @Rodhull

    Isn't that exactly what Loinhead confirmed for Fable III?
    Regular controls for the game but with the option of using it for the touch and gesture based parts?

  • Kaminari #33 2 years ago

    It doesn't bode well when the Star Wars presentation was clearly an epic fake.

  • gerald #34 2 years ago

    This needs a nunchuck-style controller with one or two button. Holding your hand in the air for 3 seconds to "press" a button will become annoying really fast. With a button you could grab and move/throw stuff.

    And Forza without gas/brakes... yeah, great.
    Edited by 1 at 15/06/10 @ 14:37

  • BlinxHDD #35 2 years ago

    Another apologetic PR-article from Digital Foundry. Come on guy.

  • funkateer #36 2 years ago

    So the conclusion here is that it actually works quite well, with latency within reasonable bounds.

    I still think Kinect has some real potential for core games:
    * Good head tracking for almost any kind of game (increasing immersion)
    * Gesture control for RTS games (RTS is still a problematic genre on consoles due to the controller, and Kinect could make a real difference there)
    * Making arm gestures in online FPS games (waving and signalling team mates etc)
    etc
    Edited by 1 at 15/06/10 @ 15:05

  • lagoonalight #37 2 years ago

    Not sure, exactly, what the move is for latency but I do know they are improving it. I would bet it will be under 100ms very quickly, otherwise, why would sony claim 22ms. I have no clue really but that doesn't change the fact IGN is now gushing over it with socom.

    For those claiming any of this crap including move is going to change gaming you are all full of fanboy bull****. Natal especially, I refuse to call it kinect, will do nothing for anybody that is important. I can already see failure all over this but then again you guys are the ones that buy your system three times I could be wrong.

    I just laugh when I see people say they see so much promise in natal. In what? ten years? five? Shovelware and that is all. The wii had VERY FEW good games to make use of its controller. The move is better than the wii controller. It DOES have games to make us of it. Resi 4 was completely awesome using the remote and there is no doubt that Socom 4 will be that much better. So there you go. I am very sure something impressive will appear for natal but until then this is all hogwash and speculation.
    Edited by 6 at 15/06/10 @ 15:27

  • ronuds #38 2 years ago

    Move has less lag than a wired controller?

    :D

  • Raya #39 2 years ago

    @cen4pgb

    "Only if you had head turning in the game mapped 1 to 1 muppet."

    Sorry Beaker. I was being sarcastic. You can't do something like that and make it practical. Or can't you see that

  • StooMonster #40 2 years ago

    I would like to see Kinect combined with some head motion tracking ... get really good 3D that way, admittedly only for the player and rubbish for anyone watching though.

  • smelly #41 2 years ago

    So .. the free xbox slim worked then...

  • busboy33 #42 2 years ago

    @paddy29:

    "I fail to see how this can even be considered even close to an eyetoy."

    Because then you can go "LOL epik FAIL EyeToy doez that 10 years ago rofl!!!1!!!". If its not like the EyeToy, then you can't make dumb-ass comments.

  • Vyggo #43 2 years ago

    Love the couch warriors everywhere that think they know more about Kinect then the people that actually played/experienced it. Can some of you set your cynicism aside for a minute and try to imagine a world where it might actually be a fun way to play games (for the intended audience disclaimer).

  • AphoticCosmos #44 2 years ago

    I think the question of whether it will interface well with the hardcore gamers on the 360 is quite moot - Kinect isn't, for the moment anyway, at a technological level where it can deliver the performance that the hardcore expects and thus it won't be seen in anything more than optional minigames in most AAA major releases.

    It is, however, a big move for Microsoft into the casual market. I don't see why the hardcore and the casual can't exist alongside each other as long as both are making money and driving technology forwards, and as far as the casual market goes I think I'd prefer this to Move or the Wiimote, not that there's anything wrong with either of them.

    Anyway, I'm pretty certain I'm going to get a Kinect because it looks like me and my new housemates will have a laugh with it next year for casual party games after getting quite inebriated. If Microsoft wants to take the Kinect and improve it incrementally over time so that it's a viable option for hardcore games, then I'm all for it, but I don't expect that to happen until the Xbox 720 is released.

  • BuckEntropy #45 2 years ago

    I can't get all excited about it yet, but I do actually like the idea of simple things like camera cues being possible just by 'looking'. It seems feasible that it could be executed intuitively enough, if not perfect right away it might not take long before it's second nature. I have to laugh at the idea of scratching your nose or rubbing your eyes in that scenario being some big problem, just because the system *can* track your whole body doesn't mean it must be an uncontrollable force of potential doom.

    Racing with an 'air wheel' is still idiotic, I don't need to try it to be certain about that (for myself at any rate), and that example has always had the reek of simple minded marketing gimmick. But there's more then enough on display to be very intriguing.

    Oh and what the hell is with the name hating? Kinect... kinetic connection... seems to get the idea across well to me. Did people actually get so attached to freaking NATAL?
    Edited by 1 at 15/06/10 @ 19:19

  • MinerWilly #46 2 years ago

    My cousins kids used to have something similar on their battered old PS2 called an Eyetoy , the kids loved it and it was alright in small doses i guess . I haven't really enjoyed much of the limited Wii time i have played on friends wii's and at the ripe old age of 34 and a gamer since space invaders on the Atari 2600 i think i prefer to use a controller . I did recently see a preview of the Sony Move by Jason Bradbury from the UK tv show called "The Gadget Show" and it looked like something I could possibly buy , the ping pong game had him going loopy as he could put topspin on shots .

    http://fw d.five.tv/profiles/Jason_Bradbu...

  • sentinal101 #47 2 years ago

    i think theres a huge amout of consumers that like the casual/familyfitness etc games but are put off by a remote of any kind and this is kinects selling point. personally i like core and casual games and cant wait to see what its like. has great potential IMO.

  • GreyBeard #48 2 years ago

    Things I want to know:

    Any sign of using Natal/Kinect to scan 3D objects into the scene hyped last year ?
    How effective is Natal as a pointing device, can you "pew pew" with your finger accurately?
    The dash UI seems to work fine, but how quick is it? The on-stage demo looked slow.
    Can it handle additional objects held in the hand? Why was that light sabre game only shown on video?

  • TRUTH #49 2 years ago

    What a load of Sh*T!!!...I hope MS realize people, esp gamers, are thoroughly disappointed in what Kinect showed. Where were the proper games, not these funfair crappy family friendlt Wii shovel-ware games!!...Good job Rare for making another shit game, instead of something that every gamer wanted.

  • BuckEntropy #50 2 years ago

    I think Rare can mostly only do what MS will let them at this point...

    But honestly what can you expect? Kinect is not a viable replacement controller for most established genres, that's clear enough. So they're first trying to sell it as direct competition to the Wii experience, and for party games it has a lot of advantages over anything else. For the core it'll only succeed as a supplemental device I think, though the ability to 'select' things with both hands and give (presumably) very reliable voice commands could finally deliver a great RTS experience?

  • White_Westie #51 2 years ago

    I am looking forward to this. It could open a large sector of gamers on the 360. That crave more adventure or strategy games that could use this interface. Think outside the box! How about settlers 7 with a good UI to just move your hands to scroll the screen and "touch " a virtual control panel etc would be awesome