E3: Molyneux showcases Natal tech

By Dan Pearson Published 1 June, 2009

Child the new dog?

Perennial showman Peter Molyneux took to the stage towards the end of the Microsoft conference this evening to showcase a project which Lionhead has been working on for Project Natal - Microsoft's new hands-free control system.

Lionhead developers have already had their mitts on the motion-sensing technology for a "few months" now, developing a rather impressive piece of software featuring a virtual boy named Milo. Before introducing us to Milo, Molyneux claimed that what we were about to see would "change the landscape of gaming forever".

Viewers were introduced to Milo as a Lionhead employee, Claire, chatted to him on-screen, using Natal's voice, face and motion-detecting technology to listen to questions, interpret mood and respond appropriately.

We're not sure how much scripting was taking place, nor how far the limits of Milo's interpretational powers can be pushed, but there was a definite sense of interaction from the wee lad as he paced about, lamenting his homework assignment. After a quick, relatively natural chat, the development was invited to help with the homework, which apparently consisted of drawing some fish.

Against a typically colourful Lionhead background scene, Claire was lead down a pier to the water's edge to fish, having reassured Milo about his drawing skills. Nearing the end of the pier, Milo tossed a pair of goggles through the screen, so to speak, drawing an "unscripted" catching reaction from the Lionheader as they hit the lens, before putting them on. Drawing closer to the water, featuring her realistically ruffled reflection, Lionhead lady was invited to "touch" the water, and proceeded to swish the currents around with hand and arm movement.

She then whipped up a quick fish drawing, on a normal piece of paper, before handing it to the camera for Milo's inspection. Taking the drawing smoothly, albeit virtually, Milo instantly recognised what it depicted, as well as its colour. Twee perhaps, but a demonstration of the system's potential flexibility.

Molyneux concluded by saying that this sort of achievement has been a "dream" of his for "20 years", "meeting a real character" who knows you. We're expecting to see Natal behind closed doors in the next little while, so look out for our own interpretation of the Milo demo in more detail as soon as we have.

Picture of Dan.

About Dan Pearson

Read more articles from Dan by visiting the Dan Pearson archive page.

You may also like...

Comments (44) Latest comment 3 years ago

Comments threads automatically close after 30 days, but please feel free to continue chatting on the forum!

  • stevetuck #1 3 years ago

    So i can download friends now... YAY!!!

  • GamesProgrammer Verified Games Team Programmer, Eutechnyx Ltd. #2 3 years ago

    if you can they will be 1200 points

  • oerhoert #3 3 years ago

    Unlikely that MS has solved all the problems of AI research in one go, sadly...

  • r4z0rbl4d3 #4 3 years ago

    Seriously, that demo blew my mind! Screw Eye Toy, this was truly next-next-next-gen in my book! Next step is a freaking Holo-deck! :D

  • Negotiator #5 3 years ago

    "Unlikely that MS has solved all the problems of AI research in one go, sadly... "

    Shut up you nob head, MS and Pete unveil something incredible and thats all you can say, MS just changed the gaming landscape forever, brilliant.

  • Razz #6 3 years ago

    Looks impressive, but I trust nothing that Molyneux has his hand in. He always promises the Earth and delivers somthing disapointing. We shall see. :
    Edited by 1 at 01/06/09 @ 21:12

  • Negotiator #7 3 years ago

    Seriously, that demo blew my mind! Screw Eye Toy, this was truly next-next-next-gen in my book! Next step is a freaking Holo-deck! :D

    You hit the nail on the head, Eye toy is a 2D camera, this is full 3D body control, face and voice recognition, this is incredible.

  • autogunner #8 3 years ago

    i refuse to believe its real, untill eurogamer tell me otherwise.

  • CallousB #9 3 years ago

    Virtual child = wrong and creepy

    Should have been some weird goblin or alien....as they are supposed to be wrong and creepy.

  • SeesThroughAll #10 3 years ago

    "Unlikely that MS has solved all the problems of AI research in one go, sadly..."

    ^This

    It won't be the sophisticated AI people are imagining... In fact, I am starting to believe MS has just pulled their own version of the "KillZone 2 in-game trailer" stunt.

    Goes without saying, it was a very good show in it's own right. Great news about the next MGS game, I hope it's coming out on the PS3 too. If not... I'd appreciate trophies on MGS4, thankyouverymuch.

  • vegard #11 3 years ago

    i need to see a video of this!

  • AphoticCosmos #12 3 years ago

    If Molyneux wanted to show off what Natal can do, he should've done it live. I really like the potential of Natal, assuming it all works, but Milo is a bit advanced [and a bit pointless] to actually be working as he says it does. Give it 20 years and what he wants will be possible, but at the moment, even with the most advanced technology, it's just recognising emotions and key words and formulating a response based on it.
    Edited by 1 at 01/06/09 @ 21:26

  • MORZTAN #13 3 years ago

    This is Microsofts Wii.

    You will see tons and tons of games with shodd controls for the next 2 years, with only a few hitters, and by then, Nintendo will have mastered motion control.
    I do not believe that the Milo demo in the way it was shown will be reality on a home console for the next 5-10 years.

    That being said, I am looking forward to the hands-on. 4:1 on that the journalists gets handed a scrap o paper with all the commands they are allowed to give Milo, unless they wish to be banished from the demoroom.

  • makeamazing #14 3 years ago

    @Negotiator, relax dude, your coming across as an ass. Seriously if it worked as well as they demo'd it I think we would all be amazed. Do you know how difficult it is to do voice/face/AI? Its really difficult. So though its exciting, I think this was a well scripted tech demo... who knows I could be wrong, but its unlikely with the very slow advances in this type of technology.

    So please go and get excited, but try to be alittle realistic about what your thinking it will do.

  • Negotiator #15 3 years ago

    @Negotiator, relax dude, your coming across as an ass. Seriously if it worked as well as they demo'd it I think we would all be amazed. Do you know how difficult it is to do voice/face/AI? Its really difficult. So though its exciting, I think this was a well scripted tech demo... who knows I could be wrong, but its unlikely with the very slow advances in this type of technology.

    So please go and get excited, but try to be alittle realistic about what your thinking it will do.

    You just saw what it can do, and its being shown behind closed doors for the press to try it out. You my friend need to open your mind because MS and Pete have opened up the box and it is good.

  • Les #16 3 years ago

    What surprises me the most is that MS shows this before Sony's EyeToy team did: They've been talking about this stuff for years.

  • SeesThroughAll #17 3 years ago

    What surprises me the most is that MS shows this before Sony's EyeToy team did: They've been talking about this stuff for years.

    And as a natural consequence, we'll see EyePet coming out long before Molyneux's concept becomes a full game, but actually fulfilling it's much smaller ambitions and expectations...

    Different approaches that will, in the end, converge to a new paradigm of gaming. Let's hope this technology advances in an interesting way.

  • LazyDan #18 3 years ago

    @Negotiator - God you almost sound like a shill. Next you'll be praising MS & Peter for synergizing real blue sky thinking.

    We didn't see exactly what it was, because it was an E3 presentation. Someone else has already mentioned it, but everyone thought Killzone 2 at 2004 or 5's E3 was the real deal... then silently 4 years later it's just a standard shooter. It's alright to be excited at the potential of something, but you just look silly if you're not realistic about it. It will not be all it appeared to be in that demonstration. You will not have a virtual friend. Make a real one.

    Or you can carry on believing everything you see at face value. Here's some advice: Don't read The Sun - you'll start believing that all immigrants are evil, Susan Boyle is a coke addict and that the Royals are all Nazis with an attitude like that.

  • makeamazing #19 3 years ago

    @Negotiator, what i saw was a woman who did a well scripted presentation to Milo... I mean she asked if he had done his homework.. not a particularly normal question to ask.

    The in game presentations were better (e.g the bit with the water, or the woman that was kicking and jumping around).. but even in the presentation area, the Mii wasnt perfect when the guy was doing the presentation. The technology just isnt there yet, and I implore you to go and look up the MS voice presentation they did for vista a few years back, it was well and truely awful. Not to diss MS, but to show you even in the best prepared presentations, that voice recognition is a bitch just on its own, before we take into account Milo.

    Also according to the EG text, PM has only had the dev kit for this a couple of months, please dont think that they have done something amazing in that short a time, yes the graphics look nice etc... but its not what they are touting it.
    Edited by 1 at 01/06/09 @ 21:50

  • Negotiator #20 3 years ago

    For the last time Eye toy is a 2D camera that is it, this is full 3D body control in real time in a game, with face and voice recognition, do not compare this to a 2D camera, its like compare radio to TV, you can't.

  • AphoticCosmos #21 3 years ago

    "You just saw what it can do, and its being shown behind closed doors for the press to try it out. You my friend need to open your mind because MS and Pete have opened up the box and it is good."

    We saw what it could do in a film. None of that was live. Odds on that it was edited to look amazing, and in actuality had a lot of flaws.

    If he was actually confident in the technology, and how revolutionary it could be, a live demo would've been done.
    Edited by 1 at 01/06/09 @ 21:51

  • smelly #22 3 years ago

    it wont work as well as shown there in real life.. it cant do.. physical impossibility.. we dont have the tools to make AI good enough.

    For starters lets just say it IS good enough to recognise even shit accents... Will it past the turring test? Unless mollyneux has invented something which knocks YEARS of AI research out of the water - then no.

    It couldnt possibly recognise everything you say to it for example - just key words it knows.

  • LazyDan #23 3 years ago

    This is beside the fact that it was Molyneux hyping it up. That should be an instant sign that it'll only deliver a fraction of the original promises.

  • Negotiator #24 3 years ago

    OMG you people are insane, you just saw this do everything, you say it can't do, live on stage in front of hundreds of people. Wake up its rea,l it works and its the future of games, live it, like it and love it, I know I do.

  • ST.. #25 3 years ago

    Really sounds like the 360 space marine demographic will lap up this one....shame as it sounds interesting.

  • makeamazing #26 3 years ago

    So PM has invented voice recognition with Advanced AI in 2 months which programmers/scientists have been working on for over 20 years... I love the guy but hes NOT that good :D

    I remember seeing a program on BBC2 about robots, and it covered voice/sight recognition and for basic images it is fine, but to do anything complex, then it struggles.The basic recognition would be enough to do some cool things, but not what I suspect Negotiator is expecting it to do.

    I think the program was a Horizon special, you probably can find it on Youtube, I recommend you check it out.
    Edited by 1 at 01/06/09 @ 22:03

  • smelly #27 3 years ago

    "live on stage in front of hundreds of people."

    were you watching a different presentation to me?

    Seesh.. dont tell me you're actually BELEIVING molyneux hype? Dont you people EVER learn?

  • BOFH_UK #28 3 years ago

    "OMG you people are insane, you just saw this do everything, you say it can't do, live on stage in front of hundreds of people. Wake up its rea,l it works and its the future of games, live it, like it and love it, I know I do."

    No, we didn't. What was shown was a demo and not even a live one at that. Let me put it this way: I can build a bit of software right now that will respond to a limited set of inputs from a source. I can even make it respond to voice commands or facial expressions if I know who's going to be doing the demo and can capture their voice / face beforehand. The test comes when you work outside of that sandbox and so far we HAVEN'T seen that.

    That being said, do we need it to work to that level? I personally don't think so. Imagine this tech in, say, Fallout 3 where you have a wide range of characters with scripted responses. The player can have a normal joypad to control their character in the way we're all used to but when it comes to interaction they can talk to the on-screen characters directly using the same basic decsion tree that's already used for that type of game.

    Couple of other examples - face or fingerprint recognition door locks in sci-fi games could actually work in real time with this very easily. Or how about a 'hands-on' version of Bioshock's hacking? That's where I see real potential with this tech, not isolated games but adding a further layer of interactivity to existing genres.

  • smelly #29 3 years ago

    "That's where I see real potential with this tech, not isolated games but adding a further layer of interactivity to existing genres."


    See i hate it when wii games do that stuff uneccesarily... so that's not for me.

  • BadBoyBonner #30 3 years ago

    I'm with BOFH_UK on this one.

    To be honest, voice recognition has been curios by it's absence in this generation of consoles. Anyone lamenting the speech recognition available since MS Office 2003 really hasn't trained it very well.

    I used it extensively through my Law Degree to quote passages from cases (as I can't touch type - and looking down reading a bit, typing a bit , reading a bit started to drive me mad from such small type faces). The voice recognition worked flawlessly after reading through the test passages a few times.

    Obviously once you have built your speech recognition profile it could be stored on your HD (or even upload to Xbox Live) allowing you to pull it down with your gamer tag.

    Something like Fallout3 or Mass Effect offer the ideal medium with conversation branches already on screen - and just about bugger all else happening while the conversation is taking place - freeing up valuable CPU cycles for the voice recognition.

    Clearly you can also do this with just the headset, as a few select games have demonstrated (mainly from UBI).

    As for Lionhead - I'd fully expect this to be in the next Fable - with the engine already in place it could happen a lot sooner than perhaps we think.

    As for going outside the programs parameters - that could easily be rectified in game with a few witty one liners "You do know it is rude to talk while eating"..."Do you have any hobbies other than mumbling" "I suggest you go back to Mr Florins elocution lessons on Porrit Street to brush up on your intonation Sir" which is where you train in the game world for any strange names/words that need to be used in the game etc etc

    What you can't do with the head set is things like the use of winking, a la "I Robot", to let your A.I. mate know what you said is not strictly the truth. "Yes officer I'll never do that ever again" while winking at Milo.

    Also in Fable III expect to see Albion's Great Gurning Competition - for the best use of Natal yet! I want paying for that one Peter.

    I can hear the Bard singing about me already "They say there's a hero with a rubbery face, who's also quite partial to pink frilly lace, he can pull his own lips right over his head........

    Edited by 1 at 01/06/09 @ 23:12

  • Gurgeh #31 3 years ago

    The Arstechnica people had a preview of this last year

    [link url=http://arstechnica.com/gaming/news/2 009/05/microsofts-big-surprise-at-e3.ars
    ]http://ar stechnica.com/gaming/news/2009/...[/link]

    "In my years of covering games I've sat through literally hundreds of briefings and tech demos at trade shows, and what impressed me about 3DV was how real the technology was. Most companies are all show and no go at trade shows, but 3DV had a camera I could pick up and look at. They had a game I could play, as well as demos of the user interface functions. It was all there, and it was all working."

    Note the last sentence.

  • curtlikesmeat #32 3 years ago

    Just my personal opinion but I have to say I'm not even remotely interested in stuff like this... it's so gimmicky.... reminds me of VR headset style things.

    Show me Morrowind with the CryEngine 3 and that will satisfy me. Flapping around infront of your TV is not immersive, if I want to flap around I'll just go play football or something (or go paint for real).

  • BadBoyBonner #33 3 years ago

    Gurgegh

    I do not think anyone on here is thinking the camera doesn't work.

    More the fact that the Milo demo is implying that Lionhead have seemingly cracked the Turing Test in a couple of months of messing about with Natal.

  • smelly #34 3 years ago

    (I'm scared of saying this in case the fanboys jump on me)

    .. but i dont see how this (natal) is going to perform any better than the eye toy?

    After all, it's just a souped up version of the same thing. The eye toy sold "okay" but it wasnt exactly the second coming everyone thought it was when we first saw it. I fear the same for this.

    may be wrong though.

  • Britesparc Verified Creative, ITV #35 3 years ago

    If this works - if it works like it works in the video - this is amazing. Utterly, unbelievably, I-didn't-think-we-were-there-yet-technologically amazing.

    Edit: incidentally, I do think the video was heavily scripted; it had to be. But it was more stuff like the paper recognition, scanning the image, etc - if it can genuinely do stuff like that so quick, if its voice recognition and face recognition is as good as all that - then THAT'S amazing.

    Edit 2: on Twitter, PM said that Milo is Dimitri, and unless I'm mistaken he's been working on Dimitri for a couple of years; so I guess that the "AI interaction" stuff might be the product of several years' worth of research, and the Natal interaction stuff just the result of two months spent playing with a dev kit.
    Edited by 2 at 01/06/09 @ 23:59

  • smelly #36 3 years ago

    @Britesparc: Totally agree...

    But this is molyneux we're talking about here...

  • come_get_some459 #37 3 years ago

    Anyone remember Hey You Pikachu?

  • ChrisOTR #38 3 years ago

    Just watched it.

    Anyone else get an inexplicable "Space Ace" and "Dragon's Lair" feeling?

    ;-)
    Edited by 1 at 02/06/09 @ 06:49

  • spammage #39 3 years ago

    Molyneux has never promised the world and failed to deliver, no, wait a minute..................

  • smelly #40 3 years ago

    [link url=http://www.vimeo.com/495 2629
    ]http://www.vimeo.com/495 2629
    [/link]

    That is all

  • onyx_elite #41 3 years ago

    I'm so unenthralled I can't even be bothered to get verbose on the subject. Anyone excited by this is seriously wet behind the ears and PM needs psychological adjustment. I just wish I could put money on this being complete nothingness that is eventually revealed to be the core of a rather run of the mill rpg with Wii-style elements to hook in all those non-gamers.

  • Les #42 3 years ago

    "And as a natural consequence, we'll see EyePet coming out long before Molyneux's concept becomes a full game, but actually fulfilling it's much smaller ambitions and expectations..."

    True. Maybe Sony listened to some of the critics (including MS ironically) and decided to not show stuff that's years from release...

  • Devilfish360 #43 3 years ago

    although technology wise this is an amazing acomplishment i still cant stop thinking that kid is gunna try and kill me when i have my back turned

  • kangarootoo #44 3 years ago

    My thoughts.

    1. There is something distinctly creepy about a virtual child. But I'll leave that subject well alone.

    2. As others have suggested, the AI development required to really meet our expectations here is pretty huge. If the rest of the AI R&D world haven't met the challenge yet, I would be surprised if a games developer has beat them to the punch.

    "OMG you people are insane, you just saw this do everything, you say it can't do, live on stage in front of hundreds of people"

    Seriously man. Ok, here is a question. IF, just hyperthetically IF, all of what we saw was scripted from start to finish. HOW would you know the difference?

    If you see a man saw a woman in half on stage in front of thousands of people, is that what you would actually believe had just happened? Did David Copperfield really make the Great Wall of China disappear, just because the world saw it on TV?

    I am NOT saying that everything we saw was a setup, but I think it is bizarrely naive to assume that everything we saw was happending procedurally and would behave just as well if any of us had been involved in the demo (what would have happened if I had asked that kid about cricket, or surfing, perhaps in a different accent?).


    The truth is that what we saw was a very impressive technical demo. But what we did not see was the bounds and limits of that technical demo. We saw it performed within a strict set of parameters, paramters for which it had been specifically built to perform well within.

    We need to see FAR more before we can really form an opinion about how good it is.