Sony: Why we turned down Kinect

Eurogamer: Did you have a stressful meeting with the execs when you pitched Move?

Anton Mikhailov: Yes. And, actually, at that time we were investigating a lot of technologies. We were looking into 3D cameras like Kinect. That started way back in 2002, so we already by that time had stopped that research. We worked with the London Studio guys on that.

Eurogamer: Why was that research stopped?

Anton Mikhailov: Research never so much stops. We break it off because we feel it's not cost-efficient any more. At some point you have to do a cost analysis and say, 'OK, this technology costs this much, it's able to do this much,' and you understand roughly what you get out of it and how much it costs you.

So when we were working with the 3D cameras we felt the cost of the camera outweighed the advantages of what it offered. The London Studio guys were familiar with EyeToy. With EyeToy, one of the toughest things is body-tracking. It's hard to segment the player from the background. You have lighting condition variations. That's why a lot of EyeToy games use motion-tracking instead of parts-tracking.

EyeToy: AntiGrav.

EyeToy: AntiGrav [released in 2005] was an interesting exception because it uses head-tracking along with hands-tracking. That's actually done by Harmonix, surprisingly enough, which did Dance Central.

Those guys said, '3D cameras solve a pretty crucial issue, which is segmenting the player from the background.' At first they were very excited and said it was great. But when you make the games what they found was while it was more robust it didn't fundamentally enable new kinds of games. The games themselves still played like EyeToy games.

Yes, there was a better sense of robustness, but the best case wasn't improved. It's only the worst case that improves. You still did very good exercise games. EyeToy: Kinetic was an awesome exercise game. The issue with EyeToy: Kinetic is sometimes in certain lighting conditions it just wouldn't work. The people that had Kinetic work had a great time with it, but when it didn't it was a downer.

With Kinect what they found... Or the PrimeSense cameras and 3DV cameras is, you can solve those lighting issues, but you still don't get a different experience. And it's very hard to apply to other sorts of genres.

That's Kinetic, not Kinect.

The EyeToy experience they felt was a bit played out. They still like it. It's not that the studio said, 'You know, we're done with that EyeToy tech.' Actually, one of the things they asked for was a one-handed controller. They were asking for something like an analogue stick so you can move around. One of the games they were working on is a magic-casting game so you use your hand to cast spells and use your other hand to walk around. There's just no way to do that walk around feature with gestures.

You can think of putting your hand forward and to the right or something like that. It's kinda cool, but it's not functional. It's fun for three minutes and then after a while you're like, well, why do I have to do this? I'd much rather have an analogue stick.

Eurogamer: None of the Kinect launch titles allow an avatar to move around in a 3D space using player gestures, do they?

Anton Mikhailov: I guess Joy Ride does a bit of that.

Eurogamer: Isn't it on rails?

Anton Mikhailov: Somebody sent out a video internally where there was a guy playing Joy Ride like this [holds up hands]. He's not moving. He's just staying like this, and he comes in third. And I was like... [whistles]. That's harsh.

But I guess so. I haven't actually analysed the games too intensely. If you just think about how you'd do it? If I want to go forward, what can I do? Well, I can put my hand forward. OK, that's pretty good. And I can move my hand to the left and right to go left and right. You get a joystick with your hand.

OK, that works, but is that something you want to do? Well, maybe for the first 10 minutes, or maybe for a short action. You can run in place, but nobody wants to run in place, and it's actually very hard to tell how fast the user wants to run.

So you lose these subtle and quick controls. We felt, what's the advantage you get? You get a magical experience. EyeToy felt magical as well, just because you have no controller. You can move and something happens. But you lose that sense of connectedness to the game. You get a ton of latency.

You sacrifice some of that for the advantage of it being intuitive. That's always a trade off. Ideally you can find an interface that is both intuitive and powerful, but it's very rare you find something that's catch-all. Even if Kinect was highly accurate and it could detect the tiniest motions, still for a sword game you'd want something in your hand because it feels weird to swing around just your hand. Your tactile feedback isn't correct. It doesn't trigger these wrist feelings. People are very physical in that sense.

For some actions, I can think of each of my finger touchings [on the palm of the hand] be a button, but that's much slower than pressing a regular button, and what's the advantage of that? So buttons are still very good for a lot of different applications. And that's just assuming perfect accuracy. In reality, the 3D cameras we surveyed and what Kinect ended up using, they're 320x240 resolution, so when you're talking about tracking fingers, or even tracking things like the rotations of your hand, you're working with 10x10 pixels. It's very hard to get anything useful out of it.

Kinect Adventures uses skeleton tracking. Just like those new machines at airports.

That's the long-winded answer of why we stopped that research. We basically got to a point where we felt we understood the limitations of the tech. Sony as a group wanted to do a motion controller that could work with a broad variety of games.

Motion controller is not quite the right term because what people think of motion control is you swing a gesture and an event triggers. What we were aiming for is a spatial input device, like a mouse, where it could map to a variety of actions.

We weren't trying to replace buttons by gestures. When you do that, you get some intuitiveness. You can swing a tennis racket and the character swings. That's nice. That lets people that don't usually play videogames play videogames.

But what you don't get is a competitive edge. You don't get that I can swing better. You don't have that tracking capability, so you can't analyse the player's motion. If you can do that, then you can do both. You get both an intuitive interface and you get something that's pretty powerful. That's what we're trying to do with Move. We wanted to bring a spatial input device, because that's something the DualShock's not very good at.

For example, if you want to create a level in LittleBigPlanet and you want to draw something over there, you have to slide your cursor there and then you have to rotate. With the Move you can just move it there and rotate the object quickly. It's a spatial task that's much better suited for a spatial controller.

For action games, take Ninja Gaiden, you're going for twitch movements. You want a very short throw analogue stick and buttons. That's better suited for the DualShock.

We realised what games we were good at, and what we were missing. We wanted to fill that space and never necessarily replace controllers. That was the rationale behind Move; get something that's complimentary to the DualShock rather than just try to replace it altogether.

1/4 Heavy Rain was one of many titles to get Move controls added.

Previous Next View all ▸

Eurogamer: What needs to happen before Sony looks at Kinect-style tech again?

Anton Mikhailov: First, it has to be higher resolution, because at this current stage the cooler things you can do you really want some precision. Some of the neat demos we don't want to disclose, we would much rather have the camera be higher-res.

The next big one is the camera must run at least at 60Hz. 30Hz is awful. When we switched from EyeToy 30 to EyeToy 60 it was miraculously better. But some people don't notice that, so that's fine.

There are a couple of technical issues we ran into. For example, rubber is hard to track. While nobody should be wearing a rubber suit, often belts and things make an empty section through your waist, which causes some tracing problems. Dark jeans, fresh denim jeans often completely bug the tracking out. Skirts are hard to track because you just have a single leg instead of two and without higher resolution it's hard to disambiguate. Baggy jeans. Shorts cause issues if you don't have higher res. Certain materials, like polyester.

Basically it's an infrared camera. Infrared light is not some sort of special light. It's just like visible light, but it's not visible. It's a different wavelength. I'm wearing a black shirt and if you look at it with a regular camera, that's black to the camera. The reason is because there's no light coming in from that colour. So there are certain objects that are infrared black. To an infrared light in an infrared camera they appear black. Those are the objects that are hard to track.

There are also some objects that are infrared reflective, just like a mirror is visible light reflective. Those tend to show up bright white on the camera, and then they come up as infinite depth. Those are the funky materials that sometimes just make the tracking bad. I don't know if that can be solved, but some cameras have fewer issues with them and some have more.

1/4 Dead Space: Extraction also has Move support.

Previous Next View all ▸

Eurogamer: I haven't heard anyone complain about that with Kinect.

Anton Mikhailov: I'll be honest with you, most of the time it's covered up by the game itself. Dance Central is a good example. In Dance Central, if you see your silhouette missing parts you know something is going wrong. But Dance Central is clever, and the reason is, I think, because all they're doing is silhouette-matching. I don't think they're actually doing any skeleton-tracking. Skeleton-tracking is the hard thing. That's the neat thing about Kinect, but it's actually the hardest part.

The reason they did silhouette-tracking is because the Harmonix guys worked with EyeToy and they know the best tech is the most reliable tech. So they take the z data from the camera and they just chop it out of depth and feed that into their game.

Some of the more sophisticated Kinect games, like Adventures, try to do skeleton-tracking, and that's when you get things wigging out. That's when you're trying to do something hard. The reason Dance Central is one of the higher-rated games is because it works pretty well.

Background subtraction is one thing the EyeToy developers said was the neat and solid tech out of it. That, we were quite excited about, but we still had this irking feeling that that cost, just to get that, is a bit high.

If you look at Kung-fu Live for the Eye, that does background subtraction as well. It's not as robust due to lighting, but they do a really good job. We always felt we can get better with just visible light to get background subtraction and we didn't need to go to 3D. The really exciting stuff was the skeleton-tracking, and that's what proved to be really hard.

Eurogamer: So you're waiting for the cost to come down.

Anton Mikhailov: Cost down, 60Hz, fewer issues, higher res. And those are all very achievable, just not at this moment.

Eurogamer: When, then?

Anton Mikhailov: You'd have to ask PrimeSense, and you'd have to ask 3DV, and whoever else Microsoft doesn't buy out by the time that...

Of course, there are other solutions we can do. There are stereo cameras, which are also looked at. Lots of other things we can even do on our own as well, so it's not necessarily that we're dependent on that research.

I feel it's a case of early tech. The same thing happened to the Wii. The Wii started out with accelerometers. They hit that point right where accelerometers started to become cheap, but still at that point they weren't very good. When Sony looked at that idea it said, 'We're not so sure about it.'

Eurogamer: Nintendo added the Wii MotionPlus to improve it.

Anton Mikhailov: When they added the Wii MotionPlus was when the gyros came way down in price. That's when we added the gyros to the Move. A lot of the time it seems like these people came out with this first, these people came out second, but the reality of it is somebody is behind the scenes doing the cost analysis and some companies say, 'Yes, we want to do this when it's expensive.' Some want to do it when it's cheap but not so good.

Nintendo errs on the path of new tech, cheap as possible, make up for it in software. Microsoft... I don't know why they did Kinect. It seems very un-Microsoft of them to be honest.

Eurogamer: Because it has lots of money?

Anton Mikhailov: That might be the case. Maybe they took the path of, you know what? We'll eat the cost of the hardware.

Eurogamer: Reportedly, Kinect only costs $56 to make. And it costs £130 to buy.

Anton Mikhailov: In our research we certainly thought the sensor could sell for cheaper, but at the same time, when you're dealing with those companies, they're not always honest with you up front. The might say the cost is X and X, and then they say, 'We've got to put this extra chip in there,' and by the end the cost is triple or double. You're like, 'We can't launch with this cost.' There's a bunch of politics going on in there as well.

If you combine the two systems it would be nice, because you can use Kinect for body-tracking and Move for precise tracking. That would be good. But if you tally up that cost, it's high.

We figured, we can do a bunch of what Kinect can do with just the Eye. The parts we can't do we can compensate for with the Move. For hands-tracking the Move is still a much better device because you get these subtle angles and the positions more precisely.

The EyeToy was a peripheral because you had the camera and you had games for that camera. It was tailored to that camera. With the Move, you can play a shooter game, an RTS, which is pretty awesome because there has been no real RTS on consoles. We can enable adventure games, sports games, party games. It just felt like we got a lot more for a lot cheaper and more robustly. It just seemed like the right choice, technologically.

Eurogamer: Move is out. What happens next? Can you improve how it works through PS3 firmware updates?

Anton Mikhailov: We can certainly update it through firmware. The hardware specs we ended up with are good enough that we can get some more improvements out of them. The camera is still a good camera. I don't know how many more software improvements we can make. The real question is, do we want to? So far we haven't had any real requests from studios to improve the accuracy. There are a couple of issues here and there we can fix, but the majority the games are not even taxing it to its full accuracy.

Games like Tumble, for example, really use that accuracy. Games like Sports Champions and The Shoot, it seems the user barrier is higher. People tend to have trouble doing the actions precisely enough rather than the Move being precise enough to pick them up.

For Sports Champions, table tennis is quite a hard game on the expert difficulty. At that point you're thinking, well, how much more precise does it need to be? We need to decide. There's room for some more precision. It's going to be up to us.

One exciting thing is going to be dual-handed interaction. The Fight is doing two hands now. There's a lot more you can do with this concept of tracking your hands completely. Having more unstructured gameplay where you grab and throw things and really interact with virtual worlds using both of your hands. That enables this 3D multi-touch stuff you can do. You can push, pull, grab things and pull them around.

Eurogamer: How will Move games change next year and beyond? What will gamers notice?

Anton Mikhailov: Internally, the studios are getting to grips with using this one-to-one motion. You've seen this in Sports Champions. They were one of the titles that worked the longest with the Move. The character control and animations are going to improve vastly because most games of this era were set up to do DualShock control. Everything was baked and scripted. With the Move you have this one-to-one control over the character – you've seen that in The Fight and Sports Champions.

There are noticeably some glitches and everyone's aware of that. It's quite good, in my opinion, for the time we had, but we can improve on that. That will be much better in the future.

People are going to experiment with the basics. Different camera angles and different setups of the HUD, things like that, just to make the experience feel closer and more connected. There's a bunch of work we can do there.

There's going to be a lot more shooter support. I was talking to the Killzone team the other week; they're having really good results. A lot of the best QA people on their team prefer Move.

Eurogamer: Why?

Anton Mikhailov: It's not everyone that's converting, but they said a lot of people from PC prefer Move. People who've never wanted to pick up analogue sticks can do all right with Move.

A lot of the great feedback we got from MAG was, 'I tried the Move, the first two days I was just awful and then I got better and now I'm as good as DualShock, and then I don't even want to play DualShock any more.'

Even though you don't get much of an advantage, it's so much more intuitive and fun to play with it. You don't feel like you're wrestling with the hardware as much. So some people switch on that basis. They figure, neither one is preferable.

Other people in SOCOM have said that because the cursor is unlocked, that gives you quicker random access over the screen, so you can shoot well. For some games it's going to be beneficial in a competitive sense. We're still working that out, but there's a lot of potential. I don't think it's going to be as clear cut as people thing. There are a lot of big advantages to shooter fans.