Levine warns against early focus testing

By Robert Purchese Published 30 July, 2008

Claims BioShock would have flopped.

2K Boston creative boss Ken Levine has warned developers of the dangers of focus-testing games too early - saying that nobody would have wanted to play Bioshock if the idea had been focus-tested years before release.

Speaking at the Develop conference this morning, Levine said that if Irrational games (as the studio was then called) had gone to a focus group and said, "how would you like to play a first-person shooter set in an underwater objectivist utopia", "nobody would have wanted to play that".

He said there had been a similar experience with the development of Thief, a game which deliberately flaunted the conventions of the top FPS games of the time like Quake and Unreal. "Here was a game where you had a sword and a bow and arrow, and you had to hide under a dresser for half an hour... It's not exactly saying, rev up the focus group, they're going to love this.

"We had people going to focus test groups and asking, 'Would you like more weapons or less weapons?'" explained Levine - the answer, of course, being obvious. "Marketing people have really improved their methods, and it's much more sophisticated now, but you can still get some really bad data out of these things."

Later on in development, when the game is coming together and the scope of potential changes are reduced, Levine does believe that focus testing has a place.

"You have to be the author," he said. "You can't move with the wind. But if everyone tells you they hate something... They can't tell you how to fix your game, but you have to be receptive to them saying there's a problem."

(This is the last story we'll do from that session, we promise - although we'll be chatting to Levine in person later on, and will let you know if he says anything interesting.)

Picture of Robert.

About Robert Purchese

Bertie is Eurogamer's Senior Staff Writer. When he's not hunting down news stories he's often found playing with his kids or gabbling about fantasy games and strange online worlds.

Read more articles from Robert by visiting the Robert Purchese archive page.

You may also like...

Comments (26) Latest comment 4 years ago

Comments threads automatically close after 30 days, but please feel free to continue chatting on the forum!

  • Greychapter #1 4 years ago

    I totally agree. I've worked on several (good) titles in the past where the concepts were focus tested and the titles got canned. The problem with it is that most of the public know what they like and are unable to imagine what the final game will be like when summed up by a couple of paragraphs of text. This system is in some way responsible for a lot of the games which feel the same - the public likes game "X" so in focus testing they lean towards concepts that are similar to "X".

  • consignia #2 4 years ago

    Reminds me of the Simpsons episode where they bring in Poochy to Itchy and Scratchy:

    So, you want a realistic, down-to-earth show... that's completely off-the-wall and swarming with magic robots?

  • Freek #3 4 years ago

    Or as Ford put it (on the subject of cars): "If i asked my customers what they wanted, they would have got a faster horse".

  • vegard #4 4 years ago

    levine is very popular on eurogamer today. if tubgirl was half as popular the world would have been a better place

  • Salvia #5 4 years ago

    I kind of agree with him on this one but he's purposely demonising the focus-testing process.

    "how would you like to play a first-person shooter set in an underwater objectivist utopia" - In games development you don't structure focus-testing like this. You show the group the game (in some form) and gather their reactions to it. You don't leave it up to them to visualise what they think the game is like.

    I'm sure Bioshock was sigend and in development before they got the focus groups in.

  • Nithron #6 4 years ago

    One of the biggest problems with basically any media industry today is that it's run by marketing types. Marketing and business people are in it to make money the cheapest, easiest way possible, and the cheapest easiest way to make media(games, films, music) to successfully sell to the masses is to recycle the last successful product endlessly until nobody buys it.

  • jaxon58 #7 4 years ago

    Focus testing isn't just about 'would you like to play this kind of game?' though.

    It's what it says it is, a FOCUS test, so you focus on one part of your game, i.e. controls, then get the group to test just that thing.
    So what he's saying is a bit of a nonsense.

  • MENTAL1ST Verified Senior Software Engineer, Picsel UK Ltd. #8 4 years ago

    Oh for fuck sake, how many more "Bioshock should have failed" articles do we need?

  • Azazel #9 4 years ago

    As many as they feel like publishing.

  • pjmaybe #10 4 years ago

    Bollocks to focus groups. Bioshock wasn't a flop, and the main reason it wasn't was because of the demo.

    Still amazed to see this same tired old news story being diced up and rehashed in different ways. Makes me actually wish that it'd bombed at times, even though I enjoyed it immensely - just so idiots like Levine had to go out and do some work instead of endlessly spouting crap like this...

  • kangarootoo #11 4 years ago

    I don't think you should ever focus test your concept in the way he is describing. That kind of hands-on playtesting is good for testing mechanics, but there are much better ways to gauge how a concept will be received and marketing people are really the ones with the knowhow to do that. I suppose that is in essence what he is saying.


    "One of the biggest problems with basically any media industry today is that it's run by marketing types. Marketing and business people are in it to make money the cheapest, easiest way possible, and the cheapest easiest way to make media(games, films, music) to successfully sell to the masses is to recycle the last successful product endlessly until nobody buys it."

    I know that is quite a popular point of view, but as often as not it is a distortion of the truth. A lot of talented marketing teams are very aware of what will sell and what won't, and that knowledge is vital is making games is your business rather than your hobby.

    You say "make money the cheapest, easiest way possible" as if that is a bad thing. What company in its right mind would want to make less money, or take a harder more expensive route to the same money? Selling games is absolutely crucial if you want to stay in business long enough to continue to make games, and knowing what will sell and what won't is a mandatory part of that.

    I know some people believe that making games should be about the art first and foremost, but that belief just isn't based in reality. The way I see it, I like making games and I want to continue doing it for a long time, and if the games I work on don't sell I won't get to do that.
    Edited by 1 at 30/07/08 @ 13:35

  • paulf #12 4 years ago

    so kangarootoo - would you be happy working on shovelware if it sold even though it wasnt at all challenging for you to make? obviously theres a balance here games do need to sell to keep people in jobs, but the medium wont progress if we keep on churning out the same old stuff. I suppose if all depends on how much influence the focus groups / marketeers have. Any industry needs people willing to take risks in order to survive, but they are few and far between

  • kangarootoo #13 4 years ago

    "would you be happy working on shovelware if it sold even though it wasnt at all challenging for you to make?"

    That is a big old strawman right there.

    "obviously theres a balance here games do need to sell to keep people in jobs, but the medium wont progress if we keep on churning out the same old stuff."

    The part where you say "obviously there is a balance" is the crucial part. I didn't say that making sales is the ONLY thing of importance, but it is extremely important in its own right as without sales nothing else will allow you to continue to make games of any kind, works of art OR shovelware.

    The industry does need to take risks, but they should never be needless risks. Working on a new concept is a fantastic thing to be involved in, but that doesn't mean that no mind should be given to finding out whether anyone out there might be interested in buying the thing.

    If someone conducted no market research at all about a new concept, I would say they were unwise. If they conducted market research (proper market research, which isn't the same as focus testing as previously discussed), found that their new concept wouldn't sell, but made it anyway... I would say they were mad.

    I absolutely want to work on an exciting product, but I also want to be able to work on the exciting product after that, and the one after that, and pay my bills.

  • Ceatlan #14 4 years ago

    kangarootoo,

    Actually I think I'd happily work on the odd bit of shovelware if it meant that I made enough money to enable me make the game that people said wouldn't sell and you say I'd be mad to make. That way if everybody was right and nobody buys it then it doesn't matter because the shovelware has made me enough money to continue, but if I was right then something genuinely exciting may have made it to the market and opened doors for others.

    You need marketing and business analysis to generate enough revenue from safe projects, to give people the chance to occasionally ignore what they say and do something completely original and off the wall.

  • kangarootoo #15 4 years ago

    I absolutely see where you are coming from, and if it were your own dosh you were knowingly chucking into the fire then I guess that is cool. I'm just not sure it makes good business sense (maybe it doesn't and that is exactly your point :) )

    There is nothing wrong with a successful company spending money on stuff that is fun for its staff to be involved with, and I guess making an interesting game that you know won't sell is really just like paying for a big company picnic, but in a slightly different guise.

    I think you have convinced me actually (at least in spirit), so long as the costs are known and you actually HAVE the big selling shovelware, maybe its ok to spend a bit of dosh on just making fun stuff for its own sake. The games dev business being the unstable beast that it is though, I'm not sure just how cheap a non-seller and how huge a cash cow you would need to have before I felt it was a reasonable thing to chuck money away on :)

  • paulf #16 4 years ago

    thats maybe where your xna, wiiware etc come in, lower dev costs and times and more room for innovation without massive financial risk

    btw i hope i didnt sound like I was criticising you in anyway, (I wasnt) I'm just interested in the opinion of someone who works in the industry :)

  • kangarootoo #17 4 years ago

    Oh hey man, I didn't take any offense whatsoever. And I certainly didn't take any of it personally. You were perfectly polite, and I certainly don't mind the odd pointed or provocative comment (I write enough of them myself) :)

    This has been a good thread, an actual proper discussion and exchange of ideas. Rare enough on the internet at the best of times.

  • kangarootoo #18 4 years ago

    P.s. you are quite right about the recent boom in what is essentially sharware on consoles that has taken off this generation. The process is still not cheap compared to the traditional PC shareware scene, as certification and dev kits are bloody pricey. I would also say that just because the relative costs seem small, a small dev can still not afford to chuck money anyway. If anything, a small dev may be a greater risk in spite of the savings stuff like XNA affords them.

    That said, a dev that works on several XBLA titles (for example) is able to spread the investment, and thus spread the risk, which makes good sense.

    One thing that does really help (and allows devs to take greater risks in these areas) is that XBLM and the PSN Store are exactly the places where gamers are happy to take chances with their purchases. Gamers tend to expect slightly more zany stuff via these channels, AND the cost of the average XBLA or PSN game is such that gamers are also more willing to take a chance on something a bit left field.

    I think both these store fronts (I'm not excluding Wii Ware from my comment 'cos I don't rate it, I just don't know anything about it) are really news for everyone, devs and gamers. Consoles have been crying out for a proper budget line of software for years. I don't mean platinum ranges, which are really just price cuts on old games that aren't selling anymore. I mean the equivalent of PC shareware or the £1.99 range on the Amiga. Cheap to make, cheap to sell, driving variety and innovation in a relatively safe environment.

  • VMerken #19 4 years ago

    *squints*

    Fo-cus... groups?

    The kind they use in the mo-vies?

    Oh dear.

  • Nithron #20 4 years ago

    I didn't say it was bad business, I was explaining the reason why the media industries basically churn out the same stuff year after year with little or no innovation taking place.

    Granted, if someone finds that a given original idea will probably not sell, and then don't bother producing it, that obviously makes sense. These things are not free, after all. As long as their marketing research methods are sound, anyway.

    I personally would like there to be a lot more original ideas in such industries, especially in TV, movies and games, however, if that would result in the total collapse of the industry in question then obviously... It aint such a good idea, is it.

    I wasn't suggesting some kind of new status quo, I was just explaining why we get so much shovelware and cloning going on.

  • Lurks #21 4 years ago

    "One of the biggest problems with basically any media industry today is that it's run by marketing types."

    Well, you mean business really. Eg people who employ people and make anything at all. Marketing, the way Levine and others are using it in this context, is about the entire market facing aspect of a business. To figure out what you should make so you sell some product and your employees get paid.

    It's genuinely absurd to think that this whole approach is wrong. Far from it, a decent marketing/strategy department ought to be able to shed some valuable light on what kind of game people want while allowing creatives the freedom to make something unique.

    Interesting I would have said a key problem with Bioshock was the fact they turned what could have been the best action RPG of all time into essentially a slightly smarter shooter. That's a critical decision though. As far as a marketing decision, what they did was genius and they made a fuckton of money. You'll have a hard time convincing anyone of the developers that it was wrong because it was *good* for them all, personally and professionally, to sell that game to the mass market.

    There ends your business 101.

  • stoopidgreg #22 4 years ago

    "how would you like to play a first-person shooter set in an underwater objectivist utopia", "nobody would have wanted to play that".

    i disagree. maybe if you just asked randoms off the street they would give it 5 seconds thought and say "nah give me mario" but if you actually asked gamers that and described a bit more about the game they would love the idea. oh and it was a dystopia, not utopia.

  • kangarootoo #23 4 years ago

    @stoopidgreg

    /pedantic hat on

    Isn't the point that all utopia are impossible to maintain, and although they start with the best of intentions, they inevitably devolve into the dystopia you describe.

    /pedant hat off
    /pedant hat off
    /CTRL-P pedant hat OFF
    /help, its got jammed

  • kangarootoo #24 4 years ago

    @Lurks

    +1

  • stoopidgreg #25 4 years ago

    but the fact remains the game is set in a dystopia, even if it started as utopia :p

  • kangarootoo #26 4 years ago

    I can't hear you. My pedant hat has slipped down over my ears.