Can you predict review scores? News

Brighton-based usability company Vertical Slice claims to be able to predict videogame review scores.

Speaking exclusively to Eurogamer, director Graham McAllister said his techniques can "inform [companies] to make a better game", pushing them closer to the "magic line" of 80 per cent.

"We just finished some research where we can start to predict a year in advance what the game is likely to get," McAllister told us. "And obviously we get more accurate as time goes on.

"This is brand new research: we're aiming to publish it this year. At the minute we're saying we can get it into these bands: low, medium or high. We're not saying we can predict an eight or a nine, that level of granularity - that's subjective behaviour and all sorts of things. That's nearly impossible, to be honest."

"People think you can't predict a game based on quantifiable data," he added. "What we can do is get these estimators. Some people will just have a hard job believing it. We have analysed the statistics to death, thorough and rigorous, and what we're saying is, 'You may not like it, but this is the best model that anyone has come up with to date.'"

Built on - and run out of - the University of Sussex, Brighton, Vertical Slice uses the brains of physicians and psychologists to compile a series of accurate, scientific tests. It's this approach McAllister feels has been "lacking in the industry to date".

The tests come from reverse-engineering 154 Edge magazine reviews (more on that later) and from something called "behavioural or sequential analysis" - otherwise known as what you say and do while playing videogames.

The latter is based on the work of marriage counsellor John Gottman, who predicted within five minutes whether the couple before him would stay together for the next five years. He was 97 per cent accurate, said McAllister, who has a PhD himself.

Thus, he said: "After 30 seconds, we can predict if the game is going to be bad or good, to a certain extent." Specifically, this means sorting them into bands: low (1-4), medium (5-7), and good (8-10).

"What's important about that first minute," he added, "is that it's the time people play a demo for. That's super critical."

Reverse engineering uses patterns of words or phrases from reviews and matches them with scores. "All the high-scoring games talk about certain aspects; all the medium-scoring games talk about certain things; and all the low-scoring games talk about certain things. And there's a very clear mapping between them."

Who Vertical Slice picks to test the games is handled just as carefully, and the net is cast much wider than usual.

"Four or five years ago people made games for gamers: geeks making games for other geeks, basically. Now you have people making games for everybody. The problem is that the games companies don't understand everybody," said McAllister. "We're very, very thorough about the people we choose to test the game."

Vertical Slice interviews those people militantly, and is investigating psychological profiling techniques to uncover bias tendencies. "No one else is doing these sorts of things. It's really detailed information on who you are testing with."

And you can't lie. "Biometrics is our big thing; we hook people up to equipment that will measure your heart-rate or skin response. If someone says, "This is the scariest game ever," we'll be able to say, "Really? Well, we don't think so." And we'll be able to prove it," said McAllister.

By adding all of those components together, McAllister said Vertical Slice can very accurately predict the outcome of a game.

"We look at what [the testers] say, we map it onto what they were doing at that time on the video, how their video was responding on the biometrics, and we say, 'Look, all these indicators are saying low engagement, this is not looking good, so what's the cause of that?'" he offered.

With that information, McAllister can suggest improvements. "We're not game designers," he said, "but we can influence the game design."

"80 per cent is just so important these days. This is the magic line, can we get our clients towards that or over it?"

Vertical Slice likes to work with companies as early as possible to make sure "they're on track to deliver a bloody good game". Most market research is done on Alpha builds, which is "too late", McAllister reckons. "We're trying to help them much, much further back".

"There's no reason why you would not want it," reasoned McAllister, "the return on investment is potentially huge. At the minute, our clients range from PS3 developers to iPhone developers."

He's not trying to put critics out of a job, either. The usability testing is designed to "complement" reviewers or "expert game players" as he flatteringly dubs them. Even functionality testers like Babel have their place. "All these people should be involved," he said.

"What we hope is that you're reviewing better games."

Pure developer Black Rock has worked with Vertical Slice before. Pure and Split/Second game designer Jason Avent reckons usability testing added 5-10 per cent to Pure scores, and is keen to work with Graham in the future.

"We'll be working with Graham in specific cases where we need independent views and testing or where we can make use of their expertise in rigorous and detailed scientific analysis," Avent told Eurogamer.

In closing, McAllister said he and Vertical Slice have nothing to hide; this is not sorcery but science.

"We're putting it out there and saying, 'Here's our model in detail; broken down, scientific methodology, data analysis, results.' Anybody can look at it, it's completely open, they can go and repeat our process and probably get the same data," he said. "It'll be interesting to see how everyone reacts, you know?"