Tech Interview: LucasArts' frame-rate upscaler

By Richard Leadbetter Published 31 August, 2010

Digital Foundry talks in-depth with creator of the 60FPS "miracle" tech.

Page 1 of 3

On Saturday, Digital Foundry covered a SIGGRAPH 2010 presentation by LucasArts' Dmitry Andreev on the process of frame-rate upscaling. This intriguing concept potentially offers all the advantages of rendering a console game at 30FPS, along with the visual smoothness and potentially even the crisper response of 60FPS gaming.

Almost five years into the lifespan of the Xbox 360, it's a fascinating insight into the kind of tricks, techniques and thinking that developers are employing in order to produce ever more impressive console titles.

If you've not downloaded the eye-opening footage of Andreev's demo showing the technique in play on the Star Wars: The Force Unleashed II engine, it's well worth a look, with both HD and standard def versions available for download, and the original presentation also online for public viewing. Be sure to have an h264 decoder installed on your machine so you can view the AVI files (Windows 7 has one installed by default).

In this follow-up tech interview with Dmitry, we go over the basics, discuss the implementation in the tech demo in more detail and talk about the potential for the technique in future console titles.

Digital Foundry: Can you give us a basic outline of your technique in layman's terms? Are we really seeing an effective 60FPS with all the advantages of rendering at 30FPS?

Dmitry Andreev: The basic idea is to build an extra frame based of the previous one [using] new information available for the current, new frame, and present it in the middle of 30FPS rendering while still working on the current frame. This way, technically, we are seeing an effective 60FPS with all the advantages of rendering at 30FPS.

Digital Foundry: You talk in your presentation about the blind spots in the human eye - how are you using them to your advantage with this technique?

Dmitry Andreev: Well, one of the points that I was trying to make with this presentation is that quite complex things can be actually very simple once you start thinking about them. Working things out by analogy, by trying to see why something happens. All those have indirect influence and that's why it is harder to see a simpler solution. Very often people ask, "Why are you doing this or that? It has nothing to do with the problem whatsoever." The point is that it does. You might not find the solution to the original problem, but very often you find some other interesting things that can lead your curiosity further, and this way you can find unexpected solutions to various problems.

Human eye and the whole visual system is one big inspiration to me. It makes you think, it makes you wonder. A while ago, after reading a book called "On Intelligence" by Jeff Hawkins, I prototyped a few neural networks with feedback connections, simulating the missing input from the blind spot. In fact, the network can predict what you would see as well as respond to certain optical illusions the way the real visual system does.

I also did a lot of experimenting, trying to put different patterns around the blind spot and try to observe what happens. That gave an idea that it is definitely pattern-based, it is localised but not around the "edge" and it is not too wide. Then I thought about what it would look like in motion and the fact that it has to not violate predictions our brain makes about the image. We would tend to notice things that change rather than things that don't change.

It is a set of ideas like this that are used to our advantage.

'Tech Interview: LucasArts' frame-rate upscaler' Screenshot 30fps1
'Tech Interview: LucasArts' frame-rate upscaler' Screenshot 30fps2

30FPS rendering has become the norm with console development as having twice as much time per frame allows for more advanced rendering techniques. The frame-rate upscaler allows for the advantages of a 60Hz refresh combined with more impressive visuals.

Digital Foundry: Can you talk about the process in which you remove the characters from the scene? Why is this necessary? Is it about reducing the artefacting in the interpolation process on the characters?

Dmitry Andreev: This is not necessary in general. As I discussed in comments, all the characters could be re-rendered at 60FPS, or could be rendered separately from the environment. It happens that they can move any way they want. So the easiest way is to have a version of the environment without the characters, and you use that version whenever we detect an artefact.

Now, with forward rendering it is easy to render the environment first, store it somewhere and then render characters on top of it. But it becomes very difficult to do the same thing with the use of deferred techniques. So when I was working on a console implementation I didn't want even to think about redoing most of our deferred rendering pipeline. It is already crazily optimised with different sorts of tricks. That's why I thought it would be easier to just somehow remove them from the existing frame and then use that to remove the artifacts.

But I must note that it is more than just a character removal. It is also to remove all other problematic regions. In the demo that you've seen it is only used for characters though. This is what I mean by saying that the talk should not be understood literally.

Page 1 of 3

Comments (37) Latest comment 1 year ago

Comments threads automatically close after 30 days, but please feel free to continue chatting on the forum!

  • Haloboy #1 1 year ago

    That's all fine and dandy but pulling a star destroyer out of the skies with crap controls will be just as frustrating even at 60fps.

    Please Lucasarts spare us a similar fate this time around.

  • sneetch #2 1 year ago

    This all sounds like witchcraft to me!

  • telboy007 #3 1 year ago

    Even with that interview I still have no idea how it works, witchcraft indeed!

  • NorUraeus #4 1 year ago

    Heh, his prediction that Naughty Dog might be among the first matched my thoughts when first reading about this too. It sounded exactly like the kind of thing I would think ND would love to test their mettle trying to do :)

  • Badassbab #5 1 year ago

    So even though the effect is great on 360 it's going to be even better on PS3.

  • Dizzy #6 1 year ago

    I hope we won't get 15 fps games pimped up to 30 by this technique.

  • Ryboy #7 1 year ago

    To long, didn't read.

    All this 60fps is bollocks, the human eye cannot detect anything over 25fps ish anyway.

  • King_of_Hyrule #8 1 year ago

    I feel like some elemental info has been cut out of his responses to keep the article from becoming too long. One sentence doesn't always seem to follow logically on the previous one, maybe that's just me though.

  • schnide #9 1 year ago

    I don't pretend to understand everything he said, but for me one thing is clear - this guy is very, very clever. I salute you.

  • sneetch #10 1 year ago

    @Dizzy
    I hope we won't get 15 fps games pimped up to 30 by this technique.

    Not even if they were 1080p pimped to 30? :)

  • darkmorgado #11 1 year ago

    All this 60fps is bollocks, the human eye cannot detect anything over 25fps ish anyway.

    Actually sir, YOU are talking bollocks. The human eye can easily perceive way beyond 25fps. If we can't, how come we notice frame drops in 60fps games? How come we can easily perceive that a 60fps game is running more smoothly than a 30fps one?

    Sigh.

  • kangarootoo #12 1 year ago

    Man, I read article like this and I think "there are some damn clever people out there". Lateral thinking impresses the socks off me. Not just people who understand complex things, but people that can make those leaps ahead by taking the smarter route.



    In other news, we read.

    "To long, didn't read."

    followed swiftly by THIS nonsense.

    "All this 60fps is bollocks, the human eye cannot detect anything over 25fps ish anyway."

    That is profoundly stupid in too many ways to count.

    Firstly you get bored by too many words, secondly you come out with a frankly stupid statement about the abilities of the human body (which shouldn't surprise us I guess, given you have already made clear how difficult you find reading).... but thirdly worst of all, you presume to make a statement that basically says that this guy's tech, the new tech that is the focus of the interview, is complete nonsense and makes no difference. Did it never for a single second occur to you that if that were true, somebody at Lucasarts would have noticed?!?

    Seriously.

  • ChuckNorris #13 1 year ago

    A solid 30 fps is all i need in an epic blockbuster like uc2. So if ND make the visuall fidelity twice as impressive by running it in 15fps and upscale it to 30, I wouldn't mind. as long as it is rock solid with negligible artifacts.

  • mrStones #14 1 year ago

    Remember where you were the day the framerate comparsion died. Don't understand how this works at all but he certainly does and thats all that matters

    can't wait untill framedrops are things only mentioned in retro articles

    *edit* backard were words my
    Edited by 1 at 31/08/10 @ 17:08

  • mAc062 #15 1 year ago

    @Ryboy: Maybe your eyes cant see over 25FPS
    [link url=http://en.wikipedia.o rg/wiki/Frame_rate#How_many_frames_per_second_can_the_human_ eye_see.3F
    ]http://en .wikipedia.org/wiki/Frame_rate#...[/link]
    Edited by 1 at 31/08/10 @ 17:12

  • Calgon #16 1 year ago

    "So early on we decided to go with 30FPS motion blur as it took our rendering team of three engineers more than a year to focus on performance and PlayStation3 SPU work."

    I'm sure there are better implementations to be found on 360 if they spent as long prototyping for 360 hardware, having said that it's not suprising that devs dont bother as much as they do with PS3 hardware because of MS' stricter dev guidelines. I think its time they relaxed them and gave devs free reign to experiment on 360 hardware as low level/close to the metal as they need to go, as long as there is a significant gain to be had. Gamers will understand come next gen why some games aren't on the BC list, its a fair trade off too IMO if they do find significant gains.
    Edited by 1 at 31/08/10 @ 17:29

  • funkateer #17 1 year ago

    "A solid 30 fps is all i need in an epic blockbuster like uc2. So if ND make the visuall fidelity twice as impressive by running it in 15fps and upscale it to 30, I wouldn't mind. as long as it is rock solid with negligible artifacts. "

    I really hope that this won't happen.
    Even though you'd get 30 frames per sec, the result would still be a 15FPS game in terms of responsiveness, which will affect playability badly.

  • Thirith #18 1 year ago

    @funkateer: Wouldn't it be possible theoretically to do a game that basically renders things very simply internally at 30 or 60 fps but that has a graphics engine at 15fps upscaled to 30 or 30 --> 60? I'm not saying it would be easy or even worth the effort, but I'm pretty sure that there have been games that calculated different things at different framerates (e.g. Rock Band AFAIK, which calculates the background action at 30FPS but the notes at 60FPS).

  • Zaiz #19 1 year ago

    Eh, more seriously, there's no hard measurement to how many frames the human eye can detect! The more movement onscreen, the more frames we'll be able to perceive. The less movement, the less frames required for you to perceive them as smooth. This is why ME2 can look smooth when characters are slowly jawing away, while CoD looks just as smooth in high intensity gun battles. And now that we have motion blur, even 30fps games can look significantly smoother.

    We can't perceive the difference in movies because they have always used "motion blur", due to the nature of film. Each exposure is 1/24th of a second, so all the movement that happened within that period is blurred together.

    Did that all make sense?

    More importantly, let's translate Dev talk!

    "We've got this fantastic technology, but we aren't using it because, uh, it would confuse the art team." Can be translated to:

    "We've got this fantastic technology, but it isn't done yet and we just want to build hype for our game."

  • sarcasmoidosis #20 1 year ago

    That man is smart. Really smart. I hope this isn't just an idea and it will actually be used.

    Also, this is probably the last confirmation I needed that there won't be new consoles until at least 2013.

  • Baranga #21 1 year ago

    For an animated comparison between 15, 30 and 60 fps follow this link:

    [link url=http://www.bo allen.com/fps-compare.html
    ]http://www.bo allen.com/fps-compare.html
    [/link]

  • Chufty #22 1 year ago

    All this 60fps is bollocks, the human eye cannot detect anything over 25fps ish anyway.

    Your eye is not v-synced with your screen, you fool.

  • Dizzy #23 1 year ago

    "All this 60fps is bollocks, the human eye cannot detect anything over 25fps ish anyway"

    The human eye can in fact track changes up to 250 fps.

  • StooMonster #24 1 year ago

    All this 60fps is bollocks, the human eye cannot detect anything over 25fps ish anyway.

    25fps ish is what you need for persistence of vision to give impression of fluid motion, it's not the fastest FPS you can see.

    Back in the day of CRT monitor, when Sony were kings of displays, before my Sony GDM-FW900 Flat Widescreen 24" (undisputedly the greatest CRT monitor ever made) and subsequent 30-inch LCDs, I had a Sony 21" monitor that could refresh at 180Hz. I used to play Half-Life 1 Death Match et al with their engines outputting 180fps on 180Hz monitor (and the mouse hacked to be more responsive) ... it was the silkiest smooth graphics I have ever seen. =)

    I wish we could get 120Hz monitors that accept 120Hz signals now; I'd buy one tomorrow, my PC could easily support that fresh rate. :)

  • StooMonster #25 1 year ago

    White Paper from BBC R&D experimenting with frame rates of 300fps
    http://do wnloads.bbc.co.uk/rd/pubs/whp/w...

  • smelly #26 1 year ago

    >I don't pretend to understand everything he said

    Thats okay, neither did the guy writing the article, but he likes to pretend.

  • funkateer #27 1 year ago

    "@funkateer: Wouldn't it be possible theoretically to do a game that basically renders things very simply internally at 30 or 60 fps but that has a graphics engine at 15fps upscaled to 30 or 30 --> 60? I'm not saying it would be easy or even worth the effort, but I'm pretty sure that there have been games that calculated different things at different framerates (e.g. Rock Band AFAIK, which calculates the background action at 30FPS but the notes at 60FPS)."

    Yes, it could be done (in fact it's common practice for a lot of games to update game logic at a higher rate than video), but with 15fps video upscaled to 30 I see a few problems:
    1) Latency will be higher, so less responsiveness (even if internally the game runs at a higher rate than 15Hz).
    2) Rendering artefacts of the frame rate up-scaling tech are likely to become much more noticeable if upscaled from only 15 to 30.

    Upscaling from 30 to 60 seems worth the effort, however. Although I'd still like to be able to turn the effect off; I don't even like the framerate upscaling effect in these TV's because of the artefacts and because it has some kind of fakeness to it I can't really put my finger on.

  • DoctorFouad #28 1 year ago

    this guy is very intelligent ! impressive invention !

    I wonder if he could get a patent and be paid for this, or else what would be his motivation to invent such thing ? any ideas ?
    Edited by 1 at 31/08/10 @ 20:07

  • cardboardMonster #29 1 year ago

    "You're not shipping with the frame-rate upscaler in Force Unleashed II despite the pretty stunning demo. "

    Shame.

  • funkateer #30 1 year ago

    "I wonder if he could get a patent and be paid for this, or else what would be his motivation to invent such thing ? any ideas ? "

    It would be a bit of a limited patent since frame interpolation is not new, so if he could get a patent it could maybe be for some of his specific ideas related to applying it in a real time games engine. But I agree he seems like one inventive guy.

  • frostcircus #31 1 year ago

    This is amazing! I haven't been this wowed by tech in a long damn time. Impressive, brilliant stuff.

  • Kerome #32 1 year ago

    Seems fairly straightforward, but the devil is doubtless in the details.

    For example: using a velocity buffer to interpolate the intermediate frame is all very well, but what about camera movement revealing chunks of the background that were invisible in the previous frame, anything with rapid movement such as characters is likely to break, alpha-blended sprites and volumetrics would probably need special treatment, animated lights would cause issues, and so on. Adding this kind of technique to an existing engine is not simple, if indeed it can be made to work for a general games case rather than a demo.

    The other thing is, it's not exactly free. You can't go from 30 to 60, you need enough per-frame headroom to make this work, say rendering your 30 fps at 90% GPU load. Which can cause dropped-frame issues as well, and would impact the amount of "smoothness" you add to the experience, and may actually hurt things rather than help.

    Cool tech, but I'm not convinced how practical it is in general.

  • kangarootoo #33 1 year ago

    @smelly

    I bet you are such fun at parties.

  • kangarootoo #34 1 year ago

    @DoctorFouad

    Lucasarts may well patent this, or aspects of it. He might get a bonus, but he won't own the patent. People invent stuff for the same reason coders write code in general, or designers design games for that matter. Some people just like solving problems, its fun :)

  • kangarootoo #35 1 year ago

    @Kerome

    "but what about camera movement revealing chunks of the background that were invisible in the previous frame, anything with rapid movement such as characters is likely to break"

    He covered this in the interview. He admitted the tech was not best suited to games that had a lot of quick camera angle changes (which is why he said games have to built with the tech in mind). He also discussed rendering the scene without the characters, applying the interpolation, and then adding them back into the rendered scene.

    Again, if a game is built around this sort of tech, the overheads will surely be considered (just like the weaknesses or risk in any engine must be considered when using it to build a game). I'm not sure LA are suggesting this can just be licensed out for easy use within any engine - it is still in development for internal projects if I understand correctly.

    Like any tool, if it is used imcompetantly, the results will be poor. We shouldn't judge something like this on the basis that we can think of silly things to do with it that would bear poor results.

  • superdelphinus #36 1 year ago

    any patent he got would belong to his employer anyway

  • gingerlink #37 1 year ago

    @RyBoy

    Last I checked, it was 72fps ;)