Driving Innovation and RTX Advances with John Spitzer, VP of Developer and Performance Technology transcript

Please welcome to the stage Vice President, NVIDIA Corporation, John Spitzer. Good morning. Thanks for being here. My name, my name is John Spitzer. Uh, I head up a team called Developer and Performance Technology at NVIDIA. Um, we cover a lot of different things. Uh, there's, like, everything from being the technical liaison between NVIDIA and game developers worldwide. We have a first-party game studio, if you will, called NVIDIA Lightseed Studios, where we don't really do games, but we do demos.

Uh, all of the materials, uh, and the assets that you'll see in the demos next week, uh, at GCC are largely done by our team. Um, we also have a performance lab where we measure performance and power across the full spectrum of NVIDIA's products, from the smallest Jets and embedded devices up to clusters consisting of thousands of GPUs running training runs and everything in between. Uh, we do technical marketing, we do, uh, digital human technology, and then we also do all of the ray tracing driver software.

I'll be talking about that in a bit. Uh, we span four different APIs, in fact: DirectX ray tracing, Vulcan, uh, VK array, uh, the optics, uh, API for doing sort of like studio rendering. Um, and then also we do the ray tracing software for the, uh, Nintendo Switch 2. So, um, uh, I've been at NVIDIA since 1999. When I joined, it was like 300 people there. Uh, it's changed a bit, and I think we weren't nearly so known in the news as we are today.

Um, but, uh, it's been a great run, and I can't imagine working anywhere else. So, a lot of exciting stuff to show. Uh, let's see, where are we? Yeah, we're on the next slide already. So, um, going to be talking about, um, a lot of things today, and I'm super excited. So, you guys probably know us best because we are building the hardware, right? So, gaming GPUs, not only for GeForce PCs, but also for Switch and Switch 2.

We build workstations so that people can build incredible assets for games and could build AI models. And so the, uh, RTX Pro 6000 server is something we're going to be, uh, covering later on in the presentation. And then, of course, the big iron that makes up these, uh, multi-million-dollar, multi-billion-dollar data centers, uh, consisting of thousands of GPUs, which are being dedicated for doing training as well as inference at scale.

So, not only do we do hardware, but we also build a lot of software to go with that hardware. And on the gaming front, we have path tracing, uh, software, ray tracing. Path tracing is sort of a faithful simulation of how light bounces around in a scene, whether it's reflected, absorbed, or transmitted through a translucent object. It is faithfully representing how that light is moving around. And by doing so, you're able to just place objects and lights within a scene, and things just work in the end.

So, as such, path tracing has really become the gold standard for real-time rendering in games. And I'll be covering that in a bit as well. Uh, not only are we investing in ray tracing and path tracing, we're also investing in AI and applying that to rendering. DLSS is probably the best example of that. It is our best-adopted feature of all time, and I would say probably our, our best success, uh, you know, at least in my 26, 27 years here at the company.

Um, but not only are we working on rendering software, but we're also working on AI models. So, Neimatron is our class of large language models and small language models that are distilled from those large models, which are best-in-class in terms of running on device so that you can have a very high-quality result, but in a relatively modest amount of, uh, VRAM that's being used, and it can be run simultaneously with the game engine.

So, um, let's kick this off. So, we're going to cover path tracing and DLSS first. So if we look back 10 years, Pascal, this was our GTX 10 series of product that was launched in April of 2016, almost exactly 10 years ago. Uh, if you look at the performance there with just a software RT core to today, where we have fourth-generation RT cores, we have third-generation tensor cores, we have DLSS 4.5, which is able to infer 23 out of 24 pixels rendered.

These are multiplicable, uh, multiplicative, that you can multiply them all together to get a scaling factor that, combined with the algorithm, uh, that is the Rester algorithm, that essentially gave a two-fold, or sorry, uh, 100-fold improvement in image quality for the number of rays used. You get a, a total multiplicative product of 10,000 times that we've improved the performance over the last 10 years. Now, we're not giving up now.

We're still not to where we want to be. We want that the real-time images look indistinguishable from reality. We want them to look like a film. But that's going to take, if we were to brute-force things, it would take probably another 100 or a thousand times more computational power. Well, we're not, we don't have that. Moore's law is dead. We are not going to see 100 improve, 100 times improvement in my lifetime in terms of silicon.

So, we're going to be relying upon algorithmic ingenuity and fully leaning into AI to cross that chasm between what's attainable now with real-time graphics in games and what's attainable in film rendering. So, I was saying that path tracing is really the gold standard today in state-of-the-art rendering for games. And there have been a number of blockbuster hits, starting with Cyberpunk. We have Indiana Jones, uh, Black Myth Wukong, um, there's, uh, Doom: The Dark Ages, uh, blockbuster hits, which are fully leveraging path tracing, and the results are amazing.

I don't know if you guys have seen Resident Evil Reququum, but it's an incredible-looking game, and I can't wait to play it. Uh, but we've also applied path tracing to classic games. So, we have, uh, Half-Life 2, we have Portal with RTX, Minecraft, and these games, even though they may be older assets, they look amazing with path tracing. It really reinvents the look of it. Now, we're not stopping there. Some of the hottest games coming out in the next year are also going to be fully adopting path tracing.

So, Pragmata, 007, First Light, Control, Resonant, Directive 8020, as well as Tides of Annihilation. So, really looking forward to that. So here at GDC, we're announcing two new technologies related to path tracing. One of which is the restart PT, or path tracing. Global illumination, it's the cutting-edge, uh, the most accurate, again, simulation of how light is transported within a scene. So that's what you see on the left side, is that you have accurate mirror reflections and a much more accurate, essentially, uh, description of how light is globally illuminating a scene.

On the right side, uh, detailed animated foliage. Now, foliage is probably second only to human rendering in terms of difficulty. And why is that? It's much more difficult to render foliage realistically than it is a landscape. Landscape is static. It's not moving. Foliage typically is moving, swaying with the wind, and the individual leaves can be moving. And it's a huge amount of geometric complexity as well as depth complexity.

Depth complexity is how many layers are represented in the scene at any given pixel. And the depth complexity on some of these could be a hundred layers, if not a thousand layers, if you're in a very wooded scene with a ton of leaves. Now, each individual leaf also can be completely unique. And so you need to be able to very efficiently trace array into that leaf so that you're able to do it with, again, a depth complexity potentially of a thousand.

And so we have a technology that we debuted back in ADA, which is called opacity micro maps, or OMMS. And this is essentially a cookie cutter. It's a bitmap that you're storing for each piece of leaf geometry. And it allows you to then have the ray tracing core efficiently determine whether or not it's hitting the leaf or it's actually passing around the leaf. So, we're announcing that, um, we are going to be working with CD Project Red in incorporating Mega Geometry into The Witcher 4.

Um, now, this is just a technology demo. It's not in any way representative of what The Witcher 4 is going to look like. But what you're seeing here is that, uh, it's a completely path-traced. Even primary rays here are path-traced. And we are, uh, simulating, if you were to expand out all the geometry here, it would actually represent two trillion triangles. Now we are judiciously swapping in various levels of detail for geometry throughout the scene and rebuilding the acceleration structure for the ray tracing on a per-frame basis.

So these are tens of millions of triangles, which are being built in at, at runtime so that we can go and get these types of visuals. Uh, and again, this is using OMMS for the leaves, uh, leaves, and then, uh, it's all animated with the wind and so on. So really looking forward to, um, to working with CD Project Red on this project. Um, Martinique, if you're interested, he's the senior director of our ray tracing driver team, uh, under me, and he will be discussing, uh, exactly what's going on in this demo on Thursday at 10:00, and that's in one of our sponsored sessions.

So please, uh, please visit him. So moving on from path tracing, now we're going to talk about DLSS. So, um, you guys are probably familiar with DLSS. Uh, we, it had sort of a shaky start, to be perfectly honest. It was little more than a research project when we announced it. Um, it had a lot of fundamental issues. One of the issues is that it required each game to have its own bespoke model trained on the images that that game would produce.

And that just wasn't scalable because, you know, obviously we want to have one model that serves every game. And so we waited until the, uh, ADLR team, this is the Applied Deep Learning Research team headed by Brian Kenzaro, who, by the way, will be speaking here at 1:30. So please come back to see him speaking on a panel. But, uh, ADLR really pioneered DLSS2 into a single model that could apply to any game. And from that point, we really saw a huge, uh, increase in the amount of adoption.

DLSS is the most widely adopted technology that we've ever had, and it's actually used more than any feature that we've ever released. Over 90% of gamers enable DLSS when they play games. Uh, so it's incredible, and we have almost 800 games supported. Now in this graph here, you can kind of see the lavender or the pink. This is, uh, what we call super resolution alone. So this is like just upscaling the image from either 2x or 4x.

Then, uh, below that we have, uh, frame generation. This is, is essentially frame interpolation where it's taking frames that are really rendered and then interpolating or, or generating one in between. And that could be, you know, one frame for frame generation, and then for multiframe generation it could be up to four. Now, with the latest generation of DLSS 4.5, we can generate, uh, 6x. So for every frame that's generated, we generate another five in between.

And this can actually be enabled without any changes to the game itself. You can enable it through NVIDIA app, essentially the control panel, for whatever games have support for multiframe generation already. And that includes dynamic, uh, MFG as well, so that it'll have a variable number of frames depending upon how much budget you have at the time. So, um, you can see that that red bar is really steep. We're seeing that, you know, within one year we were able to get almost 250 games supported.

And one of the reasons we're able to do this is because we used a technology that we invented called, uh, it's called, um, I'm draw, drawing a blank right now. Uh, Streamline. Streamline is an SDK which, once you have it integrated into the game, we can then go and back, you know, essentially integrate new technologies of DLSS into games that already have it integrated without having the game developer do any changes whatsoever.

So that's allowed us to have this incredible adoption over time. All right, this is a demo of DLSS 4.5, the latest. What we're seeing here is Blacksmith Wukong, uh, from our partners, Game Science. Uh, the, the, the image on the right is MFG 6x. So this is essentially, for every rendered frame, you're generating five inferred frames. On the left you have MG4X, which was the previous version. Now this is incorporating a second-generation transformer.

So it's incredible that not only is it faster, but also the quality is significantly better as well. Um, this will be made available in 20 upcoming games, including, uh, 007 First Light, uh, as well as Pragmata. And, uh, it will be available by the end of this month for you to enable yourself. All right, last but not least on the rendering front, um, RTX Remix, if you're not familiar with this, it is a modding platform coming from the NVIDIA Lightseed Studios team.

Uh, it is, uh, a very sneaky way of doing things, and I never actually thought it would work. Um, when my, uh, lead engineer came to me, Alex Dunn, he says, "I think I can make this thing work." It's taking a fixed-function API, let's say DX7, and it's intercepting those calls of where the light sources are, geometry like triangles, uh, textures, what have you, and, uh, intercepting all of that. And then it is replacing the standard rasterizer with a full path tracer with DLSS and all the bells and whistles enabled.

So you're able to take games like Portal and infuse it with ray tracing in DLSS without actually making any changes to the game engine itself. Um, we have over 165 titles that are supported with RTX Remix. And we have modding communities. I say communities because they focus typically on, on separate games or game classes. Uh, we have several, uh, game communities, uh, which subsist of thousands of modders who are using this platform on a daily basis, and they're doing incredible stuff with it.

So, I'm gonna run this reel, just some of, some of the most recent highlights. Yeah, it's incredible to me that, like, old games like this can really find a new life with all this new rendering technology. All right, we're going to move, uh, on to interactive in-game AI. So AI is really defining the gamer experience, and we've been working with a number of game developers over the past three years to figure out, like, what are the experiences that are going to be most compelling to gamers.

Um, we're working with Sega and Creative Assembly on their game Total War Pharaoh. That's what you see on the left. This integrates a small language model into the game that allows the user to type questions and then get answers back on how to stay engaged with the game, how to learn the new game mechanics, how to understand what's going on because, you know, it's a complicated game. You know, it's almost like an encyclopedia that you need to understand to really succeed at this game.

And this is sort of an on-ramp for gamers, and it, it keeps them from churning out as well. So I imagine in the next few years that essentially every game could have an in-game assistant to help gamers, again, get onboarded and get ramped on the game and to reduce churn. Uh, so that's on the left side. On the right side, uh, this is PUBG ally. So, um, our friends at Craftton, we've been working with them. It introduces an AI teammate.

Now it's, it's not an unfair advantage because everybody has an AI teammate. So, you know, everybody's on a level playing field, so to speak. Uh, but it is the running the full stack. It has a speech recognition. It has a small language model, and then it has text to speech, and it runs all locally on the PC, the RTX PC, with no hitching, uh, or, like, stopping or halting of the rendering sequence when you're running all of those models.

So it's, uh, really very impressive. So, I mean, uh, I was playing with it, uh, recently, and I was able to tell my teammate to go and find me some ammo and, and then cover for me. And then it would also say, "I hear shot, I hear shots coming from the southwest." And it's able to clue you into certain things. It also shot me in the back one time. Um, and I guess, like, real teammates sometimes do. And it was funny because it says, "Is this what guilt feels like?" So, never thought that an LLM would be able to do that, especially one that was that small.

Um, so ACE is our collection of technologies and models that make interactions with AI agents, with NPCs, with teammates possible. And so we have worked on some of our own models, and we've also partnered with people like Resemble AI on Chatterbox, which is a text-to-spech model to get them to run efficiently on the device on the RTX PC. So we have our own Reva, which is, uh, a very good speech recognition model. We have Neotron, uh, which I'll talk about in a second, which is, it's a great distilled model.

It's, it's quite small and, and very good for what it is. And then our audio to face, which we've open-sourced for doing, taking the, uh, speech and then animating the mouth movements and other facial movements along corresponding to the speech that's been generated. Uh, this all is within 6 GB of RAM. Now, we of course want it to be even smaller than that, but it makes it possible that people can have this experience, um, with an RTX PC.

And it doesn't stop there. Even for devices that you could not necessarily host this natively on device, because they're small, you were able to deploy them in the cloud for a very small amount of money. So we estimate that even for a PUBG ally-type experience is that if you're using the game, playing the game 50 hours a week, it would only incur like 15 cents for the total time that you would be using it. So here's an example of, uh, the whole stack working.

Let's hope it works. Behold the throne built from the obedient dead. I am its master. Unchallenged, unbroken, unending. Your heartbeat quivers like prey. Your shadow kneels before mine. Choose your death at my hand or at my mercy. All right. So, underscoring what I had said before is that with NVIDIA Neatron, we've really built this for on-device use, and we've sort of maximized the quality, uh, while minimizing the VRAMm that's required to run it.

And so this scatter plot right here, it's a little confusing because on the y-axis it's quality. So larger is better. And on the x-axis, it's actually VRAMm that's occupied by the model. So you actually want to be lower. So you want to be up in the upper left quadrant, like where we are. So high quality, small, um, it's all open source and, you know, you can fine-tune it as to your heart's content and use it as, as in whatever way you wish.

Um, and this will be available starting next week when we, uh, have it at GTC. Um, also wanted to underscore that the models are continually getting better, and especially the ones that are running on device. Um, they are approaching the quality of the cloud-only models just six months ago. Now, it used to take us almost two years to c, to catch up, but because there's this velocity, this acceleration in terms of innovations in distilling models and getting them to run in smaller, uh, RAM sizes, uh, we're seeing, like, that, you know, I don't think that these curves are ever going to converge.

The one on the top, the white one, is the cloud-native models. Uh, again, it's, uh, you're looking at date on the x-axis and you're looking quality on the y-axis. And then, uh, the green line is, uh, what we're seeing for on-device running on an RTX PC. But you can see that they're getting quite close to each other. So it's a, it's a great trend. And what it says is that, you know, even with, uh, you know, uh, RAM always being a precious commodity, uh, especially these days, um, things are going to be getting smaller and smaller over time.

All right. I want to talk, I think last, no, two more sections after this. So, but we're, we're getting close. So, uh, how do we accelerate game production? I think, like, all of you guys are interested in this. We believe that AI can improve every step of the game development pipeline. Uh, in addition to helping actually deploy the game and keeping it alive, right? So, that includes content creation. It includes game development.

For those of you who have, have been under a rock, things like cursor or cloud code or copilot, these are really, um, reinventing how software engineering engineering is being done, and it's going to have multiplicative, um, uh, improvements in, in software development. Uh, and then for live service operations, if you have millions of users out there, being able to get the telemetry and to be able to effectively engage with your gamers, right, monetize, of course, but also to keep them engaged and keep them playing your game.

All of those things can be improved with AI. Another thing we're looking at, and I, I briefly alluded to this earlier with our workstations, is that the RTX Pro server is rack-mountable. And so you can centralize all of your infrastructure for your studio in one place. And this makes a lot easier than crawling under people's desks. Especially, you know, postcoid, a lot of the workplaces are hybrid. You know, they're so many of my colleagues have moved out of town.

They're no longer in the Bay Area. They're living in Folsam or they're living in Austin or they're living in Vancouver or what have you. And so even your immediate team may not be geographically centralized anymore. Well, you can centralize your infrastructure, and even with the people who are still living locally, they can work two days at home and work three days in the office or vice versa, and they'll always have access to their workstation, and it's always going to be, you know, fast and available.

So, um, the, the, this particular graphics card has 96 GB of memory, which is really attractive if you're an artist, if you're an engineer, if you're a researcher, because you're going to be able to take full advantage of all of that space. But it also benefits QA engineers because we have a feature called VGPU, virtual GPU, which essentially mimics a GeForce environment on top of or inside the workstation. And so you can load GeForce drivers onto it.

You can have it mimic essentially any Blackwell skew from a 5060 up to a 5090, everything in between. So it'll actually cut the number of speeds and feeds. You know, it'll, like, lock off SM and it'll lower clock rates of the memory and of the core clocks and such so that it actually faithfully mimics what that particular skew is. So, and so if you have a performance issue, you don't have to have a slew of GPU cards that you're always, like, swapping in and out.

You can just go load it up vGPU, tell it to mimic a specific card, and then start your QA process. So, um, modern games, I don't have to probably tell you guys this, but it requires large-scale, uh, play tests, and it, it's, it's complicated. That means it's expensive. Uh, you know, configuring all these local environments can take time. Um, there's communication problems that you can have between QA teams and dev teams, and sometimes it might be you be vending out the QA to another team and keeping them in lockstep with your CI/CD to ensure that they're always getting the last, the latest version of the game to test.

It can be very complicated. Um, similarly, you need to have, you know, alphas and betas and, and what have you to get real-world feedback on the game to see if it's going to hit or not, or if you've caught all the bugs when you've done your internal testing. So, what we've done to help with this, uh, is that we're opening up a GeForce Now cloud play test. And this allows you to hook up your game through your CI/CD to automatically, uh, upload your games.

And then you can have strict access control lists, uh, for that, those games. And that could be, of course, broadening over time to include press, to include external QA, to include alpha, beta, closed beta, open beta, what have you, right? And so, and, and furthermore, you can stream it to practically any device. You could stream it to a mobile phone, to a tablet, to a MacBook, to a workstation, or you could, you know, have it even on a PC.

But the PC could be a legacy PC with no special GPU in it. As long as it has a 4K monitor to it, you're going to be getting that full-fat, incredible experience as if you had a real 5090 in your machine, and it's being used by a ton of our partners: 2K, Activision, Bethesda, Frontier, Techland, Ubisoft, Warner Brothers, and Xbox Game Studios. So, people have been loving this, uh, feature, and I hope you, you give it a try.

Uh, furthermore, we're distributed across the world, uh, every continent except for Antarctica. So any, essentially wherever there's people, um, and, you know, we're putting up a new data center in, in India, but we have coverage all the Americas, Asia, Africa, Europe, of course, and so you're going to be assured of having a low-latency connection wherever you are or wherever your gamers or your QA engineers may be located.

I wanted to thank, um, Mark Delora and everybody at GDC for this opportunity to be here with you today. And a special shout-out to Ike Nullley, who was my slide guy, and, uh, he put together most of these slides for me. So I really appreciate that. So thank you, guys.