Optimizing RTX Mega Geometry with Nsight Graphics GPU Trace transcript

Hi, my name is Calvin Chu and I'm a GeForce developer technology engineer at Nvidia and I'm going to be sharing how we optimize the RTX Mega Geometry SDK using Insight Graphics Pro GPU trace. Hopefully you'll find something here that will help you in optimizing your own applications. So just an outline for today's talk. We'll go over the background of what is RTX Mega Geometry SDK and the perframe algorithm to give you some context for what we're optimizing.

Then I'm going to talk a little bit about the optimization workflow we used the test scene and the iteration process and the optimizations that we did. So just first a little bit about the mega geometry SDK. This is uh mega geometry SDK and mega geometry API is also referred to as clusters for clusters of triangles. So this is a publicly available demo to encourage cluster adoption. And in in this demo we demonstrate how to do real-time path tracing with teslated subdivision surfaces using this new mega geometry API clusters.

It was released in uh version 1.0 0 was released in July 2025 with Vulcan support and the latest version includes all the performance optimizations that are covered in this presentation. It's written on top of Donut and Nvi which are some Nvidia frameworks and it supports DX12 with NV API extensions and Vulcan with Vulcan extensions. uh find the link below and this QR code. So just this is the application in action. You can see uh we've got an animated character and we support three bounces of path tracing.

Um and we are rebuilding the BVH every frame of course uh with pretty consistent build performance uh as noted in the bottom right. So what does the perframe algorithm look like? Well, we start with uh animation pass uh followed by occlusion um tessellation BVH build path tracing motion vectors and finally we apply DLSSRR for denoising and upscaling. So we'll dive in. We're going to be focusing on this tessellation part today and the teslation part is composed of this compute cluster tiling fill clusters and followed by these are the three BVH steps.

So Cass is the cluster acceleration structure instantiation and we build our BAS from Cass and finally from that we build the TAS. So diving into u the teslation part a little bit more deeply. We're going to be focusing on these two passes. Uh for the purposes of this presentation, you can kind of think of compute cluster tiling as a per surface like a hall shader in the traditional teslation pipeline. We take an input patch, a subdivision surface, and we need to compute um what clusters and how many clusters we have across that surface.

With those clusters calculated and tiled uh we need to then fill the clusters with the content. So this is per micro triangle per vertex we generate the vertex buffer and the text chords buffer. You can think of this as the domain shader. So here is our test scene. It's the most complicated scene in our sample. Kitchen set asset is from Pixar and the Amy Rain character is an asset from Blender. At the default camera position, there's about 89.6 million micro triangles.

And we're testing this on RTX 5090 with native 4K DA DLA and DLSS RR. So no upscaling. You can see this is the color output. Um, again the screen is filled with subdivision surfaces and these are what those subdivision surfaces look like. You can kind of tell that in smoother areas we don't need so many surfaces and we are going to teslate these into micro triangle geometry. So this is what clusters look like. Each cluster can consist up to 256 microtriles.

And finally, this is what the micro triangle view looks like. So, we're really talking about sub pixel triangles here. And just to show you the uh normals buffer, this is with um there are no vertex normals. This is all with geometry normals. and for each triangle and this is possible because each triangle is sub pixel sized. All right, so let's talk about profiling this thing. The first thing that I suggest everyone to do is just enable in the developer settings in Nvidia control panel give access to the GPU performance counters to all users.

This just prevents you from having to restart ends site in admin mo mode every single time. So next um before we start to profile we're going to start a do a GPU trace profiler activity. Uh first thing to do is a good idea is to close all GPU apps. This helps reduce noise. And then for these two specific settings I'd like to call out. um the max number of frames. Setting it above one can help you deal with context switches and setting lock clocks to whatever you prefer.

You can either lock to base. This will give you pretty consistent timing from run to run. It will be slower uh but it will be unaffected by things like thermal throttling and other sort of performance factors. Or you can run unaltered. This is less consistent runto run but closest to real world real world performance. So you'll launch your GP trace and you can press F11 in the application or you can uh press the collect GP trace button from inside graphics.

Also I suggest renaming your trace for your own sanity after you captured. Uh so in terms of reducing noise of the capture uh you can tell when I talked about closing GPU applications um if there is a context switch it will be marked with this uh red striped area and this can get kind of annoying when the path you care about is interrupted and this messes up the timing for this particular frame. This is why I suggest capturing multiple frames.

Additionally, running in windowed full screen and this is indicated by this like independent flip indicator if it's engaged tends to result in less noise. Um, and you can kind of tell that. So, with independent flip disengage and there's a lot of background GPU apps, you get a lot of variance in the the frame times for your counters. But then if it's engaged and you close all GPU apps, you get this very beautiful, very straight, consistent uh run.

Again, I found that overhead is pretty minimal. It's only like 3 to 4%. But if you lock clocks to base, uh it can result in slower performance than using unaltered. So let's talk a little bit about the optimization workflow. Uh so we're going to launch our application and collect our trace. Um then I suggest kind of doing a high level overview of the graphs. Then followed by low-level analysis. Um do your experiment and do some optimization then take another trace and you can kind of then do a trace compare to kind of make sure that your optimization worked and repeat.

I mean, it's obvious when something is faster, but you also want to understand why it's faster so that if you're doing further optimizations, you kind of you don't negate your previous gains. So let's first go over uh in our application when we released the initial version the beta version of our this [snorts] SDK uh this was our baseline performance of 2.58 milliseconds for cluster tiling and 2.88 milliseconds for hill clusters.

Let's I redid the test against more recent driver and 580.88 88. And these timings were taken with a in-game performance counter uh just an average over 30 60 frames or so. So, and when I open up the trace initially, uh I will look at this top level throughput. And you typically the way I think about it is that you typically want to try maxing out some category of top level throughput first because you want to maximize GPU utilization.

If something is already maxed out, it kind of means that there is a bottleneck and further optimization has to come from reducing that bottleneck or moving that bottleneck somewhere else and that can get more complicated. So an empty area in this graph generally means a GC optimiz opportunity for optimization and due to load utilization and depending on the nature of the pass you know if you have a lot of buffer loads and texture fetches or memory rights they're going to become memory bound in some way or the other um you can sometimes become ALU bound if you have very low memory usage but high calculations maybe complicated uh calculations and then or you can be RTOR bound in a ray tracing pass.

So another thing that stood out to me here was also that we have a pretty high average warp latency and this makes uh we should check out why we're stalling. I'm going to switch to actual live view of the trace that we have here. this same one in the in the slide and this is that area of uh compute cluster tiling and notice here on the right we also have a percentage view of the the metrics so we can see that at most the highest top level throughput that we get is SM at 26.7% uh we have this very high warp latency which means that things are taking very long and maybe we're not getting the expected occupancy because uh there's a lot of warps that are waiting or stalled.

And you can kind of see here if you open up there's this another section called SM warps stalled at issue stage we have a lot of warp stalled on long scoreboard which has to do with global memory fetches and texture fetches. Going back to the slides. So the next thing I like to look at is in the the shader registers view just to make sure like the shader counts um in terms of warp max warps register pressure and shared memory seems reasonable.

But I also like to look at the indep input dependencies um on the left. So this is this panel on the bottom the shader pipelines here and this is our compute cluster tiling shader. If we scroll to the right we get this 32 64. This is you know okay not not too concerning for the number of max warps. Um and there is pretty high shared memory usage even though our our shader doesn't use that much shared memory. Um and we can kind of tell from here as well.

We can look at input dependencies and there are surprising number of stalls on well atomics are expensive but there's also these local memory loads. If we look at our uh fill pass as well, we see 47% of our stalls are in from local memory loads. So that is definitely something very strange. Uh similarly, we've got pretty high shared memory usage, but the shader actually doesn't use group shared memory. So both local memory loads and shared memory loads.

It means that we're probably spilling registers or the compiler has decided to spill registers. So this register pressure might be artificially low. Yeah. So I cover this here and just to kind of highlight something local memory is actually uh per thread memory but it's coming from L2 and global memory. So this is mainly used for dynamic arrays or register spilling. Um meanwhile shared memory it's from group shared and this memory comes from L1.

Uh but it can also be used for register spilling as we'll see in a moment. So just to clear that out local memory is not very local. It's only called local because the instructions are per thread and you don't have to do any special addressing. So there's a pretty neat feature is that uh we can investigate the the source of these local memory loads. Um and what I mean by that if we go back to this so there's going to be 35% of the stall time.

Um but this is not like it doesn't get stalled on at the load instruction. So if we if we click through it automatically brings us to the HLSL and to the SAS we have on the right. Now you can see that there's no sample time happening here. But what's happening is that downstream there are things waiting for this load instruction. So this is a critical feature in identifying your bottlenecks is that often times the place where you're stalled is not where um it you're stalled on the place where you're waiting for the memory load to come back.

So we can see that for some reason oh yeah we've got endpoints here and this uh dynamic loop. So endpoints looks like it's uh some dynamic variable and this is probably causing us to load into a dynamic array. But if we actually click through we can find that m points is either zero if patch type is regular then m points becomes a fixed number 16. So this is not actually a dynamic variable and it's actually uh we can simplify this code and maybe help the compiler a bit.

So let's switch back to presentation. Ah yes and so LDL on the for the SAS instruction it's for load local and STL is store local. Yeah. So this is the code I just showed you and endpoints looks like it's dynamic but we can change this into a pool of whether we have points or not and then hardcode this dynamic variable. Then the compiler was able to unroll this and we can see that for the same pass our stalls on local memory that was at uh for this particular line was at 35%.

That line has disappeared and our total 47.6% of stalls has now reduced to 5.6%. So we definitely got uh rid of those uh particular uh lo loads. All right. Um so what happens if it's not a you know a local memory load? If we go back to our timeline here, uh we've got pretty high usage of global memory as well. Uh another way to kind of find some stalls in your application is to go through the shader profiler view. So if we for this particular compute uh cluster tiling pass, we can click through on this bottom right here.

We can click through on the shader. This will open up in the shader profiler. This button over here will send us to the top the highest value for our total number of samples. So this is uh one way to kind of step through and see where you're what where you're getting stuck at. So if we scroll down now a lot of these are going to be legitimate like you have to wait from global memory loads. Um, but we'll find Okay, here is an example where okay, I've got this a bunch of writes to a UAV this cluster shading data and for some reason I am getting stalled on a surface load.

Now I shouldn't be because I'm I'm storing but clearly there is a load instruction here with this surface load as well as you can see in the SAS here this is su surface and load. So what's going on? Well, uh, this is a strct and turns out the strruct has a bit field and so there there's two fields that are eight bits each and uh, the UAV write is going to want to write four bytes at a time. So, because this is a partial write, uh, the compiler has for correctness has said, well, maybe something else has altered this.

So we need to have a load of that particular word to make sure that we have the latest information. Now the it's a pretty easy fix, right? Just uh initialize the strct to zero and write out the str completely and then assign once instead of doing a per field assignment. And you can see that from the instruction mix. Um that's this uh most left side of this bottom area. Now once we fix that then the surface load part is completely gone and there's no more stalls.

All right. So let's try to understand after those two optimizations and a few less more interesting ones. Uh our first pass at optimizing this netted us uh a millisecond on cluster tiling and about 1.3 milliseconds on fill clusters. So eliminating the local data stores doing dynamics loops was huge and yeah it includes some optimizations I didn't cover here. So some unne unnecessary atomics and some displacement map samples when the mesh doesn't have displacement maps.

Okay, but let's go a little bit further and we're going to try to understand why this optimization worked and what exactly improved and we can do that through uh this trace compare feature. So if we go back to insight, the way you do this is you click select the first pass and then control leftclick to select the next pass. If you do that, it will show up in the correct order. First file, and you can choose which frame uh to compare to each other.

We got baseline and first pass here. And once you do the comparison, so we'll just uh zoom in on this part and you can see it will highlight the corresponding areas and just we'll do align selections. Can synchronize vertical scrolling. And well it's it's clear that uh things have shrunk. Um but another thing I like like to do is just look at the top level throughput and see what exactly got faster. So you can see that VRAN total bandwidth improved by about 30% and L1 improved by 30%.

Um, and this just helps you give some more confidence about what did you actually speed up so I'll definitely go into more examples in later passes. All right, so that's what that was. Uh I do want to point out something that you know previously you might have uh relied a lot on the number of max warps and the register pressure uh but more warps doesn't always mean faster more max warps right so it actually matters how many warps you can launch and uh even though you could potentially launch more max warps doesn't always mean that you are launching them.

Um we can also go back and see that um if you filter for sync compute warps launched this is the number of this is a rate of launching we actually have 80% improvement and we able to launch more warps but um you can see that even for cluster tiling for instance if we just highlight that area for instance we can launch 70 we're 70% more efficient at launching the warps but our max number of warps actually went down right um and for fill clusters our optimizations actually resulted in more max forps and less register pressure um so yeah it's it's always not it's not always so straightforward it helps to look at the the stall points we had less of the the local memory stalls were were greatly reduced uh between the two.

So um don't just look at max warps registers and the shared memory usage. Okay. So we're going to look at the next optimization. So the next kind of B bottleneck is that well atomics were were very expensive. So and this is in the after we applied our first pass optimizations we still have a good number of weights on atomic six 16% stalls and yeah we can click through again and we can see where exactly this is happening.

Okay, we're doing we're doing a bunch of different counters here. And to give some context, we're allocating memory essentially inside the shader. And each of this is being done per wave. Uh so I equals zero. This is happening on a per wave level, but we're getting stuck uh waiting for these atomics to come back. uh and we can before we can tell if the allocation succeeded. So one way of solving this is that instead of doing uh each wave does its own atomic, we're actually going to coales uh we have four waves at a time.

So, we're going to use group shared memory and do one atomic for every four waves instead of and there's four waves in the thread group, right? So, we're going to do one atomic in the thread group instead of one atomic per wave. Yeah. And this code looks pretty looks like this. We've got this interlock ad on the shared memory of these uh counters. We're going to do a group sync to make sure that all the waves are complete and then only on the the first wave we're going to do one global atomic for all of our waves.

Now I have seen the driver do this coalesing for you automatically if you're doing per thread atomics and it will do that by using the wave intrinsics to coales across multip multiple threads. But in this case it it can't do that because we are uh allocating across waves and so we have to use shared memory and also it changes the uh allocation a bit because uh it also means that we allocate four waves at a time. So if we were to run out of memory, we'd be slightly less efficient.

But uh running out of memory is the uh more rare case and we you know we want to prioritize performance. All right. So what did this coalesing do for performance? Uh this helped save us about 6 milliseconds. And you can see from the input dependencies view that the atomic stalls stalls on atomic operations have completely vanished. So that's great. This is where our speed up came from. All right. So we're going to go into a more difficult optimization.

This is more specific to your the algorithm and but it's a common strategy in if you have a lot of dynamic branches. And so in our particular case, we've got three types of input surfaces and they had different complexities. Uh we have what's called this pureb spline patch. This is our simplest case and we can see for a particular asset like the kitchen set, this is taking up 70% of the input surfaces. Then we have something called like a regular patch and this is in this case it's only 1%.

And then we have the most complicated um smooth patch. This is 27%. Now this is basically um we want to take one shader and we're going to make it into three shaders that are statically compiled with this dynamic branching eliminated. It also means that we have to reorganize our dispatches so that each dispatch is specialized to only those surfaces and we have to do sorting on the input surfaces in order to um separate them to three separate dispatches.

So uh the result of doing this was that okay it it required some more overhead in the initial pass because now we had to run this multiple times and for each surface type. So we got a little bit more overhead here. Um but it sped up the fill clusters by another 2 milliseconds or saving us 16%. So let's understand why that helped. So actually in an earlier version of this presentation, I did a test against 576. And this was a lot easier to understand because the previous combined shader was pretty heavy.

So it consumed 80 registers and it only supported 24 max warps. And then when you split them out into the three different shader types, then boom, all the max warps uh went up. So this is really easy to understand. Okay, great. More occupancy uh then like it's going to be faster. But in our newer driver, um the original shader actually had a greater number of max warps. And I I was confused at this by first, but I noticed that the shared memory used to join up.

So that that's a clue. Um and again, you can tell from the uh shader stalls uh after the optimizations we did already. Uh we have these uh you know shared memory loads and local memory loads. Again, this shader is not supposed to be using shared memory. So these are from register spills and in the older version that just used more registers you can see that there's no uh shared memory or local memory. Um so what actually happened here? is that the driver decided the compiler decided that okay there is some code paths that use these variables very selomly and it decided to shove these into shared memory and in particular I found that in this case this is the descriptor for this structured buffer vertex patch points and it does happen to be that it only is used in one of the three branches um so I decided to shove these into shared memory uh indexed per thread.

And then um it and when we after with specialization this the particular uh branch that used this limit surface shader now no longer had to contend with register spilling. Then these were just put back into uh normal registers here. So if we look at the trace compare view just to understand again why did this speed up um also is is kind of complicated but just from the get-go if you look at the visual graph you can see how organizing the work to different workloads and specializing the shader had an interesting effect on the top level throughput graph.

You can see how there's now this section you can see is one type of workload and this is the simpler path and then the more complicated auto value path is actually aloubound right has very high ALU usage and less memory usage um and this is just interesting from a okay in a combined shader you're going to be relying on theuler to do a lot more work in terms of the branching part, but if you're able to organize the work up front, you can get some savings.

Um, and you can kind of see that and how more of the area of this graph is now being used up. uh specifically this is was harder to compare this time because so the one top level throughput that went up was VRAMm but all the other main indicators is that L1 was actually slower uh less throughput and L2 had less throughput as well and I'm going to guess that this is because L1 and L2 were previously inflated because of the shared memory loads and global and local memory loads.

Um, and we can confirm this again that the stalls were eliminated on shared and local memories once we specialized the shader. And another indicator is that okay, even though L1 throughput was lower, more of the memory was allocated to the global buffer loads and texture fetches. So the actual memory we care about fetching, not the spilled register memory. And we can see that this is in the tagged memory section. We have 80% more bytes being allocated towards tag memory and a lot less of L1 being allocated for shared memory.

And so there were this resulted in less stalls because we could use our cache more effectively. And this resulted in better launch efficiency. So we got more warps into the air so to speak. And this also meant we could request more memory uh from VRAM resulting in the overall bandwidth improvement. All right. So let's talk about our last optimization here. And this is called the group shared prefetch optimization. So again this is very specific to this algorithm but the purebine the simple path support um the control points are shared across the whole uh thread group.

So instead of uh fetching the control points inside the inner loop. So this is being evaluated per thread per vertex. We need to go through the weights of the input surface and we need to go and fetch the control points. But again yeah all these control points are shared in this pure Bspine case. So instead of fetching it in the inner loop here, we're going to do a prefetch into shared memory. And this is going into group shared memory.

And later on, we're going just going to access it from shared memory. So what did that do? Okay, it it sped us up by 0.21 milliseconds and another 16%. And just to put into perspective, total speed up we've achieved is minus 1.6 milliseconds and minus 1.8 milliseconds on fill clusters. About 62% reduction or over 2x speed up. But we want to understand why that how did this help us? And if we look at our trace compare again, we can see that.

So VRAMm is better, L2 is better and SM is better uh for this particular pure BPLine case. And this is as we we expected. We saved a little bit of registers, but we're putting that into shared memory usage instead. And what's happening is that we're doing we're probably doing more uh loads from L1 and we're being more efficient about what we request through L2 because we don't need to do duplicate requests and that helps the uh stalls reduce the stalls and then we were able to launch more warps uh per microcond.

And another interesting stat is that uh because we are not doing those duplicate reads anymore, we actually have less van vramm reads. Um and so this saved bandwidth from having to refetch the same control points and allowed us to eventually get more varietes out. All right, so there you have it. Uh here are some conclusions. Use Insight graphics GPU trace to identify hotspots quickly, right? uh definitely rely on the input dependencies view and that is like critical to help finding the source of the stall and previously this wasn't before this you had to look at the SAS and match up the weight request from where the fetch was actually happening so this is like a huge feature um another thing to keep in mind is that yeah the compiler is constantly being optimized and changed and I got different results and 572 and 576 580.

Uh you got to consider the spills to local memory, shared memory in addition to max warps and registers. Uh this goes without saying, but uh try to improve your iteration cycle. Get a default scene that launches with the same camera view. Use shader reload and recapture multiple times. Um, get timing measurements that are automatically averaged. Use runtime switches. All right, that's all I have. Um, hopefully this was helpful and happy optimizing.