How to Build Practical Applications Powered by RTX Neural Shaders transcript

Today we're going to take a practical look at neural shaders, not just the theory behind them, uh but how they actually translate into real high-performance graphics application. We'll walk through the journey from the underlying mathematical ideas and architectural concepts all the way to production-ready implementation that run efficiently on modern GPUs. I'm uh Alexey Bakhtin of tech engineering at Nvidia and joining me is Dilu, graphics developer tools engineering at Nvidia.

Uh together we will explore how neural techniques are reshaping the programmable graphics pipeline and what it takes to make them fast, scalable, and usable in real-world systems. So, I'll begin with a high-level overview of neural shading and why it's strategically important. Then I'll introduce the neural shading SDK and how it enables adoption. Uh we'll show how neural texture compression um drives memory and bandwidth savings, and I'll close with neural materials and what they unlock to next-generation content.

And uh D will cover neural shading intrinsics and how they map to the hardware, show how I critical GPU behaviors uh that impact performance. Then we'll uh look at inside graphics a GPU trace for practical optimization insights. So, why neural shading? Uh real-time rendering is about uh approximating reality with uh highest possible fidelity within a strict time budget of uh 16 or 32 milliseconds per frame. This pursuit has uh driven over 40 years of increasingly sophisticated graphics pipelines and shader architectures.

From rudimentary 3D worlds uh worlds to the photorealism we achieve today, uh the evolution of real-time graphics has been a continuous push toward greater physical accuracy and visual richness. Take a look at uh the Nvidia's origin. We can render this scene uh real-time today, but can we render it faster? Can we reach 60, 120, 240 FPS? Can we scale this to an entire city or even full world? Can we Can the environment respond dynamically to in-game events?

Can we populate it with characters of equal or high visual quality? Can we integrate with a system or dynamic vegetation? So, and we need to be able to store, ship it, and render it. Uh real-time rendering has relied on Moore's law to increase uh fidelity year after year as hardware become faster and more powerful. As a Moore's law slows down, we must uh look for the new approach. Neural shading offers a new path forward.

So, neural shading is uh an umbrella term for any techniques or algorithm that utilizes machine learning within the traditional rendering pipeline. In neural shading, machine learning uh learning models are replaced or augment traditional shading functions with learned neural networks. Uh these networks can approximate complex light uh lighting, material, and appearance functions that are difficult to model analytically.

Let's take a look at the neural shading SDK. So, let me quickly walk you through the SDK content. Uh we provide a set of samples that demonstrate different aspects of using the neural shading SDK. This range from simple examples to show how to use uh pre-trained uh MLPs uh more sophisticated scenarios where interface and training run in parallel with graphics workload. Uh at the core of the SDK is a neural shading library.

This library provides uh MLP implementations along with the utilities for defining and executing neural networks directly inside shaders. It includes support for cooperative vectors to enable efficient GPU execution, built-in automatic differentiation for training, and flexible system for defining activation function. All of these capabilities are integrated into unified infrastructure designed to support real-time neural evaluation and optimization within graphics pipelines.

Let's take a look at the 30-seconds video illustrating training process for one of the samples we have in our SDK. So, at the beginning we initialize So, on the left side is original shader and we're trying to approximate it using neural uh neural shader that use MLP for that. So, in the middle is uh just um current render with uh randomly uh initialized that weights. Um we see only white spheres since the MLP uh just initialized it uh with a random.

And once training start, and I start video, uh the model quickly begins to learn the target function, and we observe rapid improvements in the reconstruction. After about 10 seconds, the model already produces a reasonably accurate result. Uh by around 20 seconds, the training has essentially converged. How However, if we examine the difference view, we still can see, which is on the right side, uh we still can see some slight oscillation.

Uh this occur because the training continues with randomly sampled input data, causing small fluctuation around the optimal solution. So, let's take a closer look at the shader training process. We start by applying positional coordinate to the input parameters. This helps uh the neural network better represent high-frequency variation and improves the overall utilization of the MLP capacity. Next, we generate reference image using Disney BRDF render, which serves as our ground truth.

Um we then compare neural shader output again the uh this reference and compute the loss function to measure the error. Uh from that loss we calculate uh the gradients and feed them to iteratively update and correct MLP weights. Over time, this optimization uh process uh drive the neural shader closer to the physically based Disney BRDF behavior. The implementation uses four uh our shader take uh share a large portion of common code to keep the pipeline consistent.

We start with the original Disney pixel shader, which performs three stages: preparing inputs, evaluating the Disney BRDF, and composing the final color. The MLP-based pixel shader keep the same structure but replaces the analytical BRDF with MLP inference step. While input preparation and output composition remain unchanged, the training shader runs both uh the analytical BRDF and uh neural version, computes the reference result, evaluates the loss, and calculates gradients.

Finally, the optimization shader updates uh the MLP weights and biases using those gradients. Overall, rendering, neural inference, training, and optimization are separated into different shaders while sharing common code and data flow. Nvidia is opening the door to the next era of real-time graphics by bringing AI directly to into the shading pipeline. This initiative signals a shift from purely handcrafted rendering toward intelligent learned visual computation.

The SDK provides a foundation for developers and studios to experiment, innovate, and build production-ready neural rendering workflows. You can explore the initiative and its technical foundation uh the link, and you can use the QR code for that. And the next case is neural texture compression. So, what is neural texture compression or NTC? It's a machine learning approach for storing textures more efficiently. Instead of storing every texel directly, NTC compresses the texture into compact learned latent features that capture its essential visual information.

At run time, a small neural network running on GPU reconstructs texel values from these features, computing them on demand instead of uh reading large textures from memory. Importantly, NTC is deterministic. It does not generate new content given the same latent features and network weights, it always reconstructs the same texture. So, why are we looking at NTC in the first place? Uh first, it provides significantly higher compression ratios compared to traditional GPU formats like BCN, allowing us to store much more texture information using less memory.

Second, it works very well with uh high-channel count materials. Complex assets with many material properties or packed feature channels can be compressed efficiently without splitting or simplifying the data. And finally, this leads to a practical benefits: smaller disk footprint, faster downloads, and more compact texture storage overall. In short, NTC delivers better compression, greater flexibility, and improved storage efficiency.

The core component of NTC is latent textures. Instead of storing traditional RGB texels, textures are encoded into latent feature maps that contain compact neural representation rather than final colors. Each texel stores a feature vector describing material structure and properties, capturing the essence of the texture instead of memorizing appearance. At run time, a neural decoder reconstructs the full-resolution texture from this latent features, generating detail only when needed.

By learning shared patterns and removing redundancy, latent texture can achieve very high compression while maintaining visual fidelity. So, our network has um two main parts: a compact latent texture and a neural decoder. Instead of storing a full-resolution texture, we learn a much smaller latent representation, greatly reducing memory usage while preserving essential information. To reconstruct fine details, we apply positional coding to the UV texture coordinates before feeding them to the decoder.

Uh this adds high frequency special information, helping the network better capture subtle details and repeating patterns, resulting in sharper and more accurate texture reconstruction. So, let's look at how the NTC training process works. Uh we start by applying positional encoding to the UV texture coordinates, which helps them will be captured high frequency detail. A latent code is also provided to give the network additional flexibility for represented complex texture variations.

The network output is then compared to original texture, which serves as a ground truth. Uh from this comparison, we compute a loss that uh measures the reconstruction error. Using this loss, gradients are calculated and both MLP weights and latent code are updated iteratively. Over time, uh the neural shading converge and learns to accurately reproduce original texture. So, let's take a look at the examples of this Tuscania real scene.

In this image, uh we are seeing the final render produced using BCM compression uh compressed uh textures. The entire scene fits within approximately 6.5 GB of VRAM. And now we're looking at the same scene rendered using NTC, but this time it consumed only 970 MB of VRAM. As you can see, there is a no significant visual differences compared to the original rendering. The image quality remains footprint is dramatically reduced.

Let's compare BCM uh texture compression with neural texture compression using the same VRAM budget. In this image, this scene is rendered using BCM compression using just uh 970 MB. And now here's the same scene rendered in NTC using the same amount of RAM. And finally, we can see side-by-side comparison of the two approaches. Even under the same memory constraints, NTC preserves significantly more texture details and visual fidelity while BCM introduced noticeable compression artifacts.

Uh this demonstrates how neural texture compression can achieve substantially higher quality in the same memory budget. So, neural compression provides several practical benefits. It significantly reduces disk footprint, which means more game installs and smaller patches. Important uh because textures dominate modern game storage. It also reduces download bandwidths and update times. In some scenarios, it can lower RAM usage by storing compact latent representation and decoding textures on demand.

Importantly, uh this technology is already available today through the NVIDIA RTX neural texture compression SDK. Uh you could see the link and QR code for this SDK. Conceptually, neural compression also enable higher detail materials within the same memory budget by trading compute for quality. And uh this perceptual loss function, we can achieve high compression ratio while maintaining or even improving perceived visual quality.

Now, next case is uh neural materials. So, real world uh materials serve as a core inspiration for neural materials. Instead of inventing an arbitrary representation, we learn uh from how physical materials behave in reality, how they reflect light, scatter energy, and respond to view directions. Modern techniques allow us to render very complex materials with high visual fidelity. However, achieving that level of detail in real time remains a major challenge.

The reason for this is that we're dealing with deeply layered materials. These materials are built from a complex shader graphs with many interconnected nodes and dependencies. As a result, uh working them efficiently become extremely challenging, especially when we try to render everything in real time. This highlights um an open problem. We still don't know how to simplify this. So, what are neural materials? Neural materials represent material appearance using learned neural features instead of traditional parameters like textures or analytical BRDF coefficients.

Rather than storing separate channels such uh albedo, roughness, and normals, uh these properties are compressed into a compact latent feature vector. A small neural network decodes this representation to produce the material response for a given direction or shading context. This allows many material attributes to be stored and streamed far more efficiently than the conventional texture-based approach. The main advantage is a much smaller memory footprint, enabling richer material appearance within the same VRAM and bandwidth budget.

It's also important to note that neural materials remain an active research area at NVIDIA and across the graphics community. This promises result, but limited uh production deployment so far. Um When we look at the rendered object and think uh it looks real, it's rarely due to a single shading model. Realism comes from layered multiple material components, each capturing a different type of a light interaction. One layer may represent diffuse color, another uh glossy reflection, and another subtle surface effect like clear coating or micro roughness.

Together, these responses create the appearance of realism. So, rather than a single equation, it's better to think of a material as a stack of light responses working together. Now, let's take this uh material and examine its component in more details. Um So, we start with the base uh ceramic layer, which serves as a foundation for everything that follows. We incorporate a gold vapor deposition layer into the material model in our furnace uh to capture the characteristic uh reflectance and spectral behavior of a thin metal coatings.

Next, we add a clear coat glaze, which gives it richer, more polished appearance. And finally, we add a sublayer of dust to help place the object into the real world. Uh these extra details break the perfect synthetic look and make uh the material feel physically grounded. At this stage, we're working with reference texture that contain uh 19 different channels capturing all the material properties we need. Now, let's take a look how we can represent all of this using a neural network.

So, we need to train a latent code that represent texture of the material. Instead of storing raw color maps or traditional material parameters, we compress the texture information into a learned embedding. This latent texture acts as a compact feature representation. They capture the essential appearance information in such more efficient form. Then we train a decoder MLP that takes this latent representation and reconstructs the final material properties.

Conceptually, this is very similar to the NTC network. We encode texture information into a compact latent space and to use a neural decoder to recover the detailed detailed shading parameters when needed. Um so, the latent texture we use can be very high resolution, even matching the resolution of the original material textures. However, instead of optimizing latent values per texel directly, we introduce an encoder MLP.

This network takes material parameters and map them into a latent code. This avoids treating each texel as an independent parameter, which would be memory heavy and poorly structured. Instead, the encoder learns a compact and meaningful latent representation of the material textures. Importantly, the encoder is only needed during training. Uh once the latent texture is generated, the encoder can be discarded, keeping the runtime model lightweight.

So, during training, the system acts as an encoder-decoder model. Material properties are encoded into a latent texture, and the decoder predicts the final BRDF values. The training loop itself remains unchanged. We simply insert structured latent uh structured latent bottleneck. So, let's take a look at the result neural material. So, we originally had uh 19 texture channels, and now we have just eight texture texture channels for latent textures, and you can see how material behaves in real time rasterization scene.

So, for performance, uh what we report here is uh total render time for a full 1080p frame rendered at the one sample per pixel, measured in milliseconds. As I can As you can see, those speedups range uh between 1.4x to 4.7x in this particular setup, which is fairly substantial improvement, especially at the full HD resolution. So, why we have that speedup? So, why neural materials are faster? Neural materials are faster because they change how surface appearance is evaluated.

Instead of running complex BRDF equations and multiple texture lookups, we use a small neural decoder that directly approximate the material response. This remove expensive analytic BRDF computation and replace many textures like albedo, normal, and roughness with a compact lighting texture, greatly reducing memory bandwidth. Finally, rather than evaluating material layers sequentially, the neural decoder produces a full material response in a single efficient pass.

So, neural neural neural materials reduce smart complexity, reduce memory traffic, and fuse layered shading into one streamlined computation, making them significantly faster. So, neural materials emerging research direction aimed at bringing highly complex material to real-time rendering. Um instead of storing materials as a large set of textures, they encode material behavior into a compact neural representation. This allows complex appearance and light interaction to be represented far more efficiently, reducing memory usage and bandwidth while preserving visual quality.

At NVIDIA, this is an active research area exploring how neural representation could enable higher quality materials in future real-time graphics pipeline. So, neural shading is beginning to transform industry After decades of traditional pipelines, neural approaches are opening the door to new shading paradigm. Importantly, this is no longer just a research concept. Modern GPUs already support neural rendering and necessary hardware is widely available.

The technology is also cross-platform. Support is emerging across multiple graphics APIs and ecosystems, making neural rendering a practical industry-wide evolution rather than any share feature. With that, I'll turn it over to D. All right. So, um Now that Alexey has talked a bit about the various different neural shading, texture compression, and material MLPs, we're going to take a closer look at how NVIDIA hardware is actually able to run inferencing in real time.

In this section, we're going to introduce the new Vulcan and DX12 intrinsics that aid neural shading, called the cooperative matrix and cooperative vector. We will also take a pretty in-depth look at how the GPU actually accomplishes matrix multiplications to bridge the gap between the concepts and the hardware. And then we will go over how to optimize our shader code for the best performance and how NSight Graphics might help you in this process.

Just a fair warning that this section will be very dense and we will go over a lot of important but somewhat maybe seemingly disjointed concepts pretty quickly. But I hope the key takeaway here is that you don't have to maybe remember everything that I mentioned, but maybe one or two things will stick out and you will use them in your own applications. So, when we talk about the motivations to introduce new Vulcan or DX12 features, we have to begin by examining what type of work is actually involved in inferencing and training.

Alexey already talked about this in depth with regards to neural shading, but from a very generic standpoint, training involves the following steps. They are forward propagation, where a neural network essentially begins with an initial set of weights. We can pass an input through the neural network and evaluate the final output. Once the initial output is computed, we evaluate the outcome against some kind of ground truth and come up with a loss that represents how far away we are from our ideal.

Using that loss, then we evaluate a gradient for the neural network weights, which is a set of directions that we can change each network weight in order to minimize the loss. And then lastly, using the gradients, we update the weights of the neural network in the optimization step. However, inferencing is a lot more straightforward than training and usually only involves propagating the input through an already trained neural network to get an output.

So, forward propagation is sort of common to both of these steps and since training can be done offline, inferencing typically runs live during the game, we want to pay special attention to forward propagation. If we look at a very simple case of forward propagation, we have a simple two-layer neural network example and in order to get each element of the output vector, we actually compute a weighted sum of the inputs along with the bias vector term and then apply a ReLU activation function to get each of the elements in the output.

If we discount the activation function, it becomes apparent that we're doing a lot of matrix and vector multiplications, right? The input here is a three-element vector and the weights of the neural network is just a 2x3 matrix. So, if we know that both forward and backward propagation involves a lot of matrix multiplications, then you're actually in very good hands because NVIDIA has dedicated hardware to run these operations very quickly.

On the NVIDIA GPU, the hardware that is responsible for executing all shaders is called the streaming multiprocessor and we typically shorten that name to the SM for short. The SM executes shaders in a parallelized manner by utilizing warps and you can conceptualize a warp as just a group of 32 threads that execute the same shader instruction in lockstep. The second element we need to care about is the tensor cores. In this picture, we have a diagram of a typical SM and as you can see, it's broken down into a number of sub-partitions.

This is the case for Ada GPUs where we have four different sub-partitions. And within each sub-partition, there is a warp scheduler, a register file, and some tensor cores. On each cycle, the warp scheduler in a sub-partition will select one of the available warp slots in the sub-partition and prompt it to move forward to the next instruction of the shader. If that next instruction is an HMMA that needs to make use of the tensor core, then all the data from the individual threads will be combined and fed into the tensor core once from the SM register file.

At this point, the tensor core will then operate in cooperative mode. So, when we say cooperative, uh this means that the threads might actually read and cooperate with each other's data in order to compute the final output. And this is not the prototypical behavior of SIMT programming model where each thread is its own independent context. You can think of the blue block as the tensor core taking the input matrices and then spewing out the computed result into the SM register file.

Once the data is ready, the next time the warp is selected by the scheduler, we can move on to the next instruction uh denoted by the gray lines. So, tensor cores are always executing a matrix matrix multiply and accumulate that spans the entire warp. If you were to think about the matrix example we just saw at a warp level, since shaders execute in SIMT, each thread can hold its own vector element. So, conceptually speaking, we have the weight matrix and then each thread has a different input vector and potentially different biases as well.

And the output here is a height by 32-dimension matrix. Now, if a developer were to write matrix matrix multiplication that leverages the tensor core, there are at least four different factors you'd have to keep in mind. And the first one is that the tensor core expects matrix values to be in a very specific layout by the time it's handed to the tensor core. And number two is that the matrix values are expected to be tightly packed into 32-bit registers.

So, if you have a matrix of, for example, floating-point 16s, you would have to manually pack them. Number three, you'd also have to manually load and store the matrices in shared memory, which is not readily available in the graphics pipeline. And lastly, you'd also have to manually ensure that all the threads are sharing the same AXMB matrix and figure out how to handle the case where the A matrix might not be the same, in this case, the weight matrix.

This is a lot of work for a developer to manually keep track of. So, graphics APIs like Vulcan and DX12 have exposed extensions that would make handling all of these aspects a lot easier, so developers don't have to directly worry about the details. Um in this example, we're showing Vulcan, which has the cooperative matrix and cooperative matrix two extensions. And then for DX12, they have just released shader model 6.10 in a uh in an agility SDK um and it's called the wave matrix instead of the cooperative matrix.

You can think about the cooperative matrix API as something that exposes the hardware programming model directly to the user. So, what does this mean from a developer's perspective? What is it good for? Since cooperative matrix is designed to handle matrix matrix multiplications, the workloads that work best here are um shaders that intend to build large GEMMs or networks. So, things like image generation and prompt processing for large language models are the best use case here.

This is all perfectly fine for compute shaders, but it doesn't work very well for specialized graphics pipelines because compute shaders readily have shared memory available and you have the freedom in deciding how big your workgroup sizes are and whether there should be coordination between them. Now, if you recall the mathematical example we used at the very beginning, uh you might realize that if you were writing this as shader code where each thread is operating on a unique input vector, it makes the most sense to express it as a matrix vector multiplication.

If you were to use the cooperative matrix API directly in this use case, then you would have to manually assemble the input vectors into a matrix, which then introduces even more overhead for a developer. So, of course, then Vulcan and DX12 also expose the cooperative vector API. It comes with its own set of load store at functions as well. And in this picture, you might notice that the DX12 functions are the exact same list from the matrix case and that is because you can use the same uh functions for matrix matrix multiply and matrix vector operations as well.

The compiler will just pick the right one based on what you passed in. So, the cooperative vector model makes it much easier to write inline neural networks and graphics shaders, which unlocks a whole wider range of potential outputs like what we said for neural texture compression and neural materials. Now, we're going to switch gears quite a bit and talk about the performance implication of these matrix uh sorry, of these intrinsics from a hardware level.

As developers are using cooperative vectors, there are a few places where you could run into performance pitfalls. There are three distinct flavors of prob- sorry, four distinct flavors of problems here, and in this picture there I am showing instruction divergence, data divergence, and those two things are sort of general general purpose um potential pitfalls you can run into that are not specific to neural shading, but layout conversion and layer fusion are specific to neural shading.

So, the first one uh is warp divergence or instruction divergence. So, recall that we mentioned a warp is a set of 32 threads that execute shaders in lockstep. This is kind of the reason why branching is usually very expensive for shaders. When a warp advances to the next instruction, but each thread hits a different instruction, like in this case we have an if-else statement, then what's going to happen is all the threads that hit the if statement will be masked off while we execute the if statement.

And then once that's complete, we'll mask the threads that hit the if statement and execute the else statement. Basically, what's being shown in this picture right here. When we say masked, the threads that are inactive will essentially not write their results into any of the registers. So, the instructions have become effectively serialized, which causes us to lose a lot of capacity for parallel work on the GPU. Unfortunately, if you have an HMMA instruction, the idea of thread masking is not as straightforward.

Since the tensor core uses cooperative execution model, all of the threads are required to participate in that HMMA execution. So, if all threads are active at a time, then the threads will start executing disassembly related to the tensor core. If they're not active at the same time, the threads will take a different route, and basically use the fallback FP16 MMA mechanism, which doesn't use the tensor core, and I think it's also element-wise, which is a lot slower.

So, we talked about instruction divergence, and then second issue is data divergence. If a warp's next instruction is a memory load or store, then the request will go to the LSU on the subpartition. That just means uh load store unit. Which will find the best way to coalesce these addresses into as few requests into the L1 caches possible. So, suppose that we see we have a warp, and there are two different ways two different sort of um possibilities for that warp to um well, let's say there are two different um cases for a warp with uh where it's requesting for certain um addresses to be accessed.

If the warp looks a lot more like the first case, then it's going to be then the LSU will be able to coalesce all of these requests into a single request because they fall under the 128-byte cache line. But, if your accesses look more like the second case, the one on the bottom, then that's more random and scattered, and the LSU will have to serialize these requests, which again becomes an overhead. So, the tensor core expects a single sort of weight matrix, and in this case the we know that the X matrix in the middle, the second one, where it's made up of all the inputs, is going to be packed with the input vectors.

And we also know that bias matrix is also packed with all the bias vectors. So, we know that these two are guaranteed to be the same across the warp. However, the cooperative vector programming model technically has no strict pipeline level enforcement or offline shader compiler level checks to make sure that that weight matrix is the same across all the warps. So, the general advice here is to sort uh your is to group your draw calls by material and try to sort threads manually or use shader execution reordering to prevent what we just talked about from happening.

Um there are also compiler level optimizations to help detect this problem. If the compiler can't be sure that all threads will use the same weight matrix, then a divergence peeling loop will be added. If at runtime there turns out to only be one weight matrix, then this will only execute once, although branching does incur a little bit of overhead as well. So, here's a picture that demonstrates performance degradation on a log scale based on how many threads actually share the same matrix.

As you can see, as the number of threads sharing a matrix decreases, we kind of fall off of a cliff. So, we just talked a lot about warp divergence and cache thrashing, but is there a way for you to tell or visualize when that's actually happening in your game? And the answer is yes. In order to aid developers, NVIDIA provides a suite of debuggers and profilers. Some of you may already be familiar with this um Nsight Systems is a general system level profiler that also has some GPU metrics.

Nsight Graphics is the comprehensive debugger and profiler for graphics and games. Nsight Aftermath helps debug GPU crash. And the Nsight Compute is responsible for doing CUDA kernel analyses. The GPU Trace Profiler in Nsight Graphics is especially handy in exposing how hardware resources are delegated per API call and other useful instruction rate metrics. These metrics should be used in conjunction to determine what resources are actually occupying your GPU and how much space you still have left to fit any remaining work you still want to put through.

The um one of the ways that you can immediately tell whether your application is even using tensor cores is by checking SM throughput, which gives you the metric for uh SM tensor pipe active. The instruction mix for the shader also can tell you whether you have any instructions that are planning to go through the tensor pipe. You can also visualize warp divergence through GPU Trace because it provides an active threads per warp metric on the timeline.

So, typically you'd like this value, that's the top screenshot right there, you'd like this value to be as high and as close to 32 as possible. The second method is to use the shader profiler to get a per source line per warp count. You can also visualize uh cache thrashing or data divergence in GPU Trace as well. Now, keep in mind that the GPU has four different levels of memory. Uh registers are the fastest, L1 sits on the SM, and is the next fastest.

L2 is shared memory between all of the SMs, and VRAM is global memory, and that's the slowest. By checking VRAM bandwidth and L1 cache misses, um you can also also check L2 misses to see if you're uh seeing any cache thrashing essentially. So, now we're getting into problems that are very specific to neural shading. Um the tensor core expects an opaque custom matrix layout. So, the components of each matrix are mixed up in between the threads.

This layout depends on the GPU architecture and data type, and in this picture we're showing one example of the way that 32 threads can be distributed across four different computation groups. As a result of this, the matrix of weights should be pre-shuffled in memory. And the reason is because you usually don't need to convert the weight matrix at all during inferencing. Once you have a shuffled, you can use it in the shuffled layout.

By shuffling it once and using it multiple times, we can get better cache coherency as opposed to loading from a simple row or column major matrix layouts. Now, uh this needs to be done sort of at the runtime of your application, and the reason is because as a developer, you definitely don't want to implement a lot of if-else or uh checking statements based on all of the possible GPU configurations you could be running your game on.

So, Vulkan and DX12 also provide APIs to tell you how to do the conversions. Now, the procedure for doing this is that you could query shuffled matrix size layout using these APIs, and then you can upload the source matrix onto the GPU. Then you can submit an on-GPU conversion command through the command list, and then once you reach your shaders, make sure that you're using the converted matrix in all of your shaders for your cooperative vector operations.

So, we talked about the weight matrix and its issue with shuffling, right? But, from the hardware perspective, the input vectors also need to be shuffled in order to make it compatible with the matrix. Unfortunately, you can't do this the same way ahead of time the way you can for a weight matrix because these vectors at the time of shader execution are distributed on the SM registers. Luckily, the compiler implements a way to deal with this for you.

It can automatically shuffle and unshuffle the input vectors, but the shuffles and unshuffles are quite expensive. So, what the compiler then does is it will try to detect whether the work in between the gray pairs of unshuffle and shuffles can be done in the tensor core layout. If it can be done in the tensor core layout, then those redundant pairs will be removed. This is called uh layer fusion. GPU Trace provides a method to view uh shader source line performance stats, and now it also shows layer fusion.

So, although the feature is preview and will require an up-and-coming driver version, I did want to show a screenshot of what that looks like. We can also show you for your uh fusion regions why a certain region had broken. Um so, from a developer's perspective, the idea is that you should see as um you should see regions of blue and as few red lines as possible with that green line at the very end. Layer fusion is a little bit brittle, so it can break for various reasons.

In order to prevent this, we recommend that you avoid element-wise operations between each coop vec matmul call, and try to use vector operations instead. Secondly, you should also avoid element-wise load and stores. And if you have a bias vector, consider rolling all of it up into a single coop vec matmul add call using one function for the hidden layer and the activation function all at once. Oh, not the activation, but the accumulation.

There are also some things to keep in mind when it comes to actually choosing the right data type for your matrices. These are the four main types of precision available for the cooperative vector API. There's uh 16-bit float. We have two different types of 8-bit floats, and we also have the 8-bit integer. Neural networks are typically trained in floating point, which makes that 8-bit integer at the bottom the hardest to work with.

A ReLU activation function or applying biases, normalization will all be floating point values, which means you will have to do into float conversions. This uses a lot of register space, which is a pretty scarce resource on the GPU. 8-bit floating point is easier to work with because it doesn't require any int float conversions, um which makes it less register intensive, and it's faster. But, FP8s may not represent the full range of RGB values from 0 to 255, which makes it impractical for image reconstruction networks.

FP16 is the easiest to work with because again, no conversions are necessary and precision is good enough to represent large values, but the peak throughput of FP16 is lower, so networks will be slightly slower. There are some existing limitations as well of the cooperative vector with respect to performance and I know we just talked about the various um types of uh data that you should use in your matrices and I wanted to help us visualize um register pressure in particular when it comes to uh running neural networks.

So, in this example of an application, it contains a geometry pass called parallel draw and this pass essentially decompresses the in space on the material properties. And then the lighting pass called render lights will evaluate the BRDF. From a development perspective, typically uh a developer wants to shorten the amount of time that these passes take, right? When we're looking at the GPU trace report that was taken on this application, we notice immediately that the SM throughput is fairly high and we have very good active threads per warp for both of these shaders, which means there is not a lot of branching, which is already a pretty good sign.

However, in the towards the bottom, you'll see that the SM warp occupancy row, you'll see like the green uh against the backdrop of the gray. What this is telling us is that this uh particular shader is seeing very, very low SM warp occupancy. And obviously, the more warps you have typically on an SM, the more work you can do at once. If you were to expand this metric row, we can see that it's facing a lot of register occupancy, which means that the SM is not actually able to even launch more because uh of resource limitations.

So, as a result, there are still some work being done at NVIDIA to improve shader com uh compilation stages in order to basically better set register limits for these types of shaders. That is something to keep in mind. So, we encourage you to please download the uh neural shading SDK, the neural texture compression SDK, and the neural materials, as well as uh combining it with using insight tools to help you get the most out of your NVIDIA GPUs and try to understand how they how they're working together.

I will wait for a second. Okay. Uh thank you very much and uh thanks for attending the talk. Hope that was helpful. And I did want to uh specify that we that there is going to be a survey after this, so please give us a rating and feedback for how helpful we were and whether you enjoyed the talk. Thank you. >> [applause]