Top Game Performance Tales | RDC 2025 transcript

Please welcome to the stage Peter McNeel. [Applause] It's a hike out here. It's a big stage. How's everyone doing? Everyone had a good RDC. We're right at the end of it now. We only have the uh innovation awards left after this, I think. Um, my name is Peter McNeil. I work at Roblox. I've been here for about 8 months now. And before that, I've been a community member for about 7 years. And today I'm going to be giving a bit of a talk on top game performance tales along with my co-presenter Lynch.

She'll be coming out a little bit later to talk about the uh product side of what we've been working on. This game is uh this talk is about top games performance tales. So, in this talk, we're going to briefly cover a little bit of a overview of the tools that we have and what we have available. And then we're going to get into what performance problems look like for some of the top games on the platform because we've had a killer year and some of the games have had record numbers.

And we've been getting been getting involved with these games and helping them uh tighten up certain aspects of the things. We're helping guide them with what the problems they're seeing actually mean and then they go away and do the work and fix up the projects. So, we'll be talking a little bit about how we diagnose those problems and what it means to actually get in there and use the tools that we use, which are the same tools that you can use, too, to do the same thing for your games.

A little bit about what you should be doing with your projects to fix these things. And then at the punchline of all of this, when we find these problems with games, when we find the gaps in our toolings and find the gaps with the things that we could have been doing better, we take some of what we learn and then update the tooling here at Roblox to do a better job of it. But before we get started, there's a few disclaimers and a few little things to cover first. when we talk about performance problems.

And we're talking today a little bit about dead rails and a little bit about Grower Garden. Both of these game teams have been super kind and generous to let us talk about issues and blunders and little mistakes that they've made that we've helped them fix up. Some of some of their fault, some are our fault. Um, so it's a very human thing. So, thank you very kindly to them for allowing us to talk through this. I also want to have a shout out to the other optimization folks.

It hasn't just been me helping it out. There's been a whole team of Devril, uh, other people in the performance and optimization pod, all of infrastructure. We've had some crazy times this year as we've been shooting up to our 45 million concurrence, uh, just a couple of weeks ago. And as always with these tales, we're we're kind of blasting through a little bit quickly. Some of these things took place over days or weeks, so I'm cutting a lot of corners in the storytelling.

So just forgive me for a bridging in a few places, but to start off with something to ponder. What do you think this is? Have a look at it for a second and then uh we'll come back to it later. So as mentioned, it's been an absolutely crazy year for concurrent growth for Roblox. At the start of the year, I think our maximum uh concurrent user count was only at around 12 million and then at the moment our maximum concurrent is 45 million.

And we learned all sorts of new things about operating at this crazy scale. Some of it has been updating our infrastructure, but some of it has also been helping out these uh creators with performance issues when they've run into it, too. And so when these games reach out to us and they're having performance problems, the most common thing we've seen this year uh basically comes down to these three categories. These categories are client memory crashes, uh low client frame rate, lag as the kids call it, and uh server CPU overload where the servers are using not running at like as well as it could be.

SAS mentioned when we do help these game teams out and we help diagnose what the problems are, we don't go and fix it ourselves. We we help them use the tools, we help to come up with the battle plan and then all of the hard work is really theirs. So it's really their victory. So in these three categories, just to unpack what they are a little bit more, uh crash rates are really out of memory errors. So anyone who caught my talk last year will understand a little bit more what I'm talking about here.

But when we talk about crash rates, we're talking about low-end devices. When they there's too much content on them and they cross the magic 1.4 GB threshold, basically your crash rate starts climbing up because a big chunk of your players are still on 2 GB. And so you really have to be sensitive to that if you want to target all of your player base and have a well-running experience on them. And if you're getting into diagnosing what is going on with your client memory, still the best place to do this is to actually plug in a real device, go in and use the dev console.

And the dev console still has the best memory tools for this. We're introducing more tooling that will hopefully change the state of this. But for right now, the best thing you can do is go and use a real device to check the memory. Because if you use a PC client when you're looking into client memory errors on a mobile phone, you're going to actually get different readings because we do slightly different things behind the scenes on these.

There's also other tools inside of the dev console that have uh come along this year like the lure heap inspector which lets you find unparented nst instances and there's a lot of road map on there. Lynn will be talking a little bit about this too. In terms of low frame rate, the biggest indicator of low frame rate if you're not CPUbound is going to be scene complexity. Uh we have great talks already on scene complexity.

That's pretty much what I spent the entire talk last year talking about. And so I'm just going to refer you to that. Go check out that talk if you really want to get down into what's going on with your triangles and your draw calls and your geometry inside of the scenes. And if you want to look into this still, the best place to go is the F2 menu and look at those numbers right there in that. And then the last bit when you're getting into CPU usage, a lot of people know how to check your CPU usage on the client already because they go and use the microprofiler and then know how to bring that thing up.

But what a lot of people don't know is the microprile is also capable of pulling down HTML files off of actual live running servers and seeing what's going on there. And I heard from one developer, he said, "I didn't even know I could do this. Once I realized I could pull down HTML files of the uh microprofile dumps directly off the server, I didn't go back and use the other one anymore." When you're getting into the micro profile, the biggest things we've seen this year have broken down into physics and physics costs animations, like when you're running animations on the server, scripts, which is on you.

You pretty much are in control of what goes on inside of your scripts. And then sometimes there's been a couple of times when it's actually been bugs. So developers have come to us and they said, "We haven't changed anything, but suddenly everything's running slower and there's this big chunk of extra time in the microprofiler that we didn't see before." And that was us. we we'd change something on Roblox's side and we oh didn't catch that.

So if you see something like that, come to us on the dev forum, let us know and we'll see if we can diagnose it. Getting into the actual performance tales part of today's talk, let's start with a little game called Dead Rails. If you've not played Dead Rails, it's a storytelling sandbox game where you take care of a train. It was the first popular game on Rob Roblox Roblox to take advantage of the new physics draggers.

So, it's very physicsy and tactile where you're dragging lots of things around and you run around with some friends and you shoot a lot of zombies. So, when you have a game with lots of physics and lots of NPCs running around on servers, you can pretty easily run into server CPU usage problems. Oh, we had clickthrough images here. This is what dead rails looks like. So when you do have server CPU usage problems and you we use the same tools you do, we just pull up the micro profile and we have a look at it and we get a diagnosis like this.

So when we were seeing that dead rails was having server CPU usage problems, we got this exact micro profile back from them and then we could see where the hot spot was, right? We can see that this this purple bar in the middle, this purple bar in the middle and the jagged teeth sticking down. There's four steps of it. It's like, oh, that's physics. We can see these labels and we understand what's going on there. And we've seen this pattern before when you have a lot of NPCs running around and the NPCs are driving a lot of physics.

We thought we understood this problem pretty well because when you have NPCs on a server in Roblox and they're driving animations, every time the animation updates for the character running around, it also goes and poses all of the physics parts that make up the uh character on the server as well. And so this way animation physics end up being linked and it ends up looking pretty much like this. So the solution for dead rails ended up being something really straightforward for them.

What they noticed was they're already really good at spawning in towns as you moved around through the through the uh wilderness on your train and you'd roll out into the desert and then when a town would spawn in, they'd only spawn in the zombies there and then. But what they noticed was on the screenshot on the left, you can see that there's lots and lots of zombies all over the town, but only a small amount of them are actually really within combat range of the players.

So what the Dead Rails team did is they said, you know, these zombies aren't really within combat range right now. So they went and anchored them and turned off their animators and basically did that in a radius cull themselves. And that brought down the server CPU usage considerably for for the things we were just talking about. But it didn't quite solve it. What we this got turned on for Dead Rails and they got into it and they what we noticed was the server CP usage was actually still kind of high.

It cut it down by about half but but it wasn't quite where it needed to be yet. And so what we saw was when we were looking into what was going on, there was still quite a lot of awake physics parts awake out in the desert. And so if you see it's a demonstration here, these red outline boxes are the system that was supposed to be built into Roblox to cause physics parts to go to sleep when you're no longer using them.

If everything was working correctly, all of the physics objects inside of dead rails that were well outside of player range and nothing was interacting with them, they should have been turning off just like this when you kick over the wall and then they all go back to sleep. But we turned on the the profiler to go and check out the uh where all of the awake physics parts were on the server. We're seeing about 400 of them.

And then we noticed a problem because when we got into dead rails and we're flying around, can you see where all the awake physics parts are out in this world? Because we sure couldn't. What we realized was the outlined version for seeing where the awake physics parts were wasn't appropriate for finding out specifically for dead rails because it's miles and miles and miles of desert. So, we ended up having to write some custom tooling for this.

And then we took this back to the physics team and said, you know, this was wasn't super great that we uh couldn't find where the awake physics parts were. And they went and developed a new API that we'll be shipping this year that allows you to actually go and query the scene now for where the awake or asleep physics parts are. So, we won't won't run into this problem again. Now, what the actual problem was was minor authoring errors.

So this is a dead rails table and you can see the gun there has was placed slightly inside of the table and then got overly enthusiastic and is clipping through it. That gun's not going to sleep anytime soon. So you multiply that by an entire desert worth of items and shops and loot. You you can pretty much see what the root cause of this was. So there was two paths here for dead rails to actually go and solve this. They could go through every single table and every single item everywhere and make sure that it was all set up perfectly nicely for when it popped in.

But they actually kind of just cribed the solution from the zombies and if you're out of range of an awake physics part like a pickup and no one can possibly interact with it, they just anchored it and solved the problem that way. So they didn't have to be precious about the placement of things as much. That went well. The physics in dead rails was great. We got through the uh so-called live rails incident and then about a week later they introduced yet more physics chaos.

This team, you're starting to see a pattern where they're actually quite good at uh really stressing us out with how much physics they get around and use. What they added was a very cool, very unique liquid system. So you can grab a bottle and you can pour a little liquid out and you can scoop it back up in a bottle and it adds these parts that grow and shrink and get scooped up and then they also would rain from the sky as for some reason they decided that it was a good idea to have kerosene rain from the sky all over the map and these droplets would hit the train and then it' actually travel along with the train.

So when the droplet would hit a train they'd anchor it onto the train. We had no idea what was actually causing the the huge amounts of physics problems from this because in theory it should have been fine just welding a few extra parts around the place. But then when we got around to reproducing it for ourselves, this is me trying to reproduce it with 100 trains and 100 water droplets. What we discovered was that when you resize a server part that's been anchored on uh welded onto another part, it causes the whole assembly to recalculate and then also re-replicate.

So the solution for this ended up being stopping doing that and changing the water droplet over to a purely client visual effect. So the water droplet appears but then the resizing and the visuals of it is handled just client side. And in terms of performance, keep in mind this was back in April. The the peak CC for a single game was still at 2.6 million. Uh, Dead Rails was a huge hit, hitting a million CCU. And in fact, any game that hits a million CCU is a huge hit.

Congratulations to the team for doing that. But if some of you have been paying attention, you might have some idea that this was just the tip of the iceberg of what was coming. So, have you heard of this game called Grower Garden? In fact, if you've been at RDC and attending a lot of the talks, you're probably sick of hearing about Grower Garden. And like a lot of projects when you start taking over the whole platform and in fact the entire internet uh we tend to take an interest in you.

And so we reached out to the Grower Garden teams that had only just crossed 1 million players themselves. Uh and they ended up being a really great case study because it was a brand new game. It was less than 4 weeks old and they were having problems with client crash rates, problems with client frames per second and high server CPU usage. In terms of the high CPU server usage, that one ended up being a problem they solved on their own.

They decided that it was good to go from having 30 updates per second for the server updating how much the plants were growing. And they dialed that rate down themselves by just skipping over every few frames. And that ended up being a great one. But in other ways, Grower Garden has a bunch of other unique challenges because when Grower Garden first started out and we first met them, maybe one player in five had what we call a mega garden. a garden absolutely the end of its life cycle chalk full of plants and each garden itself is made up of dozens and dozens of parts, right?

And as the next week as the game moved along, two out of five players would have tons and tons of parts making up their game. And then now when you go to a grower garden server, you're lucky if it looks as tame as this. It's like there's five players in each server and they're absolutely maxed out with plants and parts everywhere. And this leads to a interesting problem where one of the biggest challenges that grower garden actually had was stopping clients from crashing cuz the game may look like how it looks, but it's actually actually incredibly complex in terms of how the gardens are put together.

Every garden is made up of out of thousands of thousands of parts. You only have about 600 meg to play with on low-end devices and every single part costs 1 kilobyte. So when there were this led to an incident with Grower Garden with right before their big first 5 million surge and starting to enter the Guinness World Record books are certainly the biggest game on Roblox at the time. Uh 10 hours before they went live with that update.

Uh they pushed a build and they shot up from their usual average 3 to 4% crash rate up to a whopping 45% crash rate. I can't really show you the graph of that, but it was such a cliff that we're actually kind of wondering if the data we were looking at was right or not. And it took us a few minutes to figure out that, oh, this is legit and we've got a problem. So, how we solved it, and one of the things that really enabled us to solve it properly was the fact we cranked out some old devices and we plugged them in.

And as I mentioned, the only way you can actually find out what's going on with memory is by looking on real devices. And we ran the build over and over. and they used debug tools to spawn in five of these gardens so that or 1 2 3 four or five of these gardens until we could find a cut off point and really isolate what was going on. The solution ended up being that it was just a minor authoring error on their side where they ended up accidentally adding a lot of attributes and tags to parts um for something that they were trying to do on the server not realizing that you multiply that by 10,000 or 100,000 parts that that amount of memory was going to stack up.

And then that brings us to our client frame rate optimization part of what we helped grow a garden with. So, who's that Pokémon? Has anyone worked out what this thing is yet? It's a coconut. So, for reference, a Roblox sphere is supposed to have around 432 triangles. This little friendly guy with his uh an anatomically correct holes and a fully modeled interior was a very generous 12,000 triangles. And there were hundreds of them in some of the gardens.

So the actual solution for this ended up being quite straightforward that they it has since been replaced with just a ball. And there's a lesson here because sometimes finding problems and diagnosing performance problems is big and in deep and involves micro micropriilers and testing and iteration and sometimes your coconut just has too many triangles and you have to swap the art asset out. So much love to who on the grower garden team actually built this thing originally.

The coconut of doom is an internal meme now and I think it will be forever. So, just wrapping up my part of the talk today. Uh, what I just want to leave you with is five final quick tips. First of all, and this doesn't get talked about much, but when you're testing your game and iterating on your game, you need to be able to make sure your game is testable under load while you're still working in studio. It's amazing how many things you'll find to be able to be a debug when you do this.

So, we would have been stumped in Grow a Garden if they weren't actually capable of spawning in a bunch of gardens for testing. And you should probably consider adding something like that to your own projects, too. Test early and often on real hardware. As I mentioned, this is absolute only way to find out if you're having client memory issues and diagnose what's going on with them via the dev console. But there's other performance reasons and microprofiles and form factor reasons and UI and control reasons to test this.

Don't leave it to the last minute. Don't wait till you're in production. Test early, test often. It It's still absolutely a truth. Our analytics team is awesome and have been adding heaps and heaps of analytics to the dashboards over this past year and the past year and a half and kicking goals everywhere. So, one of the things that you have as a creator is access to the analytics dashboard, which gives you really good insights into what's going on in production in your game.

So the ones that you'll care about if you're having client crash rates is we show those client crash rates directly to you in the analytics. So find out where they are. Find out where they are for your project and at least have a look at them. There's a whole bunch of other insights and goodies in there as well. This also leans into point number four. Make sure you know what other tools are available. There's more tools coming and we've made a bunch of tools to all bunch of improvements to all the other tools that we have already.

So you should go through and have a look at all of these. And the last one, most importantly, if you're stuck, reach out. We have the creator hub. We have the dev forums. There's a bunch of Discord servers around the place. There's lots and lots of places you can reach out to top creators in the community and our Devril teams and look at through the documentation to find out how to use all of these tools, what everything inside of all of these tools means.

And even if you make captures in your own game, you can even post a micro profile directly on the dev forums and I'm sure someone will have a look at it and take a guess of what's going on. So, a thank you again to Rick Miller and Jandel for uh letting us share some of their stories today. Hopefully, it wasn't too painful. I think I appreciate it a lot. And I want to take I want to share that again when we find these problems and and we find the gaps in things that we could have done better on our side to prevent creators getting into these problems to begin with, we take them back to our team.

Um, and that's really improvements to the product. And so if we're talking about improvements to the product, the really the best person to come and talk to you that talk to you about that is my friend Lynn. And she's coming on stage now. Warm welcome for Lynn. [Music] Right. Thank you, Peter. My name is Lynn. I'm a product manager on studio team at Roblox. We worked really hard to turn the top performance tales in into tutorials, fixes, and tools.

The goal is simple. The knowledge from the experts like Peter shouldn't just be the secret source of a small group. It should be the tools and all the tutorials to the entire community. That's our mission and that's a power of a platform. So for tutorials, please just visit our documentation site and the YouTube channel. We just recently dropped a very informative video of how to identify memory leaks by the famous Andrew.

You will love it. Today I'm going to highlight more fixes and tools. So for fixes by working with grower garden, we realized oh wow we can there is lot of room for us to improve the memory optimization. So now at Roblox compared to five four months ago um the tags attributes and parts all take less memory. By working with statreil we realized that oh wow actually we need a new category for physics parts. So we worked really hard in our with our engine team to introduce a new memory subcategory for physics parts and it will be visible in studio explorer as well and with the feedback from all of you we have made a lot of improvements for our animation systems well stay tuned next I'm going to highlight some tools and uh features with both the new tools and improvements first microprofiler as Peter mentioned the HTML format of microprofilers.

The microprofiler dumps is a really powerful tool. Now we make it five times smaller compared to one years ago and we added the network traffic information to microprofiler. Moreover, currently now you can compare two microprofiler dumps using the fig uh what's it called? It's a flame graph diff view um in just side by side in the in your browser and which will highlight the memory and CPU usage to you. Addition to that we also introduced mode which is a new mode in the microprofiler tool itself in studio or in playtime and which can highlight the memory usage that you need to pay more attention to.

Next performance an analytics analytics is your friend. Uh by working with a grow garden we realized oh wow we need to we need to introduce better analytics to be selected and filtered by versions. So we introduced the we're going to introduce by end of year filter by version. As you can see here, the blue line here is a new version you just introd just published the latest and also the green line marks all the current versions that's alive in the migration phase which is still running.

So with that after you publish a version you can immediately identify the issues caused by the new version. We just verified with Jendo this is like a tool he definitely absolutely wanted and we think it will benefit the entire community. Also as Peter mentioned out of memory issue is a primary issue for like all the client crashes. So we introduced we're going to introduce two new charts to creator analytics. The the bottom ones will be the client crash rate.

Next, this will be a new feature in studio. We're going to introduce new overlays. So, this I'm extremely excited about this feature because it has been requested by the community and even our internal developers for multiple years since I've been here. So on the top left side is original view in studio. And then with this new overlays either both in the edit mode and in play mode you can have the highlighted heat map of the triangle density potentially draw calls or overdraws or even more.

With that you can easily identify where you need to pay more attention to. I believe if we had this five months ago, the grower garden team will immediately identify the coconuts of doom in no time. All right, last but not least, it's a small feature. We improved the fonts of the stats. It's just a small updating on our side, but I want to call it out because it will be a big read readability improvement for the creators who need to stare at this every day.

This reminds me of a model my three years old son always say, no mission is too big, no pop is too small. Um, with that, I'm going to call back actually the five tips Peter just talked about. These what I just introduced are just the small improvements or new tools we already improved or going to introduce by end of year this year. But we have a very exciting road map ahead. Actually that's very well aligned with the five tips.

We're going to have faster testing tools in studio and scene analysis tools. We're going to have uh ondevice testing either by device farm or the connect to the real devices at your hand. and we're going to introduc in remote debugging for you to identify the scripts, the bugs in your scripts. With that said, I'm going to introduce Peter back to the stage together with me for a quick Q&A. >> So, you know how this works. >> We have microphones on both side.

If you have questions, please go to the microphones and use them. Really no questions. Oh, we have one. >> Hey, I'm busy city guy joined in 2012. My question is, what is the order of operations for what to focus on first in optimization? >> Early and often would probably be probably be the first two rules that I'd go in for it. Uh, in order of optimization, you probably want to make sure that your game's actually functioning. like so if things are functioning well and then get on device and test like with real hardware first and just smoke test it to see if anything's going wrong with it.

Normally I'll be looking at frame rate. If frame rate's not good, go straight to the stat screen and if um if there's nothing untowards on the stat screen for rendering and scene complexity, you go to the micro profiler and pull that up and look for what's going on there. >> So generally just look for what is not performing well and focus on that first. Well, yeah, but it really depends per project, right? Like some projects are, you know, that you're really pushing things in terms of visual fidelity, so you're likely to be having problems there.

Other games are very physics heavy, and if they're physics heavy, you're probably going to look there first in the micropriiler. >> Maybe we take a question from that side. >> We'll go left and right. >> Yeah. So in the grow garden talk you mentioned that the attributes and tags were what was causing like the memory performance issue. Um is it like does it stack multiplicatively or like how does that work? Like how does that like kind of stack up?

Should we like afraid from having too many um collection service groups or how does that work? >> No, it wasn't a matter of collection service groups. It was a matter of there being, if I remember the details correctly, there was a loop on the server that would run through every single part and add a lot of useful for what the scripts were doing data that could have been stored inside of Luell tables. It's like but when you apply it to a part on the server in its tags and attributes that gets replicated.

So the client then has to duplicate stuff that should have been server side only. We've actually, as Lynn mentioned, we've made two improvements this year to already to parts and attributes where they take up less memory now directly as a result of what happened with grower garden. And we also have a some improvements to the memory categories coming so that instead of just seeing physics parts as a big block of like 400 meg of immutable memory that you're seeing on the client, when you get into the memory dumps, you can see that it's tags and attributes and how much they are actually taking up. >> Awesome.

Thank you. >> All right. >> Seeing now the how you mentioned the overdraws and like triangle density of tools that we will get. Do you think we'll ever get something more like a bit more deep like something like ends but like being built into Roblox? >> What do you think then? Um so the question is beyond like seeing dens density we want a more deep overlays of like >> site or like render doc where it will show you how the whole frame is generating >> I think so why not right >> it's possible uh generally our philosophy for this sort of thing is as long as we can justify connecting it back to the content you've created so if there's something out of this overlay that's going to tell you something in your scene and your uh explorer that you're going to go change, then yeah, that would justify that existence.

If it's just for I want to know more about what the engine's actually doing, but there's nothing I can connect to, it's maybe not so useful. >> Thank you. >> Hello. Um, I'm a developer from Souls RNG and um, hey, thanks. Um, what I wanted to ask is about uh, the part triangle issue from Grow Garden. Okay. What we also use a lot of parts when we build maps and what we do to like we also use a huge ton of parts to build them because we kind of we are kind of themed like a part build >> and theme.

So what we do is that um when we're done building what we what the modeler does is that they just remove the remove all of the parts and they just make it into a single or just two or three um meshes and they just make the whole map into a one singular mesh so that we could remove all the parts and >> that is much more memory efficient when you do that. Yes, it absolutely better on multiple axises to do that as long as your collision geometry is okay. >> Yeah, I think it's okay because they really >> Yeah, because each individual part has a background cost of 1 kilobyte.

So, you're tossing all of those out and replacing with the mesh. It's it's much cheaper at the moment to do it that way. We have upcoming technologies like Slim which should LOD all of this stuff and make it a bit more easy to manage so you don't have to manually go and bake things. But the general principle of meshes being a lot more efficient than tons and tons of parts, especially in terms of memory, still holds true. >> Thank you.

That's what I wanted to hear. >> Great. >> All right. >> Uh I was wondering if Roblox could release uh statistics on like the breakdown of what type of devices are used across platform because if you don't have a game that's currently out, that data is not available to you. Like for example, what percentage of uh users are on devices of 1 GBTE memory? >> That's a great question. So it's potentially will be a report to report like the overall like usage on different devices.

I will go back and ask if we can do that. Yeah, but I will take that as a creator request. >> Thank you. All right. Hi, my name is Master of the Elements. I've been a robot since 2010 and my question is we already have a lot of tools for debugging memory and rendering but one I feel is not covered a lot is network because sure meshes are really good and a lot more memory intensive or the opposite memory efficient but that adds a networking cost that is really difficult to debug.

Mhm. >> So it like I have a hard time being able to debug whether a missing mesh might be pack is actually like disappearing because I'm overloading a client with a lot of like downloads that they have to do and there's nothing for me to really debug this consistently because I have to experience it and see it myself in order to >> kind of track this. So I don't have statistics that show kind of these like where do these spikes appear?

How does this affect things? What is like the biggest networking spikes that I experience etc. and how this affects players? >> So that is essentially something like is that something that Rolex could look into you think? And is that something that has been considered? >> That's a great question. I can start by the as I introduced earlier the microprofiler dumps. You can download the HTML microprofiler dumps which included the network information already.

So what we recommend you to do is to download a bunch of dumps and then analyze that. Um maybe there are some other good ways to do so. Peter, you have any ideas here? >> Well, yeah, if there's still gaps after having having a look at the new micro profiler stuff, uh come talk to us, reach out. We'll see if we can track down what's going on. >> Yeah. Thank you. >> Hi, I'm Jake. I have a question about LOD and scene management.

Is there plans to make something obviously streaming for macro scene management? Is there something from micro scene management similar to uh chicken rocket um tools that you can do stuff like flowers fade in and out and aren't currently being rendered if they're far away? >> Um I'm not the best person to speak to about our exact road map for LOD and slim, but I'm pretty sure that what I've heard is that the new slim technology does the correct thing for smaller elements that don't visually impact the scene very much.

Thank you. Great. >> Take the last question here. >> Hi. >> Hello. Uh I've been on the platform since 2012 and I'm an event organizer. So, uh part of the creations that I do is that uh I host virtual conventions targeting several hundred players in a single server. And so there are a ton of performance issues that come from having that many players in one server. So, I was wondering if you guys had any advice for potentially tackling some of the more unique performance challenges that would come out of hosting an event with that many players, >> right?

Uh there's several things you can do there. Uh the I'm assuming that streaming enabled doesn't get you anything here because all of the players are in a tight area. >> Correct. >> You might want to look into custom replication for the time being to just really lower the the network rate of all of the updates that are coming down for all of the players. And then you'll probably want to think of some strategies on the client for how you can reduce the uh player complexity.

But really the main challenge you're going to have is just fitting all of that replication into 40 50 kilobytes a second. Uh we can talk about this offline or you can grab us outside and we can dig into the weeds on that a little bit. But with that we are absolutely out of time and out of questions and out of RDC. So thank you very much everyone. >> Thank you. [Applause] [Music]