Using Roblox Tools to Improve Performance | RDC 2024 transcript

[Music] please welcome Mr Chicken rocket to the [Music] [Applause] stage good morning everyone I hope everyone didn't party too hard last night and uh ready to hear some dense facts on optimization hi I'm Peter mcneel otherwise known as Mr Chicken rocket I've been in the Roblox Community now for about 7 years and outside of Roblox I've been making games for about 25 years and as for the name Mr Chicken rocket well my kids gave me that name because they wanted me to go play this thing called be swarm simulator with them about 7 years ago and I kept the account and so the name stuck ever since and hi just before we begin just a few ground rules please keep your arms and legs inside of the ride at all times there's no need to take pictures of the slides because this will all be uh put online afterwards and there's a really good summary slide right at the end too and we'll hold off until questions at the end like with most of these talks and there's going to be a QA room across the hall afterwards to do a bit of an extended QA if you want to catch me there and as for me I did some stuff back in the olden days I worked on Fruit Ninja and I've worked on many versions of Fruit Ninja since then um I worked on Fruit Ninja VR Fruit Ninja connect Fruit Ninja Xbox all of them basically I also worked on this game called Jetpack Joy Ride which I I I get keep getting told that was a bunch of bunch of you's childhood game and that makes me feel really old yeah back in the day that game had to ship in under 20 megabytes so that that was a real engineering feat to get that thing to even ship and more recently I was over in Sweden working on this thing which I think some of you might have heard of uh I was the engine director on Candy Crush for a few years on Roblox myself I've worked on a whole bunch of stuff and I've worked with a whole bunch of uh really excellent amazing teams I've shipped a couple of games myself and I've released some open source software uh but probably the most important thing and why I'm up here talking about this today is that I love optimizing games I love understanding how they run and how they perform and I got information that I can share with all of you about that so that leads to today's talk using the Roblox tools that exist to improve performance which really comes down to starting by Health checking your game and so why do we care why do we care about optimizing our games because roblox's long-term vision is that you don't have to optimize your games and Roblox will take care of all of this for you but the truth is that right now optimized games make more money they announced that the keynote that Roblox engine optimizations have added an extra 300 million play hours over the course of since they've added them and the truth is that more than half of your players are on mobile optimized games also get played longer so people are having a session on their mobile phone they're playing it and if the phone starts cooking in their hand and catching on fire they're not going to play it as long as they're going to play it if it wasn't doing that on top on top of that there's other things that count as optimization which is load times for your games so you've got a game it's loading up and if it's taking 15 seconds 30 seconds 45 seconds I don't know about you but it 80 HD sometimes kicks in and I'm over on Tik Tok come back to the game and what was I doing you know that's not a good result that you want for your players the good news is and the real point of this talk is that Roblox has great tooling to help you Monitor and track down and overall check on the health of your project so that you can tell how it's going to run on these devices and My Philosophy is that it's way better to stay on top of performance and optimization as you work on your project as opposed to making it a process that you try and do at the end so you let your performance fall you do an optimization pass you come back it's better to avoid all of that and just work on it as you go so that's what this talk today is all about so diving right in I've profiled and optimized a lot of games at this point I've worked with a lot of teams people send me builds all of the time and say hey why is my game running badly and I I've seen some crazy stuff but honestly there's these six points that I'm going to bring up right now most games that have performance problems have problems with one of these six points so that's excessive triangle count excessive draw call count I'm going to explain what draw calls are if you don't know if you haven't heard that term before I'll dive into that high CPU usage High memory usage long loading times and excessive bandwidth usage so I'll run you through the tooling that Roblox has for all of this today and show you how to tell where your game fits on that and some current appropriate benchmarks and standards for that if you're targeting mobile phones so to start with triangles now you're going to have to forgive me for this one because this is a very high level top level talk um I have to tell you that everything in Roblox is made out of triangles so this means your measures Your Parts your unions your terrain all of it it's all triangles that's the primary way that Roblox gets things drawn on screen the main problem with triangles however is that most people don't pay attention to it and as a developer Community we're kind of excessive with it we use a lot of triangles in our things we treat triangles as High Fidelity models and we we treat it as a mark of quality and truthfully our scenes are excessive with it so the number of triangles that you have in your scene that affects your rendering performance pretty linearly Roblox does some things to try and scale down the number of triangles that you use um based up from performance and distance and LOD but genuinely um if you have rendering and you have a 2 million triangle scene that takes about twice as long to rasterize as a 1 million triangle scene so would affects things pretty linearly so the total number of triangles you're pushing out to the scene really does affect your rendering performance so how do you check your I mean just pause for a split second the um when you're caring about the triangles in your scene what I'm actually talking about is what your camera can see right now so like triangles that are to the left of the camera or to the right of the camera or now because we have the new occlusion coling triangles that are behind things that are being occluded coling occlusion coling is amazing right like let's Round of Applause for the engineers for that one but the only triangles you care about are the triangles that are actually on scene for your current thing so when I say scene triangles that's what I'm referring to and how do you check your seene triangle counts well that's really easy it's the shortcuts that I'm showing on screen now which is uh on screen now which is control shift F2 for Studio or command shift F2 for in studio if you're on a Mac and so when you're actually looking for your triangle and you're looking for your optimization what I usually do is I go and find a hot spot in the scene jump down Pull up the rendering stats pull up the rendering stats menu like so and I look around and find the hot spot I look for when this number here this is the most important stat in all of the rendering stats even though it's quite a information dense screen is your draw scene uh numbers there which is I keep pointing at the teleprompter sorry about that so it's number of draw calls is the first first number and the second number is the number of triangles so this scene is drawing 77 draw calls for the scene triangle and it's drawing 130,000 odd triangles in this View for the triangle count there's a lot of other numbers in the rendering stats and I'll cover a few of them in a minute but this is the primary one that I use when I'm looking at people's rendering performance because it tells you the overall health of the rendering scene so how many triangles do we use like this one's doing 130,000 what's a good budget for your game and for your mobile phone with the state of Roblox as it is now and the types of device we're trying to Target this graph gives us a little bit of an insight into that so this graph tells us how many triangles a bunch of different devices over time could comfortably render and stay at 60 frames per second so all the way back on the left we have the Nintendo 64 which could render about 2,000 triangles per frame and stay on 30 frames per second cuz that it was a 30 frames per second device and all way on the right we have the PlayStation 5 which has a scene triangle budget of about 10 million cuz that thing's a beast and then as we go along we have things like the on just on the right of where I've picked the iPhone 6 and the Galaxy 6S phone that is at about a million triangles for the Playstation 3 does everyone remember the Playstation 3 or is that like your dad's gaming console right any anyway so we're at about the 500,000 point for the iPhone 6s and the g Samsung Galaxy 6S and the reason why I pick that phone as a good Benchmark to tell people hey look Target this one is because those are 2 gigabyte devices those are very low spec and 90% of your player base or more like even will have something better than that to play on so this is right down the bottom it's not the very worst possible phone but it covers 90% of your player base if you can make it run well on that Target so next up let's cover draw calls the mysterious magical draw calls what are draw calls well draw calls kill your performance draw calls are the most misunderstood part about rendering performance on Roblox because people think the total number of triangles is the only thing that really affects things it's like my scene has 4 million triangles and it's like well that's not the only thing that affects how fast it renders because draw calls are a command that gets sent from the engine to the rendering Pipeline on the device that say drop draw something so every draw call is a combination of this is massively oversimplifying so any rendering Engineers here please forgive me but this is a each command a draw called command is a combination of a mesh and a material so in this case we have a teapot which is our mesh and then the material that goes with it so a lot of games are like four to eight times over budget on this they've got 2,000 scene draw calls or 4,000 scen draw calls but honestly every draw call that you put out has a fixed performance cost in it even if it only has one triangle in it and so you can very quickly make your scene run slowly even though you're not rendering much stuff if you're blowing out your draw call Budget but Pete you're saying if we only have 500 draw calls how are we rendering scenes with 10,000 or 100,000 Parts in it and the truth of that is that there's something else going on now roblocks can let you build exactly the same thing in maybe 10 different ways you can build it out of parts or unions or meses or however however you're going to put it together but not all of those are created equal and especially not all created equal when it comes to how many draw calls those things consume so what Roblox has internally is a process called geometry instancing so sometimes some things take more than one draw call to draw or that take a single draw call or some things can get packed into the same draw call and you end up only having to use one draw aall to draw a whole bunch of things so this is our example here with the teapot um if this works works it should animate when I click right so we have this teapot moving around the positions moving the colors moving the scales moving the rotations moving and all of these things can be changed without making it cost any more draw calls than it currently is so all four of these teapots are being output by the rendering system in just one draw call that's really the power of geometry instancing and now because we're eating pretty good as developers this year we now have surface appearance color 3es these also batch though I think they need a round of applause for that one [Applause] I actually had a note there saying hold for Applause so that was great yeah so anything that batches anything that doesn't break the batching you should be using those freely so when you're doing your builds and you're like can I change the parts on these without affect change the color on these parts without affecting draw calls the answer is yeah like all of these properties can be changed and really AIDS in your ability to go and kit bash your scenes together and if you want to hear a spooky Campfire story about the olden days of game development back in the olden days on the iPhone 4 when we were shipping jetpack do you know what our scene draw call Budget was it was 40 so 500's not the hugest budget in the world but it's a lot better than what it used to be and continuing with batching batching is good but it's not magic so this is an animated slide so I'm going to press the button on this in just a second uh different me IDs with different material combinations always go into a new draw so although you might have the same mesh if you have a concrete one and then you have a glass one that's going to be two different draw calls it has to be the same mesh and the same mesh ID the same mesh ID and the same material for this magic to work um and so internally all of the different block types like wedges balls um cylinder parts and all these things internally they're just meshes too so they will start going into their own batches as well uh but where people usually get caught off guard about batching because things do batch together if they're the same thing is when you start introducing transparency funnily enough once transparency is introduced the scene has to stop and start rendering of things to basically get things sorted in the right order it has to draw from back to front and then once you have that happening you have to introduce more draw calls so I'll press play on this slide just for a second and if you watch very carefully what's going on you'll see that as you introduce more boxes and you change properties the number of draw calls is staying the same two draw calls once you introduce a decal but now that we start introducing transparency this thing that was two draw calls when it wasn't transparent is now 16 draw calls now that it is so that's just something to uh keep in mind when you're working with transparency and looking at your batching and as with everything I'm not saying don't use transparency just be aware that it can add on to your total SC draw call C when it comes to decals and texture image inst instances the these can also surprisingly add to your droal count very quickly and both of those also add to your triangle count too because anything with a decal on it or a texture instance on it has to reender that mesh so that it fits onto the geometry so that can catch people off guard as well so this is a talk about health checks how do you check on your draw calls well the truth is when you go to check on your draw calls it's exactly the same rendering stats menu that we were looking at before we had the draw calls that's that number there and and then you had you scene triangles so it's the same as before this scene here is using this scene here is using 276 draw calls and as I mentioned before there's a bunch of other numbers in this rendering stat that also contribute to your scene draw so scene is the most important one but secondary one is the UI the UI in roblo is just geometry meshes and materials the same as everything else and if you're paying attention and you have a very UI heavy scene you'll see that the draw calls for your scene UI really starts to climb up and if you're budgeting and you're caring about how many draw calls your scene is producing you pretty much just subtract the UI ones from your total budget so if your scene has 150 UI draw calls well you your actual rendering you should be targeting about 350 um probably don't have it that high but if it does start really climbing up you might want to have a look into it especially for rendering on mobile depth the depth number of of draw calls is largely ignorable uh the devices that use the depth prepass which is what this draw stat is about that's usually the PC uh it's not the mobile devices at the moment and PCs are much much more powerful than mobile phones remember we're building like a minimum minimum draw call set here so a PC can do 4,000 draw calls it doesn't care about it so having a having some extra draw calls being spent on the depth prepass isn't something you need to concern yourself about which is why it's marked as ignorable up on the slides and shadows Shadow draw calls is complicated I could give a whole talk on just how to optimize your scenes regarding Shadow and light rendering because what happens is every time a light moves every time something that's being affected by a light moves every time where Shadows are casting moves every time a player moves inside of Shadows Shadow atlases have to get rendered and when Shadow atlases need to get updated that costs Shadow draw calls which is what you're looking at with these stats and some shadow have to get rendered twice because they're part of like a big scene Cascade texture generally the the number that I say to people to look out for here is if this is a three times multiple of your scene draw calls you're in a healthy state but if it starts going higher than that so if your scene budgets 500 this is500 3,000 5,000 if it if it's really stacking up then perhaps you need to look at how the lighting in your scene is being done and remember this only applies to R rendering modes and setups that actually have shadows so this is like future Shadow map um and older compatibility Shadows things like that so here's a bit of a batching cheat sheet for you when it comes to deciding H what batches better inside of your scenes what uses less draw calls and what's better to use a lot of inside of your scene this this is sort of the tier list for it so right up the top we have using using a lot of color three changes on things using a lot of surface appearance color 3 changes on things C frame size um using meshes and surface appearances and materials is really the goast path of Roblox and is really the most optimized way to put things together but as it moves down we start getting down to the transparent things right at the bottom which need to be sorted and then they take more draw calls up so the trick with this is that I'm not saying don't use the stuff at the bottom of this list I'm saying use it sparingly keep an eye on the fact that this does affect your scene budgets look at that scene budget number all of the time and figure out if this is something that you should be doing more or less of and remember when we're talking about budgeting numbers it's worth adding in here that okay I've got a budget of 500 and 500,000 scene triangles it's better to be under that you know like it's it's supposed to be an upper ceiling so how did I find this out well the method for finding it out is really primitive total caveman stuff actually all you do is you grab a chunk of geometry out of your scene and you whisk it up into the sky and you start cloning it around with the rendering stats open because you can see this tree is made up of two different mesh parts and sorry two different mesh parts and as you clone it around you can see that this tree is still only taking two draw calls there's nine or 16 of there and it's still just two draw calls to render this tree that's because I can look at that and I can say this tree batches but if I go and look at a particle system I know that one particle system takes one dra call and looking at it on its own isolated I can see that when I add another particle system well that's two draw calls five because there's five particle systems and so on so if I had 500 particle systems visible in the camera that's 500 draw calls eaten up just by that alone remember I'm not saying don't use particle systems I'm just saying be aware that there's a cost associated with them now on to CPU usage so I've worked with games that have done all of the stuff that I just talked about they've optimized their rendering stats they've uh brought down how complex their scenes are they're really not rendering all that much and they've been very conscientious about things but they still have performance problems when they actually get on to mobile phone and they're like what's going on here why is my game not running well on mobile the render rering should be fine well the other half of it is CPU performance if your CPU performance is taking longer each frame to render than 16 milliseconds or oh longer and if it's longer to render than the actual rendering then you're going to be running at the speed of the slowest part which in this case is the CPU usage and so that leads into a term that's used all across the industry which is is this game CPU bound or is this game GPU bound and how you tell that inside of Roblox is same rendering stat that we were using before the rendering menu it has the numbers on it which is how much CPU usage in milliseconds you're using per frame and how much GPU usage you're using per frame so we can see in this one that the CPU usage is 7.3 milliseconds if we're targeting 60 frames per second that's well below that the magic number for 60 frames per second is 16.66% but CPU usage is not like GPU usage when it comes to phones phones roughly on average especially the low spec phones like my iPhone 6s is clocked in at 1 GHz right that's that's a quarter of the speed of like your average desktop PC these days so when you're looking at your PC CPU usage and it's like oh my CPU usage is 10 milliseconds per frame and it's kind kind of chunky but it's okay on the PC when you get it onto that lowp speec phone it's going to be 40 milliseconds per frame and your your game's going to be running at 20 frames per second so it's really important that you keep an eye on this and understand what you're looking at and test it on device regularly so although Roblox can't do much to scale down your CPU usage for you it does provide a set of really good tools for you to tell where your CPU usage is going this is the very most basic check to get an overall CPU usage timing check but there's better tools that you can use as well the more in-depth way to use it is the micro profiler there was some announcements at the keynote of improvements to the micro profiler they're improving it all of the time and it's really in-depth and really complex but there's really amazing documentation on that and I'm not going to spend any of your time today going into the in-depth details of how to do all of the micro profiler stuff but it's important that you're aware that it exists and you use it to get a breakdown on where your CPU performance is going so the other thing is that the micro profiler also applies to your servers when you're building games so it's not just the client CPU usage that you can dig into with this you can also connect to your server and get a profile back and look through it and look at what's going on on top of the server even in live games outside of studio so in this one what I'm trying to in this slide and what I'm trying to show you is a healthy looking bit of CPU usage and you can tell it's healthy because Roblox has recently added a convenient feature where if there's lots of time left at the end of each frame the profiler starts turning blue blue so if you're seeing lots of blue in the top graph it means I sitting around doing nothing and I'm going to turn the bar blue so that you know that this is a really optimized CPU frame so blue is good news uh but remember mobile is slower and you're probably not going to see blue there and between the in the profiler the other thing to look at is the vertical lines as the basic guide for reading the flame graph is designating single frame so you see the vertical lines breaking it up and we can see that each frame is taking a block of time chunks and so you can see the flame graph is not filling up the entire frame each frame which leads us to knowing that this Frame there's plenty of idle time on the CPU in then if you do something crazy like this where you pile up a bunch of A-frames you're going to have a graph that looks like this this is perhaps a less healthy CPU frame I apologize to the physics team in advanced you're really not supposed to to do this but this is a good way to illustrate what's going on when you're really maxing out the CPU on the profiler and you're having a look at it you can see that there's tons of threads being used which is each of the horizontal roads and rows and you can see that there's lots and lots and lots of stuff filling up each one of those rows so the micro profiler all of the documentation for what all of these tags mean is online but generally if something is using your CPU usage up it's not going to be mysterious it is going to show up in the profiler and you are going to be able to track down what it is and then the last thing I want to talk about with the micro profiler is excessive scripts are usually your fault robloxs can't scale down what your scripts are doing so if you're doing lots and lots of fun stuff inside of your scripts like doing lots of raycasts and you're not uh spreading things out over multiple frames or you're doing big powerful Loops that touch a lot of stuff in the data model you're going to see that show up in the micro profilo like this like I spawned in 10,000 coins and it's absolutely chunking out and it's saying hey there's something inside of of heartbeat here that is using up all of your script performance now on to the next bit which is memory usage so this is an important one if you ever looked at your device analytics and you see a lot of crashes on phone what crashes are most of the time in Roblox is really out of memory errors so what happens is devices have a fixed amount of memory and as your game starts using up more and more memory you start approaching a point where the operating system on that phone will first give your game a warning and say hey you're using up too much memory and the engine jumps in and tries to free up some extra memory and try and stay under the limit that it's warning you about and then if it goes past that that or you're not quick enough or if it's just in a bad mood it ejects your game anyway and that's a super bad experience for the user cuz the game just goes away this can also happen if you alt Tab out of the game and come back and you won't and you're using too much memory your game will get ejected and you will see a crash in your crash report so when it comes to using memory and checking on the health of your project most devices now as I mentioned before have at least 2 GB of memory there was a lot of talk during the keynote about a focus on renewed and renewed focus on targeting Android and 2 GB low-end Android devices and honestly that's a that's a pretty good safe bet to Target these days like I don't know if I'd personally care too much about 1 gab devices anymore 2 gbes is a really good place to Target and if you're targeting 2 GB devices the magic number when you start getting increased increased increased levels of ejection is about 66% of total memory on the device which ends up being about 1.3 gabyt so that's where we end up with a metric a a benchmark for you to say how much device memory is my client using well 1.3 GB is a really healthy spot for 2 gab devices and so the health check side of this checking on your memory on device is really really simple you just hit F9 on a console and right in the top of the bar it'll say what your client memory usage is in this case it's using about 1.2 GB uh then there's a caveat on that though right now checking that number inside of Studio doesn't give you an accurate reflection of how much memory you're using on the client so you have to actually go and load this up in a live game or even preferably on device and then go and check your memory usage there that way the advanced version of checking your memory is also on the same menu you just click a couple more buttons uh there's a developer console where you can click down you can say memory and you can click around and then get a really detailed broken down graph of how your memory is used there's good docs on how to use this thing they're improving it all of the time I'm not going to dig into it too much today but it's good for you to know that this exists you can check both your client and server memory there and announced that the keynote was a new x-ray mode for the memory profiling that I'm not sure when that's live but when it is you should have a look at it there as well when it com to memory usage why is it always u3 whenever anything happens so when I've been looking at games and the main three culprits for what's chewing up the most amount of memory in games is these three here which is textures part count and honestly audio files is the one that catches most people off guard if they haven't done any work to optimize their process so textures uh the maximum size texture we can currently upload is a th uh currently have rendering in our game is 1,24 by 1,24 even though you can upload larger or smaller size than that this is this is the large size that most people use when you have one of those set up as a surface appearance with all of the goodies attached to it like a normal map roughness map and Alpha that ends up using resident at Max in the client of 2.8 megabytes there's circumstances where the roblo engine will automatically scale that down but if you're doing uh metrics and trying to find out what a good budget is you can multiply that out and figure out well if I've got 100 fully resident surface appearance textures that's going to multiply out to about just under 300 megabytes of client memory out of that 1.3 gig budget that we were just looking at for parts this one catches people off guard all the time too A Part on the client total system impact at this point is roughly about 1 kilobyte this means that if you have 250,000 parts coming down to the client and and you can control what comes down to the client by things like streaming enabled or what's actually in the workspace but the things that actually arrive down on the client it's 250,000 Parts is 250 megab so it's like if I've got a million instances in my game well you know what your memory usage is going to look like if you multiply that out and with audio files audio files have to be resident to be played so if you have instances of audio locally on the client and they're ready to be played they are also sitting in memory so a little trick there or a tip there might be to don't put those on the client until they're actually ready to be played like if you have 12 3 minute long soundtracks and make up your game soundtrack shuffle them around a little bit because they're just sitting in memory not doing anything and as a bonus slide uh we saw this at the Keynotes but there's now an analytics dashboard that talks about memory usage on devices and especially crashes on devices in a lot more detail so I won't get into this too much it was covered much better in in the keynote but that's there now so you should definitely go to your analytics and check it out and get a more detailed breakdown of how your memory usage is going and unfortunately that's pretty much it for all the fun pretty slides in the deck we've covered rendering we've covered the graphs and now we're on to the boring bits like progress bars and how long things take to load so if you've ever wondered why some games take a long time to load and load in and some games drop in almost instantly almost all of that comes down to the number of things that need to be replicated to the client before the client is allowed to start up instances especially really bog things down so when I've done benchmarking on this what I found is that you get roughly 10,000 instances a second replicated down on client connect so that means if you have 300,000 instances coming down to the client that's going to be 30 seconds of load time you should really be careful about managing this streaming enabled is a great tool for bringing in the radius for newly connected players and you should overall be thinking about how many instances in your scene anyway on top of this if you're doing content aware loading where you have a loading bar that's looking at what's coming down in in the pipeline and then not letting the player into the game until that's cleared audio instances instances can be surprisingly naughty there as well because they have to be fully downloaded textures have to be fully downloaded before that gets cleared and I don't have a bullet point for it but other players characters also have to be fully loaded all of the stuff that's in them so if a player connects and then another player connects and you're monitoring how much stuff still has to be loaded that number will climb back up as other players come in and you have to handle their character appearances so just be aware of that uh we're on the home stretch now we're on to bandwidth usage when it comes to bandwidth usage it usually comes down to you have too much moving stuff so it's nice and easy to finish up with ser owned moving assemblies they all cost about 1 kilobyte per replicated assembly that's moving around on the server so if you have a bunch of balls rolling around and you have 50 of them rolling around that's going to use about 50 kiloby a second of bandwidth it's a little bit less than that in Roblox does a lot of things to try and smooth this out for you so you don't care after you get to about 40 kiloby a second down on the recv stat here it'll start round robin throttling some of things but that's an implementation detail that's subject to change they're improving this stuff all of the time but as things are right now the rough Benchmark is that you should be looking at planning your game around there should only be about 50 moving things moving around your player that you care about physics wise at a time and to see the stat it's really simple it's command F7 or control F7 if you're on PC and it's that stat right there you've probably all seen it before it's the recv and it shows you how many kilobytes per second that you're using there's lot I could add to this about saying if this doesn't work for you you could use custom replication or do send events to move things around and things like that but that this is not that kind of talk so maybe we can uh cover that in some other talk or some other paper somewhere and now moving on to in summary in summary what this all comes down to is be aware of the current limitations realistic limitations of where Roblox is right now and get a budget so that means covering the things that we talked about before that's 500 aim for about 500,000 inen triangles if you targeting phone aim for about 500 draw calls in scene if you're targeting phone aim for 1.3 client memory uh gigabytes of that I know I said 250,000 is my example for instances before but I honestly feel like 150,000 is a really good Benchmark because you're not blowing your load times out bringing that's 15 seconds for 150,000 instances uh keep keep it keep your replication nice and low and keep your CPU usage as low as you can because it always surprises people how much slower things are when you actually go and get them onto phone uh I know I wasn't suggesting people take photos of any slides but if you wanted to take a photo of any of them this would be the one to have the numbers to take with you and tricks because you've all taken your phone out I have a QR code all right so I did an article some of you are probably aware of it already I posted I don't know if I can call it an article it's more of a book it has chapters it's on the dev forum and it really goes into detail not just about how to go and check this stuff but it has some more prescriptive details on things you can do to improve your situation with these some building tips some optimization tips it's really quite in-depth so if you want to check that out that's on the article there and that's it we're at the end of the presentation all right do we have time for Q&A did I go over Q&A is acrossing the hall okay yeah so Q&A is not now it's going to be across in the hall uh thank you everyone [Music]