Fable Broke My App and Couldn't Fix It transcript

Not going to lie, I'm pretty excited to film this video because we're actually talking about one of my old favorite topics again, performance in the web. Yes, really. No clickbait. We're actually going to dive into some performance stuff. But, the reason we're doing it is probably why you guys are here, because the fixes that I'm working on here are things that I could not get any of the models to do. But, more importantly, the things that were broken here were broken because of those very same models.

As great as AI developer tools have gotten, and as powerful as models like GPT-56 and Fable are, there are still things they don't understand. Some of that is because these things are just really hard to get right, but some of it's because the tooling just isn't there for the agents to figure stuff out properly. And while the agents couldn't solve this problem themselves, I actually found really useful ways to wield agentic coding while I solve problems that the agents just could not understand.

What are those problems though? Well, here, I'll show you. You've probably seen the app before. It's right here, T3 Code. So, what am I talking about with that? It looks like it's running fine. Everything when you click it loads immediately. It's pretty good. Well, here's the problem. My GPU process is currently using as much as 13 to 15% of my CPU, and I'm running at 720p right now. When I blew this up to my full resolution on my laptop, or even pulled it up on my 5K Studio Display XDR, this would take up to 50% CPU utilization on the graphics process.

That's not good. I don't know about you guys, but this doesn't look like a complex 3D game to me. This looks like a very simple web app. It should not be performing that poorly. Did I screw up the React code? Am I hitting the network too aggressively? What could I possibly be doing that is causing my laptop to heat up, and is causing my GPU process to use such an absurd amount of resources? Well, these are all very good questions, and if you were to ask your agent about it, the answers you get would be, frankly, garbage.

And the actual answer ended up being way more interesting than I expected. I put a lot of time into these fixes, roughly a day and a half, and I talked to a bunch of my favorite performance people in the space, and most of them were also surprised at how absurd this problem was and how common it is. Spoiler, it was a single built-in Tailwind class. Kind of. You're going to want to watch this one if you're a nerd about the web, and at the very least you're going to want to watch this if you want to see how to utilize agents in solving problems that they are not smart enough to solve.

Before we can do that, we're going to have to take a quick break for today's sponsor, My Team is shipping so fast right now, it's kind of insane. Today alone, I merged 11 PRs into T3 code and about to merge many more. But in order to do all of this, we have to run a lot of CI, and if I'm being honest, CI is just not very good on GitHub. GitHub Actions feels like a forgotten product of a past era. And that's why I'm so thankful for today's sponsor, Blacksmith.

If I'm being totally honest, GitHub should just buy these guys because they made using their platform so much better. For running your actions, Blacksmith is two times faster. For downloading from cache, Blacksmith is four times faster. And for building Docker, it's up to 40 times faster. If you think these numbers are I promise you they aren't. I've seen it in my own projects. I've seen build times go from 20 plus minutes to sub 10 trivially.

Making the switch is easy. You change one line of code in your action and you're good to go. On top of that, there's so many little things they get right. If you or your agents want SSH access to figure out what went wrong, they have that. If you want to cache your get checkout so you don't have to fetch the whole thing every time with a sticky disk, they have that. If you want to dig through logs to see which builds are succeeding and failing and why, they absolutely have that with one of the best observability platforms I've ever seen for CI.

If your CI runs are longer than this ad, fix it today at swita.link/blacksmith. I want to go full story time mode so you guys can understand how I ended up in this position because it was quite an experience. I was doing a lot more work in T3 code over the last few days both to like maximize my feeble usage before we lose it, and also because I just had a lot of stuff I wanted to get done and finally was done with conferences for a brief like 5-day window, which ends tomorrow.

I'm back at conferences again. This video is probably coming out when I'm at Open Source. Yeah. So, I finally had time to just sit down and work for a very brief moment, and I wanted to get as much done as possible in that time. And I did. This is a screenshot of me using T3 code a couple days ago where I was pulling up and landing dozens of PRs across different projects. I still have like 10 or so I'm trying to get merged into the T3 code project, as well as a bunch of work on Lake Bed, which is so close to launch.

I'm really excited about it. But as you can see here, just the sheer volume of threads like these ones, there's like eight threads that are all in the last 10 minutes. I have a couple here that are in the last 12 hours or so. I got a shitload of work done. But the more I was doing, the more I started noticing some issues with the performance. Thankfully, for most of this I was just at home at my desk with my monitor plugged in, so it didn't seem too too bad.

But then I started doing something silly. I started using Moonlight to play games on my desktop remotely while doing Agent Code on my laptop. And while I did have pretty absurd settings, I was streaming at 120 FPS over my 10-gig network, and the games looked and ran great. It was insane that it wasn't native because the latency was really low. Huge shoutout to Moonlight. It's an unbelievable client. It's so much better than Steam Link.

It's way more annoying to set up, but once it's set up, it runs comically better. But I noticed that when I was running the game, that my keyboard started getting pretty damn hot. So, I put a bunch of time trying to optimize Moonlight and Sunshine to get a better experience there and not have my computer overheating as much, but I couldn't get it quite where I wanted. So, I closed it and went to bed. The next day I woke up and came to my computer, and it was still quite hot.

Not quite as bad, but pretty damn close. So, I started using Fable and 5/6 on my machine to trace things to figure out what's going on in a way that's causing my laptop to overheat this much. And the more I dug in, the more I realized it was my browser processes because as I've mentioned before, I usually run T3 code through the browser because I'm using it to control other machines. Generally speaking, the biggest issue with T3 code performance has never been T3 code itself and certainly not the JavaScript back end that we use for the web socket server.

It's almost always the actual harnesses that it's running because things like cloud code and codex aren't the most efficient on memory or on RAM. But all of these were running on other machines on my network. It shouldn't affect my computer anywhere near as much as it was. So, I was confused and concerned. So, I started to dig in a bit more and I couldn't figure out what was going on and I also couldn't figure out confidently enough whether or not it was my tab in Helium or other tabs because and this is probably the most important hint to get from here, the built-in browser performance tools were not helpful at all.

These tools are more focused on things like JavaScript running when it shouldn't or really slow load times or resources that aren't coming in properly. This slight spoiler for what's coming, once things are offloaded to a layer like CSS, these tools become much, much less useful. And when I went into the browser tools for T3 code, I just didn't see anything that suggested that this performance should be as bad as it was.

And it also was still snappy like navigating and doing things was great, but it was hammering this GPU process. But also again, this task manager is not tab specific. This is for the whole browser. So, if any tab had problems, it would come up here and there was no way to figure out which tab it was other than debugging one tab at a time, which is why I'm currently in a browser you don't see me using much. I'm in Chrome because I use Helium for my day-to-day work and I use Zen as my like casual browsing.

So, I need a browser that only had one tab in it, T3 code, so I could be certain that the problem was T3 code. And sure as hell, you can see here it has to be, cuz if I go to another place like google.com, the GPU process drops a bunch down to like 3% instead of the 18 plus I was hitting and 40 plus I was hitting when I was on a bigger monitor. So, all of this made it clear something was up and it was up with T3 code directly.

So, I started how I start a lot of tasks nowadays. I give a very vague description of my problem to Codex to see what it would do. And it wrote this up. T3 codes web UI can plausibly drive sustained CPU load and laptop heat. Live Chrome profiling found a web socket driven main thread stall while source analysis found several multiplicative paths that could turn active agent streams into continuous decoding history scan state re-writes persistence, time derivation, and highlighting.

And this all sounded pretty viable. So, I did what I usually do. I told the agent to build the thing and test the thing and see if it could fix the problems. It very confidently came back and told me it did. This was a huge PR where it rewrote a ton of stuff. So, I started using it and it didn't meaningfully improve the experience on my machine. This is when I realized I was in for a rough ride because it had just done like a 10,000 plus line of code change in order to optimize performance mostly focused around the network layer and how it triggered updates in the React layer for the app itself.

And none of it mattered. None of it mattered at all. And that's when I started to go deep. That's when I gave up and opened up Chrome and isolated things to here. That's when I started using the task manager to figure out what might be the problem. Like what layer is this issue happening in? And that's when I noticed in the browser section that GPU process was using way more resource than it should be. Even here when I go to the tab, the tab's only showing like 1 to 4% utilization, but the browser's pulling GPU as high as 20% even on the slow res.

Thankfully, I have been doing this since way before AI was a thing, and I had a rough idea of a handful of things that could be causing problems. In particular, anything that triggers at a high frame rate because I was on a 120 FPS 5K display. So, I asked for my agent to go through the code base and find every single thing that triggers rendering or updates on the page that could possibly have any GPU demand whatsoever.

This is the thread where I did most of this work for reference. I'm noticing some brutal performance issues in this app, both Electron and browser. If you fix in Chrome, they should be fixed globally. I have a separate PR for CPU-related issues, but they aren't the ones I care about right now. The GPU is getting pegged, and I have no idea why. I traced a bit in Chrome on the production app T3 code site, and I got an AI summary, you should start with this.

This ended up being absolute garbage. I was just curious and still loosely debugging while trying to play Palworld. So, I threw the Gemini summary from Chrome in here, and it was pretty garbage. It again was saying that scripting, style, layout, and framework were all like happening but very, very small amounts of time. So, none of this seemed like it was the problem. And initially, it called out the Ultra Think composer treatment.

This is a pretty weird thing for it to call out because the Ultra Think UI only appears when you type Ultra Think and you have Claude code set because it's triggering Ultra Think in Claude. And it does this fancy little gradient around the input box, but that only happens when you're using Claude code and Ultra Think. And I had this problem when I was exclusively using Codex and had forgotten we had this Ultra Think feature in.

And as cool as the little text box is, I would remove it immediately if it was the source of problems, and it's very obviously not. So, the fact that it's calling out the sustained GPU bug being the Ultra Think composer treatment made me realize in this exact moment just how I was because it was so far from the reality I was experiencing that there was no world in which I was going to get sold to figure this out for me.

In failed it just as bad with roughly the same prompt by the way. It also fixated on this ultra think wrapper. So at this point I realized the model was not going to solve this problem for me or even find the problem for me. So I had to flip my mindset a bit. Instead, I try to think about how can I use the model to debug as many different theories as efficiently as possible. So I asked it to make a way for me to test changes that are easy to experiment with in prod ideally with just console commands so I can see an immediate difference.

So it wrote up this fun window binding function for me that would create underscore underscore t3gpu and in this let me define custom CSS that would be applied to the page to turn on and off various features. This included things that I was suspicious of like animations, filters, shadows, the composer, blur, any media layers, and this noise layer. This is one I was particularly particularly suspicious of. So now that I have this all defined, I can yoink it.

I can go back to this broken slow build, go into the console, paste it, and now I have the ability to experiment. So as I was saying before, performance here is bad. It could occasionally get as low as like a bit under 10% but is usually way over. Watch what happens when I do underscore underscore t3gpu.applyall. It has broken the UI a decent bit like the text is appearing behind the text box, but if we look at the performance here, we are now down to 3% or less GPU process.

So it's clear something in that set fixed the problem. I don't know what yet, but something did. But now we have a custom tool that we built to go back and forth with these options and figure out what it was. And just to confirm, I have the t3gpu.reset function and as soon as I run that, GPU process spikes back up. Okay, let's start applying things. I said noise was suspicious, so let's just do the noise change. Huh.

Numbers are still pretty high. It's idling a bit lower with the noise change, but not enough. Okay, how about blur? Now the blur is off, and the process is somehow going up. It just almost hit 20%. Okay, let's try the next one. Animations. I hope it's not this one, because if it is, that means we have to turn off animations. Oh. Looks like it was. All of a sudden, the GPU process is down to basically nothing again. And now I have some actual information.

I also forgot to mention at the start, it spent 30 minutes trying to get debug info out of Chrome and just failed outright, and eventually I just told it to use computer use and look at the browser tabs instead, instead of trying to go through the Chrome extension that insisted on trying to use. Helped a bit, but still this this is a rough one, frankly. This message was a queued one where I gave it the context of a basic empty like idle T3 code instance and the massive GPU utilization I was seeing.

Said that this changes the diagnosis materially. The 31.3% is GPU process CPU with 1 gig of GPU process memory, specifically while the thread is working. Likely frame pump is now visible in code. Here is where it starts fixating on the animations. I also had told it that the animations seemed to turn off and cancel this problem out. Here it gave me another script I could run to find all the animations and manually pause them and see what happens.

It is also worth noting that DevTools changes the performance characteristics of a site. Just having it open starts to insert a bunch of things into the site and put it in a debug mode that inherently makes it slower. So, it's not the most accurate readout compared to something like the task manager that I was using. So, it told me to do that, close DevTools, and then watch the task manager and see what the difference was.

And it told me what order to apply different things in to see how it affects the performance. I then describe what I had seen, which is that all had an immediate cut to under 4%, reset jumped back to 25 plus, and animations appear to be the biggest improvement. One correction, helpers animations rule disables both animations and transition. Let's separate them in order to diagnose further. Now I could turn off transitions and animations separately in order to get a more direct answer to what was up.

It had its theories and I will again tell you guys most of these are wrong. The pulse had a little bit of the impact. This stroke for the blue context window ring did not at all it did not matter in the slightest. The pulse was the biggest issue for sure in different places though, specifically in the sidebar. And here's where we identified one of the major offenders. It was the pulsing terminal icon that appears when you open a terminal in a thread.

This little icon right there was what was causing all of my hell. I could never have guessed when I first started running into the problem here that of all the things on the screen these two little green icons in the bottom were pegging my GPU on my now $8,000 computer. And then it gave its theory for why an opacity pulse is expensive. Normally it's a cheap compositor only animation, but an infinite compositor animation still requires frames at the display refresh rate, which for me was 120 FPS, and on a high DPI display it was pretty brutal.

And its suggestion was to make it static, which I did not love. And it said if most is desirable, use a finite pulse only when it first becomes active. This is like the worst guidance I've ever heard. Like if you like the pulse, do it for 2 and 1/2 seconds instead. No, you. This is why everybody thinks that Codex is so bad at design. I hated this suggestion so much. It gave suggestions for bad optimizations to not do.

So stupid. And it also was suggested a lower frequency blink to see how it would affect performance. I tried that and it was not much better. It was a little better, but I I hated how it looked. So I was just not happy with a lot of this. And it made a PR that just removed pulsing from that one icon, even though other icons like the working completed and the work in progress in threads were also affecting this in a meaningful way.

So, again, Soul just was terrible at this. Here I start talking about other agents cuz this is the point where I was accepting of the fact that Soul was not going to help and was hoping, praying even, that Fable would do a better job. So, I gave it some of the work Fable had done. And this is probably the point where I should switch over to the Fable thread because I had given up on Soul being helpful. Also, fun fact, I did most of this work on my other MacBook over T3 Code remotely, but since I was moving to Fable and it wouldn't benefit from Codex's control options, and I also wanted a bit more resource available, I moved this over to my Framework Desktop running Linux.

Cuz it wouldn't have quite as much computer use capability, but at this point I didn't need it. I just needed to test theories and write code. So, I had this going at the same time. Also, so that like these two could run at the same time and not interfere with each other and get confused by each other's solutions. This separation was really convenient and just throwing old machines on my network and being able to hop between them just by clicking different things in T3 Code and the browser.

It's such a nice UX. I I have never been doing more work across more machines in my life. Huge shoutout to Julius for making all of this as stable as he has. The best thing I ever did was buy a Mac mini for Julius because now he's making all of this workflow stuff work great. So, here is what I did with Fable. Discovered some awful perf issues due to CSS animations that are playing the GPU really hard, especially on 120 Hz monitors.

I know the terminal icon in the sidebar is guilty of causing problems. What other animations may cause perf issues? What can we do to make them less GPU intensive? Found the full animation inventory. Here is where it started and like built up the list of all of the different animations. And again, started with three that didn't matter. It started with the loading skeleton when you first open T3 Code, which is not the problem.

I don't care. And then it suggested the ultra think rainbow and the ultra think chroma shift. As I discussed before, ultra think has nothing to do with this. Thankfully, as we keep scrolling and we leave behind the expensive ones, we end up in the moderate ones where problems actually live. There's the animate pulse, which is the terminal icon that I mentioned before, as well as a few others, like the typing dots for message timeline thread status indicators, things like that.

Opacity animation on the compositor. So, each isn't expensive alone, but every infinite animation promotes its element to its own GPU layer and keeps the compositor committing at 120 FPS even when nothing else changes. With several sidebar rows pushing at once, that's a lot of tiny layers being re-composited forever. The reason terminal icon feels guilty is probably that it prevents the page from ever going idle. Oh, boy.

And then we get into animate ping, connection status.preview.local.server.card. These did not end up being a big deal. Animate spin didn't matter. It's only loading states. And then provider update pill countdown also didn't really matter. But here is where I got some actual useful insights from Fable. It fixated on the backdrop blur. It's a thing that was funny enough added by Opus way back when we started T3 code in order to try and make it look better.

For the most part, our use of backdrop blur is not too bad. As you see, I am blurring the back background behind the text box and my performance isn't bad right now. But there are other places where we're doing this type of thing. And in particular, I don't think it's mentioned here. Uh okay, though the root TSX is mentioned. We had a noise layer that was applied to the front of the page with a very low opacity in order to make it a little grainier, to make the gray not feel so monolithic.

It's very common. And looks so much better. We've used this on most of the things we have built. But when combined with a handful of other things, specifically the backdrop blur and the consistent animation, that's when the performance started to get bad. Mind you, it took me this entire thread and a dozen and a half back and forth to get this figured out. And even when it did get mostly figured out, it did a really bad job fixing it.

This might not come through in the encoded version of this video on YouTube because compression is a But there were very clearly distinctly different gray colors in multiple places. You had the gray for the sidebar and you had a now darker gray for the main body because my filters were actually brightening the color a little before and a different gray on the bottom here for this special area where like the local checkup button and things go.

It was not super visible in most SDR displays, but on my HDR MacBook and on my HDR studio display, it was really bad. So, yeah, I was going mad with this. I ended up removing the noise entirely and tuning the color to make it look good enough, but it kept screwing up the color here. This one should be even more visible. I don't know again if this will come through with the compression or not, but it was a very clearly a lighter gray on the bottom and it was very jarring and bad.

Found it and this one isn't my change at all. The bar is the composer's lower chrome strip, the full-width row that holds the branch and checkout information. In the dark mode, it's defined as this specific mix. Let's lighten it to try and make it match. It screwed it up again. Ended up fighting it more in another thread and eventually got it to the point where you cannot see it at all. But this also resulted in the darkening of the body here, which with a little tuning I got to a point that I was okay with.

Most importantly, I got the performance down, right? Ugh. Why is that still at 18%? 20% God, did our merge not go through? I'm on the site app.t3codes. So, when we finally shipped it and I finally moved back over to Helium, I noticed the GPU process cooking still. And I was so stressed thinking I just did all of this work and it didn't fix it. I don't even have very many tabs open here. I have my YouTube stream, which thankfully as bad as the YouTube client is optimized is not the problem here.

I have the chat, which I'll close as well. I have my Codex dashboard with my usage there. We've lost a tiny bit here, but not much so far. So, let's just close that. Okay, that helped a bit. That took 2% or so off it maybe, but it's still way too high. Well, I had forgotten something. I had two other Helium windows open. I had one for Claude and I had another for my other Claude cuz I have three Claude accounts. Once I closed those, the performance was exactly where I expected it.

Do you know what that means though? What that means is an empty idle Claude.ai page uses 10% of my $8,000 laptop's GPU for each open tab. I had to sneak a Claude dunk in here somewhere, but I understand why because when I asked Claude to fix it, it gave the most stupid suggestions I've ever seen in my life. So, as most of my real engineering videos tend to conclude, I think Anthropic needs to hire some real goddamn engineers because they made me think that my fixes didn't work because their code on an idle page was that bad.

To their credit, the same thing happens in Codex a lot. In fact, this thing happened in Codex so much when I was first testing the Codex desktop app that I was concerned that the performance would continue going to randomly. They would fix it and then the next update would break it again and that kept on happening to the point where I decided to build my own open-source alternative to the Codex app to make sure this would never happen again.

And despite that, and despite having 120,000 people thank each and every one of you, by the way, having 120,000 of you guys using T3 Code, I had to stay up until 5:00 a.m. two nights in a row fixing this myself. These things aren't easy. It does not matter that models have gotten so smart. They were very helpful. This diagnosis would have taken me way more than like a day and a half of partially working on it if I had to do it all by hand.

Getting agents to build tools to help me diagnose different theories was great, even though their suggestions were absolute trash. And this is why being an engineer is so useful and having experience in these things is great. My experience didn't make the agent solve the problem, and the agent didn't understand it better than me. The agent could find things in the code base faster than me. It could build custom tools to test my theories better than me, but I was still the one who brought the real information.

And that felt weird, but it also felt good in a way to know that this experience I have fighting weird CSS behaviors in the browser without tools that are actually helpful for it is still valuable. And it's valuable enough that both major labs that are trying to replace all of us engineers with their fancy GPU runners can certainly waste my GPU in ways that make no sense. Yeah. This was a fun deep dive. I am happy I did it, and I am very happy with the resulting performance improvements that we now have in T3 Code.

If your reason for not using it before is because the performance wasn't great, it's more than fixed now. I highly recommend giving it another go. The desktop app is great. I personally mostly use it through the website connecting over the CLIs on other computers, and T3 Connect is coming soon, too, which will make it so you can connect without even needing something like Tailscale. But if you do already have Tailscale set up, I can say with 100% certainty T3 Code is the best way to work with it.

If you're curious about the final version of the PR that landed and fixed all these problems, you can go check it out on GitHub right here. Both Fable and Soul pissed me off a ton during this debugging journey, but they also ended up building useful tools to help me find the answer. But, what was really cool here is that I still had to find it myself. As great as these tools are, they are just that, tools. They don't know everything.

They can find things fast, but they have to be steered. And when you learn how to steer them and wield them for different types of work, whether that is they find and solve the problem themselves or they build the tools to help you solve it, agents can always be valuable in almost any dev work, even if not as valuable as people on Twitter like to make them out to be. Just cuz your agents don't know the solution to the problem doesn't mean they can't be helpful when you solve it yourself.

This is a fun deep dive for me, and I hope it was for you guys, too. Until next time, peace nerds.