LLM codegen fails and how to stop 'em — Danilo Campos, PostHog transcript

Good morning. Who's afraid of robots? Afraid of robots? Afraid of robots? Uh I'm not afraid of robots because they have already bloodied my nose so many times. There's no more pain that they can give to me. And that is what I want to tell you about on this fine morning. Thanks for coming to hang with me. So, my name is Danilo. I work at PostHog, and I make the PostHog Wizard. And the very strange thing that the PostHog Wizard does is it skips 2 hours of misery that you will never get back in your life, and it hands it back to you as 8 minutes of pseudo entertainment.

Now, how do we get away with this? We're talking 15,000 people every single month run this wizard, and in exchange for their trouble, they get a PostHog integration that works and that they actually like. How do we do it? I'm going to tell you all about it today. And just to underscore the point that this actually works, in the last 6 hours, we got two unprompted posts on Blue Sky and Twitter where people are actually happy.

Now, this should be terrifying, right? I got a robot out there. It's writing code for people. What if it's doing a bad job? Well, we learned all the ways that it could do a bad job. I'm going to tell you the ways that those bad jobs happen. I'm going to tell you some strategies that you can use so that your autonomous coding agents do the right thing as well. All right. Let's start with the easy one. We've got model rot.

Now, training a model takes a lot of time, but it's not even the time, it's the it's the money, right? You're not screwing around as Anthropic training a model on a weekend as a lark, right? This is serious capital expense. And the trade-off with this is that the models sit there no longer representing reality, right? They are a snapshot of the world and the web as it was, you know, 6, 8, 12, 18 months ago, perhaps. Now, this is useful for many things, but if you are fast-moving software project, and there are loads of fast-moving software projects, the trade-off of this is that the model doesn't know what the hell is going on anymore.

So, you got to deal with model rot. Now, this is fairly straightforward stuff. You've probably dealt with this sort of thing before. Does anyone here have a conviction about how you deal with model rot? Any guesses? I'm seeing I'm seeing some shaking heads. What's that? >> Rags. >> Rags is good. >> Yeah. >> Although, I'll tell you what, with the context windows being what they are at this point, you can't beat just shoving a bunch of markdown files into the context and patching the holes.

And this is exactly what we do, uh, with the PostHog wizard, is that we have documentation that is fresh, hot off the presses on posthog.com, and we allow the agent to make a selection. We say, "Hey, what are you doing? What are you integrating right here? What have we detected?" And the agent can use tools to go out, pick from a menu of fresh hot markdown that it can then just slide right into its context, get the job done, do things correctly.

Now, what happened to spur all of this was that a year ago, people started asking their very primitive agents, like, "All right, Cursor, I want you to integrate PostHog for me." And it would do a terrible job, right? It's just it's making up keys, it is making up patterns, it is inventing APIs that don't exist. And it is is not our fault, like we didn't do anything, but it was our problem. So, figuring out ways that we could serve correct, up-to-date context to the agent so that it would do the correct job is part of how we get people posting happy about what the wizard did for them.

All right. Now, these models, I mean, clearly they've been scraping every kind of project out there, and I have to guess that not all of them had great architecture, because some of the decisions that these agents make when they're putting a project together are very strange. Uh and so, what do you do? How do you deal with the fact that an agent's conception of how to put something together may be technically like workable, but not exactly ideal.

Well, me and my homies on the PostHog wizard team, we maintain a fleet of what we call model airplanes. And these are projects that have PostHog implemented in them. They've got them across a bunch of frameworks, a bunch of languages. But what makes it a model airplane is that we don't have an entire proper production application going in there. What we have is something much thinner. Something that is a simulacrum of a real application, but for example, the off doesn't work.

Or rather, the off works for anything. You can just put whatever you want in the password field and you're going to be able to log in. But the off is off-shaped, which means that we can provide these model airplanes to the agent. And then the agent knows, "Oh, cool. So when off shows up, this is a great place to put the particular event tracking that one would want to use when they wanted to track logins and identity in PostHog." And so, through the maintenance of a thing that isn't quite as elaborate as the real production application, which means also, of course, it is more token efficient, what you get is the correct shape of an integration as a pattern that the model and agent are able to complete consistently every time.

All right. So, in addition to weird architecture, the agent can find a weird path through the problem space. And with 15,000 integrations per month, it might find 15,000 ways to get a PostHog integration done. And while this would satisfy the requirements of we've automated integration, it would leave us with a very strange support burden because we would have too many different ways that PostHog was set up. It's like, "What what what the hell is this?

How do I make sense of this?" Right? This would be a problem at scale. This would be some sorcerer's apprentice stuff. So, to limit improvisation, what we do is breadcrumb the agent. We don't tell the agent up front exactly what we're going to do. You know, maybe you've seen this before even when you're using Claude code is that if you tell them exactly where you want to go, it might make a Claude code-shaped hole through the first four tasks and then just get really rock polishy with the fifth, right?

And this is not what we want for our case. And so one of the things that we do is we start off barely even telling the agent that this is what we're doing. We don't even mention really that we're doing a PostHog integration. We start with something like where are the files with interesting business value in this project? Can you find something that looks like a login or a Stripe interface or something that might indicate someone's about to churn?

All right, we go looking for the files that would be responsive to impact in somebody's business. And the funny thing is that business stuff casts a huge shadow in code. And so we can very reliably detect this kind of stuff. Now, from there, we say, "Okay, here are some cool files. What are the interesting events going on in those files? Don't write any code right now. Just let Let's think about some cool events that we might want to sprinkle through here.

What might those be?" So, we make a list of these. And we get the event names. We get the descriptions for those events. And we just tuck them into a little file. And this is the start of things, right? Like we don't even know where we're going necessarily. And so the next breadcrumb is like, "Okay, let's start to actually implement PostHog. We now know a bunch of events and we've really thought carefully about what those events might be.

And now we have documentation and everything, which we can load at whim according to the framework and language that we care about here. And so we can reliably go in there and start to make modifications to people's files and the modifications are once again not stupid and and they're not mad about it. Okay. Now, we can do all of the thoughtful stuff that we can to make the agent successful. But, the biggest threat to our agent outcomes is ourselves.

We're we're we're feeble little beings. We got a little bit of meat right here locked inside of our heads, and we have a context limit, too. We can't really quantify it, and it varies by how long ago we had some coffee, and if we had breakfast that morning. Our context is not just limited, but fragmentary. There's stuff that we remember implementing last week, and there's stuff that we forgot from last month. And so, we're making changes, and we're editing code, and we're evolving the stuff that our agent is working around, and sometimes we are dropping things that really matter.

And so, there is a point where we had an MCP tool instruction that was contradictory to a different tool. And the agent is like, "Man, I don't know what to do here. You're you're you're putting me into an impossible spot." Um we had um a situation where we were telling it, "Hey, there is a tool that you definitely need to use to conclude this setup." And the agent is going, "All right, cool. Let's Let's use the tool.

Wait, the the MCP does not have a tool by this name." And we're talking like hundreds of runs going with this missing tool. And what's going on there? So, if we didn't ask, we wouldn't know. And so, one of the things that you can do that is really handy and fairly cheap is a little bit of inference time interrogation of what just happened with your agent. So, at the end of every run, uh right at the stop hook, we ask a very simple question.

We're doing a little bit of user research, but the user is in this case a robot. And we ask the the robot user, "Hey, what could we have done better to set you up for success in this run. And then it tells us, and that's how we found out. Like, "Oh, you we We didn't give you permission to access the tool." And so, there was no tool. "Hey, you've got these contradictory directives." Um without this ongoing interrogation uh Oh, a good one is that we kept giving it instructions for JavaScript, and it was a Python project that it was working in.

Um of course, very um well, not frustrating, but, you know, we would identify it that way. Um so, the human error, big deal. You have to ask to find out. Now, there's also shenanigans that you got to be concerned about here because running an agent on someone else's machine demands a huge amount of trust, right? We we we've got this robot that could do anything potentially, and we don't want to do something uh bad or destructive to the user's project.

We don't want to put them in a worse spot. Um and one of the early versions of our wizards would actually just read .env files, which is necessary to do writes, right? You can't just write blind to a file. It's just one of the mechanics of how, you know, these agents work. But, it's also not ideal to be sending people's ENV contents up to a cloud, and it's like, "All right, cool. That's sitting in someone's damn log that you don't know about." Um so, this is obviously bad news.

But, when you're designing these things, you have fine-grained control over tool usage, right? You can decide, "All right, these tools are okay. These kinds of reads are okay. These kinds of reads are not okay." So, we really locked down what the agent was allowed to do around anything that was an ENV file. And then we were able to build it a tool that could do two things. It could check the presence of a key. Does this key exist?

And it could write to a key a new value. And that was it. There was nothing that was going up in terms of inference for this ENV file. And so as a result, we were no longer touching this stuff. Uh but again, man, you you were setting loose these robots on anybody's computer. You got to keep an eye on these shenanigans. Because even if you're kind of solving the problem you promised you'd solve, you might be doing it in a way that makes you look like an All right.

Now, this is the big one. This is the weird one. Because our whole careers, we have been rewarded by writing the code. We write the code. We write more more code. We write the clever code. Oh, I got a structure in here. This thing works. This thing is reliable. This thing is elaborate, but the performance is really good. And and man, we just got to code the out of this thing. If we code our way out of this problem, everything's going to be great, right?

That is not the world that we live in anymore. And a very funny thing about code is that if today you have written some code that you think is good, and tomorrow a new model drops, the code that you wrote has the exact same value. If anything, it might be declining a bit, right? Code has always been a depreciating asset. You write it, you might ship it into the world a little bit rotten cuz it's got some tech debt, and you got to deal with that at some point.

But meanwhile, you shipped on time, you got what you had to do. The wizard that makes everybody so happy is 90% markdown files, 8% tools for delivering and processing markdown files, And then the rest is like Agent Harness stuff, right? Plain text prose is where so much of our value now lives. When you write great prose today, and tomorrow, and even better model drops, it's going to be able to take that prose and do even more with it.

And so, an agent is an octopus, right? It can wiggle, it can squeeze into tight corners, it can maneuver itself around problems. You do not want to over-constrain the agent in its ability to get problems done, aside from the shenanigans stuff as we talked about, right? So, instead of thinking about like, man, how can I scaffold the hell out of the behavior of this agent? It's about saying, how do I step back? How do I give it enough information?

And how do I sequence the information that I give it so that it does the thing that I want it to do, and it makes people happy in the process. So, this is what I know from the robot bloodying my nose. I see on my clock here, I got a couple minutes left. Does anyone have questions about the strange adventure of building this robot that makes people happy? Shoot. >> Uh you had in the model airplane section, >> Yeah. >> you sort of talked through giving a sort of like exact R for the model which was really helpful for us to have a match to figure out like what.

Is that distributed in part of the prompt? How how how Basically, how's that structured in the mind of >> Oh, sure. So, the way that we drive context for the wizard is we use skill files that are generated from our context service. And so, that context service is going to take all of those model airplanes, flatten them into a single markdown file, and then include them as a reference in the skill file. And so we always have access to the full model airplane which the model can grep and otherwise churn through.

Oh, sure. So, yeah, this is just part of the supplemental content that is included in the skill. And so what we found was there was a range of useful input that we can include as part of the skill file. So, we've got documentation which is plain text prose, but then we also include the model airplane so that it can see the shape of a successful integration and it references all of that as part of getting the job done.

Shoot. Oh, sure. So, this uses the Claude agent SDK which we then wrap inside of a CLI and so you just run a single command and then we give you free inference by logging into PostHog. We've got this LLM gateway where we can cover all of the tokens on your behalf which was a whole zoo because sometimes Claude code would store off information in a place that we weren't expecting and then it would just break for people.

It's it's early days for doing this as kind of a service. Anything else I can tell you? Well, then I'm going to scoot out of the next speaker's way. Thank you for hanging out. It's great to see you. Have yourself a great rest of your day. >> Hey.