How I Shipped 2000 PRs Last Month — Trusting AI Agents｜Grok Bot｜Lauren Tan transcript

Hi, my name's Lauren. You might know me as potato on X and I work on Grok Bot at SpaceX AI. So, last month I did something pretty crazy. I shipped 2,000 pull requests to production. A lot of how I'm able to do this is through trust. Um and I think a lot about trust in terms of how I can trust my agents to produce high-quality work even when I'm not there. And my argument and thesis for this talk today is that if you set up your environment for your agents really, really well, you can end up with something that looks more like a personal or even team software factory where you're producing very high-quality code at much greater rates than before.

Uh but I'm not a fan of the term software factory. I like the analogy of a Michelin kitchen better where, you know, um as technologists, we're not really producing we're not mass producing a product on an assembly line. But the work that we do looks very creative. It's very um you know, it's the act of building a product. And it's art in some sense. So, you know, even though with agents we're not cooking the individual components that go into the product anymore, um we're still responsible for the final outcome and thinking about our kitchen setup.

Right? Because depending on how you set up your line cooks, your sous chefs, you know, the kind of equipment they have, the kind of training they have, you know, dishwashers and the ratio I guess of line cooks to dishwashers, um all of these ingredients go into making the final product. Um and I think this analogy is really apt. So, uh I wanted to talk a little bit about a story before we begin. Where 6 months ago, when I first joined Cursor, before we were a part of SpaceX AI, I obviously had, you know, no agent skills to to use, right?

I just joined the company. It was um a fresh code base and a fresh product that I was working on. So, at the time, um Cursor was building the replacement for the Cursor IDE, which is the new agents window. And uh before I joined, the Cursor agent window had quite a lot of performance issues. Um and my manager at the time asked if I was able to help them. Um and since I had spent some time on the React team before I joined Cursor, uh it seemed like a good fit.

Uh but when I first started, I quickly realized how manual this process was. Uh obviously I've done performance work before, um but at the rate at which pull requests were being landed, it was just it felt almost an insurmountable wall of pull requests that just kept coming in and I had no idea whether or not the performance of the app would be regressing, right? So, a lot of my early time on the Cursor team was spent looking at uh Chrome DevTools and the performance and doing performance traces and taking heap snapshots.

Uh and it was extremely extremely manual. It was it It was so manual it it got to the point where um I just got really frustrated and started to think about, you know, wait, we have agents, what am I doing? And so, I started thinking about verification skills. Where you know, what if my agent could actually run the application itself and take the traces for me, understand the traces, and find the hot spots, and basically hill climb, you know, to uh better uh performance in our application automatically.

And throughout the last 6 months uh of being at Cursor and SpaceX AI, you can really see that my productivity has skyrocketed. And, you know, I never set out to ship 2,000 pull requests a month. That was not a goal of mine at all. But, I realized that all of the skills, all of the tools, and and code base changes I were making laddered up to this idea of trust. You know, I didn't know it at the time, but um I had this thought in the back of my head head, which was, you know, I am the bottleneck, and I need to be able to take all of the knowledge that I have as an engineer and impart them into into my team of agents so that I didn't need to uh be the blocker for everything.

And um you can clearly see that it's it's paid off. So, I think it really comes down to trust. Um but, how exactly do you build that trust? And where do you start if you are, you know, wherever you are in your journey of using agents? So, uh when I started, obviously, I was in this category, you know, the one to one to five range, where um you still feel like you have to baby sit every chat uh and and conversation, and you're just constantly course correcting, you know, you're intervening, you're correcting your agent so that it does the right thing.

Um and if you're not there, basically nothing get nothing happens and the agents do the wrong thing. And I would actually argue that this part this phase of, you know, using agents is actually the hardest to get out of because it's not always very clear how exactly you get out of it. And um again, it just comes down to trust. Um you know, the reason you're unable to go from one one or one to five agents to something more like 100 um is because you don't have trust in your agents work yet.

So, if you don't have trust and you try to spawn 100 sub agents or cloud agents you're going to quickly find that you're just going to get a ton of slot pull requests and you know, a bunch of regressions and a bunch of bugs shipped and no one's going to be very happy with that. So, that begs the question how do you trust your agents more? So, for me it came uh it it really started when I uh like I said, I meant I joined the cursor team and I was starting to work on performance where I realized the need for verification.

So, when I say verification uh there are sort of levels to that. Uh because um you know, on the I guess the lower end of the scale for verification, you have things like um the verification skills that I talk about where you teach your agent how to run your application um you know, and use things like the Chrome DevTools protocol or whatever other protocol that you have for debugging and teach them how to um you know, debug the application, take performance traces um take heap snapshots and so on.

Um and then on the opposite end of spectrum of the spectrum, which is much much harder and still very much an open question, is like more formal verification, where, you know, maybe you rely on uh formal methods or, you know, languages like Lean or TLA+ uh to do, you know, verification so that you can check that, you know, your business level or business logic uh invariants are, you know, are always um true. Um and that um uh you can formally verify that your application is always in a correct state.

Uh but I would say that, you know, even if you don't have um the ability to run formal methods, and very few people really do, um that with verification skills, you can get very far. So, when I joined Cursor, uh and started to work on the Cursor agent window, the first skill that I built was the skill called control glass, which is a verification skill that teaches the agent how to um run the application and take traces, like I mentioned, and it does that through uh the Chrome DevTools protocol.

Um something interesting about that skill is uh and I kind of iterated my way to this, uh it didn't start out this way, um but the control uh the control or verification skill really has two components to it. Um the first part is obviously a CLI. So, you want your agent to be able to reproducibly um be able to run the application and collect traces and collect evidence that, you know, empirical evidence that, you know, code is working, uh that your performance bar is being met.

Um and rather than have your agents create scripts every time, um you know, that can differ between agent sessions, uh you can actually in you can actually create a CLI that's within the skill directory, and then your agents will just use that every time. And then, of course, you need to actually invest in it and make it good so that it can handle all sorts of different use cases, um and um be able to uh you know, run the application correctly.

Uh another thing that's also really important is uh this idea of a feature map. So, a feature map really is um something that I kind of coined, I guess, where uh you know, we started using these control skills with Incursor. Then we quickly realized that, you know, a Slack report would come in and a user would post a very vague screenshot, right? Like a very small slice of the UI, and just like three question marks. And the agents using our control skills had just no idea.

Like it could run the application, but it would just be guessing at what exactly the the user meant. And so, I had this idea to create something called a feature map, um which is I guess kind of inspired by like a site map. Um it essentially is a uh form of materialized memory, you know, like how exactly does your application work? What features does it have? How do it does a user reach it? Um you know, in terms of like keyboard shortcuts or what DOM elements to click on, uh you know, things like that, and what all the different features do.

Um and uh they are these this feature map is stored in the skill itself, um in the code base as part of the skill directory. And we have an automation that uh maintains this feature map as well. But when we combine the CLI and the feature map, we quickly realized that um this combination was very, very powerful because now agents could not only, you know, reproducibly control the application and take traces, but it could also understand uh requests that came in from internal users as well as external users.

And so, we very quickly realized that the control verification skills were so useful that they've become more or less uh critical infrastructure for our team, and we constantly uh maintain it. But the ability for an agent to verify its own work is extremely powerful, and uh we have um spent a lot of the time on the skill, and it's very, very powerful for building that trust. Um in addition to verification, of course, uh you want uh because ver- uh verification is really about correctness.

Correctness to me is really about does the thing does the feature or the code do the thing that you want it to do? Right? Like does the you know, the the checkout button, does it actually check out the cart? Um verification's really important for that because you can get empirical evidence that the feature actually works. But it doesn't tell you much about, you know, the performance or uh the code quality of that feature.

Um and that's where you start thinking about skills that teach agents to work like real software engineers. So, um I've built a plugin called Pystack. Um I'm not going to talk about the plugin too much today. But um a lot of the inspiration for that plug-in, uh which is a collection of skills that I've created, are inspired by the kind of workflows that I personally have used as my in my time doing software engineering for all sorts of different types of tasks.

So, debugging, you know, uh feature development, prototyping. There's a whole bunch of different playbooks and skills that ship in Piece That that teach your agents how to write code the way that you want them to. And you know, this is where uh you know, the more experienced engineers on your team can really contribute to set up a team repository of skills that um just make your agents a lot smarter. Um and when you combine those skills with uh verification skills, then you you're able to get to a point where your agents are able to not just verify that the work that they're doing is correct, but that it's also high quality.

And again, because of verification, you can collect real performance metrics. You can get real numbers and statistics and telemetry on uh the like performance of the application. Um so, I think that's a really important part to invest in. Um Another thing I think is really important is refactoring and rewriting your architecture to be more agent-friendly. Um and I almost want to say that uh this is one of the most important things you can do um as a as a software engineering team because um if you really truly believe that agents are going to be writing all the code in the future, then we need to design our code bases so that they do the right thing by default. >> Mhm.

And you'll find that there's this uh I I almost want to say like a a scale or continuum between uh you know, these different pieces of building trust in your agents where um on you know, I have like five of these points here where uh the code base is really like the best form of memory because agents love to extend existing patterns that they see. Um and you know, uh I think this is just the nature of how LLMs work where uh they're more likely to use whatever it is in their context window um to make changes.

And of course, the files that the agents read and opened are part of its context window. Um and therefore, the code base is a really important part of that. Um because you know, agents aren't going to just refactor your code every single in every single PR. They're going to just look at what's already there and just extend. The next level, I think, is about static analysis where you have linters, you have compiler diagnostics, you have uh continuous integration um and these are guidelines and constraints that you can enforce in your code base so that whenever you correct your agent, you find that, you know, they just keep making the same mistake.

Um you can add those as lint rules or even better, you can refactor your code base so that the mistake that the agent's making becomes categorically impossible. Um and then a step above that, right? Where uh where and this is where we're starting to get more into uh less of like a hard constraint and enforcement and more into the realm of guidance. You have things like rules, you have bug bot, you have skills, um which, you know, your agents will obviously sometimes and mostly use uh when they're doing their work, but there's also a chance that it might, for various reasons, you know, forget to read a rule or maybe the user that is piloting the agent ignores them.

Uh so, these aren't quite as enforceable, but also an important part of um setting up your environment so that you can really trust what your agents are doing. Uh and then finally, you have a style guide, which is really only enforceable by humans in a code review. I guess you could put the put these in your rules and and bug bot and skills as well. But if you don't, then you have this big glaring hole in your uh review process, where now humans have to, you know, look at every single line that's being changed and remember the comment and with the rate of pull request that are coming in, it just it just becomes impossible.

So, I definitely wouldn't recommend, you know, relying only on the style guide. I think the style guide or, you know, like looking at human reviews is a good place to start um in terms of what's missing, but you should really invest the time to think about the other four parts, uh you know, your code base, making things categorically impossible to better data structures or algorithms, um static analysis, and then of course, you layer that with rules and bug bot and skills.

Um on the code base front, um and in the Grokbot code base, we actually have invested into setting something up that we call Dune, which is our agent-friendly framework. So, um the inspiration for Dune really came about from a lot of the performance issues we were seeing we were seeing in um the cursor agent window. And so, a lot of lessons came out of that exploration, but um the key principle that we landed on is really that agents love taking shortcuts.

So, what if we designed a framework such that the shortcut, you know, the easy path is the right path for agents. And also one that would be a code base that is, you know, maybe pretty annoying for humans to work in uh because it's so locked down in terms of what you can do and what you can't do, but uh it actually creates the perfect environment for agents, especially ones that have very minimal context because, you know, not every contributor to your code base is going to be an engineer anymore.

You can have designers, you can have product product managers, you can have CEOs, you know, going into the code base and and shipping features. So, we want to really think a lot about how we invest and set up our code bases so that, you know, even agents that are piloted by busy people with not a lot of context can do a good job by default. And like I mentioned before, um your code base is really a form of memory for agents because they love to extend the existing patterns that they see.

And the reverse is actually also true, right? You can invest the time to set up your code base um in a way that things are, you know, bad patterns are categorically impossible, or you have like lint rules that prevent them, but the reverse is also true in the sense that um if you have existing anti-patterns, you'll actually find that these will spread kind of like a virus. Um where you have like one small workaround or a comment that explains a workaround, and you'll quickly find that agents just love to copy that.

And then in a matter of a few days or a few weeks, you'll find that the workaround has spread everywhere and it's now becomes it has become like a de facto pattern for all agents. And that's a really, really bad place to be to be in. And that goes back to what I was saying about why it's really important, you know, whenever you're correcting your your agents, that you invest the time into thinking about codebase changes and static analysis to and and of course the layering them with rules, good rules, and bug bot and skills uh because of of that reason.

Um and another analogy that I like in addition to the Michelin kitchen is this idea that, you know, your codebase is kind of like a garden where you have uh workarounds uh you know, that seem seemingly that seem kind of innocent at first, but then because of the nature of agents, you just copy that pattern over and over again, and you quickly end up with a very, you know, vibe coded codebase that is a you know, a pain in the butt to maintain and has a lot of performance issues.

So, in my opinion, the perfect agent codebase is one that's so locked down that, you know, it's again it's like really annoying for humans to to write code in, but it's so conventional, it's so standardized that uh you know, even innocent-looking patterns are just forbidden. Um and the best example of this I have is actually something that seems very, very innocent when you look at it, but when you think about it, it's actually really bad.

And that pattern is uh agents leaving comments in the code. Now, you know, when I first saw agents starting to do this, I initially wasn't Well, I I did think that a lot of them were slop, but I also thought that, you know, it actually doesn't It's It's not a bad thing, right? I guess if agents are leaving comments in the code because as humans, we left comments in the code whenever we saw, you know, edge cases or we needed to actually make a workaround or, you know, leave a note to ourselves or a colleague on a particularly tricky part of the code base.

But, what I quickly realized when we saw this happening in the Cursor code base was that agents were just using the comments around the code as justification for why it wasn't going to solve the actual problem. And instead paper over it with a band-aid or a short-term solution. So, in Dune, which is again the framework that powers Grok Bot, we made the choice to um we made the choice to actually ban comments before that reason where so that agents would not just copy that pattern and uh you know, propagate it everywhere in the code base.

So, uh my pitch here is that every team really needs something that, you know, like a role that I'm calling a gardener. Um in the same way that uh you know, with a real garden, you need someone who is thinking a lot about the uh you know, things that can kind of creep in and grow in ways that you don't want. Like, you know, you have weeds, you have uh you know, just other types of organic growth. I don't really know much about gardening that well.

Uh I But you have things that you know unwanted pests and and and stuff like that that kind of creep in into your code base. And so you want to nip them in the bud as soon as possible before they start propagating everywhere. A lot of the principles behind Dune are really centered around these three things. First of all, we want to delete tech debt that we already have for you know for reasons I just mentioned. We want to keep or enforce a single paved path for most blessed patterns.

You know there should be one conventional way to do some things. So that agents don't really need to guess and there should be enough guidance in the code base in CI in lint rules so that the agents are guided to do to to follow that path. And then finally whenever you see tech debt or bad patterns, your instinct should be I need to write a lint rule against it. You don't always have to clean up immediately because if you write a lint rule, you can at least stop the bleeding.

And which you know doesn't solve the problem entirely, but it at least prevents it from growing. So I definitely recommend you know really thinking a lot about how you can guard against anti-patterns so that they don't spread like a virus and then also spend time to actually you know get your agents to clean them up so that your code base is just constantly kept in a state where you would be happy if an agent would have copied it.

That's the kind of mindset that I would recommend having. And then I won't actually go through all the details of Dune itself. But I'll just kind of gloss through some interesting parts. So again as a reminder, Dune this architecture the client framework that we built to power GraphBot. We've invested a lot into, you know, all the things I was saying, where we have conventions. We have a lot of conventions about where code should live.

And um where and how uh code should be imported between them. So, in Dune applications, you know, there's different concepts where like, for example, features are all co-located in a single folder. Um you have an entry point um that's, you know, in the React part of the code that determines uh you can kind of think of it like a route. Uh you have transcript cards that show up in the GraphBot application. You have a host that runs on the you know, the GraphBot virtual machine, and then of course, you have your client uh which uh powers the uh overall Dune application.

And we have a lot of um strict boundaries between these things, where um just an as an example, things that run on the main process uh or the main thread in Electron aren't allowed to be run on the renderer thread. And we keep that separation very intentionally because uh of lessons we learned from Cursor's agent window, where we would sometimes see code accidentally uh get imported into the renderer thread, and you know, slow code.

And uh since uh on the renderer thread, you want your UI to be very smooth and uh performant, uh you need to make sure that you don't have any long tasks or, you know, things that take longer than 16 milliseconds for if you want like 60 frames per second, or 8 milliseconds if you want 120 frames per second. And so, your renderer has to be constantly in a state where uh it is uh you can really kind of chunk up the work um, and not do them all at once.

Um, and so, we have code within Dune that enforces this these boundaries through the import uh, independency graph. Uh, but, yeah, this is just an example of a pattern that we saw lead to really bad performance that we um, categorically eliminated through uh, the architecture of the of Dune. Um, and then, all of these other pieces aren't that interesting. Uh, but, again, the the core theme here you know, it's not about Dune but the idea that um, an agent-friendly framework of your own is actually very, very powerful.

And you can encode all of the learnings that you and your, you know, your best engineers on your team have tribal knowledge of. Um, and um, I think the lesson here is that how do you take that away from, you know, what used to be in the style guide process of reviewing code and, you know, in engineering is reviewing other engineers' work and leaving comments um, to extract, almost like extracting that knowledge and encoding that into the framework into the code base itself so that the code base acts as the memory.

Right? It's the the thing, like you're coming back to this idea that, you know, the code base is just the thing that it's the the materialized snapshot of the state in which you want your agents to extend. And you want that code base to be so pristine, so great that the next agent that comes along is just very likely to continue that pattern and keep it really, really good. Um, and if you spend enough time on this process uh, like I mentioned, you can really set up um, a Michelin kitchen or a software factory where um, because you spent so much time on you know, all of these pieces that allow you to trust your agent, whether it's in the code base, whether it's lint rules, um, whether it is uh, you know, diagnostics or rules or bug bar or skills, these layers come together um, and provide you a lot of trust.

Because now, you know, just imagine for a moment, you're working in the GrokBot code base. Um, it's super locked down, you know, it's like almost impossible to write bad code. So, you know, you can even an agent with very little context, uh, you know, even a an agent with not a lot of reasoning can come in and and actually write code that's good. And going back to my example about the Michelin factory, I think there's a lot here, right?

Where, you know, we're setting up our agent, our bot with skills and tools, you know, we're training them, uh, we're setting up our kitchen in a way that makes sense, right? For the agents and bots to do the right thing by default. Uh, you know, whenever we see, for example, in the in the kitchen example, if we notice that uh, one of our cooks or dishwashers is constantly tripping over something, of course we need to fix that.

Right? We need to problem solve and ensure that, you know, others don't trip as well because, you know, in a kitchen is a very dangerous place and you don't want to hurt yourself. It's the same mindset, I think, that we should have with our code bases. How do we set it up so that even agents, uh, without a lot of uh, knowledge can and do a good job. Um, and I think with uh, you know, GrokBot, GrokBot and Cursor play an interesting role together where GrokBot is really great at uh providing what I call the outer loop because you can connect GrokBot to lots of different uh you know, different connectors like Slack to DataDog, Sentry, uh PlanetScale, whatever services that you use.

And you can aggregate all of that information together and use that to make really good decisions for itself. Uh Some people call this like a company brain. Uh I don't really think I personally don't think you need anything that sophisticated here because agents are really good at using tools. And so, if you connect these tools to GrokBot um and you start having your GrokBots auto kick off things like cloud agents, you can actually find that it's really not uh you don't really have to invest in a lot of infrastructure to build a software factory.

Um in fact, I'm going to you know, cross cross out this this term cuz I don't like this term. Um uh I think you can set up this personal mission control kitchen for yourself through GrokBot, things like GrokBot routines which let you subscribe to you know, Slack threads to Sentry alerts that let you kick off things automatically. And when you combine all of these things that I've been mentioning, you know, your code base, your rules, your skills, uh they all compound and GrokBot uh will be able to you know, automatically respond to events that come from the outer loop and then kick off cloud agents.

Um and you can also set up Cursor automations and use our SDK uh to set up um bot additional bots as well that reuse a lot of these pieces of agent infra that you've set up um and allow them to do much more complicated tasks. So, if you if you if you've done all this, then I think you can get to a point where uh you know, uh I have some screenshots here of some of our automations and our agents in uh that that work on cursor where we are automatically re- reproducing bug reports, we're automatically opening pull requests, we are um essentially adding a lot of value to the entire team uh because all of these things compound.

So, if we kind of zoom out again, um and go back to this graph, I think that to kind of close off the talk, um if you spend a lot of time thinking about all of the pieces that you need uh to be able to ascend the trust graph, uh you start getting to a place where you can uh really trust your agents more and parallelize your work, and also empower your entire team to build on top of these uh pieces of infrastructure for your agents, um and empower everyone, you know, every engineer on your team, every builder, to be extremely productive and be able to write high-quality code.

So, the last thing I want to leave you with is actually this piece. Uh sorry, not that piece, but this piece. Um I think if there's only one thing you take away from my talk, it should be this this slide here, uh which is, you know, that these are the activities that will help you build up towards a high-trust environment. You know, whenever you find yourself correcting and intervening your agent, um you really want to think about it from these five pieces and where is the most effective step in this sequence in order to get make your agent you know much more trustworthy and of course I definitely recommend thinking about thinking about it in this order where you know you either invest the time to make that pattern categorically impossible through your code base and architecture and data structures or you start looking at things like static analysis and then you layer that on with rules and bug bot and skills.

If you do all of that and you also spend some time you know thinking about your code quality in in terms of of skills you get to a place where you trust the you trust the environment so much that your agents can just be free right and personally I have spent a lot of time for this for growth bots code base for example and this is really the secret right well it's not really a secret it's it's a lot of hard work but I hope you found this talk useful and please reach out to me on X my handle is potato with an e and I hope that you'll have a lot of fun and successfully in your own mission in kitchen.

Thanks for watching.