Build an agentic software factory: deep dive transcript

The most exciting thing in software development to me right now is software factories. Literally software that self-improves. That as you collect feedback and telemetry, the software gets better automatically. I've been using this approach for the last few months to push the boundaries of how much we can automate setting bugs and improvements directly through agentic loops and having better software come out the other side while making sure human touchpoints can always guard and take over as much as needed.

And I've open-sourced a set of skills that you can use or customize too to run your own software factory out of Codex or Cloud Coder or your choice of tools. And it's all customizable, configurable. Let me walk you through how it works. The short of it is this pretty basic loop. We want any feedback or telemetry, such as Sentry errors, error logs, performance and uptime metrics of those that are verifiably fixable. As in an agent can reproduce the issue, make a fix, verify the fix is correct, and send a pull request.

We want to just automatically delegate those to agents. Those agents can be right there on your local computer using Codex or Cloud Coder or whatever tool you prefer. And when the fix is verified, they go to a pull request, it's decided whether it needs a human review or it's okay for a pure agent review. When CI agent review and potentially human review passes, the software gets shipped and the loop starts again. Now, obviously there's some pretty important things you need to do to make sure that you don't actually break things or make your code quality worse.

That's why I'm going to walk you through the actual workflow that I use with the actual skills every day. The vast majority of the work on my open-source AgentX project is built by autonomous software factories incorporating GitHub issues, in-app feedback, internal feedback, and other sources and verifying and pushing fixes to the framework as well as a bunch of hosted agentic apps that we build and maintain for things like agentic design generation, slides, analytics, Loom alternatives, mail, calendar, whatever.

These apps are all free and open-source so you can look at the code yourself and they're non-trivial. They do things like full-on Figma-style editing and generation of designs, slide editing like Google Slides, calendar management, you name it. They're very modern. They're fully agentic. Every single thing the UI can do the agent can do. You can use them as pure visual agents or pure UIs, but the best is mixing a bit of both.

So, these aren't small apps, they're built in pretty novel ways, yet the majority of work I do is designing and optimizing and assessing the systems for building the software, as opposed to writing the code myself. And that's what my team focuses on, too. So, let's start at the beginning and show you how this works. Everything ultimately starts with input sources. If you're doing things the traditional way, product team and engineering teams will look at things like feedback submitted potentially via Slack or in your product or through customer interviews.

Also, you look at issues and telemetry. Has uptime reduced? Has the P90 response time increased? Is there a spike in new errors in Sentry? Or for an open-source project like this, are there GitHub issues and PRs being filed? All of those become the inputs that we start making decisions on. That's where our first skill comes into play. It's called factory collect. First thing it does is it collects and looks at these input sources on a loop.

That's why I have a ton of these things running on scheduled tasks. I use Codex for this, the same thing works in Cloud Code. I have a slight preference for Codex, which I'll get back to, but it really doesn't matter. Whatever coding agent you use should have a way to schedule tasks that could run hourly, daily, whatever. And the factory collect one just runs every several hours. It'll collect from all those inputs and then run them through a policy.

All of this is configured in a YAML file, so we can list out our sources and create our own, and then put them into workflows. I'll mention in a bit what some of these things mean, but it's important to know that you can add prompts to any of these things, too. So, configure them precisely or just apply prompts in natural language or both. By default, my factories have a simple fixed policy. If it's a clear issue that is verifiable, aka an agent can reproduce the issue itself, fix it, and then verify the fix, and send a PR, the policy allows, which means in Codex it spawns its own isolated work tree, does the verification, fix, and opens or updates a PR.

The reason I have a slight preference for Codex for this is because the Luna model is so insanely cheap. You can run your factory in very high volumes paying almost nothing. I create literally dozens of PRs that are merged per day, all powered by Luna on max settings, and I don't even hit the limits of my $200 ChatGPT Pro account. That's awesome. This works in Cloud, too, but you could to limits. And if your factory's doing big scale, cost does start to matter, especially when you factor in that verification is so key.

Many things need to be verified end-to-end in a browser. That consumes tokens automating screenshotting and reproducing and verifying every time. But as you might have assumed, the verification part is essential. You don't want agents just guessing and throwing up PR for whatever they think might fix it. We need to know it's reproduced, it's fixed before we get any autonomous factory PR. Now, if anything doesn't match the policy, it goes all the way over here, and it goes to humans.

Human inputs are batched separately. That's a separate automation. I'll get back to that. But when the agent does create a PR, then another skill and automation kicks in. I have an every few hours automation that reviews PRs. Now, it's important to note there's multiple layers of review here. If we go to the agent native pull request and open up a PR, we will see that your standard review bot comments will be pushed.

We have our own agent for this, but you can actually run that off of Cloud Coder CodeX using a skill and a loop as well. And the agent that sends the PR uses the factory babysit PR skill to monitor the PR. They're required to verify and address every single comment or CI failure. If they verify and agree with the comments, aka it's in line with the initial intent, and it is a true issue, they will comment back. In this case, since running from CodeX, it's commenting as me.

Or if they disagree, they just reply with why they are not going to do it. Then the separate review PRs agent looks over the open PRs, ensures everything's been addressed, ensures the CI passes, there's no merge conflicts, and then invokes an approval policy. Approval policies are customizable by you with configs and prompts, and basically decides, are we okay with this auto merging and deploying? Because it had or test, it was clearly verified, and it was deemed relatively safe.

It was a CSS fix. It was a highly unit testable fix, whatever. Or does it need human review? If it needs human review, it waits on the human review and then merge. If allowed, it gets shipped end-to-end. And that's the super basic workflow. Collect, babysit, review, ship, repeat. At any point when a policy doesn't match, set it aside for humans, either tagging them or aggregating for the humans. Now, there's also a few nuances to point out, such as when collecting, what if the issue is not clear enough?

That's where the click skill can also optionally follow up. If we need more information, we can automatically post asking for more information. Such as, can you at least give us the URL you're on? Or what time this happened so we can look up maybe in error logs. You can customize this completely. In a lot of cases I have to specify this was sent from a bot because it's coming from Codex or Cloud Code. It's just using my credentials, which means my profile.

The skill also knows to check back to see if prior clarifications were answered and now it's clear enough to take on, and then it will. Before we go on to some of the deeper automations, you might also ask, is it better to run this in the cloud? Some kind of cloud service that runs this even when my laptop's closed? Or should it be running on my local computer, which means it's not running if the computer is off or asleep?

I have gone back and forth on this like crazy. And my definitive answer right now is it's got to run on a laptop. Because to actually verify things, it needs my actual environment. Needs to run on my Mac with my end files, my keys, my browser sessions. When it's stuck, I need to be able to look at a screen and enter an OTP. Or click to login with my passkey. Or in some cases we're developing on Mac native apps. At the end of the day, I've constantly struggled to get cloud systems to replicate my environment effectively, and I just need it running on my environment.

But yes, it's super annoying when this stuff is running, especially if it's doing a lot of work, when I'm trying to do work in parallel. That's where my mind's latest solution is I have a second laptop. It's right next to me over here. And that laptop is on 24/7, always online, and running through these loops all day and all night. If it's ever blocked, it telegrams me. I need a key, I need to login, I need whatever.

I can just look to the side, log it in, and let it keep going. I can look over and check on it and see if it's doing things right. And what I love about Codex is I can see all of my remote tasks and work trees right here. They all have the blue globe icon. In fact, in Codex, I use the remote feature to actually do all of my coding on that laptop. So, anytime I have an idea and I want to spin up a branch or a work tree, I actually type it into this laptop and it runs on that laptop.

So, that laptop is frequently bogged down, using up a lot of memory, running browsers in parallel. And this laptop I can actually run and do all the chats, but my CPU and memory are never bogged down. This has been life-changer. I use it both here and with Claude Code. With Claude Code is the SSH feature, so I just have it SSH to the side laptop. I will say I like CodeX's more because with SSH, at least over my local, I can only access my laptop through Claude if it's on the same network.

When I'm traveling or when my main laptop's closed, I can still open it and see and manage my CodeX. Also, because CodeX tokens are so cheap, I like CodeX by default. Luna is a shockingly good model, especially for this type of work, especially with clear guardrails with these skills to always focus on verification, verification, verification. And also, I always use it on max settings because Luna is shockingly strong and cheap on max settings.

Opus level performance for literally less than 1/10 of the price and way faster tokens per sec. But some work is hard and needs some more oversight. That's where I use a skill called factory watchdog. For work that I know is harder or deeper, what I'll do is I'll grab the thread. You can right-click, copy, copy deep link. I'll go over here and invoke my watchdog skill, I'll give it the link, let it run. Now, I'm going to stop this cuz I don't actually want to do that right now, but what this will do is it'll watch that thread and it'll message that thread with guidance.

It'll independently verify that work. It'll make sure it's doing the right things. It'll communicate with that thread to make sure Luna is always doing the right things the right way. That oversight I find super valuable and super effective. And I found with a ChatGPT Pro plan and a Claude Max plan, I can do an insane amount of work without ever going into overages. I often use Fable for that, but the latest Opus 5.5 is probably plenty good enough.

And when in doubt, I have a separate efficient frontier skill that just reminds the model to spawn low-cost sub-agents, like with Sonnet, Luna, whatever, if it's possible to save cost. Now, that's the basic stuff of a factory, but the important part is the next part, which is what can go wrong and how do you make sure this goes as right as possible. Let's go down and look at a few more automations that run on the host scheduler.

So, Claude automations are loops or CodeX automations or loops. So, I touched on Factory Watchdog. This can be used in a whole variety of ways to make sure it's overseeing and just a whole other model trained by someone else is giving the oversight and guidance that the Bitmore GPT-based factory is doing day-to-day. But then there's another issue that drives me crazy. When you're spawning a lot of these Luna work trees and you tell them to follow the factory, there's a lot of cases where they just don't.

They just kind of forget or stop or don't go the whole way. That's what the watchdog does. I have Factory Watchdog running about four times per day and its whole job is to look over work that was done and is actionable but did not go to the next step for whatever reason. There's always a few of these per day. A PR that was created and never fully shipped, etc. So, it'll look back on the threads, it'll spot those, and it'll nudge them until they finish.

That one's been a lifesaver. I've wasted a lot of time on work thinking something was worked on when it wasn't. Now, the other big trap as well is recurring patterns. There are cases where the same issues come up over and over and the agents think they're fixing it over and over and they're just not. So, once a week I run the factory look back skill. Factory will look back over the last week and say, "Was the same feedback reported over and over even after we thought we fixed it?

Are we getting the same types of P90 reports on repeat? Or are there brittle areas? We fix it then it gets unfixed or fix it and it breaks again." We want to spot those patterns and make sure we find holistic fixes. When the look back finds a pattern, it'll delegate to a larger model like Soul or Opus or Fable. And that larger model is tasked with doing a much broader scan of the input sources, spawning sub-agents to look over extended periods of time, periods you can configure like the window and compare is the last and previous 30 days, and find those recurring issues, and make sure we find a good solution.

It'll go a lot deeper. It'll try and compare errors, logs, and customer feedback to try and triangulate a more holistic solution. And I think this helps a ton. We had cases in the agent data framework where the chat was just brittle. It just kept erroring. It was driving me crazy. There were other cases of the same sort of drag-and-drop issues kept recurring over and over in the design app. And the beauty of having these structured input sources and these structured factory flows is you can do totally automated meta-analysis.

What is systemically going wrong here, and how can we fix it, and then check back and verify that oh good, this type of feedback has actually wound down, or these errors or these P90s have actually come down. That's ultimately the goal we need to see. And having these structured connected inputs and flows lets us do that surprisingly effectively. Now, there's a lot of other nuances in these skills. For example, when collecting feedback, it already does also know to try and look back and see if this feedback has been reported before.

On a daily recurring basis, it'll also factor that in. So, if a believed fix was shipped and the feedback continued, it'll know to either A, do a deeper dive and deeper verification this time, B, ask the human for more input to further validate like do we really understand this correctly, or C, more heavily opt for a human decision. Now, at the end of the day, I spend most of my time feeling like a scientist of a biological organism, always inspecting, microscoping, and assessing if the factory's working as efficiently and as effectively as it possibly can.

Does it delegate to humans when it needs to? Does it waste as minimal human time as possible? Is it making good decisions and good fixes and good improvements? And ultimately, is it saving me and my team lots of time? So, we can build stuff like the agent native framework, all these applications built on top of it, and allow ourselves and others to do cool stuff like this action system we've come up with that you define your application behaviors in one place, they automatically can be used in UI, agent chat, APIs, MCP, blah blah blah.

This is the stuff I want to work on. I want to work on the hard problems, and I don't want to get caught in the minutia of little fixes. So, when I had an error deploying to Vercel, I just want that auto fixed. I don't even want to look at it as long as my system works. This makes me more of a systems engineer, as well as my team, to make sure we're building an autonomous system that can constantly fix and improve things.

And before we wrap, I want to give a couple specific tips as well. Number one, it really helps if the whole thing's in a monorepo. Having your factory dealing with one isolated monorepo helps a ton. And two, efficacy of your factory is highly dependent on the quality of your inputs. Spending as much time as possible building out strong telemetry, customer feedback channels, clear ability to see errors with their original source code tied to it like with source maps, and additional layers like we have thorough end-to-end tests run on beta that test performance, core user flows, etc.

And those can catch issues that we don't want to block PRs cuz they take quite a long time to run and they're kind of expensive, including not just scripted tests in browsers but also agents going in and testing regularly. Those turn into automatically filed issues and get dealt with immediately, too. So, if something does break, it's caught instantly. My third tip is to have a separate beta and production environment.

So, our internal users at Builder all use the beta environment. They redirect it automatically. Beta's always in sync with main. And when they see issues on beta, those get auto-fixed. Once a day, a production deploy is cut over to prod, and then we know production is far more stable. And if we find one of these issues leak to production, it's easy to roll back when it's only pushing out once per day as opposed to automatically all the time, every 10 minutes or so, when there's so many PRs coming from agents all the time.

And lastly, I would start small. You don't have to start with all these things. In fact, the very basics goes back to the beginning. We just want to have an easy way to take some input source, figure out a finite set of things to carve to agents. You can start really small. Clear error messages, guaranteed verifiable, only high confidence. I continue with your typical human review, etc. You can shift the balance over time, and you don't have to reinvent the wheel.

You can go to my skills repo. You can grab all these skills. And the beauty is it's not some fancy remote custom system. These two can just all run locally. So, step zero is usually just run it locally. Run {slash} factory collect. Play with the configuration a bit and look at the output. You can start with a dry run. Just have it identify what it thinks it should fix and give it feedback. Because configuration's just a YAML file, you can have the agent update its own YAML.

So, you can go back and forth with the skill giving it feedback and telling the agent to update the prompts and guidance and config options until you can run that skill locally and you like the set of issues that it's deciding to do. And then, instead of you manually doing those, go ahead and tell it to work on them in their own work trees. Tell it to post their PRs and babies at the PR. Make sure you're happy with the way it's responding to feedback.

Verify that the even feedback gets higher priority than bot feedback. This is how I started. I did the stuff very manually. Just a set of skills that let me do large and larger groups of work at a time. And then when I eventually found that I'm starting to become a robot, every morning I would run the collect feedback and ship skill. Every afternoon I would run review PRs. And I realized like I'm just clicking buttons.

Then you start scheduling tasks. You tell the skill to do the task on the schedule. If you don't have a separate laptop you can run the stuff on just run it on your main laptop. That's what you would have done anyway if you're doing it manually, so start there. It was after literally months of running the factory that I eventually realized, "Okay, in large scale the system's working really well for me. I need a dedicated computer for this." It doesn't be a laptop, it could be a Mac mini, whatever.

Just make sure it's plenty of memory. I find that with a lot of agents testing things in browsers can eat up a lot of memory. And then once you get fancier and fancier enough, you can go into these full robust automated workflows that include the automated lookbacks, which again, start by running lookbacks manually. Does not have to be automated at first. Automated recovery watchdogs. But hey, this is my flow. I'm curious what's worked for you.

I am fascinated by this idea of all these machines doing work, automating the product improvement process end-to-end. This idea of autonomously improving software is really exciting to me, but it has limits. You should do what works for you and don't do what does. You can grab, install, fork, customize all these skills over in my skills repo. There's an installer CLI that can also set up things like automations and other things for you.

And if you're digging into my stuff, check out the Agent Native Framework. It is, in my opinion, the most awesome way to build an agent today. Agents are great, but an agent paired with a custom-fitted UI that is completely seamless is, in my opinion, the awesome future ahead of us. If you're not ready to build something, go to agentnative.com, check out the apps, and use some of these apps. There's literally a free alternative to Loom, free alternative to Figma, Google Slides, whatever BI tool you use, calendar, Superhuman mail, all kinds of stuff, notion, whatever.

It's all free, it's all open source, costs you nothing to use, and it's infinitely customizable, and you can do whatever the heck you want with it. If you try this stuff, I'd love to hear your feedback and your opinions on agent factories in general. Do you like my approach? Is there a better approach? Let me know in the comments. Thanks for watching.