It's hard to escape people talking about coding agents online, but what might surprise you is how easy it is to build your own. For the past few months, I've been using my own coding agent on a lot of my projects day-to-day, and it's allowed me to do things that would be impossible with a traditional harness. So, in this video, we're going to go deep into the architecture behind tools like Cloud Code, Ph and Codex, and I'll show you how you can build your own coding agent completely from scratch using a single prompt.

Once you own the harness, you can build any features or extensions you want. [music] I'll link all of the resources and everything you need to get started in the description below. So, let's get into it. >> [music] >> So, Neo is the coding agent I built for my own use cases. It's open source, you're welcome to try it out, and it's a single one-click install. You just need to copy this command and run it in the terminal.

What's interesting about Neo is it's written in Golang, so it's very fast and easy to install. It's just a single [music] static binary, and it's very good at parallel tool call work, so you can run many operations in parallel, and you can run sub-agents in parallel as well. It's very good at this kind of work. And then, the other thing about Neo is it's very much focused on the development workflow, which we'll see in a second.

All right, so I'm just jumping over into the terminal. And so, the first thing I'm going to do is just install the coding agent. So, one of the goals here is to support any language model, so we can use Anthropic via an API key, but you can also use your OpenAI subscription or any other AI model via something like OpenRouter. So, the first thing we're going to do is type the word Neo to start up the coding agent. And as you can see, it looks pretty much like any other coding agent.

We have a bunch of slash commands and operations which we can run. And if you press the tab key, you'll be able to show the workflow panel, which I'll show you in a second. So, the first thing we're going to do is just test it out by running a code review. So, review the codebase. The first thing we do is understand the user request. But then, once we've understood the user request, we're going to turn this into a workflow, and that workflow is going to be then be visible inside the agent.

So, you can see here we've broken it down into four steps. And then, if I toggle on the tab key, you can see the steps we're going to take. So, I find this is useful because I'm often running a workflow rather than a one-shot prompt. And what you can see here, again, just going back to this parallel nature of the tool, you can see here we're running five tools in parallel at the same time. So, this is an app I'm currently building called Factory.

It's essentially a way of orchestrating a lot of large-scale coding agent work. You get metrics about what's going on. You can build work on workflows and automations, and you can dispatch all of your coding tasks to a bunch of different coding agents. You can also manage multiple different Git repositories at the same time. So, this project is all about massively scaling what's possible with coding agents. We can run, you know, five or six different Git repos.

We can run workflows, automations, agents across all of those different repos, which is kind of a really interesting idea I'm experimenting with right now. So, anyway, we're going to implement a task inside this repo and then get a pull request open so you can kind of see how this works. Okay, so I'm in the factory main branch. So, what I'm going to do is start Neo. And then I'm going to run a single command, which is called task to PR.

And what this is going to do is going to take this GitHub issue and just implement it end to end. So, this is again a workflow. We're going to read the ticket. We're going to check out a new branch. We're going to write the code. We're going to run a bunch of tests. We're going to review the code. And then finally, we're going to open a merge request to review. So, this is just an end-to-end workflow. And if you look in the breakdown here, you can see this is actually how I generally develop.

And it's how most people build code. Pretty much every developer in the world is going to follow this outline or this workflow. So, check out the new branch, read the ticket or the task, make a small plan, make the code changes, test the work, review the work, address any findings, publish the change, wait for our automations and build checks to pass, and then finally hand it back over to the human to review. So, what I really like about this is you can kind of just watch this run end to end.

Again, I'm not doing any coding here. We're just passing the task over to Neo and then Neo is running the entire flow. So I really like this way of working in general. And it allows you to scale really well because you're no longer having to prompt everything manually. You're just kind of following a relatively repeatable process. So this is kind of the way I like to work right now. You'll notice the agent has given a really nice problem description.

It's explained the changes it's made and then it's also told us about the risks and the verification steps. Again, this is all about building a reusable workflow and having high standards at every single part of the development process. And once you kind of lock in a lot of these standards and processes, it's really easy to scale your workflow. So we're just waiting for the agent to run through and basically react to some of these comments.

So I have an AI automated code review. So one thing I really like to do is have my agents comment on my change requests. So you'll notice here that the agent has actually commented on this original code review feedback item. This allows me to know that the agent has actually addressed the feedback. Which is just really useful. I've encoded this into the skill that I use for my development process. And so whenever the agent works through a task, it will wait for the code review feedback.

It will address the feedback and then it will comment back on the ticket. So you you know that the agent has resolved the issue. Okay, so as I said at the start of the video, we're going to build a coding agent from scratch in this video very very quickly. And so when I think about development of any project right now, the first thing I start with is the overall architecture. So we want to think about what we're building.

So this is going to be a coding agent and coding agents are pretty simple. They have a loop and if you kind of look down here, you can see the outline of a coding agent. Whether you're using Cloud Code or Pi or CodeX, all of these agents are basically doing exactly the same thing. There's very little difference between any of these agent harnesses if you actually look behind the scenes. We have this agent loop and the agent loop is responsible for essentially making decisions about what to do.

We send a request to a large language model, then the large language model will respond with one of two options. Either it's going to return a response to us back to the user and say we're done or it's going to run a tool call. And then a tool call allows the agents to do things like reading files, editing files, like making updates on your file system, calling bash commands and things like that. And we send that information back to the terminal or the UI.

So, coding agents are pretty simple cuz they generally command line programs. So, very very simple, very minimal UI. And that's kind of it. And then we also need our inference as well. So, we need some kind of AI model to use. So, for our coding agent, we're going to be using OpenRouter. And the reason is we can use lots of different models and it's a single interface that we need to implement in our coding agent, which makes things very easy when we're getting started.

If you wanted to support Claude, you're not going to be able to use your subscription very easily without breaking the terms of service. But if you want to use OpenAI's model, they do let you use that with external coding agents. So, I tend to use OpenAI models myself. But OpenRouter is great because it gives you access to pretty much everything you could ever want. The downside of OpenRouter is it's API key based. So, you are paying per usage, not using your subscription.

But for some of these models, like these cheaper models, it's good to experiment with them and they're relatively cheap anyways, certainly compared to things like Claude. So, we're going to use Kimiko 3 as our primary model. And we're going to copy the model ID and then update our code to make sure we're using this model when we build it. So, this entire architecture doc was generated through a skill. I've built these coding agents quite a few times.

So, what you can do is just download this architecture doc. You can iterate on it and you can edit it. But we're going to build this out in a single GoLang file. And the reason is it's going to be much easier to understand the code base. If you were working on this for real, you'd split this into different modules and files. But we're going to keep it in one single file, which will make it easy. We're just going to be able to run this in the terminal and then see the coding agent.

So, the first thing we need to do is actually break this down into tasks. So, I'm going to go ahead and open up a coding agent. What we have here is just an empty directory. We have a single document. We're going to read docs/architecture .md. So, we're going to ask the agent to read the architecture doc. And this is going to allow us to get the architecture doc into the context window. And then once we've got that, we're then going to break this down into tasks.

All right. So, we've got an understanding of what we're trying to build. And so, the next thing to do is plan out the work, break it into tasks. Because this is actually a relatively simple project, the first version of our coding agent is going to be really simple. We could probably just one-shot this. But, to kind of show you the process I would normally follow, let's break this down into tasks. So, let's say I'm going to use the plan skill, break this work into tasks, and list them out so we can start building it.

So, we're going to get the agent to essentially break the work down into a number of smaller tasks. I would typically store these in a GitHub issue because it allows you to keep track of things over time. The reason I use GitHub issues for everything or some kind of task tracker is as this gets more complicated, we might have things to kind of keep track of, bugs, all kinds of stuff we need to work on. Okay, so we're done.

We now have a task breakdown. So, we have a bunch of different tasks. I'm going to quickly scan through these. So, we have about eight tasks. So, what I'm going to do is put these into a file because it's going to be easier to read. As I said, normally I would store these in GitHub issues or linear or something like that to kind of keep track of stuff. But, let's go ahead and save this to a file. So, save these tasks to a Let's call it tasks .md file so I can review them.

So, what we have here are the tasks we're going to build. So, we have this kind of empty directory, and then we're going to basically build the go foundation. So, we're going to build the go main.go file, and then we're going to add in all of the types. Then, we are going to go through any checks, run some tests. We're then going to implement the basic provider, so we're going to implement the LLM provider. This is going to be open router only.

And then finally, we're going to go through and build the agent loop. This is the core of the agent. So, these tasks look pretty sensible to me. As I said, normally I would store these in an external task manager. Add exact match file editing. Add bounded command execution. So, this is going to be adding the bash command to our agent. All of these tasks look good. So, I'm just going to go ahead and sequence them in the coding agent.

Okay, so now that we have the tasks.md file, what I'm going to do, I've just started up a new session. We're going to read those tasks, break it down, and then just basically build the first version. So, I'm going to say read docs/architecture. md and tasks.md. Then coordinate the build of all the tasks. Use the task to PR skill to build each task. So, we're literally just going to give the agent the context again. We're going to get it to read through the tasks and then run through.

One of the reasons I don't like using markdown files for this is we're adding a lot to the agent's context window. Essentially, we're asking it to to look at all of the tasks at once. But typically, what you want to do is go through one task at a time. It's much easier for the agent to focus on smaller parts of the puzzle, and it's much easier for you to review the code if you break it down into smaller parts. But because this is such a relatively small, simple program, we're going to just run through all of them in one shot.

All right, so now we have the task. So, basically, work through all of these, create the isolated work tree. We're going to read the context, we're going to go through all of this these steps, and just kind of wait for it all to be done. Okay, and just to show that this works, you can see here we're in the directory now. So, we've written our basic minimal coding agent in this main.go file. What I'm going to do is just invoke this.

We're going to type go run main.go. And you can see here we have this basic coding agent. It looks really minimal. There's not a lot going on here. The UI is obviously not very fancy, but it's basically a working coding agent. You can actually do real work with this right now. So we can say hello. And when we send a message like this, we should just get an immediate response back from the model. It's not going to call any tools because it doesn't need to.

So now we get a response back, but if I was to say something like, I don't know, review the code, then in theory the model should start calling tools in order to do that, right? So in order to review the code, it needs to read files on your file system. So there you go. You can see it's it's doing a bunch of bash commands, listing out the files in the directory. It's running these kind of complex bash commands, which these agents are already trained to do.

And so it's just basically just reading all of the files and then it's going to send that off across to an LLM so that it can then do the code review and then send the response back to us. So what we have here is a very minimal, very basic coding agent. It may not look very much, but now that you have the foundation in place, you have everything that you need to start building on this coding agent. You can start adding more logic to this coding agent.

You can improve the the build process. You can tidy up the code, make it more modular. You can constantly just iterate on the code base now that you have the working agent. And what's really cool is once you get to this point, you can actually use your own coding agent here. This very minimal coding agent can actually be used to build out the rest of the coding agent. So once you get to this point, the agent is capable enough that you could use the agent to build the agent if that makes any sense. >> [music] >> Really the jump from this coding agent to something like Claude code is mostly just about polishing this core loop and adding more features and functionality, improving the UI, things like that.

But you can very quickly build up to something capable, very similar to something like the pie agent, which is very, very minimal. We can start adding in things like sub agent calls. We could improve the UI. We could, you know, add additional model providers, things like that. We can just start adding more features as we go. So, this is the coding agent we built. It's very, very simple. Because we're using Golang, it might look a bit more verbose and complicated than it actually is, but the core of a coding agent is really incredibly simple.

We're basically just invoking an LLM in a loop and providing tools that the agent can call. That's basically 90% of it. And then there's other things that we need to add, things like the system prompt and context management. We could read an agent.md file and append it to the LLM, for example. But once we have this basic core agent loop in place, it's very, very easy to start building up from this. So, the core part of every coding agent is this very simple loop.

And so, what we're doing is we're just running in a loop, calling an LLM over and over again until the LLM decides to stop. So, the LLM is responsible for making the decisions about what to do. And then the agent harness, our code, is responsible for executing tool calls and then giving them back to the LLM. Because if you think about it, an LLM can't access your host machine. The only way we can read a file on your local machine is for us to do it in the harness, in the code, and then we send the results back to the LLM, and then the LLM will decide what to do.

And you're only kind of just doing this loop over and over again. So, whether you're using Pi or Claude Code or Codex, all of these agents are basically doing exactly the same thing. There isn't up that much difference between any of these agents if we're honest about it. So, you can see here we pass in the system prompt. We pass in the user message and the tools that the agent has access to. Then the agent will reason about that request and decide what to do.

If we need to call a tool call, then we'll essentially just run those tools. We'll invoke the code, we'll read a file, do whatever we need to do, and then send it back to the LLM. Or we just send a response back to the user. So, if there's no tool calls to invoke once you get to the end of the loop, you just basically return the response back. And then you show it to the user in the terminal. You'll notice that we're using events a lot throughout the code.

And the reason is it keeps the agent loop very, very clean. If you look at this function here, it's actually quite big already. And you could imagine if you started adding in additional logic into this agent loop here. So, say if we wanted to add in a permissions check here, we could say something like if the user has decided to request permissions checking on this particular tool call, then we need to send a message to the user asking them to approve the tool call.

So, we could add that logic in here, obviously, if we wanted to. But very, very quickly, as you start doing that, what you'll realize is that this code gets very, very complex, very, very messy, very, very quickly. It's not very maintainable. So, this event pattern is just a really nice way to keep the agent loop clean. And then you can hook into these events. So, if you think about a Claude code hook, all you're doing is listening for events inside this agent loop and then reacting to them.

So, you know, when the agent is about to call a tool, maybe you want to run some kind of shell script or bash command, whatever you need to do. But you can basically hook in. It's very, very flexible, and it's a very elegant way to build this core loop. The other thing we need, obviously, is the ability to call an LLM. So, this is a very simple API call. We're just sending a request to OpenRouter. We're passing in things like our API key and the model name.

We're waiting for a response back from the model. And you can see here we're passing in, obviously, the API key and any other information we might need to pass in. And then the other part of this coding agent that you need to pay attention to is the tool calling. So, this is the core of any agent. Whether you're building a coding agent like this or whether you're building a real AI agent that you deploy for a customer, like a bigger agentic system, you're still using the same primitives.

Ultimately, an agent is just got access to tools, and the agent decides which of those tools it wants to invoke. So, this entire agent is a thousand lines of code, but it's actually a capable, workable coding agent at this point. There's obviously a lot of things we're missing here. We're missing things like context management. We're missing a whole bunch of other features, like reading an agent.md file or skill support, all of those things, but it's not that difficult to start adding those in.

Once you have the core foundations in place, once you have this basic reliable agent loop, it's very, very easy then to start adding new features and adding functionality to your agent relatively quickly. One of the great advantages of building your own agent harness is it just teaches you so much about how these tools work and it allows you to think differently about the tools in general. And everything you learn building your own agent harness as a local coding agent, you can apply the same ideas to bigger professional AI systems as well.

Most AI agents at this point are all built on the same fundamental ideas of the agent loop, the tool calling, >> [music] >> most of the concepts are the same. And the other thing I've kind of realized over the years is that all of the agent harnesses at this point are pretty much the same. [music] Like I don't have a massive preference for Claude code or Codex or Pi. I actually don't think it makes that much difference.

Certainly to me personally, I find these agent harnesses to be mostly interchangeable. >> [music] >> The only real difference is the model and maybe some of the features that are available within those harnesses, but they're mostly the same for my purposes. >> [music] >> So in terms of my workflow day to day, I use the Codex desktop app. I just find that to be a really great experience. You can run many different projects and you can run many different threads. >> [music] >> So I'm often working within one project on like five different tasks.

And then I'm also working across different projects as well. And just trying to manage that complexity in the terminal, even with something like a multiplexer, is not really realistic for me personally. I just find the desktop app to be a much better experience. And sometimes I'll use Claude code in the terminal because it's really great at design, far better than Codex for most of most things. But if I want to experiment with different models, different ideas, I'll use my own custom harness. >> [music] >> And that can also use different AI models as well.

So if I want to use local models or like experiment with different obscure models, I can use my own agent harness to do that.