Scaling Agents on Kubernetes with acpx and ACP — Onur Solmaz, OpenClaw transcript

Welcome. Um So, the talk is building an ICP on uh at Open Clo. It's uh also about other things like how to uh put open source agents on open source agent frameworks on Kubernetes and stuff. So, I'm I'm I hope uh to I will today share in a nice way what I've been working on in the last 2 months. Um A little bit about me, very brief. My uh A little bit about me. I've been building harnesses since a few months before ChatGPT came out.

I built a Jupiter Lab extension over OG Codex model like DaVinci Code 2. Um back in the days that eventually So, I'm currently working for a startup um that that initial coding harness turned into its current harness over time like a ship of Theseus. It got ripped apart and put back together so many times. And uh yeah, I'm a founding engineer there. I've been uh in the industry since 3 and 1/2 years. I've using Open Clo since Clo Bot uh first dropped in Discord.

I went in there. I've been following Peter since he wrote a Clo code is like computer. I was like w- this guy is crazy. Uh because uh Clo forced on it like I wouldn't give my machine to it, but he he went he ventured for it and he's paved the way for us. And when I saw uh Clo Bot in Discord, my mind was blown. The next day I installed it uh at the company uh at at in our like uh cluster. Uh it was basically talking to people on Discord and also everybody else's minds were blown.

And we were like we have the we're used to selling the enterprise for a few years now. So, that's why I started by adding in MS Teams integration in case it might be useful at some point. And I became a maintainer along the way. I was there when it was renamed uh two times. Um and today I I my focus is on agent interoperability and orchestration. Um like one of my goals is accelerate enterprise adoption of Open Claw and adjacent software and also address, you know, Open Claw's not secure.

Well, it'll be secure or Peter talked a lot about that earlier, so I don't have anything else to add on top of it. Like it's work in progress. Sorry. I'll use this. So, uh it started by I started on develop workflows right away, so I created a PR. Uh and then call it Discord driven development. Well, Telegram driven development suits better because it's TDD, so uh and then uh right after I set it personally and I realized, you know, Opus is not so reliable for complex it wasn't back in the days.

It's now better. It's agents are improving, but Codex was main my main harness and I wanted to use Codex in Discord. But it was basic I was telling I was playing the telephone game. I was telling Opus to tell Codex to do something. God knows what it's saying, you know, because wording matters when prompting, but it was working somehow and I would go uh and then look at the Codex session and then it paraphrased kind of what I was saying, but eventually, you know, uh I got some stuff done, but I knew this could be done much easier.

Um and today I'm uh running a full IDE on Discord like uh you know we have parallel workloads. You have like one to five like five channels at any point I'm working with one to five agents. Um so you see Codex one to five and then close it for testing the open close ACP feature. And then the basically that's how my that's how my channels look like. And it's very good for coding on the go because I have I'm a guy who's addicted to side projects and uh I I like to just you know uh AI is making thing making things a lot easier to do these sort of things like in parallel you have an inspiration you execute on a weekend and you just get it ready.

You ship it. ACPX which I'm going to talk about is is like similar. I built it through uh Discord. And what you do is uh you bind the channel uh Discord channel to Codex through ACP. You can also use Codex app server protocol. Harold another maintainer uh developed it. And here it is me using it before I flying to London to create an PDF about like convert the docs of ACP into PDF. And then I have to say put it in temp because open the uh Codex doesn't know about the harness and it cannot send me in Discord.

So I then I go to another channel and tell it to send it to me on that channel. So we are developers and our tools we don't have time to polish them very well but we know the like advantages and disadvantages. What is ACP? So it's not uh most people say is it like MCP? MCP for is for giving tools to the model. ACP's for uh like standardizing agent to client interaction. It's uh shoutout to Zed. I actually forgot to put Zed logo.

Um Zed is building a new editor in Rust more efficient like lower memory usage not Electron. So uh I'm using Zed also since like uh last fall and uh you know, if you use Codex on VS Code or Cloud Code, they are all building different plugins. It's not exactly It's like so much wasted work. If only you could just standardize them under one interface and you just build it once and then you you ship it. And that's what the I the idea That's the idea they had.

It's also more It's It's much less duplicate duplicated work. Um There are competing standards. Uh so agent like agent to agent protocol, that's for agent talking to agent. ACP, agent client protocol, is for human talking to agent, but agent can use the human one to talk to other agents as well. In the long run, as as these protocols get adopted, we will support all of them. We will weigh the you know, the advantages and then we will use them somehow.

But when I need So, I chose ACP because when I needed the most, I needed adapters for Codex and Cloud Code and only that had built them and Google didn't have them back back back at the time, so that's why I chose it. Uh that So, I when you're adding in functionality to Open Clo, you do uh you use do it through a CLI. Um So, I said, "Okay, let's create a CLI for ACP. Let's Let's agent call any other agent over the command line." So, that's that's how it started.

Uh And it's slowly turning into a Swiss Army knife for ACP as I will show in a bit. So, in at Open Clo, you have fire hoses. Uh we have 60 KPRs uh over 60 KPRs total. 300 to 500 per day on average are open. And basically, overnight, people woke up and decided they want they like open claw. So, we have tens of thousands of stakeholders who want to add features to open claw. And the biggest challenge of the project currently is how do you absorb all the needs and wants of these people?

And how do you balance them? How do you Like you can't please everyone. And how do you create an elegant system without creating AI slop? Uh that can cater everyone's needs. And Peter's workflow gave me an idea. So, this is also something I do. You go and you ask your clanker, what is this This is where PR comes your way. And like most of the time it's AI generated description. Like what is this? Uh if the human put thought into it, great.

But like yeah, you need to ask what it's doing. You ask, what is this the best possible fix? Most of the time it's no. And then you either continue this like he he wrote it. Like You you do some back and forth with the agent. The reason is people just run into an issue with their open claw. And it's it's going to use GitHub. And then they just say, "Please fix." And then they just send some slop your way. You know, this is You can't merge it, but you can also fully discard it.

You need to take this data point. You It's like crucial feedback from the user. Uh so, you need to put it categorize it put it in a bin, you know. It tells you when some part of the code is broken. And Vincent also talked about that a little bit. Uh and here is uh his his code extension doing that. On one side there's the one that says it's it's a good fix. Uh and on this other side it's just saying it's not bad. And it's just so mechanical and you you once you do this over and over you realize you're repeating something.

If only you could program something to automate it. So, you're automating the automator. Um so, I created I started uh to create like workflows uh in an abstract way. So, uh item comes PR and then you find the intent. You're judging implementation. You're like uh looking in if it if it has conflicts. Uh if reviews gives you like some issues that need to be addressed. So, you need to make the CI pass if it's not passing already.

Most people just don't care about that. All this mechanical work by the time the main the PR ends up in front of you, should be resolved ideally and can be resolved and that's what I'm working on. Ah, in the workflow we have the shameful uh Ralph review refactor loops. Uh I I'm a believer that, you know, when people say like give the just just run a running a agent in a loop doesn't not necessarily have to be a uh something that will create slope.

As long as you're not making it design something, but you're making it uncover shallow bugs that can be easily fixed should be fine. So, in the abstract workflow that I created uh which is actually called like for like turning to a program, you can uh tell it to uh do superficial refactors and then you can tell if it's it needs a fundamental refactor relate to the human. Um resolving conflicts also doesn't need like No, it was hard back in the days.

I don't think anyone is like resolving conflicts by hand by now. Um So, we are basically creating a standard operating procedures for agents. That's a fancy word for workflow. Um so, that's what I built into ACP-X. Uh it's a an A10-like workflow engine. Uh but it's driving a Codex session. You you see on the right. Let me show you in action. So, this was one PR. It's loading. Let's speed it up a little bit. So, there is some programmatic parts.

So, this is just replaying what it's doing. It's reproducing the bug. It's judging refactor. It's reviewing. I'm doing a review loop. And then review didn't bring anything. It It's what I do, but I make it output like JSON structured data so I can put it in like an an it's unlike workflow. And I will talk, you know, this is a general workflow engine. Engine. Um You can use it on other things as well. Um Like you need to apply agents generously on problems.

So, I see it's like an ointment that you apply to generously on any problem that can be solved with agents. You need to take yourself out of the loop and solve it with agents. Um and personal agents, I think are on a like there there's spectrum enterprise and personal agent. And normally you see the work you use at the computer and then the PC the PC use at work. And the PC use at at home used to be relatively similar, but that will not be the case with agents because at at work you will be using consuming a lot more inference.

And that means in enterprise there will be a lot more money to be made. So, that's why I'm a bit also excited about enterprise agents and open close a potential. That's why I I believe in on-demand disposable agents. If you use open clone slack or teams or discord, this is one instance. You create an app. The problem is you can't really talk to multiple instances of this um like you create an a connection, you create a Slack app to connect another agent and another name and another uh profile picture, you need to create another app.

You have to create an app manifest. And it's it's it's something that shouldn't be managed manually by clicking. And the platforms like chat apps don't have the standard yet where you can uh mult- do multi-agent provisioning, like cosmetically create different agents. This is what This is on on the screen you see I asked ChatGPT to generate the idea. You have an agent uh and then the name can be generated by an an underlying app, and you can talk to them separately.

This is not supported, and this must be supported for this uh vision to work. Until it's supported, I'm using it on another UI. Because like we are all going to start one uh agent per task, and there will be uh they will work on these tasks, and they will be creating files, editing files, and it will all be synchronized. It will be it will be uh tied with different than what you're used to with your uh personal agent.

And to do to have that, you need to have co- like a few key components. You need to have Kubernetes. You need to uh have an agent harness. Open Cloak could be one of them. Codex Cloud Code could be one of them. Could be a CP Could be uh GitHub. Uh you give read write access, and you do like state data synchronization. Like you maybe you do something that's use rsync. Some something like what uh whatever algorithm Dropbox is using.

And there are some projects that are uh taking this on. I've been working on this uh this is outside of Open Cloak. This is my day job. And uh it's an open source uh orchestrator. Uh it's the goal operator that's uh, basically handles the complicated parts. You know, there's a user experience. You want to create a concierge on Slack. Like concierge agent and you're talking to it, but you get bottleneck because you have 100 employees on Slack.

So, you need to do for for some other task, you may you may need to create a new one and then it creates and gives you like a website link. Um, I'm going to skip the like shout-outs to Cognition and Devin um, because uh, yeah, they invented the category, but I'm running low on time. So, the repo is at text cortex sprits. I'm just going to uh, demo it for our use case. We use it on error reporting uh, currently. So, if you're on Slack, you know, uh, you can ask it to dispatch an agent to debug it, you know, you're like, "I'm uh, asking any new bugs after prod release." And it's got saying something.

And then you ask it uh, to create an agent. Uh, if I could put it that agent into Slack, I would, but I can't do that. So, I have to put it in another UI. And this is an open source project. You can take this uh, UIs also uh, like a react app hosted in the cluster that you're deploying this these helm charts to. And uh, yeah, you you it starts the conversation there. It starts working on the uh, problem. This is like codex web or Devin or anything, but it's actually using like a full Kubernetes pod.

Wasteful, but I think it's the better abstraction because open core showed the power, you know, when you give a full computer to an agent, it's a lot more powerful. Um, and I I believe that as well. Um, I uh, I I think uh, open hands uses firecracker. So, I'm also uh, not so well versed on all the different virtualization frameworks. Uh So, it's I'm also learning along the way, but I have a working product that's running on Kubernetes.

Uh and you can use this uh product uh you can if you're interested in deploying internally and using Codex on the web. Uh and then you just spin things off if you like, I don't know, a back-end uh If if you have an open-source project, first of all, I can help you set this up. If you have a system like inlets of just 100 of hundreds of issues per day, I can help you process it. Um this is uh that does all the wiring around Slack and like keeping those agents like on and the user experience and then the the interoperability.

You're not locked in any in any agent. You can switch. It's all abstract the way build with ACP. Yeah. That was my talk. Uh thank you for listening. Some social links in case you want to uh get in contact with me. I just want to make clear like of my uh the OpenFlow uh side and TextCortex side. Uh so, this the last part was TextCortex about the work I do at TextCortex. Uh just to give a disclaimer. Um thank you for listening.

I guess Yeah. Well. Thank you.