A Piece of Pi: Embedding The OpenClaw Coding Agent In Your Product — Matthias Luebken, Tavon transcript

All right. I was introduced to Pi by um looking into open open claw. There was a conference a meet-up and I said like, "Okay, we're doing open claw." And I wasn't so much interested into like all the craziness things that people are doing, but I was more interested in understanding uh of how these things work. So, I was looking into a Pi and you know, uh understand the the whole world of what Pi is able to do. Um this is the one picture you need to take.

Please feel free to take more pictures, uh but all the slides and the examples are there. Uh so, that's the one slide. All right. Very quick uh about myself. Uh we're creating a small company uh Seven AI. We're building agents for organizations. Small out of Europe, uh but getting started. And uh what I really like um about Uh sorry. Uh what I really like about um uh Mario's talk is this is quote uh you probably have seen uh this this morning.

We are on the We are on the around and find out our own phase for coding agents, right? So, everything that I'm going to show you is what I know today, right? And uh I'm going to do the talk again in a couple of weeks and it's going to be most likely be different. Uh but um as as Mario was showing this morning, um he has created this minimal set, right? This this coding agent that is available um for for you for you guys to to fool around with.

And that's what I'd like to encourage you. So, coding agents and why is it so exciting for us to build more products? This is Ken Thompson, um inventor of uh Unix, and this is the famous quote by him, uh one of the quotes, "Write programs that do one thing and uh one thing well." And um I really like that because that's that's kind of like works uh to our advantage with agents. And um the best part where I show this is with Cohere.

So, this is Cohere uh Claude's desktop. Um and they're basically a bundling their coding agent into something where they feel is more applicable. Um and to be honest, I've seen very good receptions around this. And when you use it uh with uh financing tools, with their finance tools, you always need to work with Excel, right? So, uh they have this Excel skill now and there. Um and it talks to uh Excel, right? Well, it doesn't.

Uh instead, it uses a uh a set of small tools, small CLI, um uh Pandas, uh OpenPyXL, and uh stuff from LibreOffice, and package this into their own skill uh to make it uh up and running. And I think this is a great example to kind of get you going, get your thoughts going of what what is doable. Um I haven't written a book, and nobody can write a book about this, right? Because there are no patterns, right? We need to figure this out.

We're seeing some emerging patterns in the coding space, right? There's obviously tons of different coding agents, and we're seeing this, but there's no authoritative resource around this, right? So, get going. One thing uh when I was talking to Evan yesterday, uh we realized is like one architectural pattern that we're seeing is that make it easy for coding agents, right? Now, that is very broad, but think about it, right?

Like like make not don't try to be, you know, very um complex and things, but think about the the coding agent, what is it good at, and how do I build my system so that the agent is easy, make it accessible, and I have some examples. All right, this is the rough agenda for the next 10 minutes or so. I'm not going to talk too much about Pi in Open Claw. I have a few slides, slides are online, so we'll take it from there.

So again, very brief introduction of Pi, Mario, great work. Something he didn't mention is that he's joining Arendelle, which I think is awesome. It seems like great great folks working together. And yeah, it's open source, it's minimal, so it's it's just perfect to get started. And the other part that I do want to re-emphasize on is give it a try, right? We're going to talk about a little bit different, but open up Pi and ask it to build what you want, right?

It's amazing of what it what it actually is able to do by the system prompt that Mario has shown. All right, these are the extensions. So again, all the extensions you can download, build yourself, or download and yeah, tons tons to explore. All right, so let's go in. This talk is not about the coding agent itself, so using it for your daily dev works, but what can we potentially do with this? And the starting point are actually not coding agents, right?

The starting point is and I encourage you to do the the same is looking at the core agent itself. And there's other SDKs, but you know, we're we're talking about Pi, so let's let's let's use Pi. And what is an agent? An agent is actually just an LLM agent that runs tools in a loop, right? So you have some goals, you have some context information agents and of in many cases, and then you do do call tool calls, right? And you get some results, and you know, you're basically doing it doing it in a loop.

Right? That's it. Right? There's not not much more. The rest is magic, trying to put it in your use case a little bit more, in the other use case a little, in that direction. So, that's really it. Right? So, pretty please, don't like open the curtain, play around with it. Now, with agents, agent core, this looks a little bit something like this. You have an agent class. This is all TypeScript. You can, you know, ingest all all sorts of information information.

You can prompt it with different information, and also you know, you have an event system, so you know a lot of things that that that are going on. So, small example. This is a CRM lead qualifier. I don't know, I started the CRM use case for my personally, and it it just sticks around. So, terminal interface, obviously, small TypeScript application. Three three files, really easy. And you can see this, right? You have a couple of commands that you can execute, and you know, show me all leads and score them, right?

So, that's what we do. Show all leads and score them. And here you see all these, you know, things that are going on under the hood. Right? You see that that the assistant is calling tools, that you get some results, and eventually, you know, you get some input. Now, obviously, there's tons of things to do, but, you know, I've just vibe coded this away, and it's a good again good learning exercise. The system prompt, you know, as you could imagine, right?

You know, calling out the different tools, that what you do, right? So, all pretty straightforward if you are a building an agent. This is an example of how you inject here, right? So, um we said we want we we do call tool calling, right? We reach out to this and call a specific tool. But, for the agent for for steering it more, right? You know, a typical hook would be before the tool call do something, right? And in this case, we don't want to update a contact uh without, you know, checking something or I don't know.

You can imagine any types of authoritative role-based access, whatever enterprise feature in here. But, basically, you know, uh just before the tool call. There's another one, events. So, we've seen these, you know, uh the stream and you might have seen a little check mark there. Okay, the tool call was was fine and returned some result. So, again, we're subscribing to events. All pretty straightforward. And again, please give it a try.

All right. So, this is simple agents others agents SDK uh are are available. Um and now we're moving through the coding agent. Now, what's what's the coding agent? At the end of the day, it's really the same thing as we've seen uh before. It's a you know, normal agent, right? It runs tools in the loop. But, now we have a runtime and some type of shell, right? Bash is seems to be the the shell that that everyone is using.

But, we have a shell and a runtime to to start executing. And now things are getting interesting. And now the the the magic of of what you've seen with Open Claw uh suddenly shines. Uh um Peter uh uh shared this this example uh on some presentation where uh he uh sent a message to his Open Claw and uh sent a voice message. Now, at that time, Open Claw um and I still don't know if there's any like special plugin, but at that time, Open Claw didn't know anything about voice about voice messages.

So, what what it did is it uh created and used different tools. Um and in the end, one of the tools was uh FFmpeg, right? On the local local machine, and it started this. And this is was one of the tools, right? So, from the outside, it it looks like learning. But in the inside, it's actually just another tool call that is available to the agent. And that's why these things make it so interesting. So, um again, uh the example here, um now, this is a little bit more sophisticated.

But the uh important part, and and this is the extension API, and you know, please look it up online. We're We're going to do two things, or the the things that I'm most mostly interested is in in session events and UI interaction. And yeah, uh uh look it up online. But here's here's the the actual extension. Now, again, this is what you would in a coding agent, you probably just generate by asking it. But here, if if we have a look, um this is a CRM uh TypeScript uh a small snippet of it.

And basically, what we're now doing is we're doing the same example as before, right? And we have a new command called pipeline. Right? So, if you have the slash commands, and you have a new new command called pipeline. And now, we are able to we're loading all the contacts. Um and uh you see this little in um uh don't have the lines. Just below step one, uh you can see uh context UI select. Right? So, all of a sudden, we're not only interacting with the back-end systems and and sessions and and of those sorts, but we're also interacting with the UI.

And we're able to select. Right? And that that's got got me thinking. Um so, right? So, you have this this command, and again, this is now just the coding agent, right? We're not talking about the core agent last, but but this is how you would load up Pi if you just don't download the the coding agent. And now with this new extension, we have Pi. Right? And we can start selecting things. Right? So, this is a simple simple select here.

Um and you know, you you even even have drop-downs. Now, the important part here is these are extensions and the framework uh that currently Pi um has included is catered towards the use cases of a coding agent, right? So, we you know, there's a lots of work and other things to do to make this ready for others for other types of applications. But I hope you can see and understand the vision where where this is heading.

And um yeah, you know, this is all terminal, right? So, you wonder how would this look like in the web? Um it currently is not possible if you ask Pi to build something. So, I asked Pi to build something, right? And this is the web UI will be a web UI. Same command. Same selection. All based on the same extension mechanism. Now, um there's a refactoring going on to make this a better accessible and make it more clean, but I hope again it shows you a little bit of of where the where the things are going.

All right. Now, um Pi and OpenClaw um is um is a special special setup, right? So, Pi and OpenClaw what we have there um is that that now we're not only talking about like like um a single agent in a single session in a coding environment, uh but now we have a multi-channel uh environment where uh we have um you know, multiple threads going on, multiple agents going on. So, there's a little bit more to it. Um this is um and and the interesting part, right?

That's that's where where I got started is is like if you look into um you know the the packages the core packages of of of Pi all of them are used in Open Claw, right? So Open Claw has this this function run embed Pi agent and it creates a session, right? So sessions Pi itself has a great session support and it creates a session agent and streams all the information back. We have the coding agent which we just talked about.

We have agent core as the other part that we talked about and there's two other minor or major packages a Pi AI for the unified LLM abstraction and a terminal UI interface. There's Open Claw has built its own plugin mechanism and that's because you know, it's a different use case, right? And has different requirements. So you have a plugin support for a multi-channel routing different provider orchestration sub agents gateway support yada yada yada all the things that you know by Open Claw, but it's based around the core mechanics of of Pi and and and leverages it.

Cool. But one thing and that's that's that's the like the the major gist I would like to bring across is like okay, what do we do now with this? What are our other options for us to do? And this is one of the applications we've been building for a client and basically the use case is a sales process. They get requests for proposals of of an ordering another another system, right? parts. parts being sold by that company.

And we're taking all this coding agent, all all of that we're taking away, right? We're we're we're new fresh new thinking, right? And look at the process from the get-go. So, um an email comes in, right? We we we monitor basically that inbox. Then we have some gateway because what we want to do is we want to forward this to different agents. Right? So, here I have um multiple agents, right? Uh the way it's structured is we have one agent per customer.

And that agent has a general harness, right? Agent MDE um Agent MDE as an example, but you can obviously also use different ones. And that helps um understanding the role of that agent. In the specific case, it it tells uh of how to use the system and how to react to certain, you know, inputs, outputs, etc. Now, um the other one is customer MD where where we basically explain the agent like, you know, the specific customer might have, you know, specific quirks, right?

Specific um uh access, specific um um discounts, and all of that sorts. And then, right? And that's what I said like earlier I I like using sessions. Then for each case, right? We're we uh creating and reusing existing sessions so we can back and forth um um know what what was previously talked about. All right. So, email comes in. We're looking at the uh inbox and we route this to these different agents. And now we have tools.

Right? So, we have these different tools uh to talk to the CRM, to talk to the ERP, um and get the right information out of the system for this agent to look on like like behave. Like, you know, maybe it has, you know, new contacts information or or that sorts. And again, we make this available. We make it easy for the agents to access, right? And our way currently is doing this with CLIs, right? So, CLIs our agents are really good at using CLIs, so we make it available as a CLI.

We put We make sure that the data is secure. Uh we have our own sandbox, and then we're creating the drafts again. Right? So, that's the system, and I hope by this point you basically understand like logically where these things fit together, but how would this look like? Um oh, one uh final thing, right? There's always a question around okay, sandboxing etc. And and to be honest, we're on the uh just on the on the steps of of getting there, but if you've seen um Nvidia's announcement uh around um OpenClaw, their policy their open shell is really really interesting, and um um it's it's it's a way of It's one ways of securing an um an agent.

We're looking into this. Please do as well. All right. So, how does this look like? Um to to to kind of like get you an understanding of of how these things, right? So, here's the dashboard. Uh rather uh boring, but here's the in the email the inbox, right? So, again, we see the the email coming in. And yeah, we um it's one of one of many emails. Most of them are ignored, but this one is like the the the LM call said, "Okay, I'm I'm interested in this." And it is associated to a case, right?

We see the case up there. Now, this case is again is an agent session, right? Uh so, we find the session and associate it to it. Um we then create a draft. Uh so, there's tons of calls, which I'm going to show you in a second, but basically the output of all that is a draft email that the user will be able to use, right? So, our thinking is uh let them users stay in in email. Let them stay in the the inbox and drafts, and they don't even, you know, need to do a lot.

So, this is more like an admin interface they can stay in email but basically the output would be a draft generated. And how does that look behind right? We we have the the different sessions before the threads and this is the same thing right? The assistant says apologies German but now I'm looking at the articles it does different tool calls right? It gets gets results and does this in a loop to resolve right? The end effect for for the user is I'm looking at my inbox there's a new email it's associated to a case and I get a new draft which they can freely edit but under the hood we have all these agents working.

All right, that's that it it's for me again. Here here you find the slides key takeaways please coding agents are and will be a core building block for your software systems. I'm I'm betting on it a lot of people are betting on it. So please give it a try. Pi is perfect for tinkering whether you like it or not it's minimal. You can rip things apart and put things together. It's perfect. So please go tinker. Right. Thank you.