I Built An iMessage AI Agent You Can Copy (Open Source) transcript

Two weeks ago I built a custom AI agent for myself. You can message it through iMessage and it is really powerful. It has access to all my tools. It can spawn sub agents to execute tasks and it has one of the most robust powerful memory systems out there. I built this thing just for me and I never planned on releasing it. But hundreds of you guys flooded my comments asking me to open source this thing and you convinced me.

So today I am releasing it. I named it Boot because it's a proactive agent that pings you and it's also partially attribute to my dog. Now releasing this project was substantially harder than I thought. I could just publish the repo, send it to you guys and that would be it. But there was a ton of work I had to do to get this thing open source. In this video I'm going to walk through what it's like open sourcing something, all the issues I ran into, why I basically had to rewrite this entire thing from scratch.

We'll walk through the architecture and some of the design decisions so you can build on top of it and I'll share some of the things that I hope people build with this. If you're new here, welcome to the video. My name is Chris and I build productivity apps. But in this video we're talking about my first major open source project. In case you're new here, let me give you some context on what the agent is. You can check my last video if you want a more detailed version of this.

But I've been slowly hacking away at this agent for 3 months now. It started as a web version then transitioned to a beautiful native iOS app and then I scrapped the whole thing because the form factor I ended up liking was an iMessage based assistant I can chat with. On the surface it looks really simple. You text it through iMessage and it executes tasks. But behind the scenes it is really complicated. There's this crazy custom sub agent architecture so it can do tasks in parallel.

It has access to all my tools. There's a very complex and expensive memory system. Automations that run on a certain interval and a ton more. This whole thing is built on Anthropic's Claude agent SDK which is basically the thing that is powering Claude code. And one of the coolest things, at least at the time of recording, is you can use your Claude subscription with this agent. So you don't have to pay API based pricing which saves a ton of money when you use this.

And on that note, that's the primary reason I decided not to release this. It is incredibly expensive to run. Since I'm using my Claude subscription, I don't have to pay for the tokens, but I I tracking it and it's costing me about $400 a month to run this thing personally. So, if I try to turn this into a service and release it to you guys, I would probably have to charge at least $500 a month just to make sure I don't lose money.

And I can't imagine a world where people would pay that. The other reason I didn't want to release it is because it is very tailored to me. It has specific custom integrations like an App Store Connect integration so I can manage my apps and a bunch of specific skills to make YouTube scripts and Instagram scripts, all stuff tailored to me. So, if I opened it up and had to generalize the agent, it just wouldn't be as useful.

But when you guys asked me to open source it, that was actually interesting to me. Because the cost and the personalization problem really don't matter that much anymore. You would be able to use your own Anthropic subscription or modify it to use cheaper models. And you can personalize it to your use case. So, if you want a LinkedIn integration or you want to use Telegram instead of iMessage, you can go ahead and do that.

I thought this was a great idea, so I decided, "Okay, I'm going to go ahead and open source this." So, I started down that path this week, and that's when I hit a ton of roadblocks. I already have the codebase, so I thought, "Okay, I'll just go clean this up a little bit, and then I'll go release it tomorrow for everyone." But as I was trying to clean up the codebase and going through everything, I realized, "This is pretty bad." I've been hacking away at this thing for a few months.

I've tried like 10 different memory systems. I changed the database schema probably like a hundred times at this point. The code was in such a bad state, which honestly didn't matter to me if I was just using this personally. But if you guys wanted to build on top of it or learn from it, this just was not going to work. It was so bad that even Claude code was getting confused navigating through it. So, I made the tough call to just completely rewrite it from scratch.

And it was actually much faster than I thought because I already knew what I wanted to build this time. All the experimentation was already done, so I just had to execute on porting the things that I wanted over to this new version. That part was honestly not too bad, probably took me like one or two days to do it properly. Finally had this clean codebase that I was really excited to ship out to you guys, and I started working on the readme, which is basically the documentation for this library on how do you install it, how do you use it.

But the more I worked on the readme file, the more I realized that the setup process was pretty terrible. Like just to get this thing running, you need to spin up a Convex deployment for the database, you need a Send Blue account for the iMessage layer, you need a Composio API key for the integrations, you need ngrok to expose a local port so SendBird can send webhooks, and each of these things has their own little setup thing going on.

So, you need to find API keys, provision phone numbers, register webhooks. It was a nightmare to set up when I had to do it from scratch again. And I thought to myself, "Okay, if I was a user reading this readme and had to set up this project, I would probably just give up, to be honest." Then I decided to try to build a custom installer. I studied a few open-source agents, OpenClaw being one of them, and I really liked how a lot of them wrapped the setup in a single CLI command.

So, I ended up building this, and now you can just run npm run setup, and it walks you through everything. It helps you create a SendBird account, which is what's powering the iMessage stuff, provisions Convex, writes your environment variables, which is fine because this is all happening locally. A bunch of very annoying stuff is happening behind the scenes while this is running, but by the end of it, everything is set up with as little friction as possible.

I also made a custom command so that you can run the thing really easily. So, if you run npm run dev, it boots the server, it runs Convex in dev mode, it loads the debug dashboard so you can see all the agents running and everything. It provisions an ngrok tunnel, which is how SendBird is going to be able to talk to your Mac directly. It does all of this stuff in a single command, which makes it really easy for people who are just getting started.

I have never built a CLI installer like this before. The process was actually kind of fun, and I'm proud of what it looks like right now. It condensed a 30-minute setup to something like 5 minutes. I had a clean codebase, I had a really good installer, and the next problem I ran into was integrations. In my version of the agent, I hand-built these integrations. There's a Slack integration, an App Store Connect integration, and iMessage integration, but each one of these took a lot of time to set up.

If you set up a Google integration, you know what I'm talking about. You have to go into Google Cloud, set up an OAuth client, manage refresh tokens, deal with scopes. Every single integration was kind of like this little mini project, and luckily, I've done this stuff before, so it's really easy for me. But then I thought about the average person who's never done this, this would be pretty daunting. I evaluated a bunch of different options, and then I found a service called Composio.

The best way to explain this is Composio is a way for you to integrate like a thousand different tools into your AI agent with just a few lines of code. They handle a lot of the complexity, so you don't have to manage refresh tokens, OAuth clients, like all that stuff. You just have one API key, and then all the integrations are basically built out for you. Gmail, Slack, GitHub, Linear, Notion, HubSpot, they have a ton of different integrations here, and they're all already built out for you.

So, I made a branch, tried it out, and I really liked how seamless it made the authentication process. So, the way it works is when you set up Boop, it's going to ask you for a Composio API key. You type the API key in, and now the integrations will just work out of the box. There's this integrations tab in the debug dashboard for you to hook up all of your integrations, and when you click it, it's just going to work.

You don't have to deal with OAuth clients, you don't have to sign up for the HubSpot API if you want to hook into HubSpot. All of this stuff is going to work just with this one API key, which I think is going to be huge in terms of the setup process because the integrations probably would take you another two to three hours alone if you wanted to install all this stuff yourself. When I was evaluating Composio, the thing I really liked is they had a ton of the integrations I was looking for.

They even have a Posthog integration, a Supabase integration. These are things that I needed, for example, to help do support requests whenever I'm answering emails. Wired it into my agent, and this is how it works at a very high level. We have this parent dispatcher agent, and it is allowed to spawn a specialized sub agent to execute the task. The parent has access to all the Composio integrations that are linked. There's like hundreds of thousands of tools that are at its disposal in theory, depending on the number of integrations you have set up.

And what it does is it picks the best tools based on the user's request. So, if I asked it, "Hey, what do I have going on tomorrow?" what the parent agent's going to do is probably see, "Okay, for this task, I probably just need the Google Calendar integration." So, what it's going to do is take the Google Calendar integration, find the specific sub tools related to Google Calendar, and then it's going to package this up, and then spawn a sub agent that only has access to those tools.

And the reason I did this is for context. I have found that when the sub-agent only has a very small amount of tools, it is able to execute the job way more reliably, rather than if you gave it a million tools and it had to figure this out on its own. We let the parent agent do all of that work and I just found this architecture to be really good. So, I decided to integrate Composio. It made such a big difference because I also just automatically got a bunch of integrations that I didn't originally set up.

And more importantly, the setup process to get all these integrations running went down from a few hours to like 30 seconds, just getting an API key and pasting it in. Obviously, this is just a starting point. If there are integrations that Composio doesn't have, you can still modify this open-source library to go add them if you want. One big one that I have running that I personally decided not to put into this library just for safety reasons right now is an iMessage integration.

Because of the way it works, there is just no world I think where Composio can offer something like this. This is a great example of me modifying the code and then adding an integration that Composio just doesn't support. But again, they had so many integrations and I think this is going to cover like 90% of people's use cases. While we're talking about infrastructure, let's talk about storage and the database. This is how I'm storing stuff like the conversation history, the memories, all of that stuff.

If you saw my last video, you know that I use Convex for this. And a huge shoutout to Convex for actually sponsoring this video. A ton of you guys have been in my comments telling me to try Convex for a while and I'm really glad I did because it made this project so much easier. There were a lot of reasons I chose Convex, but let me go over the three that I think are most relevant for this project. Number one, it is real-time out of the box.

So, when I'm texting Boop through iMessage and it's spawning sub-agents, when I'm looking at this dashboard, all of the sub-agents that are spawning and all the tool callings, all of this is happening in real time. I can watch tools getting called. I can watch the memory system self-clean in real time, which is really cool. All live with no refresh. As someone that's used a ton of different database providers for projects, real time is really annoying to get right.

But Convex just handles this completely out of the box. The second reason is built-in cron jobs and background functions. So, the memory cleanup system I've running every night and the automations that are running every couple hours, all of this is powered by Convex cron jobs. Usually I have to set up some sort of background system to do this, but Convex already has all of this stuff, which makes setting it up for new users really easy.

Zero extra infrastructure required for it. And the third reason, which is going to be huge if you want to build on top of it, is all of the infrastructure lives in code. Which means when you're building with something like Claude Code or Cursor, it has a really good understanding of the back end and the schema. No MCP needed, no documentation, it just understands what's going on really quickly. Which is massive for open source because I think people are going to fork this thing and start hacking on it immediately.

And so having Claude Code or Cursor understand the data model really quickly is going to be important. If you're building an agent or any sort of real-time app, I highly recommend checking out Convex as a database provider. They have a very generous free tier and I'll leave a link in the description below if you want to check them out. That was the infrastructure integrations. Now, the last challenge I faced, and this is one that I just completely just forgot about until I was about to ship, is how am I going to update this system?

Let's say in a few weeks I do actually add Telegram or WhatsApp support. How am I going to make these change and push it out to everyone who forked the code base? Because again, that's how I want people to use this system. I want you to build on top of this. I want you to customize it and make it your own. I checked a few libraries and some of them have some sort of update script, but that got kind of confusing because what if I change the data model?

How am I going to do all the migrations? I was getting a bit anxious thinking about it because I could inadvertently really break someone's system if I do it incorrectly. But then I came across an open source agent called Nano Claude, which is kind of based off of Open Claude. They have a really clever solution where instead of a manual update script, a custom Claude Code skill, which is basically a set of instruction that Claude Code is able to read.

And then Claude itself handles the upgrade for you. It pulls the upstreams, looks at the changes on the main repo, looks at your customization, and tries to intelligently merge the two. I thought that the upgrade system was brilliant, so I decided to implement the same exact thing. And I think it's okay to be reliant on Claude Code in this case because, let's be honest, I feel like if you're using this library, you want to use your Anthropic subscription with this.

And you're probably going to use Claude Code to install this thing for you anyway. So, I took a page from Nano Claw and now Boop has a skill called upgrade Boop in the Claude skills folder. So, when you want to upgrade, you literally just open Claude Code in your repo and you run {slash} upgrade Boop. Parses the change log for any breaking changes and offers to migrate it for you if there are any. Previews what's changing and if it looks good, it will try to merge it all together.

I think this is a really cool way to do an upgrade script for something like this that's going to be customized like crazy. Huge credit to Nano Claw for coming up with this idea. In terms of walking through the architecture, honestly, you could probably just go to the repo, ask Claude Code or Cursor to run you through it if you want to learn how this stuff works. If you want a video of me going over this in depth, I will leave a link to the video I did last week where I basically did that.

Go check out the repo, go read the read me, go ask Claude Code a bunch of questions. That's probably the best way to learn about how this is architected. A couple of disclaimers on the repo because I want to set you guys up for success if you plan on using it. Number one is this is meant to be a starting point for you guys to build on top of. Please build your own custom integrations. Make this thing really specific to your workflow because that's what I'm doing.

Again, I have an iMessage integration, I got an Apple Script connected integration, I got all these custom skills that I've wired in. By making it super personal to me, that's what's made this agent really powerful. Number two is please be aware of the cost. This thing is using the agent SDK under the hood and I'm not going to lie, this thing consumes tokens. I do have cost tracking baked in so you can see what your usage would be if you were using API based pricing.

But for me personally, it's costing about $400 to run with the memory system taking the bulk of that. I'm on the $200 a month max plan, but if you're on the $20 month for example, what I would do is modify the system. Maybe tweak the memory system to not do the cleanup thing at night or use cheaper models for example. Highly recommend making tweaks to the system so that it can fit within your budget. Number three is this is built on Anthropic's agent SDK which is amazing because we get to use the same technology that Claude Code uses.

So, we get a very powerful agent with very little code. But the con of that is we are at the mercy of Anthropic. If they just want to cut access where we can't use our Claude subscriptions anymore, which they have done in the past to be honest, then that's up to them and we have to kind of live with that. So, if you want to build an agent or if you want to commercialize this, which you are free to do if you want, you can totally commercialize this, please be aware that you are building on someone else's platform and they could take it away at any moment.

To be honest, if that does happen, I will probably modify the library and just got the agent SDK and just build it from scratch, which would be a huge pain, but I'll do it if it comes down to that. But yeah, I wanted to be very upfront about the risks though. And number four, this is my first major open source project. Anybody is welcome to open up issues and PRs, but just a heads up, I am trying to keep the code base as light and small as possible so that people can build on top of it themselves.

So, if you open a PR and I feel like it's going to bloat the code base or take it in a certain direction, I most likely will not accept it. But you're more than welcome to try, especially if you have some security enhancements for example. If you want to fork this and you have no idea where to start, here's some recommendations I have on what to build on top of it. The first thing I recommend swapping form factor. If you're someone who doesn't use iMessage and you prefer something like Telegram or WhatsApp or Slack, go and change that out immediately.

This agent gets really useful if it lives where you live. So, that's the first thing I would probably swap. Number two is to add the integrations you need. Compose.ai has over a thousand integrations, but if they don't have something that you need or maybe they're missing something from their toolkit, please go ahead and add it. That's going to make this so much more useful for you. Another thing I would do is tweak the memory system.

I personally highly value preferences and corrections, so in my version, I have these type of memories weighted higher than a lot of other memories. If you don't care about that, feel free to tune the memory system to adjust based on your preferences. And if you know what you're doing, the last thing I do is tweak it to run on a server instead of your machine. I purposely did not add this to the repo because I needed a little bit more time if I wanted to do that right, so I didn't want to rush that part.

But if you do know what you're doing, make those modifications, run it on a server. It gets so much more powerful when it works with your laptop closed because that's actually what I'm doing, but to be honest, I don't know if I'm doing it in the most secure way, which is why it is not in the repo right now. And if you do know what you're doing, feel free to open a PR. That is a PR that I would probably merge in if you do it correctly.

I want to take a moment to thank you guys for pushing me to open source this thing. I genuinely would not have if it wasn't for your comments, so a huge thank you to you guys. Repo's in the description, please check it out. Go learn from the architecture. Go open a PR if I did something wrong that you want to correct. And most importantly, fork it and build something cool and then tag me if you do. Hopefully you guys found this interesting.

If you like this kind of content, check out my Instagram and TikTok. I post almost every other day about building productivity apps. But thank you guys so much for watching, and I will see you guys in the next one.