How I Built One Backend to Run My Company transcript

So, over the past months, I've been automating more and more of the processes within my company while simultaneously getting rid of a lot of the tools and subscriptions that I was using for automation. Tools like Zapier, n8n, and make.com. This is now all possible by building a custom backend using AI coding agents, whereas before this would just not be economically viable. So, in this video, I'm going to show you how I've set up my custom backend that runs pretty much everything and also why I think every company needs this.

And also, surprisingly, how it uses very little AI in the system itself. And if you're new to the channel, welcome. My name is Dave Abelar. I'm an AI engineer and the founder of Data Lumina, which is an AI development and education company, and we use this exact system that I'm about to show you to do over seven figures in annual revenue. So, let's dive in. So, first of all, let me start with an observation because I think, in my opinion, not enough people are talking about this.

Everyone is obsessing over the coding harnesses using the latest model, building skills, and which skills to use. If you look at social media, you open up YouTube, if you're a developer, this is all your feed is about. And that's cool, right? I use those tools myself as well. I use it to do the work for clients and to, like, literally build this system that I'm about to dive into. But, I have also feel like there's this over-obsession on the process and the tools around it that we forget what to build in the first place, if you get what I mean.

So, that is something that I have observed where if you look into how to actually get AI moving and implement AI into companies, there is so much low-hanging fruit in the form of simple custom software replacing tools and connecting tools with custom scripts that 2 years ago was simply not economically viable. Like, it just wouldn't make sense to build a custom script for that, but now it is. So, that's a that's going to be a big focus of this talk and what we're going to dive into because this is really what enabled me to build, maintain, and run the system and now also what we're experimenting with with our client base because we've done a lot of the LLM pilots, right?

Everyone wants to stuff LLMs into everything. And some of them were great, some of them have been running in production for 3 years, but also a lot of them failed. Not because really the technology wasn't there yet, but because the use case wasn't good enough for it. And we figured out later down the line that if we started with something way simpler, for example, just connecting two systems together, no AI needed, we would have helped that client way better than doing the 3-month custom LLM pilot where eventually we figured out, "Hmm, this is this is not really really working the way we've intended it to be or just we're we're out of budget." So, where does it bring us?

Well, if you look at the types of automations, things that you can automate within a company, right? Because that's the goal. We're building an automation, we're building a system that runs on its own. There are really two types of automations. That is namely trigger-based automations or automations that run on a schedule. So, every company has this, right? So, some examples over here. Let's say there is a a new payment is coming in in Stripe.

And that needs to be added to let's say to airtable or it needs to be added to the accounting system. Or someone books a call via your via your website. They want to work with you. They use Calendly. And now that is uh there is a deal prepared for the sales rep to like look into the CRM system. So, a form submitted, you get it, right? You get the the onboarding flow. But there are also work in every company that runs on a schedule.

So, every Monday morning 9:00 a.m., we need a pipeline report or every night we sync some orders in there. And now every company has a ton of these and in the future, this will only get more and more and more as we get more AI first, as we get more data driven, as we build more automations. So, the challenge in the beginning, which I also had to go through in my own company, which is small, and I am an engineer, so imagine if this happens in larger organizations and people that are non-technical, you have to go through what I call the plumbing tax.

So, in the beginning, even if you want to automate something really simple, there is a lot of work that needs to be done in order to get that right. And that's why for the past decade or so, tools like Zapier and I 10 and make were so great, because they solved the plumbing tax. Meaning, you could just go create an account, get on a subscription or even a free plan, and then just select the tools you wanted to use. So, let's say you wanted to like trigger something, a new payment in Stripe would come into an airtable database.

You would just like log in via OAuth in Stripe, same for airtable, or plug in your API key. You would select the right trigger. You would had two or three steps in a nice and fancy UI, and now there's your automation. And even I did this in my own company for the past years, because it just wasn't worth it. Even though I could theoretically automate this with Python, for example, it's just I couldn't be bothered, you know?

You need to set up a server for this to run on. You would need to build a custom back end and have APIs. You could use fast API for this. You would need a deployment strategy. So, that if something changes, then and you push it to get the whatever, it gets auto deployed. So, that's the CICD. You need authentication and API keys that need to be managed securely, secret handling. You need the the webhook endpoint itself.

Then you need a way to test, debug, and validate all of this. So, 2 years ago, this would mean you would probably spend more time on this single task trying to automate it that it would ever like cost you to do it manually. This was just always a very big thing and it still is, but something has changed because paying this plumbing tax now has actually become worth it. And you only really have to do it once if you set it up right, if you set it up pro properly.

And that's what I've done for my company so that now every time I want to add something new, I want to add a new automation, literally like 99% of the work is already done. So, let me let me show you this graphic over here. So, first time you want to automate something, like I said, you need this, everything or here on the left. And this will this will take you some time to do it well, but if you set up the right structures and use the right design patterns in here, every workflow after that that you want to automate, let's say, oh, now I instead of like connecting Stripe, I also want to connect Calendly and then put that into Airtable.

Now, that's one prompt away because the infrastructure is there. So, that is now changed and that is something that even though AI coding agents have been great for a while, it was always something that I kept postpo- postponing, postponing, postponing until a couple months ago where I said, "Fuck it. I'm going to do this. I'm going to build this platform." And now it's the best thing ever and I'm so happy and so glad that I that I did this because it has drastically changed the way how I run my business because everything is now one prompt away essentially.

And so, with this, it has also really changed my perspective on AI transformation and at Data Lumina, how we help our clients. You see, most of what you find online and what you hear about AI nowadays is what I call bottom-up transformation. This is the personal AI transformation. So, what does that mean? Well, it starts from the person level. So, one person gets faster with an AI harness, a coding tool, Cloud Code, Codex, whatever tool that individual gets, they can use that when they are behind their keyboard.

And I know with these harnesses, you can send things off to the background and it can run in the background, but still it is very much user-centric, right? So, you increase the personal productivity. And that's great. But, there is also a whole another world and that is top-down. Top-down AI transformation where you really start at the top level, start to think about how is this company built up? What are the systems, the processes, the people, and the information that flow through this company?

And how can I work that all the way down to some degree, it may be not even using a person in the loop at all, right? For some some processes. So, the company runs one back end at least in the beginning, right? Of course, it depends on how big an organization is, of course, at some point you want to fragment, but the whole idea is that there is this centralized back end that you can run things to. And every person and team connect to it.

So, it runs without anyone present. So, this is really a key distinction. This runs in the background. This runs as things happen via triggers and schedules. And now, for a complete AI transformation, you need both, right? So, you need both bottom-up and top-down. So, yes, I use tools like Codex and Cloud Code, but I also am constantly improving this back end system where I offload more and more of the work to, so it runs in the background.

So, let's talk about the architecture before I dive into the individual components and show you how this is built up. So, at the core of the architecture is what is called the event-driven architecture or event-driven pattern. And I've linked a blog post if you want to know more about what this means there. Um I'm not going to go into all of the details in here, but it pretty much means whenever something happens in your system, you want to persist the event first.

So, it gets stored in the database. You can look at the status. You can see is it executed? Do we need to retry it? Is there an error? So, we persist everything and with that also create a ledger for us to look back into. So, that's principle number one of this system. It's event-driven. Now, also everything that I'm explaining over here, you can swap that with all kinds It's um I should say it's technology agnostic.

So, all of the names and items and terms that you see in here, you could build it in pretty much any programming language. I just like to use uh the Python ecosystem. I'm an AI engineer. I have a data science background. So, that's just my take on it. So, let's look let's see at the architecture of how I've set up mine. So, first of all, I connect all of the tools that I use, whether that's Stripe, Calendly, um my website.

I set up webhooks and API calls. I have my domain hosted on CloudFlare and then the deployment of where all this lives is on a Hetzner VPS. So, I just rented a big and beefy server where I just put everything on. So, there is We'll get more into like what's running on it, but it just also contains the Grafana dashboards on it, the whole like Docker Compose stack that runs the whole back end. One server for my company that is enough.

So, that is of course protected with a firewall. I want to be very strict and protect as much as possible, so I can only access it with a VPN and with a dedicated IP address. And then, here's the high-level overview of how that works. So, Caddy, we use that for the HTTPs and the domain setups and also to let certain IP addresses through. Then, the whole central layer is built around FastAPI and Celery. So, in the Python ecosystem, these are two things two libraries that you want to look up.

We use a Redis queue, so this is where the event-driven architecture comes into play. So, this is where we queue the work, and then celery workers pick that up, and there's a whole Grafana dashboard attached to it as well, as well as Sentry for like application error logging and handling. So, a lot of terms over here, but I just quickly want to show you this high level, and then everything is persisted in a Supabase database.

So, I use the cloud version for this particular deployment. You can also self-host this, but again here also very strict IP rules set up. And then, of course, we connect to all of the external services, whether that's Stripe, Slack, Close, or CRM, or other LLM APIs. So, that is the high-level setup of this. And as you can see, this is a lot. There are a lot of components to it, and like this is pretty much just the tip of the iceberg.

But, this all is what I what I would call this falls under the plumbing tax, because all this was mostly a one-time setup. And now, now that I have this, adding something to this is as easy as I'll show you that later, but adding like a different workflow in here. That's how easy it now is to start adding things to this. Okay, so now that you understand the high-level architecture, let's dive into the components that make up this system.

And there are six broad general categories. So, that would be intake, state, execution, integrations, operations, and delivery. That is all the stuff that is running through this. And what I'm going to do is I'm going to give you some examples of file structure. So, how are things structured, right? Rather than going into individual line-by-line levels, because the thing is I don't think that makes sense, because you can build it in in any language you want, and once you understand this and have this document that I will make available to you, you can use just use this within your AI coding agents.

Cuz the thing is like the code and the implementation itself is not the the secret sauce anymore. It used to be, but now it's the architecture. So, let me give you an example of how I use this so you can understand how all of this works together and how you can potentially map it to a process within your company or to something you're building on your own. So, with our Data Lumina Academy, we literally have thousands of users.

We have a lot of free products where of course the majority of the people are but also paid programs. And there is a natural ascension path between all of that. So, you can imagine on the back end, there is sign-ups, there is payments, there is emails, there is CRM states, there's people like asking questions. So, that is all running on the back end of our Data Lumina Academy. So, if I pick, let's see, one of our products, which is the Gen AI Accelerator.

So, this is a six-week track, six-week AI engineering course that teaches my exact methodology and approach to how we build and ship all of our client projects. You'll get the exact code base to start with the boiler plates. That's what this course is about. So, if people go to this website and they decide to, let's say, to purchase this, then their whole journey starts. So, let's take you through this example of of what that looks like.

So, in here, let's see, if I go to the API, you have the webhooks, you can see I have a file called polar.py. Polar is our payment provider similar to Stripe. So, if, let's say, if a a user comes in and let's say does a purchase here, so this is, let's say, a purchase, and the purchase happens in Polar, so it starts on the website, so this is the website, all right? And then in Polar, how I've set it up is that every time something happens in that system, like an order is created or purchase is created, I've configured it to send a webhook.

So, this will be a webhook, and that hits our custom backend. And primarily, it hits this polar.py file over here, how we've set it up, it hits that. So, this goes to our let's say our own backend. Once it hits this, as you know, it is also right now because we use the event-driven architecture, it is first the event is a pretty much uh persisted into the database. So, this is beautiful database, right? So, if we go back to our architecture, what I just showed you is from a webhook to the Fast API endpoint, and then there's a Redis queue, it will be stored in the database, and then a celery worker picks that up.

So, once it's in there, from here we continue. So, this would be celery. And now, we actually execute the Python code. We execute the workflow. So, in here, we have the workflow, and this can pretty much be anything. So, the whole setup is very structured, and now in this workflow, we could have any type of process, however you want to set it up, right? So, in the case of let's come back over here, if we come in here, where we have everything nicely organized, you can see the workflows, and if I come in here, there should be the GenAI product purchase.

So, here you can see, if I click in here, let me zoom out. This I can probably show you this, yeah. So, here you can see the whole workflow that executes when someone purchases either our GenAI launchpad or GenAI accelerator. And you can see there's this schema over here, right? It's all very structured. We have a workflow folder, then a folder, and then there's an init.py, and this has a workflow schema. And this is the key to it, right?

This This initial plumbing that we've done to make sure every automation and every workflow can fit into a very rigid structure. So, not that every time you ask your AI agents to build another automation, it like the try it tries to decide where to put the file and which patterns to use. No, everything in this code base is super rigid. It all uses by identical schemas, very strict validation, and this is part of the system that we've been optimized for literally the past 3 years, and also what we use at the core of all of our client builds.

So, here you can see what happens is we first of all, there's a normalization step in there. Then, Drip is our email marketing platform. So, we make sure that we update the status within the email system. So, people will get the onboarding email, right? They get a tag, they get the onboarding email. Then, in this schema, you can see within the node config that every node is connected to another node as well. So, after the email platform from Drip, we go to Circle.

And then, also there's this description. So, this invites the academy member and applies the product tag. So, Circle is our platform where we host all of the courses. So, people need to be invited for that as well. And then, you can see we go to Close. Close is our CRM. So, this is where we keep track of everyone who enters our world pretty much from free to purchases, so we can keep track of all of the accounts. So, we create a record in there.

And then, also we track all of our sales in airtable. So, it then connects to this airtable node. And finally, you can see the final layer is a Slack notification. So, this then notifies me in a dedicated Slack channel that I've set up for this system as well to, for example, get a notification when someone purchases one of our products. So, that is just one of the many workflows that we have running. So, whenever you kick off a workflow like in between here can be pretty much anything because you're just in in the Python ecosystem.

And the sky's the limit here. And if you set up that whole beginning in a very structured and rigid way, and you also have a definition for how you create your workflows, now putting something new into this, other workflows become super easy. So, those were the trigger-based setups via webhooks, right? But we also have the schedule-based workflows. So, here you can see one cron job, which we execute via celery workers.

And you can see this runs every morning at 7:00 a.m. And what does it do? Well, it does a sales sync. So, this synchronizes two systems. And if we go in here, you can see all of the workflows again. So, now we come back into the workflow engine again, but we have a different type of workflow. So, what does it do? Well, first we have our two programs. So, we have our data freelancer program and our GenAI accelerator program, and we put all of this together into a dashboard, so we can pretty much keep track of all of the revenue numbers on a daily and also a monthly basis.

So, this now does a little bit of the plumbing and also essentially becomes your own like data platform or data layer, where you can just like on a job pull data from a system, put it into a visualization tool or anything like that. We just use Airtable for this. So, we can have some nice views that we can check out there. So, with that through two examples of what I actually run in my company right now, I've given you a behind-the-scenes look of what the intake looks like, the state, the execution of things, how we integrate with things, the operations and the delivery behind it.

I know this is high-level. On this page below, if you want to learn more about the specific types of integrations that I run and how I've set things up, you can read this page over here. Or what you can also do if you want to get a behind-the-scenes look of how this works, you can go to our website. I'll put the link in the description and you can download our proposal framework. So, there's two things. This is completely for free, so this is the proposal template that we use whenever we take on a new software development or AI project.

So, if you're interested in working as an indie developer, freelancer, consultant, anything, you want to take on paid work next to your full-time position, and you've never done something like that before, then you need to learn how to write an effective proposal. This will show you exactly how to do it. And also, on the back end, you will be put into the system that runs and delivers and sends you the emails and all of this.

So, like it's twofold, right? You can get a sneak preview on, okay, how do we run things? How do I send those emails, etc. But you also get the actual proposal template. And with that, we have come to the end of this video, but I have one question for you because I do plan to get into more of the individual components that you see over here and create dedicated videos on some of these topics because I think not a lot of people are talking about this.

Everyone is obsessing over AI coding agents, skills, whatever, and I'm pretty tired of that. I use very little skills. I just use the tools, and I I just don't understand how all of the people keep so obsessed with all of these videos. So, I want to get back to some real engineering, and let me know which of these topics you are most interested in, and then I can potentially do a dedicated video on just that. So, with that, we've come to the end.

Thank you for watching. If you found it helpful, please leave a like and consider subscribing, and then I will put a video on the screen here next that I think you will like.