Fighting AI with AI — Lawrence Jones, Incident transcript

[music] >> Um hi everyone. Uh so I'm here to talk today about how we use AI to manage the complexity of the AI products that we build um at incident.io. Uh and to share with you some of kind of the tips and tricks in the internal tools that we use when we're building um our AI SRE product. Uh but first I guess like who am I? Um so I'm Laurence. Uh I'm a founding engineer at a company called incident.io. Uh so we build if you haven't heard of us, uh we build an incident response management platform.

So we're used by companies like Netflix, Etsy, Skyscanner, and actually probably a few of you in the room. Um we page you when things go wrong and we help you run your incident. And as you're running your incident, we help you communicate with your customers. Um but like you might be thinking like where does AI actually come into this? Um and actually that like we don't just want to help people respond to these incidents.

Um our goal is actually to fully automate kind of production investigations. So whether or not it's a big incident or if you just have some ticket and you wanted to look into production, uh we want to be the place that you turn to ask us questions about actually what's what's going on. Um now it turns out we've been building this for about a year and a half, 2 years now. Um and that's actually like a really big ask and the systems that we've had to build to try and support this have been really quite complicated um and have been kind of on the edge of what you can do uh with all of the AI technology that's out there.

Um and they often pose a challenge for humans to debug them. Uh they are now complicated enough that you can't as a human really tractably dig into how these things are performing. Uh you need assistance to help you. Um so for example, this is a kind of uh one of the investigations that we would actually produce for you uh when you have an incident. Uh we will end up running uh right at the start of the incident this investigation which will go through hundreds of telemetry queries.

It's going to look at your logs, your metrics, your traces, any historical incident data that we have, um and it's going to try and cross-reference this with your code base and go, "Hey, like, I'm pretty sure that the problem is this and you should probably do this to fix it." But, I want to pause here and go like, how would you, if you were building this system, how would you actually figure out how to tell me if this was a good or bad report?

How do you know if it's right? How do you know if it's wrong? Um and there's a lot of things that you might do. Uh you might jump into the incident, you look at everything that happened, you might look at the postmortem if there was one that was written. Um but, all of this might actually take you a really long time to do. In fact, it normally takes you like an hour or so to get a real full understanding of this incident.

Uh and it's only at that point that you could then look at this investigation and go, "I think it's right." or it gave me the information that was really, really useful. Um and as I said, behind this investigation is like hundreds, if not thousands, of prompts. So, how on earth do we scalably understand how this system is performing, especially across all of our customer accounts when they all have very different things going on?

Um you end up with a lot of stuff, a lot of AI, and you've got to use AI to try and actually tractably get a handle on this. Um so, I actually did a talk uh a year ago um at LDX about becoming AI engineers, uh where I went through some of the core constructs that hopefully like a lot of you in the room, given that we're an AI engineering uh conference, are familiar with. So, things like prompts, evals, scorecards, traces, datasets, backtests.

Um this talk is going to be about, if you assume that you have these constructs put together and you're building these complicated AI systems, um how can you use AI with the internal tools that you use to understand them to get a better handle of how your system is performing? So, yeah. Um in this talk, I'm going to talk about how you can use AI to help you manage and curate your eval datasets and making it easier for you to work with them, uh making it easier for coding agents to actually work with your eval tool.

Um I'm going to talk about like probably what was the biggest unlock for us when we were building these systems, which was starting to translate the UIs that we built to try and debug them um into downloadable file systems, which has actually helped us massively using tools like Cloud Code and Code X to dig into how the system is performing. And then I'm going to talk about how you can build kind of repeatable analysis pipelines using AI agents to run through them.

Um but first, evals. So, evals for me are AI unit tests. So, each eval takes a prompt and it goes, "Here is an input data." It runs the prompt, it gets the output, and then it has some grading criteria that says, "Does this eval pass? Does it fail?" And for us, evals live in YAML files right next to all to like our go prompts. So, we do everything in go incident.io, including all of the AI work that we do. And this is how we prove when we make a change to a prompt before we ever go and merge it that the prompt is actually going to do the thing that we want it to do.

Um so, for us, this is what a prompt looks like. This is kind of a contrived prompt. So, I would hope no one actually has this in production anywhere. It takes a message and it tries translating it into pirate speak. So, really simple, but silly. But, what we do for evals is if this is the prompt, we would then define on the left some grading criteria for this prompt where we'll go, "There are two things that we care about.

We care that the result actually looks like pirate speak, and we are going to care that the meaning is preserved between the input and what we actually produce as an output." So, this is actually what we're going to use to tell us if the eval passed or failed. And then we have the evals up on the top right, which is just in a YAML file where we go, "Here are three different test cases." And we'll run through them, and you can see the results of us actually running this on the bottom right.

So, this works, and it works really quite well, but it does have some problems. And I'm assuming that several people in the room have kind of come across these themselves. So, first, um like evals are really, really fiddly. Setting up realistic test data for your evals, if you want to actually understand how this stuff is running, is is difficult to do. And especially in our environment, our production evals are including almost an entire incident.

So, you can imagine a full incident report, and that's the only thing that can trigger the bad behavior. This is kind of hard for you to end up pulling them down and putting them in your eval test suite. Um they just become extremely unmaintainable very quickly. Uh now quite early on we created like this little button that we have that allows you to like steal an eval from production. So, if anything was going wrong inside of our like AI interactions, you could go in and you could just pull that down, put it in the code base, and you could run the eval against it.

Um but the thing with this is like production evals aren't like great. If you think about evals as kind of like a unit test uh for your prompts, you want a unit test suite to be reasonably understandable. Like an ideal unit test is very focused and just says I expect it to do this thing. You don't want to have like 2 MB of all the YAML associated with it. It's just really really hard to work with. Um and what we found was as these YAML files for the evals were growing really really large, uh our coding agents weren't able to work with them.

So, if you want to do a quick like read and like modify the eval suite, you'd be boosting that into the context, and you'd quickly hit your context limit, uh which is obviously a problem cuz then you can't work with it effectively. Um so, what we ended up doing was we ended up creating a small CLI tool that we call eval tool um that was designed to allow agents to leverage our eval suite files. Um so, it's just a small CLI that can go what test cases do you have in here?

I want to edit one, I want to replace one, I want to add one. Um and it was by doing this that we allowed agents to work effectively with our eval tooling, and that's why we were able to create this runbook to the right, which is actually a runbook that's designed for um a coding agent to use. So, either a runbook or a skill, it depends on how you want to package it. Um but the cool thing about this is that now that agents can work with the evals, you can end up in a situation where you just ask your coding agent to go, "Hey, I've got a problem here.

Can you look at this prompt? I want it to do these things." And the coding agent's going to turn up, and it will create an eval case where it proves that the thing is failed and then it will go modify the prompt so that the eval now passes and then it will go through this run book and one of the most important stages for us is checking at the end that the change that you've made to the prompt hasn't ended up breaking any of the other evals that you had in your test suite and we also have like a final pass that tries consolidating the prompt as well cuz if you end up doing this repeatedly you end up with a prompt that is massive and really really difficult to maintain so you kind of want every time you make an adjustment to try and simplify as well.

Um so this is actually like like really well for us. Um and you can see here this is like me using it in core code where you can just point it at the eval and say hey have a look at the prompt. This is a real prompt for us which turns human queries into log queries for a low-key system and it ends up racing through and goes and adds a new eval it checks that it passes with a certain number of repeats and then it gets to the end and it's like yep I think I've added it kind of the pass rate is acceptable you can go ahead and get going.

Um but the problem with this is like this solves one problem um and the problem that it solves is that if you know what the prompt is that you want to change you can now change it very reliably and that's very useful if you're working on these tools but one of the biggest problems that you have now is that if you're building these systems you'll know that they're not just one prompt anymore. Um in fact most of the production AI systems you will use on a daily basis are many many many prompts.

Um and to illustrate this problem I've taken our chatbot. So this is the chatbot that you interact with inside of an incident and what I've done is I've created a graph of all of the different prompts tools agents and everything in the hierarchy that powers an interaction with our system. Um so you can actually see there's like 10 different agents there there's 50 I I don't even know there's it it's actually bigger than this I couldn't fit it on the screen.

Um there's a lot of stuff so even if you think that you know even if you've got a bad interaction that came in from a customer you don't necessarily know which part of your system is actually the problem and which part to go change. Um so even if you have this eval red green cycle, you're going to struggle to know where to go to fix it. Um and this gets even worse for a system like our investigations. So, if you think about trying to run through this process to debug what's going on an incident, we have a ton of stuff that goes on uh inside that system and you can see all the steps on the left.

Um and each one of those steps unpacks into the trace that you have on the right and like really it's not about the details here. Um it's more about each one of these green blocks actually expands into possibly hundreds of different prompts and hundreds of different tool calls. And at any point, if you make a slight subtle error, you can't then easily trace through the system where the error originated even if it ends up resulting in you having totally the wrong picture on what you think that the incident was and your RC is totally wrong.

Um so like we built these UIs so that we could help humans look at them and they've been really good for humans to look at them, but I guess going back to what I was saying before, um we just don't have enough time to go through this stuff. So, the problems that we had was we have all these UI tools, but agents can't properly use them. So, how do we get to a place where like the agents can use the tools properly? Um and I think Anthropic stumbled on this with Claude Code where um they found kind of when they released Claude Code that these agents are fantastic at using file systems and just going through this data using standard tools.

Um so we kind of thought like, can we just download all of the UI that we have as a file system? Um and that's kind of what we've done. So, now for each different AI systems, you're able to download all of the content as a file system and we drop that into a sandbox Claude Code uh at which point you can just point Claude Code at it and go, "Hey, I've got a problem here. It's behaved in the wrong way." It can see everything that went into all the prompts, it understands the structure cuz it's self-documenting.

Um and then it can tell you because you have access to the code base as well, uh exactly where you should actually be making the modification to try and change it. And then you can lean on that red green cycle from before to try and modify prompt if you need to. Also, there's more stuff that you can put in this than you might think. There is really not much of a limit as to what you can put into ASCII. Um so like traces like this can get translated exactly from how you would present them in the UI to a text file which then the LLM can consume in a really nice way.

So yeah, this is something that has turned the way that we debug our application into we hear that there's a bad experience. We end up downloading that interaction into a sandbox called code. You sit there in the session and you go, "Hey, like have a look at this. Tell me what you think has gone wrong. Like like what is your interpretation of the problem?" And then you go, "I really want it to do this instead. Like what part of the system would you change?" And then it will work its way through the hierarchy of all those tools and prompts that you just saw and it will be able to tell you where you should be making the modification.

And then all from that session because you have access to the code base, you can just go, "Hey, can I make that change?" And then you can prove it using the Eva runbook that I mentioned before. So yeah, like we've implemented these file system packages for a load of different AI interactions for us now. So it's really easy for us to just drop this in Cloud Cove and just get going. But we now have another problem, right?

Cuz whilst you can do this on an individual basis, we are running thousands of of investigations across hundreds of our customer accounts. And we're doing that daily cuz we need to know if this system is getting better or worse. So what you can see here is we have what we call a back test which is essentially a batch of investigations that we run on a daily basis against our account and against a load of our customer accounts as well.

And eventually you just get this rolled up number which is like, "Oh, cool. 86% accurate RCA on our account." Which is which is great, but like this doesn't really tell you why the number went up and it doesn't tell you why it went down. And if you want to improve the system for someone, you're going to struggle. So what we've actually done is we've allowed ourselves to download kind of all of these investigations into a file system that we can then provide into an analysis pipeline that again is leveraged or is run using Claude code that can end up running a structured analysis like with markdown playbooks that help you run it repeatedly and reliably each time.

Um so what that actually looks like is we created this repo this repo called scrapbook and inside of scrapbook we have this like very structured flow that explains exactly how a coding agent should go through all of the information that we've gone and downloaded, how it should understand these investigations and the process that it should go through to actually run them. Um now the key things that like I think are very important to these flows are you start and you actually parallelize out all of your agents.

So you start maybe 25 agents in parallel and they can all individually build their analysis of an investigation and then you go into the next stage of the pipeline where you do some cohort clustering and you look at like meta points around like what are the same types of failure, how do we go wrong in different ways. Um and by clustering it together you end up with actually like a really really useful report that doesn't just tell you how this has gone wrong but it tells you why is your AI system performing well or badly on this customer account and actually like what should you do to try and fix it or improve the system.

Um and like this is this is like something that we've done several times over for several of our systems now and I think it generalizes really well for anyone who's building this type of thing. Um so the points that make like a really good pipeline for this you should leverage sub agents to do that parallel per entity analysis. You should store all of your analysis in files inside these downloads so that you have like incremental analysis built as you run through it so that you can start and resume the analysis if you have need to.

Um and then you want to combine this analysis with the code base that is actually powering the system. So that if it finds a problem it can look in the code base and go hey I think that this is the problem and this is the place and it can actually do some analysis to go I think that you should change it like this. Um and then at the end because you have this all loaded in your code session you can just go fix it and ask the coding agent Claude Code, Code X, whatever you use to actually go make the change and then you can use that eval red green process to actually confirm it works.

Um and then like yeah, this is a PR that is created after you do something like that where because the back test showed a couple of investigations that were going wrong and I knew exactly what what the problem was, um I could have a chat with you about a feature that we might change in the system. Um and then we can deploy that and we can test that out in production and see how the thing goes. Um so yeah, that's it. Uh so um the key thing for me is like these patterns do generalize.

So for any of you in the room who are building kind of complicated AI systems um and you're finding it really hard to understand them or debug them or evolve them, um you you really need to be using AI just as effectively in your internal tools to try and understand these systems and grow them um just as you are in the products that you're building yourself. Um so yeah, make sure that you prioritize any of the debugging tools that you have so that they work really really well with the coding agents that you're leveraging on a day-to-day basis.

Um file systems are exceptionally good agent context. Like we could have put an MCP on top of this or use like human use uh agents. Uh it wouldn't have been half as effective as this ability to just download in bulk all of the information that you need so that the coding agent can grep through it and find the details. Um and then yeah, anytime you are performing complex analysis, look at creating an AI run book for it instead.

Um it will save you literally days or maybe weeks of your life. Um and then one final point for me, um like we are hiring. Uh we we're in London. Um we have just done a fairly big raise last year and we're looking to expand the team so that we can build some of these systems. Um so if any of this work looks interesting to you and you're interested in being on like the edge of building some of this AI like AI SRE product, uh then just get in contact and let me know.

I'd love to chat. All right, thank you. >> [applause] [music] >> Mhm.