Stop babysitting your agents... — Brandon Waselnuk, Unblocked transcript

Hello everybody. It's good to see you. I'm Brandon. I work at Unblocked. It's a great place. Uh and my goal is to make it so that you don't have to babysit your agents anymore. Um I'm sure we all have a different take on what that means. What I think of is care and feeding. Basically, agents, whenever you spawn on it by typing Claude in your CLI, let's assume whatever tool you may use, they exist and it's like a brilliant software engineer has just spawned and it knows nothing about what it needs to do, it knows nothing about your org, it's completely zero context in its head.

So, typically what happens is people have to move through building that context, which we'll go through in the beginning. But first, what I'm going to cover today, pretty quickly, is three myths about how you can stop babysitting your your agents. And then three lessons that we learned the hard way building a context engine at Unblocked. So, our product basically provides this context for agents and we'll tell you a bit about how we built that, techniques to care about, and then I'll show you a repo we actually constructed at our workshop yesterday that will be open source at the end of the week, which has one component of what's in a context engine, which you can lift for yourself and bring into your org if you like.

So, not long ago, you were the context engine. If you think about that when you're writing code, you thought about everything, you knew everything, you figured it all out, you were dealing with that. And now what's weird is you're in a weird state where you are actually the context engine for your agents. So, a useful way to think about this is how did you build context when you showed up at a company? So, day one, you had probably nothing, but you're really smart, you finished school, I don't know, maybe some self-learning.

Then over time, you accumulate context by doing stuff at work, meeting market, meeting your team, being like, here's a PR, getting it rejected. All these good things built up a lot of your capabilities. And then finally, you became very good at your job because you ask good questions and you knew how to gather accurate context and shred stuff that it wasn't helpful for you. This is where we're at right now. Most people here, if you look at the bottom, is in the you are the context engine stage because you're either dealing with the early phases of AI, which was just fancy autocomplete, or you're in an agentic IDE where you're triggering every job.

What we see with all these businesses that work with us in their AI adoption is it's usually at a varying level of this. This has been adapted from Basim Eldis' work. Um if you check him out later, great engineer. But basically, this is the type of ladder that we're dreaming for and far on the side is like a dreamy future that maybe Codex figured out, I think. I didn't see Ryan's talk, unfortunately. But everyone else is kind of trying to get there.

In order to move through this, you want to get to the curated context layer. That is typically what a lot of teams are doing by creating static repos, so static stores of a bunch of context that says key things about their company. These can include, of course, Claude MD files, agents MD, that those types of tooling. But usually, people start to put a bunch of other key corporate context into an area that agents can access to pull data from.

The issue with that is those are static content pieces, of course, so someone has to maintain and up them, or sorry, update them. As well as they don't have uh the availability of, you know, actual raw runtime data. There's just a bunch of information that engineers obviously need that don't go into these static layers that you're starting to see, which typically look like a file system. Wow. Context engine. What this needs to do is basically have all that static content, of course, but also be able to at runtime, when a query comes through from the software engineer, typically, how do I implement this feature?

At runtime, it's able to pull all that static source across your entire corporate knowledge corpus, essentially, whether it's any many SaaS apps, different systems of records, and pull in the runtime singles in order to analyze, reason across all those surfaces, all those different data stores, and then run exhaustively to actually find all of the things that are important, and then send a token optimized, aka small, response to the agent with all the details it needs to then execute its next steps.

So, you'll see through this, but typically that means getting the best context up front makes all agent choices and actions after that even better. So, if you give it, for example, a key research packet of like, "Hey, I want to do a new integration." And it you drop a packet so that it creates a good plan, and that information says, "Here's our patterns. We use factory pattern. We do these things." Like, all the things about your organization, it's then able to trigger its background agent jobs to go grep your code base to do the things it needs to with higher accuracy, which means it's more token efficient, and it gets the job done faster.

This is the problem from our our friend Kaparthy. The gap is not intelligence at this point. It is context. Though Mythos sounds pretty cool. Um, so the problem is people think that access is the answer, but it is not understanding. So, providing your agent tools with MCPs, with pipes to different sets, allows it to access that, but you have to remember you day one when you showed up at work, you don't know where things are, and you definitely don't know what you don't know.

So, there's probably some service over there you've never heard of before in your life. You're working on a thing. Your agent's like, "Ah, I did it. I wrote the whole code from scratch." And then your senior engineer is like, "Hey, bro. We have a service." And denies you. So, one thing you'll see is we actually triggered this task, that's a real prompt. You'll see data later in this slide about the outcomes. We did it with Unblocked, a context engine, and then without one, but it had an MCP access to each SaaS tool that was required to get the job done.

And I'll show you the differences. The short story is you kind of get this. The naive run, which just had the MCP access, basically passed all the code checks. It compiled. But the senior engineer was like, this is totally wrong and what it tried to do would have broken our entire system if we had shipped it. So, three myths about building context. The first, if I do naive rag over my docs, that is context. Unfortunately, that does not work.

Naive rag picks a bunch of things, it puts a data store there, and the agent can then crawl across that data store, but it typically falls down because there's something on the satisfaction of search. This is a known phenomenon in radiology, but the short story is, if you get an X-ray because let's say you have a lung problem, the radiologist will scan, and when they find something, they're like, oh, there's something on your lung.

They stop looking. Cuz they think they found the answer to the symptoms you have told them, and this is very bad in medical health, of course, cuz there may be other things wrong. So, what happens with an agent is if you say, make a Zendesk integration, it will go, it might call an MCP, and the first piece of data it finds, it goes, oh, this this must be the pattern. It stops looking. So, the issue is if it's not exhaustive, it will not find the actual root cause, or it may not find the correct best way to implement, and just a bunch of other problems can happen, and basically by the time the agent output is done, you read it as an engineer and go, no, and then you're in a doom loop where you're like, let me correct you, it's actually over here.

I'm going to point to file, and so you're baby sitting again. If I just connect enough MCPs, I'm done. I think I've spoken to that. They're there, they're pipes, that's great, but they don't provide an understanding or reasoning across it. And then finally, we did think this for a while. We dreamed of the 1 million context window. It's here. I don't know if anyone's ever whacked it full with something and then tried to get the agent to do anything.

It can't. Um it basically just can't reason over that much data. It's just not super helpful. It just sits there. There's no entities and relationships, and there's all these things that we need for these agents to be most effective. Um so, So, bigger context window does not solve it. There's a bunch of compute reasons why even if we got to 100 million in a context window, it's still not going to help other than needle in haystack problems.

If you're obviously like find find Waldo. So, basically this is what we see. The classic waterfall. Code that compiles is what the agent can see. But typically today, they miss all of this cuz it can't see it. It doesn't know if it's there. It would have to run for so long wrapping in a session to actually get your factory patterns or other things across your code base that you'd burn up a jillion tokens. And then when you close that terminal window, bye-bye.

You got to just do it again. And no one wants to repeat this cost. So, this is why you need a context engine. It understands who you are and what information actually matters. So, a key component of this is a social graph because you use that as a pivot point. Because if I ask how to do the Zendesk integration, the context engine should know which code bases I work in, where my PR history is, who I work with, and what I mean when I say that.

Because at a large org, we deal with companies that have 20,000 members at our like customers of ours, it's very different. So, you need to be able to reason that. And by the way, a context graph is an incredibly useful technique for building these things. We'll talk about it. It should resolve conflicts. I don't know how many times I've looked at source code that's running in main and we go, "Yeah, that's the source of truth." But there's a Slack conversation where the CTO says that was implemented wrong.

Which is right? Context engine must be able to settle that debate. And but by the way, a con a graph of social graph helps with that cuz if you see the CTO saying in the Slack thread, that's wrong, the CTO is probably right. So, the context engine reasons about that and goes, "Well, the code says this. Slack says this. That's the CTO. We should probably tell the agent what the CTO said. And of course, provide it the source code, etc." So, it passes what's truth.

So, it handles truthiness. That's a tough problem. We have a lot of techniques in our product to solve it. It is not fully solved. It should respect permissions of governments. This is pretty basic. It's one of the reasons this is delivered over MCP is you can carry the OAuth model through for data governance and a bunch of other reasons that matter in scaling businesses. I mean, as soon as you're 20 plus, typically this matters cuz some data should not be accessible to others.

So, when you build your engine, you do not want to put everything in there, especially when you think about we ingest Slack conversations and Microsoft Teams combos. So, if it's you, we will return responses from private chats. But if someone else asks a question, we will never show them private chats that aren't theirs. That's just one easy way to think about it. Oop. And it should deliver the right context, the right model, at the right time, in a token optimized way.

This is how ours works. So, the short is we ingest a bunch of data sources, it sits in our engine, we have six key differentiators, which I'll go into in a sec. And then on the output side, there should be many surface areas where agents and humans are able to interact with the context engine. One of ours is simple, human engineers in Slack just chat with it and ask it questions all the time to get data they need. I'll show you an example.

Um but then of course, agents. As you move to background agents, they need a context engine in order to run headlessly, or to run in the background, or run in the cloud, because they have to be able to ask questions of a machine, not a person. Cuz otherwise, you're not going to wake up to a PR, you're going to wake up to a am I allowed to use this tool? That's not helpful. So, these are the six. We're going to move at pace, but the short is I've talked to a lot of these, but these are kind of marketing terms.

Unified system context, so it should be able to reason across all of your systems of records. It has to be able to do targeted retrieval. Conflict resolution as described, there are many times where the docs and this and that are conflicting, so how do you settle that? That data governance problem, so secure access model. Personalized relevance, building social graphs, knowing who you are, who you work with. And finally, of course, token optimization.

This is becoming a pretty big issue. A lot of benefits you get on token optimization is just by having a context engine because you don't rerun those grips for the agent to know, but also with an engine, if you reason across everything, it's then able to compress the response into exactly what the agent needs and only send that back as an answer. That task I talked about, I'm not going to lie, we asked Claude to do the comparison, so it made these bar charts and numbers, but it did pull out all the key points of this.

That same prompt, one was naive, it had all the MCPs it needed to get the data, and the other had our context engine only. This is the difference across key principles of engineering. But these are funny. It like didn't catch that we used Bedrock as a a fallback. It shipped like bugs. There was one that broke the custom callers. The short story is, if you're working at any form of scale, again, 20 plus, agents are just going to try to mock things, and it'll look like prototype.

It's not mergeable. If you get a context engine, when I put up this PR, our senior engineer for the one with the engine, basically gave me a nitpick and was like, "Yeah, you can merge this. Just just fix that." Great. These are some of our hard lessons. We did try to optimize for access. Now, understand, we're like, "The bottle will handle this." Like, the agent's totally going to figure it out. It'll collapse into Methos or whatever.

It It hasn't. It's been years, so we were like, "We have to solve this problem in another way." And I think that's correct based on actually Anthropic's launch last night with Claude agents and Ryan's talk this morning. You have to get context into the harness. And an engine is the way to do that. We had conflicts instead of solving them. The agent would actually just pick. We found conflicts at first cuz we're like, "It can't be that bad." It's that bad.

So, solving solving conflicts is an important problem. And then finally, this is a really fun one. We thought as good answers happened, we actually got feedback loops on those, so we were caching them for latency. If you cache a good answer, basically, it's like when you write docs, right? The moment you write it, it's no longer valid because things are changing. So, if you cache a correct answer, and then tomorrow someone asks the same question, and you answer it, you probably lied to them now because things probably changed in the 24-hour clock.

So, the system is not the same. Obviously, some questions, I'm sure, are stable, but this led to a lot of problems. So, I highly recommend against even if it's optimized. This is where AI-forward teams are. They're using context engines in all of these cases. So, I know we're all engineers, but I'm sure that we support others in our orgs like a ask engineering channel. Our context engine is sits in every customer's ask engineering channel.

It detects if a question is asked. It scores confidently, and then it responds automatically. So, when support teams, sales teams, whoever is like, "Hey, what's running in prod? What's this?" The context engine just answers them and deals with the issue just like it would answer an agent asking for data. Um so, there's a lot of ways like that use case I just talked about. But then ticket enrichment, triage, incident management, obviously working with your agents in coding, these are all great ways that you get tons of leverage out of getting one of these into place.

Teams then customize them. So, most fork, we have like a cookbook. They take that repo. The cookbook's full of skills. Um but then you devise your skills with your standard operating procedures. You obviously can build your own workflows, whether you do that as a skill or some other technique. And then, of course, custom agents. All of these can leverage that same context engine. So, you just get a huge amount of leverage.

This is what I'm trying to leave you with. An agent should write code that feels like it was written by someone who's been on your team for years. Like, that's just like we should expect that by now, and this is one of those techniques to get you there. I can do a brief Q&A, and I can give a demo. I have 3 minutes left. So, what's your preference? Demo. Cool. So, this is the tool I talked about at the workshop. We're going to open source it.

This is one component of a key of a social graph. And so, when you run it, this will be available, I think it's Monday. We're going to actually open source the closed source right now. There's a whole setup, but we had a bunch of teams hack against it and ship code. But basically, it will build you one of these. I'm going to zoom in. This is our engineering org. As you can see, there's different nodes and edges, a basic graph, but it's a social graph.

Rachine is a goddamn machine. So, hit that's how much he ships is by the size of the node. If we go look at this is algorithmic, procedurally generated, so you'll see the tool. But in short, you can see on the right who he's working with. Sorry, the screen is like that small. Great. Kind of worked. Um so you can see who he works with, whose code he reviews, what area uh he worked in. We use labeling with an API key from Anthropic.

Oh god, now it looks terrible. Let's zoom in. Little better. Um sorry. It's a fun tool. But basically, this expert graph would allow you to when a query comes across, we go, "Oh, you're Rachine. You work with these people. Great." It'll pivot on that data. It'll then zoom into the code bases that he's dealing with. So when you as an engineer ask maybe not the best prompt of all time, but you're like, "Hey, I got to get this done.

There's a bug. Got to fix it." It'll know who you are. It'll pivot on that. It'll probably find the bug that's like correct. And again, it'll use this component to do a bunch of things and make decisions as it traverses in order to reason to give the agent the exact answer that it needs. So this this tool like generates this for you. You just point it at your code repo. It'll do a construction. Um but it it does things like creates experts across like various areas in the business.

So in our libs or services. Quick quick scrolling, apologies. We got time. You can check a heat map grid of who works with each other, like who reviews what, who authors what. You can check peer tables, you know, who Andre works with, etc. So this data available in a context engine, very useful. And again, this will be open source. I think you got a badge scan. I'll I'll just email you all as soon as it's open source, if that's cool.

Another quick demo, Ghosty, my my So in this one, I actually used our MCP and I just straight up said, "How do I can make a new First Class integration to Zendesk?" I just said use the MCP. It would probably have picked it up, but I want to make sure for this demo that it did. It ran It did chose to use our research task tool. You can see that it constructed a query. So, the agent did that based on the shape of our MCP.

So, it wrote the right query. Ran that. It did effort high for reasoning. It got the data back. Then it triggered its explore agents. This is key cuz now they're exploring the right place after this research packet came in. Great research results. Did the thing. Did the thing. Wrote me a plan. So, if you look at this plan, you do not know my source. That's fair. But, if we just scan it like it found all the things that matter like registering our provider obviously.

We have a factory pattern. Um I'll just pull through but like the library modules client. Like this is like one hell of a plan is the short story. And it's like pretty correct. I would probably prompt this a couple more times to get it totally right. But, at any time while it's executing, it's able to keep calling our MCP. So, typically what we see is use the engine for planning. Run execution. And then basically as you get to code review, leverage your engine again.

Cuz that engine is very good at code review and it's extremely good at planning. That's all the time I have. So, thank you all very much. I appreciate it. Uh I'm at a booth at G16. So, come by if you have questions.