What Is Harness Engineering? transcript

Hello everyone. Welcome back to my Code to Care series. Today I wanted to talk about one of the buzzwords we see out in AI, which is harness engineering or AI harnesses and explain what these what this is referring to. And you know, the AI community and I apologize for this comes up with new buzzwords every every few months, but sometimes they refer to something new so we have to come up with a term to describe it. So I will describe it for for you.

And I'll just do this with history. If you recall when chat GPT came out originally, there was this concept that got developed called prompt engineering. And companies were looking for people with prompt engineering experience and it became a little mini discipline of how to prompt something like chat GPT. You know, you might give it a goal, a user persona, how it would evaluate when it's done, what you're looking for, that kind of thing and there became all these checklists on how to sort of prompt an LLM to give you better results than just kind of winging it.

So that whole concept became prompt engineering and that was really defining the user prompt and sometimes the system prompt. What instructions, what background, what tone you might want the LLM to to respond with. Okay, so that was what we did for the first year. And then um we came up with this concept of context engineering. Context engineering. And the idea here was not only are you providing the LLM's prompts, but you're also providing it other context.

And with that other context, you can expect reduced amount of hallucinations. If you remember that was an issue early on, and you could feed it some of your private data that the LLM knew nothing about when it was trained. So, um uh so this is where rag got invented, retrieval augmented generation. So, how to pull like private data uh into the LLM, um how to pull other kind of important uh facts, guidance, policies, things like that so that the LLM answered um with full information.

So, an example in our industry, of course, is AI summarization of patient charts. So, an LLM knows nothing about Don Woodlock's chart, uh but you can get an LLM to be part of the process of answering questions about the chart by these approaches. You basically feed the LLM some context that it can use at the point of inference to answer questions. And this whole field became this context engineering. How do you pull from your private data?

What do you pull? You have context windows that are only so big, and how do you how can you be smart about providing context to the model that allows the model to perform accurately, not hallucinate, you sort of be on task. So, we spent about a year doing that, context engineering. Uh and then finally, we're in a phase um that people are calling harness engineering. My engineering. Hey there. I'm just popping in to say that I'd love to hear your comments and feedback on this video.

I read all the comments, so let me know what you think. Let me know what suggestions you have for my next video by putting in some comments below the line here. Thanks. Harness engineering refers to providing an environment for an agentic AI experience to work. So, in the harness, you have a tool loop. So, the AI system can keep looping until the task is done. Um you have tools or MCP servers. So, these are different APIs that can be called and resources and stuff like that that allows the AI to perform well.

You can have skills. So, one of the things we invented since context context engineering is skill files. So, if you want the agentic AI to do all sorts of things, instead of trying to explain every scenario, you say, "I have a directory of skill files. If you need to schedule an appointment or summarize a chart or review codes for billing or whatever, just check the skill files, read the relevant skill and use that as part of your your agentic loop." A memory system sometimes.

So, allowing an LLM to have a long-running task without forgetting. So, a long-running conversation with a chatbot, let's say, but it remembers what you did weeks ago type of thing in a smart smart way. So, some of these harnesses kind of have memory systems. And then another example is hooks. So, you might have different little things of code that before a tool is called, call this hook to make sure that I want that tool to be called.

That kind of thing. And so, all this is a series of kind of frameworks and code that surround the LLM. The LLM is still the same in all these cases. It's still just a LLM where you send it a prompt and get an answer, believe it or not. But, the harness is sort of creating an environment of tools and the loop and skills and stuff like that around the LLM so we can perform more sophisticated tasks. So, the purpose of all this is getting the most out of an LLM.

And uh and essentially we've just advanced our capability to get the most out of these uh LLMs. And the latest is building these harnesses, uh these environments that kind of can can work an LLM very well. And I'll just leave you with two examples. So, if you have You've probably heard of Claude code and Codex. So, these coding agents are an example of a harness. They have a loop, they have tools, they have they know how to use skills, they have a memory system, they have these things that surround the normal models like Claude or or um uh GPT uh uh five models.

Um so, that's an example. And then the other example is kind of the personal agents like open Claude or Hermes or whatever. They have the same thing. They're a harness that has the loop, has tools, has knows how to use skills, knows how to memorize uh things in order to get a job done. Each of these just use an ordinary LLM, but the harness around it is what makes it powerful. And so, when we're building systems today in in AI land, let's say, it's both about making the models better, but now making the harness better.

And a good harness can really add the power that a model needs to perform. Okay, so I hope that was uh clear and interesting. Uh and until next time, thanks. Hey there, I hope you liked this video. Um I've added a next video at the end of this. So, um so take a look at that if you uh if you enjoyed this. And if something resonated with you, please drop a comment at down below here. I read every comment myself and I really appreciate hearing from you.

Thanks.