Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft transcript

Hi, I'm Onela. That's Joel. And we built Ace, a live AI voice tutor that runs a full lesson start to finish reliably. The trick is LLM is not in charge. If you have shipped a multi-step agent, you know this moment. It's near the demo, then a real user gets in and halfway through the agent decide it's done. Or skip a step, or even loops. The demo never show you that. And the first fix everyone reaches for is prompt this harder, add more holes.

But reliability was never a prompting problem. It's a control problem. Think of it this way. The model is the talent, and the harness is the director. The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six. So we stop asking it to. A lesson is a small state machine with intro, teach, check, grade, advance, and wrap. Each step sends the model a neural contract.

Do this one thing, return it. The harness validates what's comes back, advance the state, and decide what's next. The model never decide where we are. That's the design. Joel is going to show you the harness thing. >> Yeah. So, when we think about the frontier models of today, let's take for example um Opus 4.7 Cloud from Anthropic, you see that oftentimes people leverage the model for essentially everything. For the thinking, for the processing, right?

And for everything in between. While that can be good, it's not always effective in situations like ours, where we are building a live AI tutor that is speaking back and forth with students. Right? For something like this, we had a need to actually build something that is reliable, something that is cost-effective, and something that is fast. Right? So, this is where the idea of leveraging the concept of harness engineering has come in.

Where instead of having a model that is really intelligent, sort of go through everything for us, we will build all of these steps that are needed and provide only the input required for the model to execute a specific scenario. So, when we were building ACE, we actually thought very deeply about state machines. Right? What is the step right now? And what is the possible steps that could come after? And within each of the steps, what are concrete things that we can provide to the model so that it is confined to that specific action, it is confined to that specific step at that particular moment, and only execute what needs to be done.

So, by doing this, instead of having a very heavy model like a 4.7, we were actually able to rely on something like a Haiku 4.5, which is a much smaller model, doesn't have as much reasoning capabilities, but because of the harnessing around it, it's still able to perform at the level in which we expect, saving money, saving time, and saving latency. So, let's go ahead and play this recording, which will show us logs um about a particular lesson.

So, as you can see in this video, especially on the right side, we see logs on all of the different harnessing that are happening. [clears throat] Right? So, for example, we see that there's harnessing for a section, which provides input to the model about exactly what to speak about, what to do. We have harnessing about drawing on the whiteboard. We have harnessing that deals with clearing the queue. We have steps to how to end the lesson and everything in between, right?

So, everything that would allow us to actually build the lesson in a way that is reliable even if there's a new scenario that comes in. We try to incorporate that in our state machine. We try to incorporate that with within the lesson, right? So, again, the model all it worries about is given an input, it knows which action to take and it provides the output of that action. Right? And so, the model never really um has to think.

It proposes, but ultimately it is the harness that decides. And so, for his A specifically, there are three things that we wanted to think about. Like, when is the lesson done is one, right? Did the student actually get it right? Like, did they actually learn in the way they were supposed to? And what comes next, right? And so, everything that comes within those three categories, all of the different questions, all of the different actions that the models needs to take, we have engineered that outside of the model, right?

So, again, it's an input, the model receives it and gives us an output, right? And so, this is very, very important and we have found this to be very remarkable. Right? So, again, the the this is applicable to really everything, right? It's applicable to something like A that is a voice model. It's applicable to coding agents, to Ops Run Books. Um it's applicable to onboarding flows, right? The same rule applies, right?

We want to find a way to not let the model think, but building abstractions around it. So, a good way to to remember on whether you should use this abstraction is essentially to think about the reliability of your agent, right? If it's somewhat of a coin flip, then you want to take the control flow out of the model. You want the model to not make as many decisions as it should and instead build those decisions around the model and simply feed an easy input so that the model can easily produce an output, right?

So, don't let the model talk, right? Or actually let it talk, but don't let it drive. So, we're Joel and Ornella. This is Ace, and if you have any questions, please let us know. Thank you. >> Thank you.