Introduction to MCP Evals transcript

This video is an introduction to MCP eval. I'll [music] share my personal view on how I think about evaluating LLM MCP tool use focusing on a threelayer approach which I will explain soon. Then I'll talk through a bunch of metrics that you might want to track if you're doing this. And we're going to start with a hands-on demonstration because a lot of us learn best through actually seeing the stuff happening. What I've got here is a configuration file. this JSON file.

This is going to define an evaluation test suite for my unit converter MCP server. I'm running across a few models. And down at the bottom, these are my actual eval. When we ask convert 0 Celsius to Fahrenheit, I want to trigger a tool call and I want the answer to include the correct answer. Uh here's another example where I'm converting radians to degrees and I want to check that the answer contains this string. Uh I also have these abstension evals.

So when we ask certain questions, we don't want to trigger tool use and we want to test that as well. Let's get this running. It's a pipi package. I'm going to run it with uvun arbiter execute and then the name of this arbiter example eval. and it spins up and runs across a few different models in parallel because we want to test all like the ability of different models to use our MCP tools because users might be coming from different models and you can see they are performing differently.

For example, we're getting some failures here from GPT 4.1 whereas Claude is performing a lot better and GPT 5 mini is performing a lot better as well. And we can see on the right this tools equals false and tools equals true. This is tracking if the tool is used when this uh when this command is submitted. We're also tracking the time. That's an interesting metric. And here is the final result. And clearly we'd want to focus on these parts here where we're getting some failures on certain models.

And we'd want to look at what those failures are. The way to do that would be to actually look at the result. So we've got these results here in this JSON file. So I could open this up and this has the full result of our entire run including all of the actual questions that were asked and all the answers. Looking at the failure cases, this one was failed because we're judging based on a string. If the uh LLM answer contains the ground truth answer.

So when we ask convert 8 radians to degrees, we're looking for this exact string to appear inside of the answer and it doesn't because the model has rounded this. What this indicates to us is that if this is the ground truth we're actually expecting and we want to see this presented to the user, maybe we need to change our MCP server. Perhaps we could return some metadata that says always output the answer to like six digits of precision or something like that.

And if I keep looking through all the fail cases, uh we're seeing similar results. It's all these fail cases are when the mod judge mode is contains like string contains. And you can note that you know the tool use is expected here and we are getting a tool use. So we can judge that our server is properly you know getting triggered by this query which is great. If you want to use this tool arbiter for your own MCP server, it's open source.

You can install it with pip. So pip install arbiter mcp evals like this. You could also use tools like pip x or you could use uv uv like that. That's how I did it because I use uv. Um and you can try this out yourself and see what you think. Full disclosure, this is my own open source library which I built and I did name it after Halo. Okay, having seen that quick demonstration, now I'm going to solidify these concepts and also explain more aspects of MCP evaluation.

But if you want to use this library, I've got a link down in the video description, my Arbiter library. I've also got these slides, which I'm about to show you available as a PDF document. I'll send you these if you sign up for my newsletter. I just send one email a week, so you can sign up. You can reply to the welcome email and ask for these slides and I'll send them your way. There's also some links in here to helpful tools and resources for MCP evals.

Keep in mind that MCP eval is a very deep topic. There's a lot of people who've thought quite hard about this. There's a lot of academic research about this. This video is meant to be an introduction. And if you want access to these slides, as I mentioned, you can sign up for my newsletter. In terms of testing MCP servers, I break this down into three areas. The first is unit and integration tests. And these test code correctness.

For me, these are preconditions for MCP evals. They're general software practices. Next are the eval part. That's what I demonstrated at the start of the video. MCP use with LLMs. Testing things like behavior, quality, accuracy, tool calling. And next is monitoring. And this part of testing MCP servers is something that's done in production. like when your MCP server is actually out there in the wild being used somehow.

Being able to track real world usage metrics and evaluating models based on those. Notice how these are three very different things. Unit and integration tests would be built right into your MCP server and you can do like something like piest. Evals are a tool like we showed at the start of the video and then monitoring is something that needs to be baked into the application and looking at dashboards stuff like that.

This image shows a representation of LLM evals. It's from Anthropic's blog post. And this is the interesting evaluation part where we're going to have some prompt and test it against some use cases. And then we're going to iteratively refine that prompt based on the information provided by the evals. The evals are going to guide us towards doing prompt engineering for LLMs. And at the end of this, we're looking to ship a polished prompt. applying this to MCP.

I've just added some little pink annotations. This is my own stuff. This is not from Anthropic anymore. And what I'm saying is instead of doing prompt engineering, now what we're testing against is the tools themselves, the actual MCP server implementation and using MCP eval to get our tools right. This could be how the tools actually work like in terms of the logic or it could be the way that those tools are um described, the descriptions and their argument names and their variable types and all of that stuff.

And the goal here now is to ship polished MCP tools. My first piece of advice is to keep it simple. This can be done with basic guardrails and success metrics. For example, this LLM here has access to some tools and we can set up some really basic guardrails in this system to check how these tools are being called and to make sure these outputs are what we expect. Another way to keep the these eval simple is to break down agentic behavior into small pieces.

If we've got some complicated agent like this guy who can use some retrieval uh stuff, he could do some memory stuff, he can do some tool calling stuff. Let's break each of these down and we'll individually evaluate these. will evaluate this part of it, this part of it, and this part and break this down into chunks and look at each of them individually. That idea came from Ian Webster. He's at Discord and he has a great talk down here iterating on LLM apps at scale.

Uh he has another quote from that talk. Enterprises won't deploy LLMs until they can measure and mitigate security, legal, and safety risks. This is what evows allow us to do. They allow us to actually measure these risks and then take the right actions to mitigate them. Now when it comes to MCP evals, I can break these things down into three layers. Earlier I had kind of a three layers of testing MCP servers. This is specifically with the eval component.

How this breaks down into these three layers right here. What are they going to be? First one tool correctness. Does the tool do what it claims to do? Is it giving the correct behavior? Next is agentic usage. Are the LLMs able to actually choose the right tool to use? And when they use them, do they pass in the proper parameters? Notice how these are different. The tool correctness one is more with respect to what's actually going on on the server code.

This is also something that could be tested with um unit testing to some degree, but we'll see how MCP evals allow us to test tool correctness really um easily using LLM as a judge. And lastly, production fundamentals. This to me is the stuff that's like latency and cost and reliability. And a lot of this stuff should be tracked in production with monitoring. However, we can do some stuff before production in terms of latency and cost.

Uh for example, the arbiter tool I built looks at the token usage and it looks at the latency like how long it takes to execute each prompt um on each model. When it comes to grading evals, we've got three options. codebased, LLMbased, and human grading. All right, let me roll this back. The first thing is codebased grading. This would be like you have an answer. This output comes from the LLM, and we could say, is that output exactly equal to what we expect?

That's probably pretty rare, but some cases we might want that. The one I implemented in Arbiter was this keyphrase in output, like is the answer inside of the output from the LLM? The LLMbased grading is LLM as a judge and it it's just a prompt that says in fact that's what I have here. So this is our LLM based grading example right here where we're like okay given this question is 42 the answer to life the universe and everything the golden answer is yes according to hitchhiker's guide of the galaxy and then we evaluate this answer we we we get a generation this answer here is an output that's going to be generated now this answer which will just be like some string will go into here this answer aspect of this prompt and this eval part with the question and the golden answer that'll come down here it'll get put somewhere in this rubric And based on this prompt, this uh this is the prompt which we feed to an LLM judge in order to decide if this answer is appropriate for the question.

Does it does it, you know, fit with the golden answer or not? Is it correct or is it incorrect? The LLM will decide that for us. And this prompt here, this template came from Anthropic's blog post which I linked to um right here. And the last example is human grading. This is like a person looking at the answer and deciding themselves if it's appropriate or not. And this can be done with sampling. It can also be done with reviewing cases, you know, which the LLM determined was a failure and manually looking at those.

By the way, if you're learning AI engineering, then you should look at my AI engineer road map. I've gone ahead and created video lessons for all of the key topics in the course. Each of these is an individual video lesson and each of these lessons runs at least 10 minutes. So, there's a lot in here. You can go to sassenodes.com. You can sign up for this. And when you visit this page in September, you'll see little video tiles for all of these different lessons.

And you can sign up and start tracking your progress towards learning AI engineering. Here are the application metrics, which I think are most important. The first one is task success rate. Based on a bunch of tasks, a bunch of eval questions and answers. How many of those were correct? This could be based on the LLM as a judge or the rule-based criteria we discussed in the previous slide. Next is tool invocation precision and recall.

And this one for me, the way I think about it is like looking at when the tools were called, if that was the right thing to do or not, because maybe MCP servers are too greedy. They're calling tools too often or they're not calling them enough. And I want to start with recall actually. Was the tool called when it was needed? And imagine we the tool call was called on every single task. So we have a big bunch of tasks in our data set, a bunch of prompts and answers.

And let's say we called the tool every single time. Well, of course the recall will be 100% because we always called the tool. It was always there when we needed it. But now the precision is super low. Was the tool call correct? Was it the right thing to do to call the tool? And that gets pushed down low. So these two are kind of like a balance beam and we want to optimize both of these things to be as good as we can.

The next metric is argument validity rate. This is just simply how often tool calls are made with proper schema and parameters. When the right arguments were passed into the MCP tool from the server, if this is low, it means we need to describe our, you know, arguments better or maybe add type hints or annotations or something like that. And I've got two more. The first one's latency. So looking at the P50 and P95, this is like the time that 95% of cases fall below this.

This P50 is the time that 50% of cases fall below. And we can track both of those things. And then we can see when we make changes to our server, how does it affect the latency of the overall, you know, asking a question of the LLM? How also does it affect the token cost? Imagine we make a change to our MCP server that outputs a lot of text that all that text needs to get processed as tokens. So, we're going to increase the cost and that's something we can evaluate against.

And lastly, retry and failure rates. This is stuff like server errors, timeouts, disconnections. This is going to be like maybe our server, our actual production MCP server is under a lot of load and it's giving like 503 errors. These, by the way, these are the types of things that are really useful to track in production. So, production metrics from actual users and you bake in monitoring into your system in order to track those things.

This slide is on user feedback. And in general, when evaluating software, getting feedback from your end users is really important. So whether or not this is feasible for you with your MCP server, I'm not sure. But things that I would think about are very simple ways to get feedback. Providing a thumbs up or a thumbs down from users is a really easy way for users to interact and give you something. reporting an issue is a lot more difficult, but if a user is having a big problem, giving them an easy way to report that as an issue is a good idea for you and that can tip you off to these things.

Another of my favorite ways to get user feedback is through AB selection. I love this because giving people an option between two things is really easy to wrap your head around. Like, do I like this one better or that one better? When we get into more than just two options, it can be more difficult to select. And we see Open AI doing all of these things with chat GPT. These are baked right into the interface. All three of these mechanisms for collecting feedback.

I've got two slides left on practical considerations for you as an MCP developer. The first one is when you're building the server, set up and run eval scripts. I showed my arbiter example. I'm going to show more examples in the next slide. And then actually look at the results. If we're not like if it's in a CI/CD pipeline and it just kind of runs all the time and we want we want to actually look at the results on like a case by case basis. look at the fail cases and do something about those.

Next, in production, log metrics and feedback. Build this into your system somehow so that for example, if you're running MCP on your server, like you're hosting it somewhere, build some metric logging and get a way somehow to collect some sort of feedback if you can, and then connect that to a dashboard so you can actually look at this and see what's happening. Here's an example of a human review dashboard where a person will look through some behavior of the system and inspect for that conversation like how did this go?

Was this good or not? And they can do things like accept it like this was good, this worked or they can reject it and they can go to the next one. And they this sort of like humans should be looking at the results, how the system is behaving and actually deciding how it's working. And this will create a really good rich data set to to then make our product better. And this example here, this LLM data review mock dashboard, this came from a talk right here, how to construct domain specific LLM evaluation systems.

I'd highly re recommend looking at that if you're interested in um doing LLM evals or MCP evalu then the document it'll link you right to this talk and here's some tools for doing MCP eval there's no shortage of these things openai has an eval API you can consider using that you can look it up it's very easy to find and they have a blog post on a use case of MCP evaluation so you can read through this and start submitting stuff and you can actually use OpenAI like this and it gives you a a report like that inside of OpenAI.

Next is Prompt Fu. This is visualized over here. Prompt FU is from that Ian Webster guy. That's where I learned about it in his talk about um from Discord about how he does LLM evals there. I would highly recommend checking this tool out for like serious eval stuff. Deep eval is visualized down here. I don't know anything about it, but it came up in the search results. So, you could consider checking that out. And then MCP eval.

This one here is a node library. It gives you a GitHub action that you can use. You can sort of bake this in. And this is good for node. If you're developing a node and you want really custom evals. And then I built Arbiter, which is a lot simpler than many of these tools, but I think that there's a lot of power in just a really simple wrapper around doing evals like I demonstrated. There's also plenty more of these things out there.

And that's it. If you've got this far, please consider subscribing and give me a like on this video. If you enjoyed this and if you learned something, let me know if you'd like a dedicated video on doing MCP valves with Arbiter or these other tools which I mentioned here and I'll create that and link that at the end of this video if it's available. Otherwise, I'll leave you with a few other related topics so you can continue learning AI engineering.

Thank you for watching and namaste.