Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i transcript

Hello everyone. Um this is the last talk of this session. So hopefully it's going to be short. I know that you guys had to go through a long day. So try to keep it short and light for you all. Um I'm going to present myself. Um I'm Ali. I'm the director of AI and ML at G2I. Um I have zero experience in ML. So I don't know why they put the ML in my title. I'm a software engineer uh at heart. And to prove that I have more than 50 abandoned side projects in my machine.

So uh you can know. So uh I'm going to make a disclaimer. The the title of the the presentation is a little bit misleading. Uh as I was working on it, I realized that it would be better if I presented my journey uh into benchmarks and what I learned instead of trying to find a dichotomy of the the bad, the ugly, and and the good. So um let's start with um I want to grab your attention. And I invite you to look at this.

These beautiful three screenshots are a single prompt on one of the benchmark tasks. As I was looking at it, I was like how can an engineer write a task like this? So I said, "Nah, it's impossible. No one writes prompts like these ever." But I wanted to double-check with my engineers. So I took three of our best engineers. I showed them the prompt and I said, "Would you ever write a prompt like this?" And the answer was no.

And they're right. They shouldn't. And so at that point I'm I was like what is a what are benchmarks anyway? Uh I needed to take a step back. I needed to look more. I needed to understand. And so as I was researching, I faced a wall of keywords. Um graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon. So, I was like, either this is too complicated or um there's a lot of jargon and a lot of um words to to to work through here.

So, um I worked through it, worked with my team. I have a lot of good researchers in the team, and we uh kind of like nailed like simplified it to the most basics. Um and so, the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents. Agents provide solutions. Those solutions are verified uh and graded through verifiers and rubrics. All of that is wrapped in a harness that's that's preventing it from um from the external factors.

And if it all goes good, uh we have um trajectories, scores, and um metadata that we can use um to to to verif- to basically uh rank um models. And so, the equation is simple. If prompts and instructions are great and verifiers and rubrics are doing their job while the harness is preventing um or creating an environment that is good for a benchmark, we should have amazing results. Um but, that's not the reality. So, what what what went wrong?

So, the first thing is when looking deeper in benchmarks, uh most of the instructions are unrealistic. Um I did a quick research on SweetBench Pro, and um there's 481 words per instruction in average. That's a two-pager per task. That is not how people write prompts. And to illustrate more of that, um I took a couple examples here. The first one I looked at I I call the leaky prompt. It's a go um task that's basically um that's trying to match in some rejects and doing test it's on on some rejects.

So, in the first screenshot here, um the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that. The second one is is even worse. Um it's basically providing a complete interface of the implementation. Basically locking the LLM from any kind of uh creativity and it's forcing it to do it that way. So, that's the leaky prompt.

The second example, it's the the not economically valuable prompt. Uh this is from Sweet Marathon. And this prompt is well-formed. It's It's It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust. So, I don't know if any of you ever tried to do that, but I don't think it's a good idea. We should not do that. All right, moving on. The second problem, weak verifiers.

Um so, the screenshot here is is a uh is the work that Deep Sweet um did uh to compare their uh their bench against Sweet Bench Pro. And um let me just fix here so I can see the numbers. In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh accepted wrong implementation in one hand and more than 20 24% of the tasks uh rejected um correct implementations. And so, I kind of went again, dug a little bit, and I extracted one of the tasks, and I started looking at it.

Um and and here's here's what's happening in the example of uh re- rejecting um possibly rejecting good good answers. So, in this example, the test is is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the variable name this way? So, this test is cornering the LLM and basically uh causing uh those false negatives. In the other example, it's base the test is basically checking functions that are unexported.

So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it. So, this is what a weak verifier looks like. All right, moving on. Re- reward hacking. So, what's happening is models are becoming increasingly increasingly able to optimize and figure out solutions to hard problems by going around the problem. So, instead of actually trying to fix the to to apply a patch to a task, they try to go and find dot git folders, or they look up the internet for any kind of traces that would allow them to um to do the task.

And this first graph here shows like shows that as models evolve, they are now more smarter and smarter in being able to do reward hacking, but that's what we want. We want LLMs to be smart. The benchmarks are lacking behind and they're not preventing from from that to happen. Um More in detail, as you can see here, the more you go in time and the more you have new versions, the delta of um of um reward hacking is increasing.

So, the conclusion here is there's a quality gap and it's causing a trust gap. I have not met an engineer in the last 6 months that would choose a model or choose um an LLM based on the leaderboards. Um they look at them. There's a lot of hype, but then they move on and they test things by themselves and they apply that. So, how do we close the gap? Um in the last 2 months, we've been working with our team at G2i to basically try to define a framework, uh a set of principles that would allow us to build tasks for benchmarks that are um better than what we have today.

The first one, human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details or try to guarantee self-containment when the task itself is is expressing too much uh too much details. The second principle is holistic graders.

Behavioral tests in one hand and then precision what were needed. This is very similar to how we approach um tests in engineering. We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So, for security issues or business logic, we want to have the whole stack units unit test integration tests and then end-to-end tests. But, for the rest of the the rest of the the software, we don't want to have 100% coverage because that's um not efficient.

The third principle, production grade. The tasks have to be tasks have to have value um and they have to be economically valuable. Um it is one thing to have a test a task that is failing the LLM proven that the LLM is not there yet. It is another for it's another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that." Currently, we don't have that. So, production grade. The fourth principle, contamination free by design.

We want to do novel tasks only and we want to make sure that we keep private holdout sets. This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos. So, our approach here is that it should always be novel. This way, it's contamination free by design. And the fifth and last principle here is information about leaderboards.

The benchmark needs to tell a story and needs to help people make decisions. Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't to you why. And so, we want to basically put the x-axis back on um on on the first page. Uh the idea here is that there's um there's a lot of um data that we can extract from those these runs, and unfortunately, they're not being put in the forefront. And people have to dig uh a lot and do their own experiments to get to those data points.

And so, finally, uh initially, I wanted to have a kind of a a lofty like ending to this, but I think I I I pivoted to something more interesting. Uh this is a call to action to software engineers. Um benchmarks are not hard. We need to look under the hood. And we need to understand them and join the Discord because engineers' input is valuable. And thank you. >> [applause] [music]