Markdown source

Transcript: Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Source Video

Local Cache

Transcript

All right. All right. First of all, thank you so much for for coming. I'm actually rather surprised. Um A lot of times like you're you're working on the stuff and you're like cooked up in a room and you're like no one cares. And then it's like so many people showed up. So I just thought that someone cares. Um so anyway, so the the title of my talk today is evals are broken and you should use them anyway. And a lot of this talk is just like a straight-up critique of like the way we do evals these days. And I kind of want to help you out.

I kind of want to give you a way out of this. It's like you have this like interesting technology and you can use it, but there's like so many ways to like mess it up. So I want to help you out. So my first claim is that people are wrong about evals. Actually, let me let me correct myself. Most people are wrong about evals. And I want you to be right about evals. I want you to I want you to use them. I want you to like I want you to be able to build with them, interpret them, use evals in your own agentic flows, um leverage them in any way sense that you can.

Um so that's that's that's basically the point of the conversation. So to be right to be right about something that has like a lot of nuances that can go in like many different directions, the final question is like how are people wrong about that thing, right? So there's basically two camps of people who are wrong about things, right? So there's two camps of wrong on evals. The first camp is a camp of objective metrics. So the objective metrics camp is this. Like there are people who would like look at a list this dashboard, right?

And they will interpret this as something akin to like like it means something as in like GPT 5.4 is effectively the same as like Gemini 3.1 Pro preview. Believe me, they're not the same. Uh there's a lot of these models which feel like show up like with similar numbers and these numbers at a certain point it's like this is the whole thing is a hoax. Like you you won't believe it at all. So, there was this tweet this came out like just this morning and it was a critique of meta where meta came out and it was just like classic benchmark maxing. Just like we're doing best on the benchmark. Everything's great.

And I I assure you if you try a lot of these models like it just just won't won't hold the hold the test of like actual real-world evidence. The other one the other camp the other way where people are wrong is that they could too far the other way. So, this is basically like the taste camp. Um So, people in the taste camp are kind of like this. They just like they're they're an archetype. They're they just they think that it's like it's everything's about like vibes and everything's about like you know, like it's like if you ask them like why do you like operas and they'll say things like I like talking to her.

Like they'll they'll like anthropomorphize it. And um and this is also not the right either, right? So, I think the truth is somewhere in the middle that the emails are not the end all be all. They're also not completely useless. There are right ways to use them. There are wrong ways to use them. Um so, in order to do that I'll give you like three stages that will help you like uh use them really well. So, the first stage is like you can like leverage emails from other people. The second stage is to like use emails to improve your own agents.

And then the third stage is like build actually build your own emails for specific use cases. In the interest of time, I could personally talk about emails for hours, but like I can only talk about level one and two and I think those would be most helpful for most people in the audience. Um so, I'm going to give you like a few heuristics uh to interpret email. So, the first heuristic is that whenever a model apps comes out with like a number, just don't believe them. Just don't. Just like these are approximations. Coming back to the tweet like just like just don't believe the model app eval numbers. Like there's somewhat of approximation.

Sometimes they're good, sometimes they're not. Um there's this tweet that's like pretty cool one where Nikunj said that like a lot of like AI researchers and engineers routinely dismiss evals. They don't really They don't really think of it as like something that's like the numbers to be taken that seriously. And it to some extent it's a matter of like actual trying and preferences. And I think I think that is like somewhat more accurate. Um Wait a second. Yeah. So the second heuristic is that like you want to stay current, but you don't want to be the earliest adopter. And why am I saying this?

So if this is like Epochs Index is basically like the aggregate score of like different models on like evals. And if you notice like in the last like 2 years, every single couple months like the frontier lab the frontier model is changing. And it's changing so fast. Like it's just like it's so hard to keep up with this stuff. And I work for I work on this. I've been doing this for a living for years at this point. And I I even even me I have like preferences changing so fast.

So like I think that like when when you're when you're working through these things like the way I would recommend is that like let the thing come out first. Let things settle on fire for like a couple weeks. And then if the if the thing still stands the test of time, I think at that point you should like do your model switch and like try something rather than like always trying to be on the cutting edge. Like the the people who have to always try the new model and try the new things like they'll be me. But like I do this for a living and you don't have to.

Um the third the third heuristic for evals is that like you need to look for very very new and very precise evals. And the reason this is necessary is that like a lot of evals are like at this point they've become like standardized. They're actually kind of old. Like they're they're not useful for you. So like this is a this is a blog post from OpenAI where they straight up said as we bench verified no longer measures frontier coding capabilities. Um, and I think like if to a lot of people in the AI research community, that was like very obvious.

It was very obvious that as of the leave edge doesn't measure frontier coding capabilities because it had like it would have problems like solve the Fibonacci sequence. It would have problems like, you know, do matrix multiplication or something. And it's just like it doesn't apply to like real world software engineering. So, you want to have something like like very new, but also like actually legit. Uh, and then it takes some discernment to figure that out. So, that's like the first part. But the second part is like, okay, now that we know that like, okay, there we have a few heuristics of like how to like use evals.

Like, how do you use evals to improve your agent upon them? And I think this is a part where I kind of lean into like the the core philosophy of this conversation where like you want to think of evals as like an engineering problem, but also as a philosophy problem, right? So, the engineering problem is obviously hard, but the philosophy problem is also very hard. Uh, the philosophy problem is that you you want to you want to you have a problem and you can't exactly approximate the the search space of like where where the problem could go where the problems could fail. Um, it's it's sort of somewhat easier-ish to do it for coding problems.

But even then, coding problems have like an infinite search space. They can go in any direction. So, you want to you want to build evals that like are somewhat more approximate representation of the actual thing that you're dealing with. And for us, like to give some context in Cointreau, so I work at Client. Client is an open source coding is a company. We have a we we have a very interesting product. I encourage you to try it out. So, in Client's journey, one of the things that we dealt with is like in the last year, one of the things we found is that like there were like a few evals available.

At the time, it was like we were very rudimentary. Every every other company was very rudimentary, as well. And our thinking was like, okay, like if if there's like if there's like so few standardized emails available and also they're not effective like they really are not measuring what it is that you're trying to do in your day-to-day programming job, like what do you do? So, our stance was and this was the stance of the Codex team and a lot of other teams that we've talked to. Uh why this why this was our stance just like this this email just completely ignore them. They're completely unnecessary.

Should probably probably wasting your time and it's just like I don't know who who will be appeased by them. And then last year we we uh we came this idea that like okay like listen I think I think I think we we got to up the ante and we got to like we got to have some measure. We got to try emails. And if no one else is doing it, we'll do it ourselves. We'll build actual emails from scratch um that would like actually test real-world programming problems uh of users.

So, we got like we went through a lot of like a massive data sets of like people who had opted in to share their coding usage of client with us and we offered them money and we we got a lot of this data set of like okay this is what the problems that people actually doing then spent a lot of time parsing through that figured out like an actual data set like these are the problems that people are solving and then just like completely cleaning it all up like doing a lot of like really hard manual labor trying to make like very decent problems that can be solved with say uh client or any other coding agent.

Um the hardest part for us when we were building emails is that like if you're building emails for anything that's like rudimentary like if you're if you're building email for say an LM model, you have a very simple like one-shot use case of like how many toes does a cat have and then the LM can just be like I don't know 11 or whatever. I don't I don't know how many toes a cat has. But um a single turn email is very easy to do because it has like a binary answer. It has just like a very limited search space of what the answer could be.

But when you're working with an agent, that can't be the case. You're working with an agent, you can give an agent a problem like, "Hey, uh I have this new MCP server. It's probably not working. Like, how do you how do you like make it work for me?" And that's usually how a lot of you guys talk to uh Cloud Code or whatever agent you're using, and myself as well.

So, in this like it's very hard to gauge because like the agent like reads through files, searches through docs, uh installs the environment, sets things up, runs Python scripts, does all of that, and then in the end like runs some tests, and then maybe the whole thing works. Like, it's so we're we're trying to grade the second thing. We're trying to get grade like all these things that will take a lot of time, and then figure out like, "Oh, did it actually work or did it not? Did it work, but like broke other things?" Like, you So, that's why it was like harder.

So, in the same time, uh some very very awesome, smart, bright people from Stanford University came up with Terminal Bench, which does the same thing, where they came up with like 89 coding problems, which are which are just like very approximate, decent representation of like real world programming problems. So, these could be things like, you know, um race conditions, uh database issues, um like um other stuff. Like, it's like figure out this infra issue.

And I think that that was built The Terminal Bench was built to like use with any coding agent CLI, so you can like you can actually test and run things um like really fast with like uh CLI, and then it will take like a couple minutes to run. So, some of these tasks would take up to like 30 to 40 minutes. Uh and and that's how you know they're legit because like you the agent does a lot of things and just like runs into our code, and sometimes just goes crazy, and and yeah. Um so, uh so, we started using that.

And the way to use that, like the way to use Terminal Bench, is that like you think of an eval problem as like an evaluation suit, which has a set of problems. And this one has 89 tasks. Some others would have more. And what you want to do is like you want to give it an environment. You want to give it an isolated environment, where you just like you Let's say say have a task like, "Hey, figure out this race condition for me in this repo. And the race condition is that like this thing is not working.

So, you want to be able to give that the eval run an isolated environment like a virtual machine. In that virtual machine, it has the whole setup, it has the repo, it has everything. And then you install whatever agent you have, in our case client, cloud code, codex, whatever you want to use, you can do that. Uh to do that, it's it's not that it's like hard, but it is also not trivial. And Harbor is another software that came from Loda Institute um where they made this thing where let's say you have 89 tasks.

One way to do evals is that learn like each of these tasks in sequence and then like do all the setups. Another way to do is like have a very standardized configuration defined in infrastructure where each of these 89 tasks have like the proper Linux machine, uh the proper RAM CPU usage, um and then like being able to like isolate those environments and then run those 89 tasks in parallel on infrastructure. So, you could use a couple different things for the infrastructure here. Uh you could use Daytona, you could run it on your Docker machine if you have like very powerful machines.

I'm sure if you can handle those that much compute, sure, but I wouldn't. Uh we use Model. Uh Model, we're very thankful to Model, they've helped us a lot. Um so, shout out to them. And uh yeah, so in this case like Harbor basically lets you split up like the 89 tasks and then they all run in parallel. So, that way your limit the limiting factor is basically the slowest task. Um yeah. So, um yeah, so the slowest task is the limiting factor. So, the process is this. You you you get a score, you first do a run uh on the 89 task, you get a score, you evaluate all the failures.

So, let's say you get like say 50 failures, right? Out of the 89 task. What you want to be able to do is you want to portfully allocate those failures. You want to say uh you want to run like uh another agent which goes through the traces of all the failures. So, the trace would be like this massive file which has like every single LM call that the that the that the agent did and then be like, okay, this one this specific problem failed because it didn't run past. This failed because the retry tool was broken.

And once you properly allocate those failures, you figure out, okay, these are the small levers that I can pull. If I pull those levers, like I can make like massive improvements to my AI agent. So, with your testing is like you're basically testing like um three things. You are testing the model itself. Um like if you have a very decent model, like somehow like you could have a horrible harness, you could have a horrible agent, but like the model just like overshoots so hard that just like you you you know, you you get a great score. You're testing the harness. You're testing your coding harness.

So, like you could you're testing say if you're using cloud code codex. So, sometimes you'll find I'm sure I guarantee you some of you have noticed that like let's say Anthropic's models could potentially work with the cursor, could potentially work with droid with with with other coding agents, but for some reason it just seems to work so much better with the cloud code, right? And I think that that that is like the testing the harness that like is the harness actually really leveraging the best best of the model. And the third problem another problem is saying if you're solving stupid problems, it doesn't matter if you score 100% all the time.

So, you really got to make sure that like um um the the problems are saying which um the Luddite Institute has done a pretty great job of. So, for us it was the case like this like um we we basically like um had like this original score which was like much lower like 43%. Um we made changes to like CPU, we made changes to memory in front of the containers. Um we raised timeouts, we improved the thinking behavior. Sometimes we'd ask the model to think more.

Sometimes asking model to think more actually interferes with the quality of the response because it goes in like it gets like a stroke and it just like goes in like circles and it's like it'll just like I am a model. I am a model. It'll just like keep doing it for like uh like uh 2,000 tokens. So, yeah, so like you you you you got to think through all of that. And yeah, so for us like we have like a huge like internal benchmark for all kinds of models, open source models. So, like we just like keep like a list of like trying different versions and stuff.

We encourage other people to try that as well if you pretty helpful. So, whatever whenever you get like zones improvements, you get like basically three zones of improvements when you get an original score. The first one is the obvious flaw. It's like sometimes your harness really has like very obvious flaws of like there's this bug that can straight up crashes the harness. And those obvious bugs you got to fix, right? Sometimes you are not like you're getting rate limited or whatever. Fix those, that's fine. I think the zone two is the most critical one where you actually do nuance improvements.

And these nuance improvements are things like there are certain prompt engineering techniques that apply to entropic model families that just straight up would not apply to Codex model family. That would be very different from Gemini model family. And those are the nuance of like why is it that this is a model that's so good that so many people are saying it's so good, but for some reason it just isn't working for me. I think those that's that is the essence of like working with agents and hill climbing. That like you figure out those nuance improvements of like tweaking your prompt, making it larger, making it smaller.

And then zone three is the danger zone where it's like you're straight up overfitting. So, you're overfitting in the sense of like you're just straight up cheating to get the highest score and then you can like make a tweet about it. Don't don't a lot of people don't it don't do it. Like I wouldn't do it. I mean, never mind. Anyway, so yeah, so anyway, so this was like this is like basically the rough outline.

So, the final the final wording for me would be like basically regardless of the kind of problem that you have, I want you to like find a benchmark and like like build the eval and just like hill climb on it. So, hill climbing means that like you get a score and then you improve the score off your harness on the eval. And you have to do both. Like you can't just like have like a good number and be happy with it. Like, you you you you you you got to both pass the vibe check. Like, does it actually feel good to use this product in this model?

And at the same time, you also have like a very very very decent uh score, hopefully. Um if a new thing comes out, you you you do your absolute best to like give it the right judgment. Uh for us, like, one of the things that we learned was that like we were very decent on Anthropic model families, not so much on, say, Gemini model family, not so much on, say, uh Kimini model family, which again are very decent models.

Um so, when we when we started hill climbing, we learned that like, "Oh, if we support these models, we have this like entire swaths of people who love these models, and they can start using us." And I think that in some attraction of that would also apply with you. Um so, my final my final note to you guys is that like, you know, I've I've done some hot takes or whatever, and if you work for for some of the companies that I've said not so nice things about, I still love you and everything, and uh it was it was um I work at Client.

So, if you find these problems fascinating, and if you want to learn more about these, like, uh this my Twitter, so like you can feel free to reach out to me, DM me about like if you want to work on problems like these, like, by all means, like, I can put a word for you. If you want to learn more about evals, if you want to learn like how do you like I have a problem that's like a completely orthogonal to everything you're defining for coding agents, like, how do you work on that? Uh so, feel free to reach out to me, and I'll I'll respond to you.

And um once again, it's very very kind of you to give me your time. Thank you so much.