Markdown source

Transcript: 20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Source Video

Local Cache

Transcript

Hello. So, today we going to try to ask us to to ask us the question what AI model is state of the art. And I guess this is a super important question for everyone because of course for our applications in research or when we deploy a product, we want always to have the the best performance out of our models. But the problem is that state of the art is a bit a confusing concept and people maybe have different vision on this. So, we'll just try to see a bit like first what what how people approach this question and how they try to answer this question.

And usually there are two main methods that people try to to use to know what new model is state of the state of the art. The first one is simply, you know, they go on the internet and check some public leaderboards, see what what is the best model on on the public leaderboard. And another method is actually just perform some internal evaluation. And again see based on their internal evaluation like what model is the best. The problem with these methods is like in most cases if you apply them naively, you will always find like a kind of lazy solution which is just to to use a large foundation foundation model.

So, we're going to just try to see a bit with these methods like what people try uh tend to do like a bit um quick and can can can be done better. So, the first method again is like just simply checking a public leaderboard. So, for example, if you take the use case of like uh image editing, let's try to find the best image editing model in this case. So, usually the first step is just find some leaderboard. In this case, you can use like Design Arena which is a very famous one. And then you just pick the top one which is ChatGPT image. And then you feel that you are happy.

This is the best model for your use case. In general, it's like kind of a good solution. Like you get like a the first reasonable reasonable model at low effort. But the problem is that um you don't know exactly a lot of things about how the users will interact with your model and so on. So, you can actually make a much better choice. So, the first problem is that if you look at many leaderboards, but not a single one, you will see that each public leaderboard will have a different ranking. So, here maybe it's a bit small, but you can trust me.

There are three leaderboards, LM Arena, now called Arena, Design Arena, Artificial Analysis, and they try to rank image editing models. And if you try to draw a bit the difference between the this leaderboards, you will see that it's not the same ranking. Like the top model is not the same. Also, like relatively, if you compare model two video between each other, they will be different. For example, there is one model, Human, that goes from rank 10 on Artificial Analysis and is ranked five on like Arena. So, the idea is that each leaderboard has a different perspective, and sometimes there are also some models that have duplicate entries.

So, it's a bit noise, a bit what is the information you want to get out get out of it. There are even some models that appear in the leaderboards and they are not in some others. So, it's hard to get the main information. And when you check actually in details like the Elo scores, so which is supposed to be the quality scores you use to know what is the best model, you will see that even these Elo scores, they are very different, meaning that for some leaderboards, it will be between 1,100 to 1,300, but for some it will be completely different range. So, relatively, models, we don't know how strong they they between each other.

And usually, the main solution for this is not to trust a single one, but you really to look at multiple one. And when you see that there is a lot of difference between like different rankings, it means that probably there are like some models which are approximately equivalent. It's not because ChatGPT image is ranked top top one on one on one leaderboard on the leaderboard that it means that it's the best overall. Another problem is in most cases you will have applica you will release and use your model for specific application. What we've seen before is like some aggregated score over a lot of different task. I don't know.

Uh for example, we we can see like removing objects, changing background, editing text. But we can actually build some leaderboards for each of the specific use case. And this is also some leaderboards from like uh design AI, I guess. And I think there is a problem with the Okay. It comes back. Um and you can see that actually, if you draw the difference for each specific use case, you will see that again the ranking they are completely different. And ChatGPT image is never top top one in this in this ranking.

There are always some new model and some models which are super good at removing objects, some models which are super good at doing some other things. That is there is no model consistently outperforming the others. And uh there are very different models working well for the for different target use cases. And this is normal because this is just due to to the fact that, you know, some models they have been, you know, for example, trained more on some specific task than others. And the solution for this is when you check like public leaderboard, you should always try to target what your use case will do in the end.

Like if you focus on removing objects, look at this leaderboard and not the others. Another problem is that usually leaderboards they are not really statistically significant for your specific use case. So, here I try to show two two different things. So, the first thing is on how many samples these leaderboards are built. And if you check on the left, for example, artificial analysis, this is all these different things they are built on, you know, few thousand samples for each of them. So, it's not much if you compare to the the load of inference you have for many applications, it's probably super low.

For some of our our models that we have, we have millions of friends per day, so probably we'll get more information by just like just training the model on our API rather than just looking at this at this leaderboards. Another thing is yellow scores. Usually, you can also compute what is the win rate of each model.

So, when you build these these rankings, what you what you do is you actually make models battle against each other and ask people, "Okay, what is the best What is the best model between the the two two two a lot of users?" And what you can see is actually the win rate usually there are no models which are close to 100% win rates. It means that most of the models they lose at least 40% of their of their battles. And if your use case is in this 40% of the battles, it means that you will just if you take the best models, you will just take the wrong model.

So, again here it's important really to average on more samples and always like have like evaluation which is close to the final setup the final use case conditions. Now, we can check also the second solution to to to try to know what is the state of the art for AI model. And the second solution is to do just internal benchmark. One way to do it is what I see the most actually in the image and video generation research and so on. People just do manual inspection. They try a couple of prompts, a couple of models, and they they get a feeling intuitively a bit what is the best model.

Another thing that sometimes people do is they just like run some benchmark automated benchmark out there and then try to see okay, based on this benchmark, which is the one that that has the best performance. So there yes, basically then you you just select the preferred model. So it can be I don't know for example the third model or the one with the highest score. The problem is that So there are a couple of problems with this and we can start this with a little game. So here I'm just going to show like three images.

And maybe one question for you is how many people in the room prefer the first prefer the first image among these three? So the this is a question like in general you can ask a lot of questions. So does it adhere the prompt? Is the what image do you prefer and so on? But the prompt was I I think a little guy and a parrot or something like this. And yeah. Okay, first. Who prefers the second image? Okay, couple of people. Who prefers the third image? Okay. So what is great here is that we have seen that people have different preference.

So it's important to see that if you do like manual inspection, you will be super biased to your own preference. So it's very important to not trust on your preference because then you have big surprise that actually it's not the the models that are preferred by by everyone. Now we can do it again. Same question, who prefers the first image in this case? I think the prompt was like probably a man eating some soup with pasta we with pasta or something like this. Okay. Who prefers the second image? Okay, great. And who prefers the third? Okay. So, that's also super interesting because I've seen some people changing their minds.

So, always on the left it was the the the Sid V model, middle Flux one, and uh on the right like uh with some models we developed one image. And the idea is like also you are super biased toward the few samples that you look at. So, when you do manual inspections, you are two times biased by you and by also the number of samples the specific samples you you look at. So, so in general, the idea is like you should never only trust the the the the manual inspection. It's good to get a feeling, but it's not enough.

You should always ask many people uh to do it, and human evaluation is usually great, but you have to scale it um properly. Another problem is that when you do not human evaluation, but more like proper like uh automation evaluation with uh with metrics, sometimes you have like non-consistent results. So, for example, this is a bit small, but you can trust me. We ranked like eight models regarding some metrics like a very standard metric which is called CLIP score, and sometimes people when they try to evaluate image models they they check this metric first.

And you can see actually that if you check like the rankings for uh the three metrics we looked like CLIP score on different datasets, it changed all the time. And these metrics are supposed to be between zero and zero and one or zero and 100, and actually the variations between models they are super small. So, it means it means that it's hard to know from this metric what is the best model. What you should do is actually first having some clear understanding of what the metric does, and also use multiple multiple of them. So, here for example, this is another type of metric. When you know you you know your use case.

For example, you know to you you know you want to be the best at text rendering. You there are a lot of text rendering metrics that would be better to evaluate your models. So, here you can see again like the ranking is way more consistent. You have always the image being the first and P image being the second model. And also the variations, they are way more significant. So, the models are supposed to be zero uh the metrics are supposed to be between zero and one. And there are like clear difference uh between like every every model. So, yes, in general, very important understanding metrics.

People usually tend to just use some metrics and see, "Okay, I did my benchmark and then I stop here." But it's important to understand what you actually measured with this. And now a a last problem, which is actually common to the the first and second method to that we that we've seen before is that usually quality is driven by compute. So, here this is Chat pretty image. And for their evaluation, uh Design Arena, uh I think or maybe it's LM Arena. Um they did like 27 uh 26 K battles. So, it means they generated 20 um 26 K images. And each of this image takes 1 minute to generate.

So, here I summarized all this information. Si- 62 seconds. Uh 26 K evaluations. And in total to do this 26 K evaluations, it takes 20 days of compute. In terms of cost, it's 5 K just 5 K just to to to run this evaluation. And in terms of energy, it's approximately, you know, 556 kW kW hour. So, I know that people might not have the order of magnitude of what it represents, this amount of energy. So, just to give like some idea, I checked my Strava and checked how much energy I was consuming by running a marathon and actually it represents 400 marathons just to generate all these images. So, it's a lot.

I'm tired after one marathon, so I don't want to do 400 for sure. Now, there are some alternative. You can use some different models. So, of course this is a model that we've we've done that does like time generation editing of images in less than 1 second. And for the same amount of evaluation, it takes only 7 hours. It takes also like it uses also way less money, so 265 dollars. And instead of running 500 500 marathons, I just need to run four marathons. So, if three of you want to run a marathon with me, it should be enough to to to do this.

So, again the idea is like people tend to just look at quality, but it's important not to look only at quality, but also at efficiency because sometimes the the additional gain you get with quality is not worth the efficiency the the the compute cost. So, to the question what model is state of the art, the answer is there are multiple state of the art model. And the tool I prefer for this is usually the Pareto plots where basically on the x-axis you have one efficiency metric. For example, on the left it's latency for the generation of an image. On the right it's the price for the generation of an image.

On the y-axis you have some quality score, let's say the L score. And here you can draw the Pareto fronts in red and you can see that there is not one single state of the art model, but there are actually multiple of them and there are like three or four. And you can see that even though the quality score is not there is no big variations, it's always always between 1,000 100 and 1,200, there is a big difference in terms of efficiency. So, you can be like really like times times I don't know, 20 times faster just by using the different model.

Even better, if you know the specific task you want to do, you can do it like the Pareto front not with quality metric which focuses on general general capability, but really based on quality metrics which is for the target use case. So, this is some Pareto fronts focusing on text rendering. And here for example, we optimized a lot like the flux two model or flux two flex models. We worked with BFL for for this. And you can see that you can get way faster. You can still be on the Pareto front for the specific use case of text rendering. So, is benchmarking dead? The idea is it's not dead.

We can do it properly and get a lot of useful information out of it. Yes. And if you use it in a better way like by taking all these you know rules when using the evaluation, you will usually usually not find a large long way a large foundational model, but more like a lot of small performance models that will be very good for your use case. So, I just listed a couple of takeaways which are like evaluate on many samples. You look at the user use case conditions. Use multiple benchmarks or efficiency which are key things to keep in mind when evaluating models. And how to reach like state of the art models in general?

Like this is what we are doing at Puna. We are actually building a lot of what we call performance models with that are served behind hand point hand points. We have the fastest for example image models, video models that can run between 1 seconds to 5 seconds. And but we also try to give a lot to the open source with a lot of open source contributions with the package to show you how to compress your models on your own. Also a lot of materials on all the best research papers for efficiency or even like some efficiency cost. So, thanks for your attention.

I think you are out of time, but if there are any questions, happy to take them. Okay, perfect. Then you have a question? Uh sure, so actually there are multiple like I mean, you know it as well, there are a lot of family of compression methods. So, of course you can guess like quantization things we do a lot, and we do it a different quantization for every specific module in the in the model, which is super important. Uh we can do also some pruning where we just remove some components which are not important.

And uh for all these image and video models, something that works quite well is working on the step that the the denoiser um like when you generate a video or an image, you usually use like 20 to 50 steps to generate like um uh the the content, and you can actually reduce it a lot either via distillation of or caching methods. So, you can instead of doing like 50 times the computations using the same backbone, you can do it way less, I don't know, 20 times or even like four times, depending on how aggressive you want to be.

Uh So, we have a like in our package, we have a lot of open source uh algorithms for good caching. Uh but we have also some internal, you know, algorithms that we have for the models we serve behind the the endpoints. But uh yeah, there are really advanced caching methods and so on, but yeah. Sure. Thanks. Okay, then