--- title: "Transcript: Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod" category: "transcripts" videoId: "ILdE7FaAjVA" sourceLabels: ["YouTube transcript", "Cached transcript markdown"] wordCount: "1823" --- # Transcript: Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod ## Source Video - [YouTube](https://www.youtube.com/watch?v=ILdE7FaAjVA) ## Local Cache - `raw/sources/youtube-transcripts/ILdE7FaAjVA.txt` - 1,823 words ## Transcript [music] >> Audrey, um I am from RunPod. Um this is an intro to RunPod. Can I just get a quick hands to see how many people have already heard of RunPod or maybe even used RunPod before? Okay, newbies for everybody. Great. Um So, RunPod, we are a cloud AI infrastructure company. So, we have the hardware, we have the GPUs, and we make it easy for developers to deploy um models. And that can be your own private model, it can be an open-source model from Hugging Face. Doesn't matter to us. You bring your code and we'll bring the rest. Um just really quickly, what problems does RunPod solve? Like, why are we even here today? Um infrastructure can be hard, managing it. I think about back in the day before we had AWS, um Google Cloud, when everybody would have to have on-prem servers and manage those, maintain those. That [clears throat] is something that we don't want to have to do as developers. Those are things that we happily um have moved away from and given off to DevOps, and now it's even it's even more abstracted for us. GPU access is slow and opaque. So, um I don't know if you if anybody has tried to buy a GPU recently, we're in a global supply crunch. Um it's a bit like in COVID when everybody went to the store and bought all the toilet paper because we didn't know how how long they would need to be at home for. We're a little bit in that right now. Um but we expect the market will recover um as customers, companies, people figure out a little bit better, uh get a little bit better at um estimating what kind of compute they need. Um and then last, builder primary focus should be building. So, again, um we want to build we as software developers, uh we bring bring the value through the applications that we build, um not from managing the infrastructure. And I think RunPod has a pretty unique story. These are These are our founders, Zenon and Pardeep. Um so, they had a couple of GPU rigs in their basement in 2022, um failed crypto mining, and then so they're like, "What are we going to do with our GPUs now?" Um so, they prototyped what is now the foundations of RunPod. They posted on Reddit and said, "Hey, anyone want to use these GPUs for free? Just give us feedback on it." And that is literally how our company has started, and we have been um revenue generating ever since. Um and the reason why I want to tell this story is um not because it's it's very like bootstrappy, but because um the origin origin story of RunPod has always started with uh builders and getting feedback from the community, and that is still true today. So, I won't promise that we'll be perfect, but um we are definitely very engaged with um our users on on Reddit, um on Reddit, on Discord. So, um we're always trying to stay engaged with y'all. Um just at a glance to give you an idea of RunPod, we have over 500,000 developers on our platforms, 30-plus data centers across the world, including um Europe and the EU. Um and we've just passed a significant revenue milestone for us, 120 million in annual recurring revenue. Uh these are just a few of our customers. Um you might be surprised to see some of the AI cloud-native companies on here, too, but um they come to us for the same reasons that most of our customers come to us. It's um because they need flexible and reliable GPU infrastructure. This is a really high-over high-level overview of um different ways you can build on RunPod. So, I would say our core at our core, um pods, it's our sandbox virtual environment. We spin up a container for you, um allocate GPUs to it, and we manage the rest. You just bring your um Dockerfiles, you bring your code. Serverless, um it's our auto-scaling product. So, when you're thinking more about like bursty workloads or batch workloads, um serverless is really great because um instead of being always on like a container is, serverless um your workers spin down, and when they're idle you don't pay for anything. Clusters, um if you were doing some heavy-duty training, there's a place for you as well on RunPod. Um multi-node clusters with high-speed networking. And then the hub, which I'll I'll I'll switch to in a second, um it's kind of like our central repository for AI repos. Um these are already preconfigured, pre-vetted. Um we have a couple of examples of listings by RunPod for popular models, but also our community um contributes to them as well. So, there's repos that you can fork, you can watch, and then um you can star and deploy on RunPod. Um so, today we're going to be talking mostly about serverless. Um so, serverless is best for real-time inference. I talked about the auto-scaling that comes with it. Um why teams use it is mostly because they don't need to um preempt and figure out how much compute they need ahead of time. Um You can set you can configure the number of max workers that you want to scale up to. You can set limits for caps, for spending caps. And you can also configure workers that are always on, so they're um already have your models downloaded, and they can respond to requests um immediately. For a lot of teams, serverless is the fastest way um if you want to start deploying a production-ready API. And now I'm going to switch over and just show y'all really quick how easy it is to get started and deploy something. Okay. Where are we? Okay. So, right now I'm um going to do everything via the console so that it's nice and pretty for you guys to see, but we also have um CLI support. We have skills um to help work with RunPod, everything that's ready for your agent so you don't have to read our documents, but since we're all humans here today, I'm going to show you via the console. Um we'll start in the hub, which is if you're just trying to explore and see what's out there, what is something that you can get up and running right now, the hub is a great place to start. So, like I mentioned, these are already vetted open-source listings for um AI repos, and I am going to pick the LLM, um and I'll just open the underlying repository as well so you could guys can see what it is literally just a GitHub um repo. It tells you how to get set up for it. Um we can see there's already the Dockerfile here. It's already preconfigured for you. It's got some defaults for you um depending on the listing. You can um pass in different environmental variables um to configure it how you wish. But I'm just going to go ahead and click deploy. And I have a model that I wanted I Let me see. I was going to just pick one. Looks well. This is going to download it from Hugging Face. I'm just expand the advanced options and look for the max model length, and I'm going to bump this up for the context window, and leave everything else as the defaults, but there's settings for max loras. Um all of these configuration options get passed as um as flags to the uh vLLM serve. And I'm going to spin it up as an endpoint here. This might take a minute since or two since this is the very I just created it. I've got to initialize my workers. Let's check out. So, the default configuration here is it's going to deploy on some H100s, and A100s are the backup here. I have my pricing. This is fraction of a cent per second. Um like I mentioned before that this is only going to be charged for a while the worker is actually running and handling a request. Max workers is where I can bump this up if I want to have my workload scale up up to 15 workers at a time and I can set um some active workers, ones that I want always to be on that I don't want um the container to ever spin down. And I can save that. Okay, so how do I how does one interact with the serverless endpoint? Um this is just an API uh HTTP endpoint right here. Um we provision this endpoint for you. You can send requests to this. Your customers can send requests to this. If I just hit run and I'm going to add a few Let's Which which should we ask the LLM today? Say one. Okay. How did I'm I'm American, so how did Bid Then get its name? I don't know. >> [clears throat] >> Um okay, well these requests are queued. Let me check on our workers. Okay. We have a handful that are in initializing. Um this is the container's being created. That's the model being downloaded. Um getting ready and the ones that are running, they've already finished. These are probably going to be the ones who are going to pick up those requests that we just added. I've got telemetry about um it's blank right now, but the number of requests, execution time, delay time, so you have observability into how your endpoints are operating. >> [snorts] >> And let's see. Okay, it's already done. >> [snorts] >> I got a request back and it sat in the queue for about 41 seconds. Um that's going to be a little bit longer than all of the subsequent requests because of some of the cold start time that I talked about, like downloading the model, um initializing the first container, but um execution time only about 1 and 1/2 seconds, so yeah, that was probably less than 5 minutes to get started and get something deployed um on serverless from a hub listing. Does anyone have any questions? This is is a very short and sweet intro. Um we have another session later today at 4:00 um and that one is going to be focused on our Python um SDK and that one is going to be completely via the terminal. Um and I'm going to walk you through how I can spin up uh and deploy my code on my code as a remote remote function onto a GPU um and uh deploy in the end and make it like a production-ready endpoint here as well. Okay. But that's all I got for today, so yeah, thanks thanks for coming. >> [applause] [music]