Markdown source

Transcript: What if the harness mattered more than the model? - Aditya Bhargava, Etsy

Source Video

Local Cache

Transcript

Hi everyone. My name is Audit, Hi everyone. My name is Audit, pronounced Audit like a tax audit. Um pronounced Audit like a tax audit. Um pronounced Audit like a tax audit. Um and I'm here to talk to you about what and I'm here to talk to you about what and I'm here to talk to you about what if the harness mattered more than the if the harness mattered more than the if the harness mattered more than the model? model? model? A little bit about me first. I'm a staff A little bit about me first. I'm a staff A little bit about me first. I'm a staff engineer at Etsy. I'm also Etsy's IC engineer at Etsy. I'm also Etsy's IC engineer at Etsy. I'm also Etsy's IC initiative lead for agent of commerce. I initiative lead for agent of commerce. I initiative lead for agent of commerce. I wrote Grokking Algorithms, which is an wrote Grokking Algorithms, which is an wrote Grokking Algorithms, which is an illustrated book on algorithms. illustrated book on algorithms. illustrated book on algorithms. Uh and I also do illustrated posts on Uh and I also do illustrated posts on Uh and I also do illustrated posts on ducktyped.org. ducktyped.org. ducktyped.org. Um Um Um and I want to talk to you about and I want to talk to you about and I want to talk to you about what if the harness mattered more than what if the harness mattered more than what if the harness mattered more than the model? the model? the model? By which I mean By which I mean By which I mean what if we should be paying more what if we should be paying more what if we should be paying more attention attention attention to building expertise in harnesses? to building expertise in harnesses? to building expertise in harnesses? Um Um and my controversial take here is and my controversial take here is and my controversial take here is I hear a lot of people say, "Oh, the I hear a lot of people say, "Oh, the I hear a lot of people say, "Oh, the models are so good that you can just models are so good that you can just models are so good that you can just keep the harness simple. Just give the keep the harness simple. Just give the keep the harness simple. Just give the model a few tools and it'll do the model a few tools and it'll do the model a few tools and it'll do the rest." rest." rest." And I've been hearing this so often that And I've been hearing this so often that And I've been hearing this so often that it has now it has now it has now uh started becoming the wisdom of the uh started becoming the wisdom of the uh started becoming the wisdom of the tech industry. tech industry. tech industry. Uh Uh Uh to which my answer is, "Yes, to which my answer is, "Yes, to which my answer is, "Yes, but that's moving in the wrong direction but that's moving in the wrong direction but that's moving in the wrong direction because that is making us reliant on because that is making us reliant on because that is making us reliant on fancy proprietary models that can't be fancy proprietary models that can't be fancy proprietary models that can't be run run run locally. locally. locally. So, what if we instead talked about the So, what if we instead talked about the So, what if we instead talked about the other direction? What if we focused on other direction? What if we focused on other direction? What if we focused on open-source models that can be run open-source models that can be run open-source models that can be run locally? locally? locally? What if we focused on building a harness What if we focused on building a harness What if we focused on building a harness that is so good that we can get the that is so good that we can get the that is so good that we can get the performance of a cutting-edge model performance of a cutting-edge model performance of a cutting-edge model through a local open-source model? through a local open-source model? through a local open-source model? So, what if the harness mattered So, what if the harness mattered So, what if the harness mattered more than the model? What if our focus more than the model? What if our focus more than the model? What if our focus should be on the harness? should be on the harness? should be on the harness? Um how much does a harness matter? Um how much does a harness matter? Um how much does a harness matter? Right? This is something I have been Right? This is something I have been Right? This is something I have been researching a lot in my spare time. researching a lot in my spare time. researching a lot in my spare time. Um Um there's a really interesting paper that there's a really interesting paper that there's a really interesting paper that talks about Harness Bench, talks about Harness Bench, talks about Harness Bench, which is a benchmark for harnesses, which is a benchmark for harnesses, which is a benchmark for harnesses, which is something we haven't seen a lot which is something we haven't seen a lot which is something we haven't seen a lot of yet. of yet. of yet. Um Um it's got 106 tasks. Um it's got 106 tasks. Um it's got 106 tasks. Um it's testing these different models in it's testing these different models in it's testing these different models in different harnesses. different harnesses. different harnesses. Uh but the point is Uh but the point is Uh but the point is that it's running the same setup. It's that it's running the same setup. It's that it's running the same setup. It's doing the same evaluation, doing the same evaluation, doing the same evaluation, same model, same model, same model, but testing different harnesses. but testing different harnesses. but testing different harnesses. Um and the results are pretty Um and the results are pretty Um and the results are pretty interesting. interesting. interesting. So, scores range from 52.4% So, scores range from 52.4% So, scores range from 52.4% to 76.2%. to 76.2%. to 76.2%. So, more than a 20-point difference, So, more than a 20-point difference, So, more than a 20-point difference, and only the harness changed. and only the harness changed. and only the harness changed. Um and the really interesting thing is Um and the really interesting thing is Um and the really interesting thing is that for weaker models, the harness that for weaker models, the harness that for weaker models, the harness matters more. matters more. matters more. Um if the harness matters more than the Um if the harness matters more than the Um if the harness matters more than the model, that's great news. model, that's great news. model, that's great news. Uh because if the model matters more, Uh because if the model matters more, Uh because if the model matters more, then we're dependent on a handful of then we're dependent on a handful of then we're dependent on a handful of companies who are able to build and companies who are able to build and companies who are able to build and train these models. train these models. train these models. Uh but if the if a good harness can Uh but if the if a good harness can Uh but if the if a good harness can compensate, and can make a weaker model compensate, and can make a weaker model compensate, and can make a weaker model perform better, perform better, perform better, then you can build our own harnesses, then you can build our own harnesses, then you can build our own harnesses, and that's something that any of us can and that's something that any of us can and that's something that any of us can do, do, do, and we don't have to depend on paid and we don't have to depend on paid and we don't have to depend on paid models. models. models. Um so, I have been spending a lot of Um so, I have been spending a lot of Um so, I have been spending a lot of time time time and doing a lot of research specifically and doing a lot of research specifically and doing a lot of research specifically into into into what would it take to build a really what would it take to build a really what would it take to build a really good harness. good harness. good harness. Um Um and and and I realized early on that none of the I realized early on that none of the I realized early on that none of the existing tools or frameworks existing tools or frameworks existing tools or frameworks of were able to do what I wanted. And I of were able to do what I wanted. And I of were able to do what I wanted. And I realized that what was actually needed realized that what was actually needed realized that what was actually needed was a language-level solution. So, this was a language-level solution. So, this was a language-level solution. So, this is my other controversial take in this is my other controversial take in this is my other controversial take in this talk is I think building a good harness talk is I think building a good harness talk is I think building a good harness actually requires language-level actually requires language-level actually requires language-level support. support. support. Um and this is kind of where my research Um and this is kind of where my research Um and this is kind of where my research has gone in the last 6 months and I've has gone in the last 6 months and I've has gone in the last 6 months and I've spent spent spent a a a good part of that 6 months good part of that 6 months good part of that 6 months building a new language called Agency, building a new language called Agency, building a new language called Agency, which is a language for building agents. which is a language for building agents. which is a language for building agents. Um Um and just to talk about the terminology and just to talk about the terminology and just to talk about the terminology there real quick because the terms agent there real quick because the terms agent there real quick because the terms agent and harness and harness and harness aren't aren't aren't super crisp in the industry yet. They're super crisp in the industry yet. They're super crisp in the industry yet. They're not super well defined. Um but in this not super well defined. Um but in this not super well defined. Um but in this talk I'll kind of use them talk I'll kind of use them talk I'll kind of use them interchangeably. But when I talk about interchangeably. But when I talk about interchangeably. But when I talk about the agent, I really mean the agent, I really mean the agent, I really mean the harness and the model. the harness and the model. the harness and the model. So if you take Claude code for example, So if you take Claude code for example, So if you take Claude code for example, Claude code is an agent. It uses Opus, Claude code is an agent. It uses Opus, Claude code is an agent. It uses Opus, which is the model, and then everything which is the model, and then everything which is the model, and then everything else is the harness. else is the harness. else is the harness. Um Um but you know, when we talk about Agency, but you know, when we talk about Agency, but you know, when we talk about Agency, it's really for building it's really for building it's really for building agents, but the thing it's providing is agents, but the thing it's providing is agents, but the thing it's providing is the harness. the harness. the harness. Um Um Agency is really good. Uh you know, it's Agency is really good. Uh you know, it's Agency is really good. Uh you know, it's still new. still new. still new. Building a language is a lot of work and Building a language is a lot of work and Building a language is a lot of work and it's only 6 months in. it's only 6 months in. it's only 6 months in. Uh but Uh but Uh but I love this quote by Alan Kay. Simple I love this quote by Alan Kay. Simple I love this quote by Alan Kay. Simple things should be simple, complex things things should be simple, complex things things should be simple, complex things should be possible. should be possible. should be possible. And that's something I love about Agency And that's something I love about Agency And that's something I love about Agency is that is that is that I think you know, when people build I think you know, when people build I think you know, when people build agents, there are a lot of primitives, a agents, there are a lot of primitives, a agents, there are a lot of primitives, a lot of things that every agent lot of things that every agent lot of things that every agent will need to be able to do will need to be able to do will need to be able to do and those things should be simple to do. and those things should be simple to do. and those things should be simple to do. Um and there's a lot of complex things Um and there's a lot of complex things Um and there's a lot of complex things that we want to be able to do with that we want to be able to do with that we want to be able to do with agents and it should be possible agents and it should be possible agents and it should be possible to do those things in the framework of to do those things in the framework of to do those things in the framework of your choice and with Agency it is your choice and with Agency it is your choice and with Agency it is possible. possible. possible. Um Um So now we have a clear goal, which is So now we have a clear goal, which is So now we have a clear goal, which is try to build the best possible harness. try to build the best possible harness. try to build the best possible harness. And we have a great tool to do it, which And we have a great tool to do it, which And we have a great tool to do it, which is agency. is agency. is agency. Uh so, the rest of this talk I'm going Uh so, the rest of this talk I'm going Uh so, the rest of this talk I'm going to talk about building a coding agent in to talk about building a coding agent in to talk about building a coding agent in agency. agency. agency. And we'll see the same agent evolve over And we'll see the same agent evolve over And we'll see the same agent evolve over seven different examples. seven different examples. seven different examples. We'll see the same model, the same task, We'll see the same model, the same task, We'll see the same model, the same task, but the harness will improve each time, but the harness will improve each time, but the harness will improve each time, and we'll see the harness improve the and we'll see the harness improve the and we'll see the harness improve the agent's performance each time. agent's performance each time. agent's performance each time. Uh Uh So, you know, like I said, I've been So, you know, like I said, I've been So, you know, like I said, I've been doing a ton of research on this. My doing a ton of research on this. My doing a ton of research on this. My goals with this talk are to give you goals with this talk are to give you goals with this talk are to give you like the basics of building a harness. like the basics of building a harness. like the basics of building a harness. When we talk about building a good When we talk about building a good When we talk about building a good harness, what does that mean? You know, harness, what does that mean? You know, harness, what does that mean? You know, you're starting from zero, what are the you're starting from zero, what are the you're starting from zero, what are the basics? basics? basics? Um Um I'll cover a good bit about agent safety I'll cover a good bit about agent safety I'll cover a good bit about agent safety because, you know, when we talk about because, you know, when we talk about because, you know, when we talk about building agents, building agents, building agents, what you really want them to be able to what you really want them to be able to what you really want them to be able to do is take action on our behalf. Um But do is take action on our behalf. Um But do is take action on our behalf. Um But the big problem with that is how do you the big problem with that is how do you the big problem with that is how do you get how do you give agent capabilities get how do you give agent capabilities get how do you give agent capabilities so they can take actions, so they can take actions, so they can take actions, but but but not so much capability that they do not so much capability that they do not so much capability that they do something unsafe. So, when we talk about something unsafe. So, when we talk about something unsafe. So, when we talk about agent quality and capability, agent agent quality and capability, agent agent quality and capability, agent safety goes hand in hand with that. safety goes hand in hand with that. safety goes hand in hand with that. Um At the end, I'll do an introduction Um At the end, I'll do an introduction Um At the end, I'll do an introduction to some of the more advanced concepts to some of the more advanced concepts to some of the more advanced concepts like sub-agents and self-optimization. like sub-agents and self-optimization. like sub-agents and self-optimization. I think that's where, you know, more I think that's where, you know, more I think that's where, you know, more interesting things come in, and we interesting things come in, and we interesting things come in, and we really start to talk about like really start to talk about like really start to talk about like what are all the possibilities, you what are all the possibilities, you what are all the possibilities, you know, because when you talk about know, because when you talk about know, because when you talk about building a good harness, there are so building a good harness, there are so building a good harness, there are so many things to explore. many things to explore. many things to explore. And so, I hope you stick with me till And so, I hope you stick with me till And so, I hope you stick with me till the end there because that's where we the end there because that's where we the end there because that's where we get into some really interesting stuff. Um so this is the code we're going to Um so this is the code we're going to ask our coding agent to edit. ask our coding agent to edit. ask our coding agent to edit. This is a code to calculate the median This is a code to calculate the median This is a code to calculate the median of an ordered of an ordered of an ordered list of numbers. list of numbers. list of numbers. Um Um and if it's an odd length list, you just and if it's an odd length list, you just and if it's an odd length list, you just take the middle element. If it's an even take the middle element. If it's an even take the middle element. If it's an even length list, length list, length list, you need to take the middle two elements you need to take the middle two elements you need to take the middle two elements and take the mean of those two elements. and take the mean of those two elements. and take the mean of those two elements. But right now this function doesn't do But right now this function doesn't do But right now this function doesn't do that. that. that. So there is a bug in this function. So there is a bug in this function. So there is a bug in this function. There's a test There's a test There's a test that shows the failure um that shows the failure um that shows the failure um and we're going to ask our agent and we're going to ask our agent and we're going to ask our agent to fix this code. So let's go ahead and start. Let's just So let's go ahead and start. Let's just dive in. dive in. dive in. Um this is the first example. This is Um this is the first example. This is Um this is the first example. This is agency code now. agency code now. agency code now. And you'll notice that And you'll notice that And you'll notice that agency looks a lot like TypeScript. It's agency looks a lot like TypeScript. It's agency looks a lot like TypeScript. It's heavily inspired with by TypeScript with heavily inspired with by TypeScript with heavily inspired with by TypeScript with some some some Python syntax thrown in. Python syntax thrown in. Python syntax thrown in. Um so the entry point is this node main. Um so the entry point is this node main. Um so the entry point is this node main. I have a simple prompt. The Python I have a simple prompt. The Python I have a simple prompt. The Python function median in demo/median.py function median in demo/median.py function median in demo/median.py has a bug. Please fix the bug. has a bug. Please fix the bug. has a bug. Please fix the bug. And I just passed that prompt into the And I just passed that prompt into the And I just passed that prompt into the LLM function. I get the result and I LLM function. I get the result and I LLM function. I get the result and I print the result. print the result. print the result. Um Um And of course this isn't going to work And of course this isn't going to work And of course this isn't going to work because it can't read or write to this because it can't read or write to this because it can't read or write to this file. So let's run this example file. So let's run this example file. So let's run this example real quick. Okay. So of course as expected it says Okay. So of course as expected it says you know, in order to help me fix the you know, in order to help me fix the you know, in order to help me fix the bug I'll need to see the existing code. bug I'll need to see the existing code. bug I'll need to see the existing code. Um Um so this is just the model. Very little so this is just the model. Very little so this is just the model. Very little harness and of course it can't do harness and of course it can't do harness and of course it can't do anything. So, the most obvious anything. So, the most obvious anything. So, the most obvious improvement improvement improvement to any harness is give it some tools. to any harness is give it some tools. to any harness is give it some tools. So, So, So, real quick, let's talk about how tools real quick, let's talk about how tools real quick, let's talk about how tools work in the agency cuz it's very simple. work in the agency cuz it's very simple. work in the agency cuz it's very simple. Um I'll define a function. In this case, Um I'll define a function. In this case, Um I'll define a function. In this case, just for an example, I'm going to define just for an example, I'm going to define just for an example, I'm going to define a greet function a greet function a greet function that greets a user. that greets a user. that greets a user. And then I'll make an LLM call that And then I'll make an LLM call that And then I'll make an LLM call that says, "Use your greet tool to greet says, "Use your greet tool to greet says, "Use your greet tool to greet audit." And then pass in that tool. audit." And then pass in that tool. audit." And then pass in that tool. Um Um and every function can automatically be and every function can automatically be and every function can automatically be used as a tool in agency. So, there is used as a tool in agency. So, there is used as a tool in agency. So, there is nothing else to do. nothing else to do. nothing else to do. Uh agency creates a JSON schema out of Uh agency creates a JSON schema out of Uh agency creates a JSON schema out of this function and sends it to the LLM this function and sends it to the LLM this function and sends it to the LLM call. call. call. The doc string here will get used as a The doc string here will get used as a The doc string here will get used as a tool description. tool description. tool description. So, let's go ahead and run the second So, let's go ahead and run the second So, let's go ahead and run the second example. example. example. So, NPX agency NPX agency example since actually 01 NPX agency example since actually 01 cuz we're going to talk about tools cuz we're going to talk about tools cuz we're going to talk about tools first. first. first. So, howdy audit? So, howdy audit? So, howdy audit? And how can we know that it ran that And how can we know that it ran that And how can we know that it ran that tool and then just generate an example, tool and then just generate an example, tool and then just generate an example, right? So, I'll just also show real right? So, I'll just also show real right? So, I'll just also show real quick. quick. quick. Um Um agency writes out logs and has this agency writes out logs and has this agency writes out logs and has this great log viewer built in. great log viewer built in. great log viewer built in. So, I'll just expand this last run So, I'll just expand this last run So, I'll just expand this last run and you can see here it's making this and you can see here it's making this and you can see here it's making this tool call to the greet function with tool call to the greet function with tool call to the greet function with these parameters. The tool responds by these parameters. The tool responds by these parameters. The tool responds by returning howdy audit, and the assistant returning howdy audit, and the assistant returning howdy audit, and the assistant just passes that straight back to us. just passes that straight back to us. just passes that straight back to us. Um So, that's tools. Now, let's look at using that's tools. Now, let's look at using that's tools. Now, let's look at using the tools the tools the tools in Agency. So, again, this is in Agency. So, again, this is in Agency. So, again, this is you know, the next step of our agent, you know, the next step of our agent, you know, the next step of our agent, which is give it tools. which is give it tools. which is give it tools. Um Um here is an example where we're giving here is an example where we're giving here is an example where we're giving the agent read and write tools. the agent read and write tools. the agent read and write tools. Again, read and write are just functions Again, read and write are just functions Again, read and write are just functions that are built into Agency, and because that are built into Agency, and because that are built into Agency, and because every function can be used as a tool, I every function can be used as a tool, I every function can be used as a tool, I can just pass this straight to the LLM. can just pass this straight to the LLM. can just pass this straight to the LLM. Uh except Uh except Uh except giving the agent the ability to read and giving the agent the ability to read and giving the agent the ability to read and write arbitrary files on our file system write arbitrary files on our file system write arbitrary files on our file system is really unsafe. So, by default Agency is really unsafe. So, by default Agency is really unsafe. So, by default Agency won't allow you to do this. Agency has a won't allow you to do this. Agency has a won't allow you to do this. Agency has a human-in-the-loop feature that would human-in-the-loop feature that would human-in-the-loop feature that would require some sort of approval. require some sort of approval. require some sort of approval. We're not providing it anywhere, so this We're not providing it anywhere, so this We're not providing it anywhere, so this code will crash and print an error. So, code will crash and print an error. So, code will crash and print an error. So, let me show you what that looks like. Great. So, interrupt standard read was Great. So, interrupt standard read was not handled. not handled. not handled. Your interrupt message, "Are you sure Your interrupt message, "Are you sure Your interrupt message, "Are you sure you want to read this file?" You can see you want to read this file?" You can see you want to read this file?" You can see the file name it's trying to read, and the file name it's trying to read, and the file name it's trying to read, and it's basically telling us that we didn't it's basically telling us that we didn't it's basically telling us that we didn't approve this interrupt. And it's giving approve this interrupt. And it's giving approve this interrupt. And it's giving us a pointer to the guide, right? Tells us a pointer to the guide, right? Tells us a pointer to the guide, right? Tells us how to use handlers. us how to use handlers. us how to use handlers. Um Um so so so we started with just a model, we tried we started with just a model, we tried we started with just a model, we tried tools, and the next step is to make the tools, and the next step is to make the tools, and the next step is to make the tools safe. So, we're going to do tools safe. So, we're going to do tools safe. So, we're going to do exactly what that message asked us to do exactly what that message asked us to do exactly what that message asked us to do and use a handler. and use a handler. and use a handler. So, now, here's the same code So, now, here's the same code So, now, here's the same code that you know, passing through that read that you know, passing through that read that you know, passing through that read and write and write and write uh those read and write functions as uh those read and write functions as uh those read and write functions as tools. tools. tools. Um but now it's wrapped in a handle Um but now it's wrapped in a handle Um but now it's wrapped in a handle block. block. block. Uh and it has a handler function down Uh and it has a handler function down Uh and it has a handler function down here. here. here. So, now when those tools call or raise So, now when those tools call or raise So, now when those tools call or raise an interrupt, an interrupt, an interrupt, this code is going to execute. this code is going to execute. this code is going to execute. So, it'll So, it'll So, it'll print the information about that print the information about that print the information about that interrupt. interrupt. interrupt. Um, Um, Um, it'll ask the user for input using the it'll ask the user for input using the it'll ask the user for input using the built-in input function. And if the user built-in input function. And if the user built-in input function. And if the user approves, approves, approves, the tool call is approved. the tool call is approved. the tool call is approved. And I know I'm saying interrupt, and I'm And I know I'm saying interrupt, and I'm And I know I'm saying interrupt, and I'm not really defining what that means. not really defining what that means. not really defining what that means. Um, and I can get to that a little bit Um, and I can get to that a little bit Um, and I can get to that a little bit later, but it's essentially a way to later, but it's essentially a way to later, but it's essentially a way to pause execution at some point. pause execution at some point. pause execution at some point. Um, and ask for human input. Um, and ask for human input. Um, and ask for human input. And so, interrupts are a great feature And so, interrupts are a great feature And so, interrupts are a great feature in agency, and all the functions in the in agency, and all the functions in the in agency, and all the functions in the agency standard library agency standard library agency standard library that that that mutate code or mutate something, or have mutate code or mutate something, or have mutate code or mutate something, or have some sort of destructive action, or read some sort of destructive action, or read some sort of destructive action, or read sensitive data data, sensitive data data, sensitive data data, all all all throw an interrupt first. throw an interrupt first. throw an interrupt first. So, So, here's the safe version of our agent here's the safe version of our agent here's the safe version of our agent where where where the tools the tools will raise an interrupt by default, will raise an interrupt by default, will raise an interrupt by default, and now this code is going to ask the and now this code is going to ask the and now this code is going to ask the user, do you approve? user, do you approve? user, do you approve? Let's run this example. Okay, are you sure you want to read this Okay, are you sure you want to read this file? file? file? I'll just say yes. I'll just say yes. I'll just say yes. You can see it's reading. Now it's You can see it's reading. Now it's You can see it's reading. Now it's trying to write to that file. So, you trying to write to that file. So, you trying to write to that file. So, you know, know, know, that's an example of that's an example of that's an example of >> [snorts] >> [snorts] >> [snorts] >> us >> us >> us making the agent safer by asking for making the agent safer by asking for making the agent safer by asking for user input. user input. user input. And this is better, and it is safe, but And this is better, and it is safe, but And this is better, and it is safe, but it's also very slow. it's also very slow. it's also very slow. So, the next thing we want to do is make So, the next thing we want to do is make So, the next thing we want to do is make the agent autonomous, the agent autonomous, the agent autonomous, while still keeping it safe, and that's while still keeping it safe, and that's while still keeping it safe, and that's the tricky part, right? How do you give the tricky part, right? How do you give the tricky part, right? How do you give it it it just enough capability just enough capability just enough capability so that you don't have to manually so that you don't have to manually so that you don't have to manually approve approve approve every single action every single action every single action without giving it without giving it so without giving it without giving it so without giving it without giving it so much capability that it's like able to much capability that it's like able to much capability that it's like able to do something destructive with your file do something destructive with your file do something destructive with your file system. system. system. Um so agency has a really cool feature Um so agency has a really cool feature Um so agency has a really cool feature to handle this. It that's called partial to handle this. It that's called partial to handle this. It that's called partial function application. function application. function application. >> [clears throat] >> [clears throat] >> [clears throat] >> Um >> Um >> Um it's a concept borrowed from functional it's a concept borrowed from functional it's a concept borrowed from functional programming. It's a really fancy term programming. It's a really fancy term programming. It's a really fancy term for a pretty simple concept. for a pretty simple concept. for a pretty simple concept. So, here's a signature of the read tool, So, here's a signature of the read tool, So, here's a signature of the read tool, right? right? right? Takes a file name and a directory. Takes a file name and a directory. Takes a file name and a directory. Um and when you read a file, it's going Um and when you read a file, it's going Um and when you read a file, it's going to look inside that directory for that to look inside that directory for that to look inside that directory for that file name. file name. file name. And with partial function application, And with partial function application, And with partial function application, what we do is we take that read function what we do is we take that read function what we do is we take that read function and call the partial on it. and call the partial on it. and call the partial on it. And give it a directory. And what we're And give it a directory. And what we're And give it a directory. And what we're doing here is we're locking the doing here is we're locking the doing here is we're locking the directory argument to demo. directory argument to demo. directory argument to demo. Um Um the LLM isn't going to be able to change the LLM isn't going to be able to change the LLM isn't going to be able to change that argument. that argument. that argument. It's not even going to know that that It's not even going to know that that It's not even going to know that that argument exists. When we pass in the argument exists. When we pass in the argument exists. When we pass in the read tool now, it's just going to see read tool now, it's just going to see read tool now, it's just going to see one parameter, which is file name. one parameter, which is file name. one parameter, which is file name. It's going to pass in a file name. It's going to pass in a file name. It's going to pass in a file name. And it will read that file from this And it will read that file from this And it will read that file from this directory. directory. directory. So, we've locked the directory So, we've locked the directory So, we've locked the directory parameter. Now, the agent can only read parameter. Now, the agent can only read parameter. Now, the agent can only read files files files from and write files to this directory. from and write files to this directory. from and write files to this directory. Uh Uh partial function application is a really partial function application is a really partial function application is a really great way to constrain the capabilities great way to constrain the capabilities great way to constrain the capabilities of your agent. Now, no human input is of your agent. Now, no human input is of your agent. Now, no human input is needed, but it's still safe. needed, but it's still safe. needed, but it's still safe. So, let's see that in So, again, run NPX So, let's see that in So, again, run NPX So, let's see that in So, again, run NPX agency agency agency examples. examples. examples. PFA is partial function application. PFA is partial function application. PFA is partial function application. So, here So, here So, here it's thinking and you can see it's thinking and you can see it's thinking and you can see it read the file. it read the file. it read the file. Didn't need to ask us for permission. Didn't need to ask us for permission. Didn't need to ask us for permission. And now it said, you know, to fix this And now it said, you know, to fix this And now it said, you know, to fix this issue, it's kind of giving us a new issue, it's kind of giving us a new issue, it's kind of giving us a new code. code. code. But it's not updating the code. It asks, But it's not updating the code. It asks, But it's not updating the code. It asks, "Would you like me to update the file "Would you like me to update the file "Would you like me to update the file with this corrected function?" But it with this corrected function?" But it with this corrected function?" But it didn't do that itself. So, didn't do that itself. So, didn't do that itself. So, you know, this you know, this you know, this version of this agent version of this agent version of this agent is the best one we've seen so far is the best one we've seen so far is the best one we've seen so far because it does give us the correct because it does give us the correct because it does give us the correct solution solution solution without any human input needed, but it without any human input needed, but it without any human input needed, but it hasn't actually fixed the code. hasn't actually fixed the code. hasn't actually fixed the code. So, the next step is to get it to So, the next step is to get it to So, the next step is to get it to actually fix the code. actually fix the code. actually fix the code. Um and that's where the feedback loop Um and that's where the feedback loop Um and that's where the feedback loop comes in. comes in. comes in. Um so, there's a very popular Um so, there's a very popular Um so, there's a very popular agent pattern called react. And react agent pattern called react. And react agent pattern called react. And react stands for reason and act. And stands for reason and act. And stands for reason and act. And essentially, you're asking the agent essentially, you're asking the agent essentially, you're asking the agent to work in this loop. Where you reason, to work in this loop. Where you reason, to work in this loop. Where you reason, act, act, act, observe the consequences of your action, observe the consequences of your action, observe the consequences of your action, and then decide what to do next. and then decide what to do next. and then decide what to do next. So, in this case, reason, read the two So, in this case, reason, read the two So, in this case, reason, read the two files, files, files, act, run the tests, act, run the tests, act, run the tests, observe, if the tests fail, observe, if the tests fail, observe, if the tests fail, read the error, read the error, read the error, and then reason again. So, and then reason again. So, and then reason again. So, look at that test failure and think look at that test failure and think look at that test failure and think about what you want to do next. And just about what you want to do next. And just about what you want to do next. And just do this in a loop do this in a loop do this in a loop until until until the tests pass. the tests pass. the tests pass. So, this is a really key pattern to So, this is a really key pattern to So, this is a really key pattern to making the agent better. making the agent better. making the agent better. And so, now let's see the same agent. And so, now let's see the same agent. And so, now let's see the same agent. The only thing we're changing is The only thing we're changing is The only thing we're changing is the prompt the prompt the prompt and giving it the ability to run these and giving it the ability to run these and giving it the ability to run these tests and now the agent should tests and now the agent should tests and now the agent should work work work better. In this case, I'm going to also In this case, I'm going to also print all the tool calls just so you can print all the tool calls just so you can print all the tool calls just so you can see what's happening. see what's happening. see what's happening. Um Well, should maybe Well, should maybe All right, hang on. All right, hang on. All right, hang on. It's always there. There we go. Okay. There we go. Okay. Now, let's see it in action. Now, let's see it in action. Now, let's see it in action. Um Um So, now So, now So, now you can see it's calling the tools. It's you can see it's calling the tools. It's you can see it's calling the tools. It's reading those two files, like we said. reading those two files, like we said. reading those two files, like we said. It's running the tests. It's running the tests. It's running the tests. The tests fail. The tests fail. The tests fail. It's writing to the file. It's running It's writing to the file. It's running It's writing to the file. It's running the test again the test again the test again and confirming success. So, this is a and confirming success. So, this is a and confirming success. So, this is a really important really important really important stage in the harness development because stage in the harness development because stage in the harness development because now now now we finally have an agent that we finally have an agent that we finally have an agent that successfully successfully successfully figures out the task, checks figures out the task, checks figures out the task, checks that the code is failing, that the code is failing, that the code is failing, modifies the code, and checks that the modifies the code, and checks that the modifies the code, and checks that the code now passes. code now passes. code now passes. Um Um So, let's talk about how we have So, let's talk about how we have So, let's talk about how we have improved the harness so far. improved the harness so far. improved the harness so far. So, we started with just the model, So, we started with just the model, So, we started with just the model, couldn't do anything. couldn't do anything. couldn't do anything. Then we added tools, Then we added tools, Then we added tools, but the agent wasn't safe. Then we added but the agent wasn't safe. Then we added but the agent wasn't safe. Then we added handlers, which added safety, handlers, which added safety, handlers, which added safety, but needed human input. but needed human input. but needed human input. Then we added partial function Then we added partial function Then we added partial function application, which was safety without application, which was safety without application, which was safety without requiring human input. And finally, the requiring human input. And finally, the requiring human input. And finally, the last one that we just looked at was a last one that we just looked at was a last one that we just looked at was a feedback loop, which added reasoning. feedback loop, which added reasoning. feedback loop, which added reasoning. So, the agent made the change and So, the agent made the change and So, the agent made the change and confirmed that the change worked. So, I have two more examples. So, I have two more examples. This one we're going to go in a slightly This one we're going to go in a slightly This one we're going to go in a slightly different direction. And this is where different direction. And this is where different direction. And this is where we're getting into some of the more we're getting into some of the more we're getting into some of the more advanced concepts. advanced concepts. advanced concepts. So, this one is sub-agents. So, this one is sub-agents. So, this one is sub-agents. Um everything so far we've done has made Um everything so far we've done has made Um everything so far we've done has made the same task better. But with the same task better. But with the same task better. But with sub-agents, we're asking, "How do you sub-agents, we're asking, "How do you sub-agents, we're asking, "How do you make the agent do more things?" make the agent do more things?" make the agent do more things?" Um Um and let me show you and let me show you and let me show you what this code looks like real quick. this program, I'll start by just showing this program, I'll start by just showing you the prompt we're giving it because you the prompt we're giving it because you the prompt we're giving it because it's slightly different. it's slightly different. it's slightly different. So, fix the failing test in test So, fix the failing test in test So, fix the failing test in test median.py. median.py. median.py. Then research Jensen's inequality for Then research Jensen's inequality for Then research Jensen's inequality for medians with the Wikipedia agent and medians with the Wikipedia agent and medians with the Wikipedia agent and explain it to me in the network form. explain it to me in the network form. explain it to me in the network form. So, So, here is a case where we're actually here is a case where we're actually here is a case where we're actually asking asking asking the agent to do a little more, the agent to do a little more, the agent to do a little more, right? right? And this one is interesting because And this one is interesting because And this one is interesting because this is our main agent, this is our main agent, this is our main agent, but for tools, we're not giving it the but for tools, we're not giving it the but for tools, we're not giving it the read and write functions. Instead, we're read and write functions. Instead, we're read and write functions. Instead, we're giving it two sub-agents as tools. giving it two sub-agents as tools. giving it two sub-agents as tools. And And And I really like this pattern in agency I really like this pattern in agency I really like this pattern in agency because I've seen a lot of frameworks because I've seen a lot of frameworks because I've seen a lot of frameworks make sub-agents their own concept and make sub-agents their own concept and make sub-agents their own concept and make them kind of hard to grasp. Um in make them kind of hard to grasp. Um in make them kind of hard to grasp. Um in agency, a sub-agent is just another agency, a sub-agent is just another agency, a sub-agent is just another function. And you just call it, use it function. And you just call it, use it function. And you just call it, use it like a tool just like you would anywhere like a tool just like you would anywhere like a tool just like you would anywhere else. else. else. And that consistency has a lot of And that consistency has a lot of And that consistency has a lot of benefits. Makes it really easy to write benefits. Makes it really easy to write benefits. Makes it really easy to write and reason about. and reason about. and reason about. So, here's the coding agent So, here's the coding agent So, here's the coding agent sub agent. sub agent. sub agent. Um Um same one as before. You know, there's a same one as before. You know, there's a same one as before. You know, there's a system message. system message. system message. This sub agent has its own LLM call, This sub agent has its own LLM call, This sub agent has its own LLM call, gets its own set of tools. gets its own set of tools. gets its own set of tools. >> [snorts] >> [snorts] >> Similarly, you have the Wikipedia agent >> Similarly, you have the Wikipedia agent >> Similarly, you have the Wikipedia agent that has its own tool, which is the that has its own tool, which is the that has its own tool, which is the search tool. search tool. search tool. Uh which is a tool built into this Uh which is a tool built into this Uh which is a tool built into this agency standard library that lets you agency standard library that lets you agency standard library that lets you search Wikipedia. search Wikipedia. search Wikipedia. Uh so, agency has a big standard library Uh so, agency has a big standard library Uh so, agency has a big standard library that lets you do all kinds of things, that lets you do all kinds of things, that lets you do all kinds of things, including this Wikipedia search tool. including this Wikipedia search tool. including this Wikipedia search tool. I'm also I'm also I'm also using the callback feature to print the using the callback feature to print the using the callback feature to print the tool call information. tool call information. tool call information. And now, And now, And now, let's see this example in action. And let's see this example in action. And let's see this example in action. And this is an example where this is an example where this is an example where we're actually asking the agent to do a we're actually asking the agent to do a we're actually asking the agent to do a little bit more. little bit more. little bit more. And we're giving it And we're giving it And we're giving it two sub agents to accomplish its task. two sub agents to accomplish its task. two sub agents to accomplish its task. So, here you can see it's calling those So, here you can see it's calling those So, here you can see it's calling those two sub agents. two sub agents. two sub agents. Uh it's calling them in parallel, so Uh it's calling them in parallel, so Uh it's calling them in parallel, so the Wikipedia agent is doing the search, the Wikipedia agent is doing the search, the Wikipedia agent is doing the search, the coding agent is modifying the file, the coding agent is modifying the file, the coding agent is modifying the file, and and that fixed the error, and now here's a that fixed the error, and now here's a that fixed the error, and now here's a limerick. limerick. limerick. So, sub agents are cool because they let you sub agents are cool because they let you add new capabilities add new capabilities add new capabilities without without without bloating context. bloating context. bloating context. Um Um and this is a good pattern because it and this is a good pattern because it and this is a good pattern because it allows your agent allows your agent allows your agent when it has a lot of different tools at when it has a lot of different tools at when it has a lot of different tools at its disposal, its disposal, its disposal, it allows your agent to do it allows your agent to do it allows your agent to do the right thing more consistently. the right thing more consistently. the right thing more consistently. Often when agents get confused and fail, Often when agents get confused and fail, Often when agents get confused and fail, it's because it's because it's because they have too much context blow, but they have too much context blow, but they have too much context blow, but also because they have too many also because they have too many also because they have too many unrelated concepts in their context. And unrelated concepts in their context. And unrelated concepts in their context. And they have too many tools that are they have too many tools that are they have too many tools that are unrelated. And so they have a hard time unrelated. And so they have a hard time unrelated. And so they have a hard time picking the right tool to use. In this picking the right tool to use. In this picking the right tool to use. In this case, we make that problem a lot case, we make that problem a lot case, we make that problem a lot cleaner. cleaner. cleaner. For the top-level agent, we just say, For the top-level agent, we just say, For the top-level agent, we just say, "Pick the sub-agent you want to call." "Pick the sub-agent you want to call." "Pick the sub-agent you want to call." And then the sub-agents have those tools And then the sub-agents have those tools And then the sub-agents have those tools that are kind of grouped in a way that that are kind of grouped in a way that that are kind of grouped in a way that makes sense. makes sense. makes sense. Um Um So now, let's look at the last example, So now, let's look at the last example, So now, let's look at the last example, which is self-optimization. which is self-optimization. which is self-optimization. Uh Agency has a Japa optimizer built in. Uh Agency has a Japa optimizer built in. Uh Agency has a Japa optimizer built in. It actually has a few optimizers built It actually has a few optimizers built It actually has a few optimizers built in, but the Japa one in, but the Japa one in, but the Japa one uh is a really interesting one. uh is a really interesting one. uh is a really interesting one. And And the way the way the way the optimization the optimization the optimization feature works is you mark any variables feature works is you mark any variables feature works is you mark any variables that you want the optimizer to optimize. that you want the optimizer to optimize. that you want the optimizer to optimize. You can just use this optimize modifier. You can just use this optimize modifier. You can just use this optimize modifier. And then you run the optimizer And then you run the optimizer And then you run the optimizer and give it a goal. and give it a goal. and give it a goal. So, So, you know, in this case, you can see that you know, in this case, you can see that you know, in this case, you can see that the two initial prompts I have the two initial prompts I have the two initial prompts I have are very basic. I'm giving it hardly any are very basic. I'm giving it hardly any are very basic. I'm giving it hardly any information. information. information. And then the goal I give it is, "Fix the And then the goal I give it is, "Fix the And then the goal I give it is, "Fix the bug in median.py bug in median.py bug in median.py using TDD." using TDD." using TDD." And And the reason this is such a great step the reason this is such a great step the reason this is such a great step for our harness is for our harness is for our harness is we're no longer doing trial and error. we're no longer doing trial and error. we're no longer doing trial and error. We're no longer just trying stuff to see We're no longer just trying stuff to see We're no longer just trying stuff to see what works. Instead, now we have a way what works. Instead, now we have a way what works. Instead, now we have a way to systematically measure and improve to systematically measure and improve to systematically measure and improve performance. performance. performance. So, I'm just going to go ahead and run So, I'm just going to go ahead and run So, I'm just going to go ahead and run this code. this code. this code. Um I'm actually going to also reset. Here, I'm going to Here, I'm going to modify this code. Cool. Cool. So now I won't I won't show the full run here because show the full run here because show the full run here because the optimizer takes a while, but you can the optimizer takes a while, but you can the optimizer takes a while, but you can see first it's like see first it's like see first it's like checking what checking what checking what prompts it has that it needs to prompts it has that it needs to prompts it has that it needs to optimize. It's going to run the agent to optimize. It's going to run the agent to optimize. It's going to run the agent to establish a baseline. So the baseline establish a baseline. So the baseline establish a baseline. So the baseline objective is 0.2 and it's going to try objective is 0.2 and it's going to try objective is 0.2 and it's going to try to improve upon that. to improve upon that. to improve upon that. So in this case I did run the optimizer So in this case I did run the optimizer So in this case I did run the optimizer earlier just so I could show you what a earlier just so I could show you what a earlier just so I could show you what a full run looked like. full run looked like. full run looked like. Um Um and so here is one where you can see and so here is one where you can see and so here is one where you can see it achieved the objective in the first it achieved the objective in the first it achieved the objective in the first iteration, so it was actually quite iteration, so it was actually quite iteration, so it was actually quite quick in this case. quick in this case. quick in this case. Um and you can see that it's Um and you can see that it's Um and you can see that it's rewritten rewritten rewritten this prompt. Um this prompt. Um this prompt. Um So the reason this self-optimization is So the reason this self-optimization is So the reason this self-optimization is so great is because we're not guessing so great is because we're not guessing so great is because we're not guessing and checking. We're systematically and checking. We're systematically and checking. We're systematically measuring and improving, measuring and improving, measuring and improving, which is a big leap forward. which is a big leap forward. which is a big leap forward. So this is the ladder we just went So this is the ladder we just went So this is the ladder we just went through for improving the harness. through for improving the harness. through for improving the harness. Um Um you know, you know, you know, first step was nothing, just a model first step was nothing, just a model first step was nothing, just a model can't even read the file. Then we added can't even read the file. Then we added can't even read the file. Then we added tools, tools, tools, but in a way that was unsafe and not but in a way that was unsafe and not but in a way that was unsafe and not allowed in agency. allowed in agency. allowed in agency. Next we added handlers, Next we added handlers, Next we added handlers, which added safety, but which added safety, but which added safety, but made things slower because they required made things slower because they required made things slower because they required human input. human input. human input. Then we added partial function Then we added partial function application, which was safe and fast. application, which was safe and fast. application, which was safe and fast. Then we added reasoning, which gave us Then we added reasoning, which gave us Then we added reasoning, which gave us higher confidence in the output of the higher confidence in the output of the higher confidence in the output of the agent. agent. agent. Uh and then we switched tracks to Uh and then we switched tracks to Uh and then we switched tracks to subagents, added more capabilities, subagents, added more capabilities, subagents, added more capabilities, and finally we did self-optimization. So and finally we did self-optimization. So and finally we did self-optimization. So we got measured improvement and not we got measured improvement and not we got measured improvement and not guess and check. guess and check. guess and check. So, So, you know, I hope this gives you an idea you know, I hope this gives you an idea you know, I hope this gives you an idea of of of how you can build how you can build how you can build harnesses and then slowly improve those harnesses and then slowly improve those harnesses and then slowly improve those harnesses. And I hope, you know, you try harnesses. And I hope, you know, you try harnesses. And I hope, you know, you try doing that and come along on this, you doing that and come along on this, you doing that and come along on this, you know, experimentation with me know, experimentation with me know, experimentation with me to see if we can build a better harness. to see if we can build a better harness. to see if we can build a better harness. Um Um because what of the harness matters more because what of the harness matters more because what of the harness matters more than the model. You know, what if we can than the model. You know, what if we can than the model. You know, what if we can build a harness that's good enough that build a harness that's good enough that build a harness that's good enough that we can now start using local models and we can now start using local models and we can now start using local models and still get the performance we need for still get the performance we need for still get the performance we need for the tasks we want to do. the tasks we want to do. the tasks we want to do. So, what language level support does So, what language level support does So, what language level support does Agency provide? You know, we used Agency Agency provide? You know, we used Agency Agency provide? You know, we used Agency for all these examples. How did it help? for all these examples. How did it help? for all these examples. How did it help? Um simple syntax for defining tools. Um simple syntax for defining tools. Um simple syntax for defining tools. Again, you know, the simple thing should Again, you know, the simple thing should Again, you know, the simple thing should be simple. Every agent is going to need be simple. Every agent is going to need be simple. Every agent is going to need tools. Let's make it easy to use them. tools. Let's make it easy to use them. tools. Let's make it easy to use them. Tools for safety, very important for Tools for safety, very important for Tools for safety, very important for agents. So, interrupts handlers, PFA. agents. So, interrupts handlers, PFA. agents. So, interrupts handlers, PFA. True pause and resume execution. I True pause and resume execution. I True pause and resume execution. I didn't cover this a ton cuz I didn't didn't cover this a ton cuz I didn't didn't cover this a ton cuz I didn't want to make this talk too long. Um but want to make this talk too long. Um but want to make this talk too long. Um but one of the really killer features of one of the really killer features of one of the really killer features of Agency is that Agency is that Agency is that when you raise that interrupt, it's when you raise that interrupt, it's when you raise that interrupt, it's actually going to pause execution actually going to pause execution actually going to pause execution and return control back to the user. But and return control back to the user. But and return control back to the user. But in a way where you can resume execution in a way where you can resume execution in a way where you can resume execution later. Uh later. Uh later. Uh and as far as I know, and as far as I know, and as far as I know, very few languages allow you to do very few languages allow you to do very few languages allow you to do something like this. And so, typically something like this. And so, typically something like this. And so, typically most frameworks you'll use, if they have most frameworks you'll use, if they have most frameworks you'll use, if they have a human in the loop feature, it has a a human in the loop feature, it has a a human in the loop feature, it has a lot of restrictions. There's a lot of lot of restrictions. There's a lot of lot of restrictions. There's a lot of things you need to do to work around things you need to do to work around things you need to do to work around that. Agency is completely different. If that. Agency is completely different. If that. Agency is completely different. If you wanted, you could, you know, raise you wanted, you could, you know, raise you wanted, you could, you know, raise interrupts inside of a for loop, inside interrupts inside of a for loop, inside interrupts inside of a for loop, inside of a tool call, inside of a sub-agent. of a tool call, inside of a sub-agent. of a tool call, inside of a sub-agent. Um, and it would correctly Um, and it would correctly Um, and it would correctly pause execution, pause execution, pause execution, go back to the user, and then resume go back to the user, and then resume go back to the user, and then resume execution at that exact point inside execution at that exact point inside execution at that exact point inside that subagent, inside that tool call, at that subagent, inside that tool call, at that subagent, inside that tool call, at that exact point in that for loop. that exact point in that for loop. that exact point in that for loop. Um, Um, which is really cool. You can serialize which is really cool. You can serialize which is really cool. You can serialize the execution, you can come back to it a the execution, you can come back to it a the execution, you can come back to it a week later, week later, week later, and you don't need to change anything and you don't need to change anything and you don't need to change anything about all the code we have been looking about all the code we have been looking about all the code we have been looking at. It just works, which is really cool. at. It just works, which is really cool. at. It just works, which is really cool. Uh, it also has these built-in Uh, it also has these built-in Uh, it also has these built-in optimizers, which are great, you know, optimizers, which are great, you know, optimizers, which are great, you know, which kind of get you to a more which kind of get you to a more which kind of get you to a more analytical, mathematical way of doing analytical, mathematical way of doing analytical, mathematical way of doing the improvements. So, if you want to try the improvements. So, if you want to try the improvements. So, if you want to try it out, please do. You can PM install it out, please do. You can PM install it out, please do. You can PM install All you need is the package and a API All you need is the package and a API All you need is the package and a API key, and you're set. key, and you're set. key, and you're set. Um, Um, main takeaways, you know, an agent is a main takeaways, you know, an agent is a main takeaways, you know, an agent is a model plus a harness for this talk, at model plus a harness for this talk, at model plus a harness for this talk, at least. Uh, better harness equals better least. Uh, better harness equals better least. Uh, better harness equals better performance, especially for weaker performance, especially for weaker performance, especially for weaker models. models. And then, how do you build a better And then, how do you build a better And then, how do you build a better harness? Give it tools, make it safe, harness? Give it tools, make it safe, harness? Give it tools, make it safe, make it autonomous, make it reason, make it autonomous, make it reason, make it autonomous, make it reason, give it subagents, and have it give it subagents, and have it give it subagents, and have it self-optimize. self-optimize. self-optimize. Thank you so much. Again, my name is Thank you so much. Again, my name is Thank you so much. Again, my name is Aditya Parikh, check out agency Aditya Parikh, check out agency Aditya Parikh, check out agency agencylang.com, agencylang.com, agencylang.com, uh, and then please follow me on Blue uh, and then please follow me on Blue uh, and then please follow me on Blue Sky, if you wish. Thank you. Take care.