Markdown source

Transcript: Respect The Process - Andrew Dumit, Watershed Technology Inc.

Source Video

Local Cache

Transcript

Hi everyone. My name is Andrew Dumont Hi everyone. My name is Andrew Dumont and I work on AI engineering at and I work on AI engineering at and I work on AI engineering at Watershed, the sustainability air Watershed, the sustainability air Watershed, the sustainability air platform. platform. platform. At Watershed, I work on AI for building At Watershed, I work on AI for building At Watershed, I work on AI for building product carbon footprints and in turn product carbon footprints and in turn product carbon footprints and in turn measuring the emissions associated with measuring the emissions associated with measuring the emissions associated with all the things that a company buys and all the things that a company buys and all the things that a company buys and sells. Sustainability, the vertical that sells. Sustainability, the vertical that sells. Sustainability, the vertical that Watershed is in, is one with a ton of Watershed is in, is one with a ton of Watershed is in, is one with a ton of expert judgment calls spread throughout expert judgment calls spread throughout expert judgment calls spread throughout it and this has made building and it and this has made building and it and this has made building and deploying agents both exciting and deploying agents both exciting and deploying agents both exciting and challenging. challenging. challenging. In this talk, I'll go deep on one of the In this talk, I'll go deep on one of the In this talk, I'll go deep on one of the tasks within sustainability that we tasks within sustainability that we tasks within sustainability that we spend quite a bit of time working on and spend quite a bit of time working on and spend quite a bit of time working on and share learnings from our work deploying share learnings from our work deploying share learnings from our work deploying coding agents on that task and why do do coding agents on that task and why do do coding agents on that task and why do do I doing so has required that we respect I doing so has required that we respect I doing so has required that we respect the process. the process. the process. So just to give a bit of context on So just to give a bit of context on So just to give a bit of context on sustainability as a vertical, we need to sustainability as a vertical, we need to sustainability as a vertical, we need to answer questions like what are the answer questions like what are the answer questions like what are the emissions attributable to one bottle of emissions attributable to one bottle of emissions attributable to one bottle of wine or which method should be used to wine or which method should be used to wine or which method should be used to allocate the emissions to various allocate the emissions to various allocate the emissions to various co-products when there are multiple co-products when there are multiple co-products when there are multiple sellable co-products that come from the sellable co-products that come from the sellable co-products that come from the same industrial process. same industrial process. same industrial process. In these cases, there are many ways to In these cases, there are many ways to In these cases, there are many ways to get the right answer the wrong way and get the right answer the wrong way and get the right answer the wrong way and there are also many right answers that there are also many right answers that there are also many right answers that experts will disagree on. experts will disagree on. experts will disagree on. And so, you have to verify the process And so, you have to verify the process And so, you have to verify the process in addition to the answer because the in addition to the answer because the in addition to the answer because the answer is really only justified in so answer is really only justified in so answer is really only justified in so far as it the process that produced that far as it the process that produced that far as it the process that produced that answer is correct. answer is correct. answer is correct. One example of this comes from a study One example of this comes from a study One example of this comes from a study in 2020 where six experts were given the in 2020 where six experts were given the in 2020 where six experts were given the exact same data on the exact same bottle exact same data on the exact same bottle exact same data on the exact same bottle of wine of wine of wine and despite having all access to the and despite having all access to the and despite having all access to the exact same things, they came to answers exact same things, they came to answers exact same things, they came to answers that varied by up to 50%. Their expert that varied by up to 50%. Their expert that varied by up to 50%. Their expert judgment were all correct in a sense, judgment were all correct in a sense, judgment were all correct in a sense, but purely validating the answer here is but purely validating the answer here is but purely validating the answer here is not sufficient to know that the answer not sufficient to know that the answer not sufficient to know that the answer produced by a system trying to mimic produced by a system trying to mimic produced by a system trying to mimic those experts is itself correct. those experts is itself correct. those experts is itself correct. So with that context on sustainability So with that context on sustainability So with that context on sustainability broadly and what kind of things that we broadly and what kind of things that we broadly and what kind of things that we need to answer, let's zoom into a need to answer, let's zoom into a need to answer, let's zoom into a specific task that we'll be talking specific task that we'll be talking specific task that we'll be talking about for the remainder of the talk. about for the remainder of the talk. about for the remainder of the talk. That task is to help users edit complex That task is to help users edit complex That task is to help users edit complex graphs. Each of these graphs represents graphs. Each of these graphs represents graphs. Each of these graphs represents the supply chain of a single product. the supply chain of a single product. the supply chain of a single product. For example, on the right here, you can For example, on the right here, you can For example, on the right here, you can see the graph represents dark wash see the graph represents dark wash see the graph represents dark wash jeans. jeans. jeans. Upstream of that is the assembly of that Upstream of that is the assembly of that Upstream of that is the assembly of that jeans from the denim thread, labels, uh jeans from the denim thread, labels, uh jeans from the denim thread, labels, uh and zipper, along with the energy, and zipper, along with the energy, and zipper, along with the energy, transportation, and packaging that's transportation, and packaging that's transportation, and packaging that's required to move uh move that through required to move uh move that through required to move uh move that through the supply chain, and create uh and run the supply chain, and create uh and run the supply chain, and create uh and run those industrial processes. those industrial processes. those industrial processes. Each graph here represents the entire Each graph here represents the entire Each graph here represents the entire flow of all of those things moving flow of all of those things moving flow of all of those things moving through the supply chain, and is through the supply chain, and is through the supply chain, and is comprised of thousands of nodes, each comprised of thousands of nodes, each comprised of thousands of nodes, each with rich metadata describing the with rich metadata describing the with rich metadata describing the materials and processing at each step. And so, when we first tried to solve And so, when we first tried to solve this problem a little over a year ago, this problem a little over a year ago, this problem a little over a year ago, it worked decently well on one graph. It it worked decently well on one graph. It it worked decently well on one graph. It did have some problems, but we gave our did have some problems, but we gave our did have some problems, but we gave our React agent highly specified tools for React agent highly specified tools for React agent highly specified tools for exploring and interacting with the graph exploring and interacting with the graph exploring and interacting with the graph via function calls. via function calls. via function calls. And even while it worked, a few problems And even while it worked, a few problems And even while it worked, a few problems were already apparent. Um it lacked were already apparent. Um it lacked were already apparent. Um it lacked consistency where it could struggle to consistency where it could struggle to consistency where it could struggle to explore sufficiently, and those tool explore sufficiently, and those tool explore sufficiently, and those tool calls on a single graph took a lot of calls on a single graph took a lot of calls on a single graph took a lot of context as it read deeply across these context as it read deeply across these context as it read deeply across these nodes with lots of metadata. nodes with lots of metadata. nodes with lots of metadata. But then, when we tried to scale it up But then, when we tried to scale it up But then, when we tried to scale it up to many graphs, or frankly even just a to many graphs, or frankly even just a to many graphs, or frankly even just a few graphs, it absolutely broke. The few graphs, it absolutely broke. The few graphs, it absolutely broke. The lack of consistency was greatly lack of consistency was greatly lack of consistency was greatly magnified with the agent taking one magnified with the agent taking one magnified with the agent taking one approach in one graph, a different approach in one graph, a different approach in one graph, a different approach in a second graph, and approach in a second graph, and approach in a second graph, and completely forgetting about the third completely forgetting about the third completely forgetting about the third graph, or even to handle it at all. graph, or even to handle it at all. graph, or even to handle it at all. And exploration itself also became a And exploration itself also became a And exploration itself also became a huge bottleneck because it took many huge bottleneck because it took many huge bottleneck because it took many tool calls to explore and operate across tool calls to explore and operate across tool calls to explore and operate across these graphs at scale. these graphs at scale. these graphs at scale. And it also worsened the context problem And it also worsened the context problem And it also worsened the context problem because these tool calls really gobbled because these tool calls really gobbled because these tool calls really gobbled up context both on the edit side where up context both on the edit side where up context both on the edit side where it actually needed to like make some it actually needed to like make some it actually needed to like make some changes, and on the exploration side changes, and on the exploration side changes, and on the exploration side where it needed to figure out what where it needed to figure out what where it needed to figure out what changes to make. And worse, the agent changes to make. And worse, the agent changes to make. And worse, the agent then really started to hallucinate then really started to hallucinate then really started to hallucinate different parts of the schema as those different parts of the schema as those different parts of the schema as those contexts got eaten, and despite those contexts got eaten, and despite those contexts got eaten, and despite those specialized tools, this led to retries specialized tools, this led to retries specialized tools, this led to retries and ultimately errors. and ultimately errors. and ultimately errors. In data terms, the task now comprised In data terms, the task now comprised In data terms, the task now comprised tens or hundreds of graphs, and tens to tens or hundreds of graphs, and tens to tens or hundreds of graphs, and tens to hundreds of thousands of nodes that the hundreds of thousands of nodes that the hundreds of thousands of nodes that the agent had to operate over. agent had to operate over. agent had to operate over. So, time passed, So, time passed, So, time passed, and coding agents got way better, and we and coding agents got way better, and we and coding agents got way better, and we thought these will work great. So, thought these will work great. So, thought these will work great. So, naturally, we swapped them in. And naturally, we swapped them in. And naturally, we swapped them in. And swapping in a coding agent gave us like swapping in a coding agent gave us like swapping in a coding agent gave us like three really important outcomes that we three really important outcomes that we three really important outcomes that we were missing before. First, it gave the were missing before. First, it gave the were missing before. First, it gave the agent the ability to delight users. The agent the ability to delight users. The agent the ability to delight users. The agent could start to find clever ways to agent could start to find clever ways to agent could start to find clever ways to solve these under-specified solve these under-specified solve these under-specified and kind of not fully formed problems. and kind of not fully formed problems. and kind of not fully formed problems. It could also create fancy It could also create fancy It could also create fancy visualizations on the fly by writing visualizations on the fly by writing visualizations on the fly by writing code to do that. And it could even code to do that. And it could even code to do that. And it could even answer related questions because it had answer related questions because it had answer related questions because it had kind of the full power of the coding kind of the full power of the coding kind of the full power of the coding agent in this environment. agent in this environment. agent in this environment. And what from our end we saw the agent And what from our end we saw the agent And what from our end we saw the agent could explore and now edit way more could explore and now edit way more could explore and now edit way more efficiently than before. It could write efficiently than before. It could write efficiently than before. It could write loops over graphs and nodes. It could loops over graphs and nodes. It could loops over graphs and nodes. It could write scripts to unpack and summarize write scripts to unpack and summarize write scripts to unpack and summarize the node content underneath it all. the node content underneath it all. the node content underneath it all. Basically, following the same pattern of Basically, following the same pattern of Basically, following the same pattern of data exploration that goes on in kind of data exploration that goes on in kind of data exploration that goes on in kind of agentic data science workflows. agentic data science workflows. agentic data science workflows. And beyond that, it also gave it the And beyond that, it also gave it the And beyond that, it also gave it the flexibility and power to do stuff flexibility and power to do stuff flexibility and power to do stuff outside what we even designed it for, outside what we even designed it for, outside what we even designed it for, which was really exciting. And we're to which was really exciting. And we're to which was really exciting. And we're to this day consistently finding entirely this day consistently finding entirely this day consistently finding entirely new use cases via new questions users new use cases via new questions users new use cases via new questions users ask and new things that they want to try ask and new things that they want to try ask and new things that they want to try with this agent. with this agent. with this agent. We were really excited. We put it out We were really excited. We put it out We were really excited. We put it out into the world. We started to write a into the world. We started to write a into the world. We started to write a bunch of evals for it. And we quickly bunch of evals for it. And we quickly bunch of evals for it. And we quickly learned that unconstrained code is quite learned that unconstrained code is quite learned that unconstrained code is quite scary. If you're watching this, you've scary. If you're watching this, you've scary. If you're watching this, you've probably had that experience with cloud probably had that experience with cloud probably had that experience with cloud code where it's gone a bit haywire code where it's gone a bit haywire code where it's gone a bit haywire towards achieving whatever goal you set towards achieving whatever goal you set towards achieving whatever goal you set it out to do. It reached for something it out to do. It reached for something it out to do. It reached for something you didn't think it could or should you didn't think it could or should you didn't think it could or should reach for or even had access or found a reach for or even had access or found a reach for or even had access or found a way to access something you didn't think way to access something you didn't think way to access something you didn't think it could. it could. it could. Um for us, this looks like the agent Um for us, this looks like the agent Um for us, this looks like the agent will find creative ways to pretty much will find creative ways to pretty much will find creative ways to pretty much solve any problem. In our case, we saw solve any problem. In our case, we saw solve any problem. In our case, we saw it write Python when we expected it write Python when we expected it write Python when we expected TypeScript and and instructed it to TypeScript and and instructed it to TypeScript and and instructed it to write TypeScript because it found Python write TypeScript because it found Python write TypeScript because it found Python on the virtual machine that we had given on the virtual machine that we had given on the virtual machine that we had given it. it. it. Or it would directly edit these graph Or it would directly edit these graph Or it would directly edit these graph artifacts underlying the data without artifacts underlying the data without artifacts underlying the data without leaving any lineage behind by just like leaving any lineage behind by just like leaving any lineage behind by just like directly modifying um parameters or data directly modifying um parameters or data directly modifying um parameters or data rather than actually like writing code rather than actually like writing code rather than actually like writing code to effectively do that. to effectively do that. to effectively do that. Second, the agent actually started to Second, the agent actually started to Second, the agent actually started to gaslight users sometimes saying it had gaslight users sometimes saying it had gaslight users sometimes saying it had made edits when it hadn't. made edits when it hadn't. made edits when it hadn't. Um it would write code that did that it Um it would write code that did that it Um it would write code that did that it thought was going to do make like have thought was going to do make like have thought was going to do make like have the effect that it wanted to. And then the effect that it wanted to. And then the effect that it wanted to. And then it said, "I'm done. Everything's it said, "I'm done. Everything's it said, "I'm done. Everything's Everything's working. It's exactly what Everything's working. It's exactly what Everything's working. It's exactly what you said." And the edits were not you said." And the edits were not you said." And the edits were not actually made. And so it it kind of lied actually made. And so it it kind of lied actually made. And so it it kind of lied to users or at least let them to believe to users or at least let them to believe to users or at least let them to believe that the edits were done as they that the edits were done as they that the edits were done as they expected it to be done. expected it to be done. expected it to be done. And then lastly, once the agent had And then lastly, once the agent had And then lastly, once the agent had completed its task, it was now much completed its task, it was now much completed its task, it was now much harder to review its work. Um manual harder to review its work. Um manual harder to review its work. Um manual review of code is not something that are review of code is not something that are review of code is not something that are is in our users' wheelhouse. They are is in our users' wheelhouse. They are is in our users' wheelhouse. They are not software engineers or may or may not not software engineers or may or may not not software engineers or may or may not have had experience with code before. have had experience with code before. have had experience with code before. And when the agent is right for the And when the agent is right for the And when the agent is right for the wrong reasons, which happens uh or can wrong reasons, which happens uh or can wrong reasons, which happens uh or can happen frequently in our in happen frequently in our in happen frequently in our in sustainability broadly, it's really hard sustainability broadly, it's really hard sustainability broadly, it's really hard to check its work without going and to check its work without going and to check its work without going and reading that code directly. And so this process is not the answer. And so this process is not the answer. Story is really not new. Um a recent Story is really not new. Um a recent Story is really not new. Um a recent paper uh in from 2026, the Open Proof paper uh in from 2026, the Open Proof paper uh in from 2026, the Open Proof Corpus, showed that there can be quite a Corpus, showed that there can be quite a Corpus, showed that there can be quite a big gap between the correct final answer big gap between the correct final answer big gap between the correct final answer and and the correct proof. Though that and and the correct proof. Though that and and the correct proof. Though that is, you know, potentially closing. But is, you know, potentially closing. But is, you know, potentially closing. But And this is in math where there is like And this is in math where there is like And this is in math where there is like a fully verifiable answer. And so this a fully verifiable answer. And so this a fully verifiable answer. And so this We view this problem as even worse where We view this problem as even worse where We view this problem as even worse where you can't fully verify the answer. And you can't fully verify the answer. And you can't fully verify the answer. And this has been found in the agents what this has been found in the agents what this has been found in the agents what reward hack um even as something as reward hack um even as something as reward hack um even as something as complex as the Erdos problems, um you're complex as the Erdos problems, um you're complex as the Erdos problems, um you're going to get many, many pages of of going to get many, many pages of of going to get many, many pages of of errors in your proof which have which errors in your proof which have which errors in your proof which have which lead nowhere um to impossible bench to lead nowhere um to impossible bench to lead nowhere um to impossible bench to um other kind of areas in math where um other kind of areas in math where um other kind of areas in math where this can where they can The authors of this can where they can The authors of this can where they can The authors of this paper, Beyond Correctness, conclude this paper, Beyond Correctness, conclude this paper, Beyond Correctness, conclude that this can pose significant risk for that this can pose significant risk for that this can pose significant risk for critical applications. Now, all this is critical applications. Now, all this is critical applications. Now, all this is not to say that we shouldn't use coding not to say that we shouldn't use coding not to say that we shouldn't use coding agents. We don't want to constrain how agents. We don't want to constrain how agents. We don't want to constrain how the agent reasons. We get so many the agent reasons. We get so many the agent reasons. We get so many benefits from these powerful models, but benefits from these powerful models, but benefits from these powerful models, but we also can't perfectly verify the we also can't perfectly verify the we also can't perfectly verify the answer in our case. So we don't even answer in our case. So we don't even answer in our case. So we don't even know when the final answer is definitely know when the final answer is definitely know when the final answer is definitely correct. Uh because these expert correct. Uh because these expert correct. Uh because these expert judgment calls are ever present our judgment calls are ever present our judgment calls are ever present our space. So, the main lever left is space. So, the main lever left is space. So, the main lever left is validating the process and making sure validating the process and making sure validating the process and making sure that we are following it in a way we that we are following it in a way we that we are following it in a way we expect. And to do that, we frame it as expect. And to do that, we frame it as expect. And to do that, we frame it as constraining the effects, not the constraining the effects, not the constraining the effects, not the expression. expression. expression. So, our task begins with a user request. So, our task begins with a user request. So, our task begins with a user request. And from this, the agent can then go off And from this, the agent can then go off And from this, the agent can then go off and freely write code to execute on that and freely write code to execute on that and freely write code to execute on that user request. user request. user request. However, we require that all the However, we require that all the However, we require that all the critical code, really the stuff that critical code, really the stuff that critical code, really the stuff that edits the graph, goes through a filter edits the graph, goes through a filter edits the graph, goes through a filter of this typed SDK that we've put of this typed SDK that we've put of this typed SDK that we've put together where we can lint and check for together where we can lint and check for together where we can lint and check for errors. errors. errors. And then we own the final execution step And then we own the final execution step And then we own the final execution step to guarantee that we create these typed to guarantee that we create these typed to guarantee that we create these typed objects that we can commit as the actual objects that we can commit as the actual objects that we can commit as the actual edits to the graph. And these are then edits to the graph. And these are then edits to the graph. And these are then faithful to the process. And all of this faithful to the process. And all of this faithful to the process. And all of this together together together means that the entire process is valid, means that the entire process is valid, means that the entire process is valid, traceable, and replayable. And if it's traceable, and replayable. And if it's traceable, and replayable. And if it's not, we can reject and retry and send it not, we can reject and retry and send it not, we can reject and retry and send it back to the agent to let it know that back to the agent to let it know that back to the agent to let it know that something has gone awry. something has gone awry. something has gone awry. So, let's go a little deeper on the So, let's go a little deeper on the So, let's go a little deeper on the typed SDK and the deterministic typed SDK and the deterministic typed SDK and the deterministic execution. So, first, our SDK is the execution. So, first, our SDK is the execution. So, first, our SDK is the only door. What we give to the agent in only door. What we give to the agent in only door. What we give to the agent in our case is a TypeScript SDK with all our case is a TypeScript SDK with all our case is a TypeScript SDK with all the edit primitives the agent needs to the edit primitives the agent needs to the edit primitives the agent needs to make changes to graphs, as well as make changes to graphs, as well as make changes to graphs, as well as explore and interact with the kind of explore and interact with the kind of explore and interact with the kind of graph objects. graph objects. graph objects. This for us enforces which fields are This for us enforces which fields are This for us enforces which fields are editable versus which are derived from editable versus which are derived from editable versus which are derived from other fields, so the agent can't create other fields, so the agent can't create other fields, so the agent can't create conflicts with itself where it edits conflicts with itself where it edits conflicts with itself where it edits just the target thing that it cares just the target thing that it cares just the target thing that it cares about, but not the thing that creates about, but not the thing that creates about, but not the thing that creates that target thing. And it also that target thing. And it also that target thing. And it also guarantees that it objects that we guarantees that it objects that we guarantees that it objects that we expect it to so that we can actually expect it to so that we can actually expect it to so that we can actually like make use of the outputs of this in like make use of the outputs of this in like make use of the outputs of this in a deterministic way. a deterministic way. a deterministic way. Now, of course, this does require Now, of course, this does require Now, of course, this does require teaching the agent how to actually make teaching the agent how to actually make teaching the agent how to actually make use of this SDK, but this is the same use of this SDK, but this is the same use of this SDK, but this is the same kind of pattern that we see with kind of pattern that we see with kind of pattern that we see with teaching an agent to like write code in teaching an agent to like write code in teaching an agent to like write code in your code base where you tell it how it your code base where you tell it how it your code base where you tell it how it works. You can, you know, do the prompt works. You can, you know, do the prompt works. You can, you know, do the prompt and all of that kind of stuff. The agent and all of that kind of stuff. The agent and all of that kind of stuff. The agent also has full access to the docs and the also has full access to the docs and the also has full access to the docs and the code underlying the SDK if it really code underlying the SDK if it really code underlying the SDK if it really needs to go and read that as well. needs to go and read that as well. needs to go and read that as well. Just for a brief example on the right Just for a brief example on the right Just for a brief example on the right here, you can see an example with a few here, you can see an example with a few here, you can see an example with a few pieces of the SDK on display. We import pieces of the SDK on display. We import pieces of the SDK on display. We import um from our API. We The agent has to um from our API. We The agent has to um from our API. We The agent has to then define kind of a top-level edit then define kind of a top-level edit then define kind of a top-level edit function with a well-defined name. It function with a well-defined name. It function with a well-defined name. It then can use uh find nodes by exact name then can use uh find nodes by exact name then can use uh find nodes by exact name to find specific nodes that it's looking to find specific nodes that it's looking to find specific nodes that it's looking for in the graph. It can write for in the graph. It can write for in the graph. It can write assertions that we give it to make sure assertions that we give it to make sure assertions that we give it to make sure that um its code fails early um as well. that um its code fails early um as well. that um its code fails early um as well. And then it can use uh specific mutator And then it can use uh specific mutator And then it can use uh specific mutator functions like set rate and edit node to functions like set rate and edit node to functions like set rate and edit node to actually interact with the graph in a actually interact with the graph in a actually interact with the graph in a way that will produce those kind of way that will produce those kind of way that will produce those kind of typed objects that we care about. typed objects that we care about. typed objects that we care about. So, that's the SDK, but the actual So, that's the SDK, but the actual So, that's the SDK, but the actual guarantee here is when we own the guarantee here is when we own the guarantee here is when we own the deterministic execution. So, even with deterministic execution. So, even with deterministic execution. So, even with the typed SDK as our entry point, that the typed SDK as our entry point, that the typed SDK as our entry point, that really only guides the agent towards the really only guides the agent towards the really only guides the agent towards the desired end state. And the real desired end state. And the real desired end state. And the real guarantee comes from the final script guarantee comes from the final script guarantee comes from the final script that we orchestrate on agent completion. that we orchestrate on agent completion. that we orchestrate on agent completion. This run executor script um This run executor script um This run executor script um it is called when the agent has kind of it is called when the agent has kind of it is called when the agent has kind of completed running, and we think that completed running, and we think that completed running, and we think that we're in a like almost completed state we're in a like almost completed state we're in a like almost completed state or a ready to be completed state. And or a ready to be completed state. And or a ready to be completed state. And what that script does is it starts by what that script does is it starts by what that script does is it starts by linting the agent code, where we can linting the agent code, where we can linting the agent code, where we can immediately send it back to the agent if immediately send it back to the agent if immediately send it back to the agent if something is wrong. Better to kind of something is wrong. Better to kind of something is wrong. Better to kind of fail early rather than fail later. fail early rather than fail later. fail early rather than fail later. We can also detect conflicts. This is We can also detect conflicts. This is We can also detect conflicts. This is where sometimes the agent in one part of where sometimes the agent in one part of where sometimes the agent in one part of the code edited something in another the code edited something in another the code edited something in another part edited the same thing or something part edited the same thing or something part edited the same thing or something that depended on the first one and that depended on the first one and that depended on the first one and didn't realize that that happened. didn't realize that that happened. didn't realize that that happened. Because we have these guarantees, by Because we have these guarantees, by Because we have these guarantees, by giving the agent this well-defined giving the agent this well-defined giving the agent this well-defined structure, we can detect those conflicts structure, we can detect those conflicts structure, we can detect those conflicts for it so that we can send it back to for it so that we can send it back to for it so that we can send it back to the agent. the agent. the agent. We then run the agent edited code, and We then run the agent edited code, and We then run the agent edited code, and that means that we can then validate the that means that we can then validate the that means that we can then validate the output artifacts. Of course, if the code output artifacts. Of course, if the code output artifacts. Of course, if the code itself fails to run, we can send that itself fails to run, we can send that itself fails to run, we can send that back to the agent, too. back to the agent, too. back to the agent, too. And then finally, once we have the And then finally, once we have the And then finally, once we have the validated output artifacts, we can validated output artifacts, we can validated output artifacts, we can create a well-structured review artifact create a well-structured review artifact create a well-structured review artifact that makes it easy to review what the that makes it easy to review what the that makes it easy to review what the agent did without having to actually agent did without having to actually agent did without having to actually ever go read the underlying code that ever go read the underlying code that ever go read the underlying code that produced it. To give you a flavor of produced it. To give you a flavor of produced it. To give you a flavor of what that looks like, here's a little what that looks like, here's a little what that looks like, here's a little snippet from an emissions report where snippet from an emissions report where snippet from an emissions report where the graph edit function impact analysis the graph edit function impact analysis the graph edit function impact analysis ran on 50 graphs. ran on 50 graphs. ran on 50 graphs. Um there were two functions that it Um there were two functions that it Um there were two functions that it applied that produced 749 edit actions, applied that produced 749 edit actions, applied that produced 749 edit actions, and it ultimately in this case reduced and it ultimately in this case reduced and it ultimately in this case reduced the overall emissions by 45.6% the overall emissions by 45.6% the overall emissions by 45.6% and we can even go to go to see there and we can even go to go to see there and we can even go to go to see there are two code there are two edits here. are two code there are two edits here. are two code there are two edits here. Um one kind of did a big chunk and the Um one kind of did a big chunk and the Um one kind of did a big chunk and the other one made a very small change. other one made a very small change. other one made a very small change. Maybe you'd go look at what this Maybe you'd go look at what this Maybe you'd go look at what this actually did, and then you can even go actually did, and then you can even go actually did, and then you can even go at a graph level and say, "Okay, for at a graph level and say, "Okay, for at a graph level and say, "Okay, for this graph, like what was the option one this graph, like what was the option one this graph, like what was the option one and which nodes did it edit within that and which nodes did it edit within that and which nodes did it edit within that graph to get a sense of um what actually graph to get a sense of um what actually graph to get a sense of um what actually happened under the hood?" So, you can happened under the hood?" So, you can happened under the hood?" So, you can see all of this without actually having see all of this without actually having see all of this without actually having to go read the low-level code. to go read the low-level code. to go read the low-level code. So, to return to those three problems So, to return to those three problems So, to return to those three problems that uh we started with, now that we that uh we started with, now that we that uh we started with, now that we have these kind of two broad concepts have these kind of two broad concepts have these kind of two broad concepts that are part of the harness, um that are part of the harness, um that are part of the harness, um when the agent will find creative ways when the agent will find creative ways when the agent will find creative ways to solve every problem, this is now a to solve every problem, this is now a to solve every problem, this is now a great thing, but we've limited it a great thing, but we've limited it a great thing, but we've limited it a little bit. These invalid actions that little bit. These invalid actions that little bit. These invalid actions that the agent will take on its route to the agent will take on its route to the agent will take on its route to solve whatever problem we presented are solve whatever problem we presented are solve whatever problem we presented are actually prevented with the type dust actually prevented with the type dust actually prevented with the type dust SDK as the only way to properly edit the SDK as the only way to properly edit the SDK as the only way to properly edit the graph, and then we can guarantee that graph, and then we can guarantee that graph, and then we can guarantee that the graph was only edited through the the graph was only edited through the the graph was only edited through the SDK by owning the deterministic SDK by owning the deterministic SDK by owning the deterministic execution and the validation of that execution and the validation of that execution and the validation of that written code. written code. written code. The second problem about the agent The second problem about the agent The second problem about the agent saying that it made an edit when saying that it made an edit when saying that it made an edit when actually that the edit didn't go through actually that the edit didn't go through actually that the edit didn't go through in the way that it or the user expected, in the way that it or the user expected, in the way that it or the user expected, these false reports are caught via that these false reports are caught via that these false reports are caught via that deterministic execution so that we can deterministic execution so that we can deterministic execution so that we can surface those back to the agent um to surface those back to the agent um to surface those back to the agent um to make sure that it's doing what it make sure that it's doing what it make sure that it's doing what it claimed. claimed. claimed. And finally, our users never have to And finally, our users never have to And finally, our users never have to look at the code because the look at the code because the look at the code because the deterministic execution system produces deterministic execution system produces deterministic execution system produces artifacts in an expected form, and we artifacts in an expected form, and we artifacts in an expected form, and we can build verification against those to can build verification against those to can build verification against those to catch both the kind of good outcomes and catch both the kind of good outcomes and catch both the kind of good outcomes and the errors that are now surfaced clearly the errors that are now surfaced clearly the errors that are now surfaced clearly to both the user um and the AI. So, even to both the user um and the AI. So, even to both the user um and the AI. So, even when it hasn't maybe gotten exactly the when it hasn't maybe gotten exactly the when it hasn't maybe gotten exactly the right answer or the one that the user right answer or the one that the user right answer or the one that the user expected, it's easy to follow its logic expected, it's easy to follow its logic expected, it's easy to follow its logic and kind of get to that um and loop back and kind of get to that um and loop back and kind of get to that um and loop back on itself. on itself. on itself. Now, the harness is all well and good as Now, the harness is all well and good as Now, the harness is all well and good as it's kind of adds constraints, but I it's kind of adds constraints, but I it's kind of adds constraints, but I also want to emphasize that even with also want to emphasize that even with also want to emphasize that even with those great guarantees, you still need those great guarantees, you still need those great guarantees, you still need to hill climb to get better at the task to hill climb to get better at the task to hill climb to get better at the task itself. Since we started tracking how itself. Since we started tracking how itself. Since we started tracking how well the agent did at these complex well the agent did at these complex well the agent did at these complex edits, where each task here um in our edits, where each task here um in our edits, where each task here um in our eval set is a collection of graphs and a eval set is a collection of graphs and a eval set is a collection of graphs and a goal, goal, goal, um we've been able to improve our um we've been able to improve our um we've been able to improve our outcomes from about 43% to 92% on our outcomes from about 43% to 92% on our outcomes from about 43% to 92% on our set of internal evals. And to achieve set of internal evals. And to achieve set of internal evals. And to achieve that, all the typical approaches were that, all the typical approaches were that, all the typical approaches were still effective. We did a ton of prompt still effective. We did a ton of prompt still effective. We did a ton of prompt improvements, like rewriting the system improvements, like rewriting the system improvements, like rewriting the system prompt and all the kind of skills that prompt and all the kind of skills that prompt and all the kind of skills that the agent had access to. We've added few the agent had access to. We've added few the agent had access to. We've added few shot examples to both teach and coach shot examples to both teach and coach shot examples to both teach and coach the agent how to make use of the SDK and the agent how to make use of the SDK and the agent how to make use of the SDK and work with these different classes of work with these different classes of work with these different classes of tasks that we expect. tasks that we expect. tasks that we expect. We also had to make the tools, the SDK We also had to make the tools, the SDK We also had to make the tools, the SDK itself, and the tools the agent had itself, and the tools the agent had itself, and the tools the agent had access to better fit for use by access to better fit for use by access to better fit for use by improving their kind of ergonomics and improving their kind of ergonomics and improving their kind of ergonomics and how obvious they were for the agent to how obvious they were for the agent to how obvious they were for the agent to use. We also broke the task down through use. We also broke the task down through use. We also broke the task down through doing this, breaking up the um overall doing this, breaking up the um overall doing this, breaking up the um overall problem to a plan and execute kind of problem to a plan and execute kind of problem to a plan and execute kind of standard loop, as well as teaching the standard loop, as well as teaching the standard loop, as well as teaching the agent some of that expert judgment that agent some of that expert judgment that agent some of that expert judgment that is endemic in our space, making that is endemic in our space, making that is endemic in our space, making that easier for the agent to operate on it easier for the agent to operate on it easier for the agent to operate on it and to and to and to elicit that expert judgment from the elicit that expert judgment from the elicit that expert judgment from the user. But the important thing here is user. But the important thing here is user. But the important thing here is even though we've kind of improved the even though we've kind of improved the even though we've kind of improved the accuracy here, which is amazing, and accuracy here, which is amazing, and accuracy here, which is amazing, and it's great that we'll now kind of get it's great that we'll now kind of get it's great that we'll now kind of get better at that first time a user goes better at that first time a user goes better at that first time a user goes through the full workflow in the task, through the full workflow in the task, through the full workflow in the task, um even when the agent errors, or even um even when the agent errors, or even um even when the agent errors, or even our ground truth is, you know, one of our ground truth is, you know, one of our ground truth is, you know, one of those points on the possible range of those points on the possible range of those points on the possible range of expert judgments, we can still guarantee expert judgments, we can still guarantee expert judgments, we can still guarantee and validate the um process itself. and validate the um process itself. and validate the um process itself. So, to bring it all together in a domain So, to bring it all together in a domain So, to bring it all together in a domain full of expert judgments, you must full of expert judgments, you must full of expert judgments, you must respect the process, too. And for that, respect the process, too. And for that, respect the process, too. And for that, you really need guarantees about the you really need guarantees about the you really need guarantees about the process. process. process. We all know coding agents are necessary We all know coding agents are necessary We all know coding agents are necessary for complex tasks, and they will for complex tasks, and they will for complex tasks, and they will continue to be necessary and extremely continue to be necessary and extremely continue to be necessary and extremely useful. But those power that power comes useful. But those power that power comes useful. But those power that power comes with many risks that are like good to go with many risks that are like good to go with many risks that are like good to go and clear eyed about. For tasks with and clear eyed about. For tasks with and clear eyed about. For tasks with complex data and judgment calls, we now complex data and judgment calls, we now complex data and judgment calls, we now think that the harness should do a few think that the harness should do a few think that the harness should do a few important things. First, it should give important things. First, it should give important things. First, it should give the agent well scoped primitives to the agent well scoped primitives to the agent well scoped primitives to interact with the kind of external interact with the kind of external interact with the kind of external system that it's actually supposed to system that it's actually supposed to system that it's actually supposed to work with. Um free freely operating code work with. Um free freely operating code work with. Um free freely operating code can, you know, lead to a lot of the can, you know, lead to a lot of the can, you know, lead to a lot of the things that we talked about earlier. And things that we talked about earlier. And things that we talked about earlier. And this really allows the builders of the this really allows the builders of the this really allows the builders of the system to allow and disallow specific system to allow and disallow specific system to allow and disallow specific things and exercise the full power of things and exercise the full power of things and exercise the full power of these coding agents while still trusting these coding agents while still trusting these coding agents while still trusting the process that they're working in. the process that they're working in. the process that they're working in. I think the harness should also maintain I think the harness should also maintain I think the harness should also maintain full control of the final execution. full control of the final execution. full control of the final execution. These very smart agents may declare These very smart agents may declare These very smart agents may declare victory in an unexpected way from what victory in an unexpected way from what victory in an unexpected way from what you or your user really want them to you or your user really want them to you or your user really want them to declare. And so it's important to be declare. And so it's important to be declare. And so it's important to be able to constrain and validate that the able to constrain and validate that the able to constrain and validate that the agent has actually done what it said agent has actually done what it said agent has actually done what it said it's done. it's done. it's done. And finally, you should use that And finally, you should use that And finally, you should use that deterministic final outcome to produce deterministic final outcome to produce deterministic final outcome to produce outputs that are easy to validate even outputs that are easy to validate even outputs that are easy to validate even for non-coders. The code is kind of just for non-coders. The code is kind of just for non-coders. The code is kind of just the means to an end. the means to an end. the means to an end. And then of course, the kind of standard And then of course, the kind of standard And then of course, the kind of standard context and prompt engineering will of context and prompt engineering will of context and prompt engineering will of course remain critical to better scoping course remain critical to better scoping course remain critical to better scoping and defining that task and making it and defining that task and making it and defining that task and making it clear what the agent needs to do. clear what the agent needs to do. clear what the agent needs to do. That's it for me. Again, I'm Andrew That's it for me. Again, I'm Andrew That's it for me. Again, I'm Andrew Dumit at Watershed, the sustainability Dumit at Watershed, the sustainability Dumit at Watershed, the sustainability AI platform. Thank you again for your AI platform. Thank you again for your AI platform. Thank you again for your time and attention.