---
title: "Transcript: Agents Building Agents - Alfonso Graziano, Nearform"
category: "transcripts"
videoId: "aHhB3sjGjkI"
sourceLabels: ["YouTube transcript", "Cached transcript markdown"]
wordCount: "4742"
---

# Transcript: Agents Building Agents - Alfonso Graziano, Nearform

## Source Video
- [YouTube](https://www.youtube.com/watch?v=aHhB3sjGjkI)

## Local Cache
- `raw/sources/youtube-transcripts/aHhB3sjGjkI.txt`
- 4,742 words

## Transcript

Hi everyone. Today we will talk about agents, building agents. So, how we are leveraging coding assistance and spec driven development to build reliable and secure agents. First of all, just a couple of notes about myself. I'm Alfonso. I'm a tech lead at NearForm. We are a services company. I am currently working on some of our AI agentic projects. I'm supporting multiple teams adopting AI native engineering and I'm also a O'Reilly author of the book Learning AI Native Software Engineering. So, we are seeing in the in the industry that right now everyone wants AI agents for workflow automation, search, you know, every use case that you might have in mind.

But, AI agents come with a sometimes a very high cost. They have hallucinations. They have an entire new set of problems. Of course, they are non-deterministic. So, today we will see how we are building and improving iteratively AI agents. And of course, as you may guess, how can we do that, right? AI is very powerful and very good at building any type of software. And given that AI agents is just one type of software, as you may guess, we are using AI to build AI. Um As I was mentioning, there are a lot of problems, a lot of different problem classes with building AI agents.

Uh non-determinism is one of those, latency, cost, hallucinations, a lot of stuff really. Uh but today we will see how to solve them at least partially by leveraging agents and the repeatable process that we developed over time on our projects. Just as a very very quick refresher, um we can say that an AI agent is basically just an LLM, which is like the brain of the agent. Then the LLM is connected to a bunch of tools. It has access to a bunch of context and basically lives into an agentic loop. So, that's the that's the gist of it. Of course, there is way more into AI agents. There is everything around observability.

How do how do we ensure that, you know, it's doing what's what's necessary? So, everything around evals that we will discuss in a minute. But like just from the first principles, we can say that an agent is an LLM inside a an agentic loop, which is connected to tools and can retrieve context. That's it. Now, there are a couple of classes of problems that we will analyze today. So, the first one is bad performances on the evals, and we will see what the evals are in a minute.

And the second one is bad performances on live data, which sometimes is similar to what we have on the evals, but you know, live data coming from real users with, you know, real expectations from the system can be a lot wider and sometimes a lot messier as well. So, let's analyze the first the first failure mode. So, we have bad performances on evals. Before we analyze this failure mode, I just want to introduce you to go to the context of a golden data set. Basically, a golden data set is a file or a set of files that we develop together with the subject matter experts when we are building our AI agents.

And basically, this file defines what is the input that the system should retrieve um, and should get sorry, and what is the expected output that we should get. Now, in the naive case, the expected output, as we can see here on the right, can just be like impossible, or like a number, or a text, so it can be a lot of things. In real-world scenarios, the expected output can be, for example, I do expect the system to call this tool, or call this tool with this parameter, or call this tool in this chain, right?

Because maybe I want to do a retrieval and then do an update, you know, so how we define the output is very different in every in every agent. And the idea is that we want a way, basically, to ensure that our agent is working correctly. So, that's why we're building the golden dataset. You can see the golden dataset as a test suite, but in a non-deterministic scenario.

So, when we build the golden dataset, we also have to build a scorer, or a set of scorers, that can basically, go through the golden dataset together with the LLM, and then can give us, basically, a number in the end, which is saying, "What is the current accuracy of the of the system?" So, that, you know, we have a baseline, we can look for regressions, and we can further improve the system over time. Now, let's let's assume that we have a very simple hello world agent. In this case, I'm using Mastra. It's a very very simple agent, as we can see.

I've called it mad agent, but in reality, in the golden dataset, we have multiple types of questions, right? But, let's assume that this is our very simple agent, right? So, it doesn't aim, it has just couple of instructions, has has a model, of course. As you can see, it doesn't have any tools yet. So, it's like the the bare minimum. Um we also built a naive evaluator. So, if you remember from the golden data set, we only have inputs and outputs. And what this very naive evaluator is doing is checking whether the um LLM output contains the ground truth output that we are expecting.

Of course, once we run the eval suite here, the results are very poor. So, as we can see, um we have a pass rate of 18%, right? Uh just because a a lot of questions are simple enough, like additions, multiplications, right? Are simple enough so that the actual um LLM has this knowledge in its training data, and it doesn't need access to any external context or like any particular tool to answer these questions. So, 18% of the questions can be answered by the weights of the LLM, the rest can't. So, we need a way to actually improve this pass rate.

Some failure modes that we can expect are well, first of all, the agent is missing the right tools, right? So, let's assume that I expect from my agent to be able to surf the web, fetch web pages. If I'm not giving those specific tools to my agent, of course, every eval that is expecting that, it's going to fail miserably. So, first of all, we are maybe just missing some tools, or uh we don't have the right system prompt. Because the system prompt basically contains all the instructions, uh all the behaviors that we should expect, and that we shouldn't expect from our agent.

So, in a lot of cases, um a lot of the optimizations will be just to tweak the system prompt and update it so that um you know, our agent has all the information it needs to work on our domain. And then, um of course, context retrieval, which is maybe our agent doesn't know where to look for information. Of course, usually context retrieval is implemented as tools, right? Um maybe it's, I don't know, an accurate search in our system. Maybe it's a fetch on the web, you know, all these kind of things.

So, we need to give enough um let's say tools uh to our agent so that it can retrieve all the context it needs, of course, in a secure way. Now, the question is can an AI agent improve another AI agent autonomously or semi-autonomously? And um Karpathy, Andrej Karpathy, which is one of the most well-known AI scientist on the planet at the moment, implemented something which is called auto research. Basically, auto research is a uh loop which um updates the code, updates the hyper parameters, updates the the actual Python code of machine learning algorithms.

And it actually shows that a coding agent tweaking the code of machine learning um of a deep learning algorithm can actually improve the results, right? So, that's the um that's that's the loss that we can see. And basically, every uh dot here in this chart is an example is a task that the system is doing um by changing some of the parameters. And we can see that the baseline has a specific accuracy. And while we go um while we go with the number of experiments, accuracy is improving. And therefore, of course, the loss um the the loss is decreasing.

Now, the question is, can we do the same with AI agents and not just like machine learning models? The answer is yes. So, I built something which is called AutoAgent, which is basically the same idea, but applied to um actually like AI agents, right? So, basically this is a loop which is able to run the evals and then um update the code of the system, try new system prompts, create new tools, doing everything autonomously, and then check again whether things improved or not. And it actually works quite well. So, here we have a few iterations, and this is the AI agent This is the naive AI agent stat agent that we have seen earlier.

And in that agent, as you can see here on the bottom left, um we have the baseline accuracy, which was 18%, and we managed to reach up to 83% um in like something around 10 uh 10 iterations. Now, of course, that's a um that that's a very naive case, right? Because we started with something which didn't have any tools and nothing else. So, it's relatively easy for the system to improve this agent. But, uh we also improved um some evals by 10% on a production agent that was already humanly optimized.

So, actually the the machine uh the the agents the the coding agent found new ways that humans didn't find um to improve the agent, and we got plus 10% on some of our internal benchmarks. Now, how does it works? Actually, I I gave already like a few insights of how this works, but let's see it in details. So, basically, the core idea is that we have a coding agent, in this case Claude Code, but it can work with multiple coding agents. It builds the agent, so it writes the code of our target agent, and then the target agent is giving feedback to our Claude Code or whatever coding tool we're using as evals, right?

So, it's giving some information as okay, here I got some regressions or these evals are not passing. Also, Claude Code can read the traces, the thinking traces, or like the full trace of the target agent to see whether something broke or something is not working as expected, so it can do like self-improvement.

Of course, there is the human in the loop, and the human in the loop is usually in structuring the initial agent and giving as much context as we can to our coding agent about what it can do with other relevant files, what it cannot do, like just to give an example, updating the golden data sets or the scorers just to let the evals pass is not a good idea, so we want to enforce we want to tell the we want to tell the AI agent to not do that, right? So, as humans, we can we can steer a little bit our our coding agent so that it can optimize the target agent.

Now, there are a few steps, right? The first step is to create an optimization job. Of course, it's very easy, everything is a markdown, we all love markdown now, so we have we can define the objective, we can define target repository, the metrics, and literally everything, right? So, in this file, we can give as much context as we want to our coding agent so that it can actually run and improve the target agent. And then the second step is to run the loop. So basically, we run the evals once, so we we we generate our baseline data. So we understand what's working, what's not working, you know, the system will generate a first report.

And then we run the loop, the optimization loop. So we can see that in the first case, we get a regression, so basically the system works by creating an hypothesis. So it's tackling one class of problems at a time. It's updating the the agent and it's running the evals again. And then it's saying whether we got a regression, an improvement, or something else happened and so the agent can decide whether to roll back the change or move on. So in this case, in the first iteration, we got a regression, so we want to roll back. Um then we see a 5% improvement, which is great.

So in this case, we are getting we're creating a new hypothesis, we get a 0% improvement, unfortunately, but of course the agent is keeping track of all the hypothesis and so in this case can go and create a new one and we can see a 12% hypothesis and and so on and so forth, right? Of course, we define the number of iterations that we want to run. Um of course, the way we create the baseline data is, you know, we just run the evals once and we generate basically a document and this is super helpful because it gives the coding agent the information about the current status of the system.

Then this generates a baseline report which contains, you know, all these informations. It contains what are the cases, it contains summary, what's working, what's not working. So basically, our coding agent it's going to have a clear picture of what's happening right now. And so, it will try to understand, okay, how do I improve this system from this moment? And so, from now on, we can start with the first iteration. So, every iteration starts with a new branch. We create an hypothesis. Um the system changes the agent to implement that hypothesis. We run the evals. We run our eval suite. And we generate a reports.md file, which contains everything that happened after we run the evals.

We update the memory file, uh which is like a global memory file across all the runs. And if the metrics improved, then we continue from this branch. Um if the metrics didn't improve or we have a strong regression or something bad happened, uh then we roll back to the previous branch. Of course, um the generated hypotheses are based on what the agent reads when it starts the investigation, which is the memory file, uh the reports file, so it has access to pretty much everything. And that's what happens when we run multiple iterations, right?

So, let's assume that in the first case, I create a branch, I create an hypothesis, uh I run the evals, everything goes well, so I get an improvement, so I am fixing one failure mode. Um therefore, I continue from that branch. I create a new one so that I can keep track of every change. Um I create the new that new branch. I try new hypothesis. This time, it didn't really improve, so maybe I get a regression, maybe things didn't improve really. Uh so, what I do is I roll back to the previous branch, so the first branch where we had a real improvement in the system.

And then from here, I create a new branch, and I try a new hypothesis, and so on and so forth. At the end, what we're building is a full change log of all the changes we have done, um every improvement in improvement or regression as we can see here on the right. We are seeing all the hypothesis that we are doing and of course, if we want we can also see all the hypothesis because every hypothesis is building a report. So, let's assume that something didn't really go well in terms of evals. Maybe the evals didn't improve after an hypothesis, but maybe that hypothesis was promising, right?

Maybe the agent was onto something, but it just didn't implement the system the change in a correct way, right? What we can do is as humans, we can just go back into the hypothesis, read, understand what the agent was trying to do, and then maybe steer it in the right direction next time. Of course, this is the real task on a on a real agent that we did have.

And as we can see here, the baseline accuracy was 67% um but then in something around 10 iterations, we managed to reach 86% in our evals without actually cheating because it found edge cases, it improved the system prompt, it improved the tool descriptions to catch more edge cases, and it also fixed some tools logic. So, that's a real agent that is now running in production and actually this system found ways to improve it more and more. Now, the second class of problems that we are going to tackle today is not the evals data, but bad performances on live data.

By live data, we mean data that we collected by our real users, beta testers, subject matter experts, you know, all the people that are actively testing and using the system and they're giving us a feedback. Just as an example, you can see here on the right, was this response helpful? And you got yes or no, and then you get a note. And basically, we leverage this data to uh further optimize our agents. Now, how do we do that? Uh this is the full flow that we will see in details. So, first of all, of course, we will need our users to actually use the system. Uh then, we collect all the traces, right?

So, we collect all the informations from our users. Our users can either give us a feedback that we've just seen, so thumbs up, thumbs down with the with the comment, or uh our subject matter experts can annotate the trace. So, basically, they can go on the platform that we are using, they can look at the trace, look at the input, look at the output, uh look at basically like all the tools that have been called, uh look at the agent behavior, and then annotate the trace by giving us a feedback about how the agent performed and how it should have performed.

Once we collect a a a relatively good number of traces, once we have um enough information, we basically run an agent workflow that we built. So, basically, we analyze all the traces with both the negative and positive feedback, but we are more interested about the negative feedback here. So, we analyze all these traces. We do clustering of the failure modes, right? So, based on all the feedback that we collected, we try to understand, okay, given all this five feedback, what are like the five, six, seven failure modes that we have here? And then, we analyze these um we analyze these clusters of failure modes with our subject matter experts.

So, we say, "Okay, based on those 10 traces, we can note that the agent in this specific case performed poorly because of reason X, for example." We validate all this with our subject matter experts. And then we do have a fix proposal, which gets generated and implemented by our coding agent. Then we implement this and then we ship it to production to see whether it actually improved on real data. And of course, we use the traces that we do have as regressions. So, we test our fix against our traces first of all. But let's see that in detail. So, first of all, the user tests the system.

So, in this case, we have a very simple example, "How can I optimize a React component?" Here we have the answer. Here we have, you know, the information about the component, whatever, right? So, that's the first step. Second step is we are going to collect all the tracing information. So, we are collecting everything from user interaction, the tool usage, the how much time the AI took to respond, you know, how many tokens we burned, everything. The second step then is the user, once we collected the user behavior, we do expect the user to give us some feedback.

As I was mentioning, thumbs up, thumbs down, and then a a comment saying what went well, what didn't, and what's the expected behavior of the system. And then of course, the other option that we were discussing is that the actual subject matter experts can annotate the trace. So, in case you don't have a feedback from the real user, I can ask the subject matter experts to take a look at at trace, invest their time, and give us feedback as they were users. The third step is now to collect all the traces with feedback and download them locally in a JSON or whatever format you're using.

As you can see here in this case, we have 114 traces, and we're using those traces to then do the cluster analysis. How do we run the analysis? Well, this is actually a skill that we built. Basically, the skill it's going to instruct the coding agent properly to go in the go in the JSON file, look at all the traces, uh run some clustering, um then we do an adversarial review of that. So, basically, um once we have all the clusters, we do also a little bit of root cause analysis because, of course, our coding agent has access to the actual code of the AI agent, so it can do some root cause analysis.

It can try to understand what happened, uh especially because it has also access to the details of every trace. So, it can go within the trace, understand why the agent answered that way, you know, trace it down to the root cause. Um and then what's going to do is going to um propose possible improvements. And what we get at the end of this process is a full markdown report, which is very helpful for us. Uh it's going to give us like the positive feedback, the negative feedback, the negative rate, you know, we are just trying to collect as many data as possible.

And so, we get this very complete um executive summary and this very complete report, which tells us what are like all the clusters which are not working as the user was expected. So, in this case, uh we have that we have a small markdown formatting failure, which is saying URLs are not hyperlinked, and there are some formatting inconsistencies. So, it's linking some of the traces with the trace ID. Uh it's mentioning the user feedback. It's showing like some root possible root causes and it's telling us, "Okay, how do we fix it?" Right?

Once we have this full report, our work is not done yet because this report will contain all the possible failure clusters reported by users, but now what we have to do is to triage everything and validate with subject matter experts. Once we are doing Once we do the triage and the validation, then we prioritize and we understand what we want to fix now, fix later, don't fix. Um and so, either we fix it with the agent, so we give to the agent, you know, all the context, we give it the failure mode, we give it the traces, and we ask our agent to fix it.

Or we discard it because sometimes it can be the the failure cluster can be a false positive or can be an intended behavior or maybe the user gave us um a feedback which is not really useful for us at this moment. So, you know, there is a little bit of human judgment here. As I was mentioning earlier, all the failure modes that we are founding during this investigation step, they will become part of the golden dataset that we mentioned earlier and the eval suite is updated to spot those regressions.

So, in case these failure modes for whatever reason is introduced uh later on in the code, we will spot it very fast because now it lives into our golden dataset and our scorers. In terms of how often the report gets generated, it really depends on your use case. Uh we found out that one per sprint is good enough, but really depends on how much data you have, right? So, if you have 100 traces, it's you might want to generate that at 100 traces, but if you have like 10,000 traces with feedback, might be too much, so you might want to, you know, generate multiple reports. So, it's really on your use case.

We found out that once per sprint is actually reasonable. Uh in multiple use cases, what we found is that a coding agent, when instructed, uh has been able to fix an entire suite of issues like the one that we have seen earlier, uh with just one prompt. So, once we give enough context and once we give the coding agent the ability to uh test its changes against some regression tests, uh then everything becomes way, way easier for the for the coding agent.

And then, uh in some complex cases, uh we have paired this approach with AutoAgent, uh and so we basically pointed out AutoAgent to some of the failure clusters, um and we let AutoAgent build some draft PRs for us. Now, of course, both AutoAgent and this pipeline to improve um agents based on live data, they're possible thanks to Harness Engineering.

So, basically, the core idea is that we are leveraging a set of things so that our coding agent has a powerful enough harness so that it can change the codes, validate its own changes, which is like super important, uh and then propose new changes if the updates that it did didn't lead to the to the outcome that we were looking for, right? So, just very, very quickly, Harness Engineering is the idea of building the environment around our coding agent so that they can work reliably. Uh so, we are giving all the constraints, all all the tasks, the feedback loop, we are giving the governance to our um coding agents.

And we are doing that with a few things, right? Which we've already mentioned. The first one is spectrum environment. So, for example, when we are trying to fix some um failure modes, every failure mode becomes a spec. Um we are creating a spec for the expected behavior of the agent, and then we are implementing that. Then we have quality gates uh like linting uh our unit tests. We have our our evals. Um we have LLM uh code review, so a bunch of stuff. Uh everything related to context engineering, so giving the right context to our agent. And then, of course, observability. Um because we need to know what's happening, right?

Um if we don't know what's happening when we ship in production, we are basically blind, right? So, we want to ensure that we know what's happening so that our coding agent can fix any bugs or anything once once it's found. So, um I hope that this has been interesting or helpful. I hope you're going to implement some of those techniques in your agents. And in case you have any questions about this talk, uh please feel free to reach out on LinkedIn. I would love to have a chat with other people which are building agents. I know how hard it is. I know how interesting it is.

So, in case you have like anything that you would like to talk, uh I would love to get a message on LinkedIn. So, I hope you enjoyed this talk, and wish you a great day. Cheers.