---
title: "Transcript: I2Cbiws9J10"
category: "transcripts"
videoId: "I2cbIws9j10"
sourceLabels: ["YouTube livestream transcript", "Cached transcript markdown"]
wordCount: "91792"
---

# Transcript: I2Cbiws9J10

## Source Video
- [YouTube](https://www.youtube.com/watch?v=I2cbIws9j10)

## Local Cache
- `raw/sources/youtube-livestream-transcripts/I2cbIws9j10.txt`
- 91,792 words

## Transcript

Mhm. >> Mhm. >> [music] >> Mhm. >> [music] [music] >> Ladies and gentlemen, welcome to the AI Engineer World's Fair. We're delighted to have you with us as we continue exploring the ideas, technologies, and people shaping the future of AI. Please join me in welcoming your MC for today's program, developer relations engineer at Replit, Ralph Schaub. >> Good morning, San Francisco. How's it going, guys? Welcome to AI Engineer day four. Wow. I'm so excited to be here with you guys today. So, this is our biggest AI Engineer event ever. We have about 7,000 attendees. We also My God, when I look at this room. Uh yeah, it's You guys, you made it to day four.

So, let's give it up to you guys. All right. I live in a tiny village in Switzerland, and I can't believe that I can fit my entire village right here in this room. Uh speaking of Switzerland, uh who's here from Europe? Wow, so many of you guys. And I see that you're strategically placed under the AC. So, Please, guys, widen America. Stay cool, all right? Anybody local to the Bay Area today? All right. So many of you as well. Nice. Okay. All right. To be honest, guys, I'm super excited to be in San Francisco. I think there's no place to uh build for AI and talk about AI.

And what I think I think you guys are the coolest people in the world because you're constantly leading in the future. Um you guys drive self-driving cars, but you also have food delivered by drones. And what I notice nowadays is that nobody closes their laptops anymore. I don't know. Uh everybody [clears throat] seems to be agent maxing. Speaking of maxing, you guys enjoyed yesterday's talks? Yeah? Let's give it up for our speakers from yesterday. All right. >> [applause] >> There was so much to talk about and to think about yesterday. So, we had Tariq from Anthropic talking about Fable 5 and uh the process of discovering new models' capabilities.

We had another Tarik from Sonar who talked about uh code verification. But, what I know what I noticed and what I heard from speakers is that it's still very important to stay in the loop while working with frontier uh frontier models. And today, guys, it's going to be about harness engineering. And we have a great lineup of speakers for you today. So, we have speakers from uh leading organizations such as Anthropic. We have also um Stanford. We have uh folks from who are going to talk to you about new protocols. They're going to talk to you about how to separate the task from the model.

And uh and also uh they're going to talk to you about how to scale AI systems. And for having seen some of these talks, I can tell you guys that you've been that you're here for a treat. All right. And after the keynote, we have breakout sessions and we have so many tracks for you guys to choose from. So, we have tracks such as uh software factories, generative media, memory, and so on. And please, take also the time to go and check out our expo. We have so many cool swags. Our partners are there.

Uh go uh talk to them and and uh and and please, if you if you need to take any swag, it's going to be This is today. This is the last day. So, uh it's now or never. Uh speaking of partners, please, a big shout out to our presenting sponsor, Microsoft. All right. Let's give it up for Microsoft, guys. Also, let's give it up for all our lab and platinum sponsors. Please, keep it up for also for our gold sponsors, for our silver and and and bronze sponsors. This is super important, guys. Without the support of each one of these companies here, this event wouldn't be possible. All right.

So, again, we're going to talk about harness engineering, but before we introduce our first speaker, what I want you to know guys is that you as attendees, you have huge power of over the quality of the talk that you're going to be attending today, okay? So, our speakers have been doing this forever. They know their stuff inside out, but what we want them to do is to feel the energy in the room the second they hit the stage, all right? Okay? So, we're going to practice that for a second.

So, at the count of three, I want you to imagine as if I was a speaker, and I want you to welcome me as this rockstar, okay? So, 1 2 3 All right. >> [cheering] >> Nice. So, let's give it up for our next speaker, okay? So, without further ado, please let's introduce our first speaker of the day. She is a partner at Amplify. Please join me welcoming to the stage Bar Yaron. >> Now joining us on stage is a partner at Amplify, Bar Yaron. >> Fantastic. You did a great job practicing. I feel very very loved. Um let's get started. So, like you just heard, my name is Bar.

I run a survey every year on the state of AI engineering. And the funny thing about running a survey on the state of AI engineering is that the field changes as you make the slides. Just in the past week, we've had frontier releases treated like national security events, Meta reportedly exploring selling AI compute. By the time I get off stage, maybe something else will happen. So, if I miss a major announcement while I'm up here, please come find me after. But, that's exactly why we run the survey every year to cut through the noise, take a moment, step back, and understand what AI engineers are actually doing.

Uh for the first time this year, we were thrilled to partner with Notion and Vercel to run this survey. Very quickly on me, uh this is the least interesting slide. I'm an investment partner at Amplify. Very lucky to invest in companies built by and for AI engineers. And I'll make the same promise that I make every single year, which is short time on bar, long time on bar charts. So, let's get right into it with lots of bar charts. First, let's talk about Well, maybe raise your hand. Did you fill out the survey? This is a very large group. Okay, yes, I see you in the front.

Um if the answer is you, thank you so much. If the answer is not you, I will find you in 2027. [laughter] But, genuinely, this only exists because a thousand of you gave your time, so thank you. We had 1,048 respondents this year, which is a lot of AI engineers. And to be precise, this is not just AI engineers, as I'm sure you see at the conference. Every year, we see that AI engineering is more of a discipline than a job title. It touches founders, CTOs, engineers, product people, folks across company sizes, and experience levels. And that range shows up in experience, too.

Um for the third year running, we see the same pattern, which is skewed towards senior engineers, but newer to AI. Of those with over 10 years of software experience, over half have 3 years or less of AI experience, which tracks. Uh these are very experienced engineers learning a new paradigm in real time. And the newest cohort, the ones who just started uh engineering, the median new engineer has nearly as much AI experience as the median 10-year software veteran. So, the newest engineers have never known software without this. But doing AI doesn't mean one thing. We talked about all these different titles, all these different roles.

Before we get into models and agents, I have a more basic question, which is when people say they're doing AI at work, what are they actually doing? So, first up, like to start with the modalities. We asked, "Which modalities are you actively building with at work?" Can anyone take a guess? Text dominates. I know, hold your applause. Um but one piece of this chart that I always find very interesting and I always look at is the ratio of nope, I'm not using this modality to I'm not using it, but I do plan to. I call this the intent to adopt ratio.

Of the people who are not building with a modality today, how many say they plan to use it? And audio has the strongest intent to adopt this year. Among AI engineers who are not building with audio today, a whopping 56% say they plan to adopt it in the AI applications they build. And this is not a brand new signal. Last year, audio also had the highest intent to adopt across modalities, but 37%. So, audio continues to take the lead and have high interest, but that interest is accelerating.

Now, there has been an audio swing, but if we look at what changed most from the last year in the survey, the biggest jump is actually in people using image generation. The share of respondents using generative AI for images and feeling really good about it doubled from 18% last year to 36% this year. Makes sense if you look at what we launched in the the window over the past year plus survey time, we've had models Nano Banana, Nano Banana 2, ChatGPT Images 2.0. The products have gotten much better. What used to feel like an efficient way to generate cursed hands is just increasingly becoming a part of real work.

Audio may have the strongest intent to adopt, but image generation shows us what happens when a modality crosses that threshold. So, I'm excited to continue watching these adoption curves every single year. I think we're going to see a lot this year. Uh now, models. If you've Who here spends time on Twitter? All right. Yes, I would imagine this is a very Twitter-pilled crowd. If you spend any time on Twitter in this in this circle, you've seen a lot written about open-weight models these past few months. And I think we'll see it even more in the next year. Um so, we asked, "What models are you actually using in production?" 94% use closed models.

45% are using open-weight models. But, here's the thing, you know, open-weight models are not replacing closed models for the most part, at least not yet. The respondents using open-weight models, over 90% of them are also using closed models. So, they're looking like an augmentation. Teams are mixing and matching. We also asked, just to double-click on this, for the top three considerations when choosing a model. If you're choosing a model, what is important to you? Um and despite the airtime of the open versus closed, it's not what drives model choice. It was a top three consideration for only 5% of the respondents. What matters is actually more straightforward, it's quality.

Quality dominates, followed by agentic capabilities like tool calling, and cost tied right with it. We'll money, money, money, we'll get back to that. Um one thing that I found very interesting is that reliability is not near the top. Only one in five named reliability. That doesn't mean teams stopped caring about reliability. Uh there are different ways to interpret this data. My guess is that it's more likely to become a threshold requirement and the models they're choosing are reliable enough so the decision moves up the stack outside of certain circumstances to quality, capability, cost. But we can talk after. All right. So here's where the model story all comes together.

Like I said, teams are not choosing one model and calling it a day. Earlier I showed that 87% of teams are using more than one model. Uh the model that's the opposite of standardization. Uh and the way that they choose models for given tasks varies. Most popular is routing by task type. Some run multiple models, compare outputs. Some route based on cost. Uh but models are good at different things. What was interesting was that more than half of respondents said that their organization is starting to standardize on fewer AI tools. They're trading standard is it flexibility for standardization. A share of those are mixed.

They say they're standardizing on some layers while staying flexible on others, but the headline here is that there's we're in the early great standardization of the platform and tools, not the models. All right. This is the slide where anyone who's opened an AI bill in the last year starts nodding. So it turns out that infinite intelligence still comes with a usage-based bill. Once teams are managing many models and AI workflows, the next question becomes cost. Cost is now a first-class engineering constraint. We see this in the data. 40% of respondents say that cost regularly shapes how ambitiously they use AI. And another 36% say that it sometimes does. Well, this is pretty straightforward.

So, all in about three out of four respondents are adjusting their AI usage based on cost and maybe the fourth has a company card. That might be surprising or maybe it's obvious, but 12 months ago it was not. Token maxing is cool. Being able to find real use cases is amazing, but cost is becoming a real big part of the product decision today. And it shows up in monitoring, too. What folks are monitoring in production includes cost and token usage as the number two thing they watch for. It's being monitored like an SLA right under quality itself. Which brings us to the biggest line item of them all, agents.

We've been talking about agents for a while. This year, as you've seen, as you'll see today, as you've seen in previous days, you're going to talk a lot about harness engineering, they're escaping demo world. So, we asked respondents what level of tool permissions their agents typically have. And this is where agents start to look more real. There are two things happening at once. First, and I don't think this is surprising relative to last year, there are far more teams using agents. This year, 95% This seems high to me. 95% say they're using agents, roughly double last year. Second, amongst the teams that are using agents, those agents are much more likely to have write access.

Last year, 52% of folks building with agents said their agents could actually write data. This year, that number is 89%. So, when you combine these two shifts, more teams using agents and more of those agents having write permissions, the share of all the respondents, and again, the survey, using right-enabled agents is up more than three times relative to last year. So, this is really the big shift. Agents are no longer reading, summarizing, drafting, they're taking actions inside of systems. And that raises the obvious question, how are we controlling all of this? Um with pretty blunt instruments. Uh very There are many ways that folks are controlling agents today.

The top two are human-in-the-loop approvals and gating permissions, which are the right instincts, but kind of the same toolkit you'd use to manage an intern. Below that, the results scatter. Task decomposition, retrieval, memory, sandboxing, people are trying everything. Nobody has settled the control layer for agents. Um memory and persistent context is one that I'm watching very carefully right now. I think it's going to evolve a lot in the next year. And when agents fail, or when people complain about agents failing, uh to be more precise, it's usually the thinking, not the plumbing. So, uh you know, like 2/3 say that hallucination or losing context mid-task is what frustrates them the most. All right.

So, agents are out in the wild, which makes it a good time to look at what everyone's actually running underneath. So, let's take a peek at the stack. Um we asked, "What is the biggest challenge in your stack?" Every single year that I ask this, they answer The number one answer is evals. Um so, evals lead here, same as always, but by a very thin margin. Like, that margin is getting smaller. And I'll say the quiet part here, which is that 96% of the people in the survey in this room have a problem with the stack, just can't agree on which one.

Um so, if you're deciding what to build next, if you're interested in infrastructure, that scatter is the map. And the leading challenge, how to evaluate your AI outputs, requires many different methods, but as always, the vibe review is number one. So, there are So, some consistent things that we'll see if they change over the time, but they they have not changed. Okay, this is interesting. So, across eight layers of the stack, we asked, "What do people build versus buy?" Um again, maybe the corporate card is is going to play a part in this, but there is a wide range and mix for every layer of the stack and a few clear takeaways.

So, the first is that inference and model serving is the layer that people buy the most. Many people don't want to build inference infrastructure, and fair enough. Uh prompt management is the opposite. 61% build it themselves. Um apparently, everyone's prompts are special. And this is true of a lot of the product logic, prompts, rag, evals. They tend to stay in-house on a relative basis. Fine-tuning is the clearest not yet. Like, most people don't have it at all. And uh folks are pretty locked in. So, those who bought aren't looking as much to build. Those who built aren't looking as much to buy.

Uh but those are those are the core takeaways from the usage in our stack. So, many of you work on teams, and like we said at the start, these range from solo founders to large enterprises. What is this doing to teams? And remember, this is a builder-heavy sample, but among builders, the vibes are good, which, you know, I'm sure if you look to your left and your right, you're feeling that. The vibes are pretty good. 97% report a net positive effect on their organization. The top effect isn't really just speed. It's cheaper failure, more experimentation, more prototypes, more bets. It didn't just make engineers faster, but it made trying things nearly free.

And so there's some happy campers as a result of that. But it's not free free. You know, there's no free lunch as nothing is. So the same tool that increases experimentation also increases review burden. Both can be true. And um you know, over nine and 10 respondents are feeling negative downstream effects in some way. The most common ones being widely you know, widely discussed at this conference uh online and anywhere that you see AI engineers, erosion of deep technical skills and understanding of the code base. And these are consequences of cheap code generation. And the org chart is really feeling it.

So many folks, 81% are saying that AI is blurring the line between their role as engineers and product design and marketing. These stats shocked me. Um where you feel it the most is shipping software, once exclusively the engineers domain. I know folks talk about vibe coding and how that's accessible to more folks than ever before in different roles, but today over a third of teams have non-developers shipping features, which was pretty wild to me. Mostly smaller, mostly internal, but 17% say that non-developers are regularly shipping customer-facing features across the stack. And even when non-developers aren't shipping, a third of teams see them building really useful things, prototypes, front-end mocks, and more.

So shipping software is not gated on being an engineer. We knew this, but uh the extent to which it's being pushed is is higher than I expected. All right, so where does all of this go? We always ask people to place bets rapid-fire. So let's talk about those results. Um So, present tense first. 76% say AI boosted their job satisfaction. So, that's good for most of this crowd. I hope you're uh as uh Elphaba and Glinda say, I hope you're happy now. Um that's great. But, 59% fear today's AI code creates long-term liabilities. Only a third call software engineering a solved problem.

Although uh when I have conversations with folks, sometimes the way in which they define software engineering is different. So, you can read into that stat as you will. Um happier, faster, but embracing the maintenance build is the TLDR, and people are unsure what's going to happen with hiring. And for the 5-year bets, we have 67% expect a leading lab will declare AGI in the next 5 years. Note the wording. We said will de- We asked about the press release, not the achievement. So, will they declare it? Yes. What does that mean? Not sure. Uh Only 9% bet on transformers being state-of-the-art in 5 years. Most are unsure. Uh but that was interesting.

And then, my favorite, will there be more AI compute in space or on land? 36 yes, 38 no. The most divisive question in the survey is about outer space. I promised you a lot of bar charts, and that was a lot of information. So, a review or our 2026 wrapped. Impact is overwhelmingly positive. Image gen doubled, or happy image gen doubled, while audio has the highest adoption intent, the same as last year. Cost really became a first-class constraint, and we see that everywhere. In monitoring, in how ambitious folks that are going out and building AI products are behaving. Open weights augment, but they don't replace.

So, we're seeing a multimodal future with a consolidation of the stack. Agents got right access more than ever before, tripling relative to last year, while the guardrails stayed pretty primitive. And inference is the buy market. Everything closer to product logic tends to relatively stay more in-house. It is a very exciting time to be an AI engineer. I cannot wait to see how the next year unfolds. So, you can find the full report in the link up here. Every chart, plus some cuts that we didn't have time for today.

Um I won't ask you to fill out a survey about the survey, but if there's something that you want on the books for 2027, something you're curious about, you can come find me here on the internet. I'm easy to spot. Thank you so much. Uh we will see you next year, or for 36% of you, maybe in orbit. Thank you. >> Please welcome to the stage the Professor Emeritus at [music] Stanford University, John Ousterhout. >> [applause] >> Good morning. It's really great to be here to talk about the network side of AI applications, and in particular, to make the case that latency matters, and it's probably going to be mattering more in the future.

But I just want to say this is a talk is unusual for me. I've never before given a talk where there are fog generators in the auditorium. It's a really San Francisco experience, I guess. So, it's it's well known that AI workloads depend on really great networking performance in order to achieve their own performance. Of course, that's because the workloads are so large that they have to be distributed across machines and then you have to communicate between the machines. But, what I want to talk about today is it it seems that those workloads are changing. And so, I hope to do three things over the next 15 or 20 minutes.

First, to convince you that in fact workloads are changing and that whereas the workloads used to be completely dominated by a large transfers where throughput is the key metric that matters, that we're seeing more and more smaller transfers where the latency is crucial. The second thing I hope to do is to convince you that legacy protocols like TCP and RDMA are poorly suited to this environment. They weren't designed for this environment and unfortunately, they suffer from very high tail latency when you mix small messages with large ones. I'll talk a little bit about why that's the case.

Then third, I'd like to introduce Homa, which is a new protocol we've developed at Stanford that actually was designed in a clean slate redesign to handle data center workloads like these. And in fact, it does quite well on those workloads and can reduce tail latency by an order of magnitude or more. So, I'll tell you a little bit about Homa. So, let's dive in. First, workloads. Historically, AI workloads have consisted of enormous transfers between machines. That's all that really mattered. Gigabytes of data for things like weight gradients and and so on. In these workloads, what you really care about is throughput. How many gigabits per second you can pump through the pipes.

And these are relatively easy workloads for networks because if it takes a while to set up the connection and start the transfer, it doesn't matter. The transfers go go for so long that all that really matters is the throughput. And so in these environments, TCP and RDMA perform pretty well. Uh by the way, when I say RDMA, what I really mean is Rocky, RDMA over converged Ethernet, which is the underlying transport that's used by RDMA for most purposes today. So anyhow, the old workloads, big transfers, throughput matters, uh the legacy protocols work pretty well. However, it appears that the workloads are changing.

They're becoming more granular with smaller chunks of computation and smaller exchanges of data. And this seems to be particularly true in the world of inference, and also in agentic workloads. Not so much for training, workloads are still massive transfers. And so what's happening is that more and more there are small message exchanges, typically for things like metadata and coordination, such as checking to see if a particular entry is present in a KV cache that's distributed, or doing barrier synchronization at the end of periods of compute. And for these workloads, what really matters is latency.

That is, what's the round trip time to send some small piece of data across the network, do a little bit of computation, and get a small result back again. And in fact, it isn't just just latency or average latency that matters, what really matters is tail latency. That is, you'd like to know that if we send a whole lot of small messages, all of them will complete quickly. So for example, we typically measure things like 99th percentile latency. And if we have high tail latency, that can limit the overall throughput of the system. So here's an example.

Suppose a common thing is to take a workload and split it up across several nodes, which do intensive computation using their GPUs for some period of time. And then once they've all finished their computation, you do some small exchange between the nodes to exchange data, metadata, and then they'll go on to the next round of computation. And while that exchange is happening, that synchronization is happening, the GPUs are sitting idle. So, if even one of those exchanges takes a long time, it turns out the whole process stalls. You need all of those exchanges to complete before you can go on to the next phase of computation.

Now, if the computation phase is say 5 seconds, and it takes a few milliseconds for the exchange, you know, not a not a problem. And that's historically what it's been. But now with the AI centric workloads, where you're trying to pump out tokens relatively rapidly at a regular rate, the periods of computation are getting down into the sort of the millisecond time scale. And if it also takes milliseconds to do that synchronization, then you're wasting a significant fraction of your GPUs resources waiting for the the synchronization to occur. So, I'm curious. I'd like to just do a a quick audience poll here.

Is there anybody here where you have reason to believe that the latency of small messages is impacting the overall throughput of your applications? If so, can you just raise your hand? See is there anybody out there today? Actually, more hands than I expected. So, quite a few people out there are raising their hands. I think this problem is likely to get worse as the trends continue. So, what's going on? Why is tail latency bad? Well, typically the cause is congestion resulting from incast. So, incast is when several nodes all decide simultaneously to transfer data to some destination node. And if they all send large messages, well, the links are the same everywhere in the network.

So, three nodes can transfer three times as fast as one node can possibly receive. And so, what happens is that packets accumulate at the last hop going to that destination in the top of rack switch at its egress port for the destination node. Then, if some other node decides it wants to send a short message to that same destination, the short message gets stuck behind the long ones in the queue there. And actually as that causes delay and in the worst case, so many packets arrive that the switch runs out of buffer space and it has to drop packets and then there are your timeouts and retransmissions that make everything even worse.

So somehow we need some way to reduce the congestion in those queues. Somehow we have to get the sending nodes to stop sending so fast so the queues don't just build up without limit. So the way this is done historically virtually all network protocols before Homa, including TCP and RDMA, congestion control is the responsibility of the sender. So senders somehow have to figure out that congestion is happening and then they have to slow down their rate of transmission. Now you you might wonder why are senders doing it cuz the congestion is way over at the other end of the data center network. How does the the sender find out?

Well in the in the old really old days, the way they would find out is the queues would overflow and packets would get dropped. The sender would detect the packets got lost cuz it wouldn't get acknowledgements back and it would assume that means there's congestion and then slow down its rate of transfer. That's really expensive so today there are better techniques that mostly involve the switches providing information. So a top of rack switch when it sees that the queue length for an egress port has reached some threshold, starting to fill, long before the queue overflows, it starts marking all of the packets to pass through with what's called early congestion notification ECN marking.

And so when those packets pass through to the receiver, the receiver sees the marking in the packets and then when it communicates back to the sender next, for example to send an acknowledgement, then it includes that marking that goes back to the sender and now the sender sees the sender realize, oh there's congestion someplace. I've got to slow down my rate of transmission." So, that's the basic idea. Unfortunately, getting this right is really hard. Really hard. It's very hard for the congestion to figure out exactly how to set its rates cuz it gets one bit of information. There's congestion someplace. And there are multiple senders all sending to the same destination.

They're all trying to make adjustments simultaneously. How much do you cut back? And how do I know when I can ramp up again? And even worse, it's really hard to do this in a way that's stable because there's control lag. That is, it takes time before the sender finds out that there's congestion. And in fact, using this process, it typically takes several round trips for the sender to gradually adjust its rate to get just the right rate to match the available bandwidth. But by the time you do that, in a network that things have changed. New transmissions have started or old ones have finished. And so, these systems tend to never stabilize.

They're constantly oscillating between sending too much and sending too little. And this problem has been around for a long time. It's been known in the research community for more than 20 years now. There've been tons of papers published on it. There have been some improvements made. That's undeniable. But we're still a long ways from anything that works well. And the problem is with the fundamental nature of it doing the the congestion control on the sender side. It just doesn't work very well. So, you end up with a lot of queue build-up. And in fact, you can see the only way to find out that there's congestion is if there's queues.

And so, by that point, we're already experiencing delays. So, that's a problem. Oops. There's one other problem with TCP and RDMA also is that their their basic data model is a byte stream. Just a stream of bytes with no differentiation in it. So, if you send a series of messages, say, through a TCP socket, they get serialized into that stream. And on this slide, I've, you know, I've shown the messages appear like they have different colors in the stream. Well, there are no colors in real life. TCP has no idea where the message boundaries are. And that also makes life hard. For example, you don't know how much more data is coming.

If you knew how big the message was, you'd know how much more is coming. And you can't prioritize short messages, which we'd really like to do, get the short messages through faster. And you can end up with what's called head-of-line blocking, where somebody sends a series of messages to the same destination, and they send two really large ones, and then a small one after that, they get stuck behind them in that stream. And so it gets delayed, and again, you have tail latency issues. So all in all, TCP and RDMA are just not well suited to this environment. So what do we do?

Well, what I'd like to do next is tell you about a new protocol called Homa that we've developed at Stanford, which was based on a completely clean slate redesign for network transport. If you could start from scratch and rethink how you do transport for data centers, how would you do it? And it turns out in Homa, virtually every major design decision is different from TCP and RDMA. TCP for all the amazing things it's done is just not a good match to today's data centers, nor RDMA. So what Homa does particularly well is to manage a combination of large and small messages, and to make sure that the messages have short messages have really low latency.

So this started off as a PhD dissertation for one of my students, Ben Moazeni, and then the results were so great that I decided to make it my personal project to see if we could get it out of the lab and into production. Uh as you may know, I'm I'm not like most professors in that I love to code, and so I turned this into my my own programming project. I created a kernel module for Linux, and currently working through the process of getting that upstreamed into the kernel. It's available on GitHub for download. So let me tell you just a little bit about how Homa works. I want to mention three things.

First, it's message-based, not stream-based. In fact, the fundamental unit in Homa is a remote procedure call, which consists of two things: a request message sent from a client to a server, and then a response message returned back from the server to the client. So, the key thing here is that Homa knows about message lengths. They're buried in the transport all the way down to the bottom, and this has a bunch of advantages. First, it allows us to predict the future. As soon as the receiver gets the first packet of a message, it knows exactly how much more data the sender wants to send, and that's so precise so much more information for doing congestion control.

Second, Homa prioritizes shorter messages. It uses SRPT, shortest remaining processing time first, to try and prioritize shorter messages. And third, because messages are all independent, they're not serialized into a stream, every message independent, shorter messages can bypass long ones, so they don't get queued behind long messages. The second thing about Homa that's different is that it controls congestion from the receiver. And when you think about it, this makes sense because the congestion happens primarily at that last downlink to the receiver. And so, the receiver has way more information. In fact, with Homa, as soon as it gets the first packet of a message, it knows exactly how much more is coming.

So, it has essentially complete information about congestion, and it can therefore respond to congestion much more quickly and much more precisely. The way things work with Homa is that when a sender has a message to send, it breaks it up into packets, but it only transmits the first few packets, those are called unscheduled packets, to the receiver. Packets after that are called scheduled packets, and they only get transmitted when the receiver asks for them. So, the the will send grant packets back.

They'll paste them out and send those back to the sender over time telling the sender, "It's now time for you to send me the next chunk of data." And the receiver can delay those grants. So, for example, if the receiver has 10 messages that are incoming, there's no point in sending grants to all 10 of them because then you'll just get congestion in the in the top of rack queues. So, it can use the grants to reduce congestion. And then it can also use the grants to give preference to its most favorite messages, which would be the shorter ones. So, it's a way of of implementing SRPT by favoring short messages.

The third aspect of Homa is that it takes advantage of the priority queues in modern switches. So, modern data center switches have more than one queue at each egress port, typically eight. And they can be used in a priority mechanism where packets get transmitted preferentially from the highest priority queue. So, I've shown only two queues on the slide here, but typically there's more than that. You can specify in a packets using the various fields of the packet, you can specify which queue it should go into. And so, Homa dynamically makes those choices in a way to give priority to shorter messages.

So, if we go back to the in-cast example from a few slides ago, all of those long messages will pile up in the lowest priority queue. But if there's a short message coming, it will use a higher priority queue. And so, it will immediately bypass all of the queued packets from the short from the the longer messages and get through to the destination more quickly. So, how much of a difference does this make? And here's a on this slide I've got one sample benchmark that I use as part of my tuning and evaluation of Homa.

It consists of a workload of a bunch of machines on a network that are exchanging messages back and forth of different sizes ranging from very small to very large. And on this graph, you can see on the X-axis is the message length from about 50 bytes up to a megabyte. The Y axis shows you the round trip time for messages of that length. So, this request This uses request and response messages that are the same length. You can see TCP in green, Homa in blue, and the Y axis is is round trip time, so lower is better. And for each protocol, I've got two curves. One curve is the P50 curve.

That's the median latency for messages of this length. And then P99 is the 99th percentile, i.e., tail latency for messages of this length. So, I want to point out two things. First, the P99 for short messages is dramatically better for Homa. So, with TCP, it's more than a millisecond tail latency. Homa is less than 100 microseconds, about 13 times faster. Second, interestingly, you might think that because Homa favors shorter messages that long messages suffer and get worse performance. It turns out that's actually not the case. Even on the longest messages, Homa is almost a factor of two better than TCP.

I don't have time to explain that today, but it has to do with the fact that Homa uses run-to-completion approaches, which are much more effective than fair than the fair scheduling used by TCP. So, just to wrap up, the role of short messages in AI appears to be increasing. I think I think it's likely that it's going to continue to increase. We'll see over the next year or two if that happens. And I just want to pose a question to you, you know, as you're running your applications and measuring performance and seeing what the bottlenecks are, ask yourself, is high latency for short messages affecting your throughput?

If the answer is yes, then just know there is a solution available. You should give Homa a try. You can probably reduce your tail latency by an order of magnitude or more. And by the way, this is Homa is basically my life mission right now. I've sort of semi-retired from Stanford, and the reason I did that is so I can spend 100% of my time hacking on Homa. So, I'd be delighted to work with you and help you if you decide you want to experiment with Homa. If you need help getting started, answer questions, bug fixes, whatever, you know, I'd be happy to work with you to try and make you successful with it.

So, if that is interesting, feel free to contact me. My email's on the slide, or you can Google me, too, and find me over the internet. So, thanks very much for listening, and hope to hear from some of you. >> Please welcome to the stage the core contributor and the lead maintainer at DSpi, Maxime Rivest [music] and Isaac Miller. >> [applause] >> Wow. Isaac, myself, all of the DSpi community are so grateful to be here today to get to talk to you about AI programming, DSpi, and the unreasonable effectiveness of separating the task from the model, its harness, and all of the implementation details.

When you think about it, in programming, if we want to repeat a task often, we make it a function. We believe this same should be true for AI programs. Functions are awesome. Functions are reusable, composable, testable, and optimizable. To make a function, you give it a name, you define some inputs, some outputs, and then you have some implementation logic inside of it. You get to reuse your functions dozens of times. You can optimize it, but you can also compose it into bigger programs.

One of the really nice things about functions is that you can also package it and distribute it, and someone else can use it, and they just need to know about the contract on top of it to use it, and they can treat it as a black box. DSPy brings all of these properties to AI programs. And so, DSPy is an open-source software in Python that lets you, like I said, bring these properties to your AI workflows and AI programs. And it gives you all of the toolings you need to do that. Why do you want that? Well, we have been inventing a lot of terms in our fields in the last 3 years.

It's growing fast. We have new models coming every other week. We have new techniques, new strategies. And if you're like me, you want to try all of them. But, will any of these new specific techniques coming out at a different time really help on your task, on your job? Well, these are all just implementation tactics. And you want to put them inside of clear contract. If for your repeated AI task, you define an input interface and an output interface, you get to play in the internals. You get a lot of agility. Let's make it concrete for AI.

So, my first AI program I made when I discovered for DSPy was that I had some invoices from my farm, and I wanted to extract them to do my taxes. I wanted to extract the tax values from there. Then, another AI program I did is that on my keyboard in my computer, I have a a little command that reads my keyboard shortcuts, reads my clipboard, and will correct the grammar for me. Sometimes, I actually want it to also rewrite for clarity. So, I have another program that takes text, just rewrites it for clarity, put it back in my keyboard keyboard, and that's a command.

And then I can like have a lot of agility and bring it different places. Inside of that, I can change it however I want. A new model comes out, and I can change that. It's super easy cuz my interface is fixed like that. I'll skip that one. But, they're not uh restrained to very easy things and small input outputs. You can be very ambitious with AI programs. So, in these examples, you could have your entire inbox and a new email coming in, and you want to compose a new drafted reply. We can do that in DSPy with RLM, recursive language models. This is an idea that came from around our community.

Or, more like things we probably all do, agentic engineering or vibe coding, you can give it a spec, a repository, and you get a PR. Those are repeatable tasks. And so, as I have been telling you, when you fix that boundary, you can focus on the how on the top, and then inside of it, you can I have a little chat with just a simple prompt. You can iterate on that prompt. Agents come out, you change it to be an agent. Tools gets invented, you add tools. And then we get into loop engineering, you put that inside of it too. Anything on the outside of it doesn't change.

Your integration and and anything else doesn't change. And when you have such a hard boundary, you can also start to automatically optimize. But, how can you automatically optimize with just that simple signature? This is not enough. This is not enough to specify your task. And even before ChatGPT came out, the creator of DSPy had started to land on this idea that you need three things to specify your task. And if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details. So, the first one is what should happen. This is instructions.

The signatures that I've been showing you are part of that. Here on the screen, you see the beginning of a real script in DS9. You set your model at the top, you configure that, and it's fully independent of the signatures here, where you have natural language instruction to extract all taxes and um and if it's illegible to output zero. Then you say it, I'm going to give you an input, it's going to be a string. I want you to give me an output, and it goes it's going to be a string and a float. This is natural language expressing my needs. This is very powerful and efficient.

If you think about it, if you have a friend over coming to play a board game with you, and you give them the instructions and they're ready to play. But if you want to do like AlphaGo or AlphaZero, and you tell them, you're just going to learn from example, you're going to have a long night. And then, the second one is what must happen. There are some constraints you have that they have to be listened to. They have to be enforced. The best way to do that is with code. So, I want you to go to the third line, the fourth line. You have self.extract and self.recheck.

You can see we're doing a predict on the extract taxes, and we're doing a chain of thought on the extract taxes. The first one is a vanilla program. The second one makes it do some reasoning. Now I'm taking them inside in the forward, and you can see in the if not pred tax, this is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning. I mean, I got to get my taxes right. And then, another requirement I have is if the value is below zero, throw. I want to show that to a human. I don't want to let you go.

This will not change. Like, even if I have AGI, I would hope it doesn't make mistake. But, whatever is in the predictor, if they make these mistakes, I still want these things to be true. So, the last one is what good look like. And when I was young, was on the farm with my dad. And I asked him, "How do you know that this tree is a maple?" And he couldn't tell me. He couldn't give me the instruction on how to know this tree is a maple. And he certainly couldn't give me code on how to know this tree is a maple.

And so, through time, with example, I learned how to know that a tree is a maple. But, this is not limited to things like classifying plants. It's also for all of the long tail in your specifications that are things that are more latent. These are sometimes the reason why you would do internship, and you would have a mentor and a mentee. You're looking at a lot of of examples, and there are long tails of successful behaviors that you have to see and learn. Now that you have all of these, you have expressed fully, you have all these three languages you can put together. You have the specs, the code, and the evals.

And now, your goal is fully specified. And so, you can start optimizing. You can use things like Jepa on your metrics and on your program, and you can start optimizing. At the beginning of the S-I, the ChatGPT didn't exist. The models were not good enough to optimize. And so, we were using code to find few-shots examples to make the base models uh act in the proper way. Then, models got better. And so, we could automatically optimize instruction. And in the future, we are starting to be able to be liberated more and more from the implementation details, and delegate that away.

And at the end, our hope in the DSpar is that you can stick to all of that and then just the news and the implementation details will be automated for you. Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have. Thank you. >> [applause] >> Thanks, Max. So, we've given you a pretty big abstract overview of specs, code, and evals. But, these aren't things that are just res- restricted to the academic sphere. These are used in production by some of the biggest enterprises for massive gains.

And we see two main benefits when you use DSpar in the enterprise. First, is that your implementation becomes cheaper. When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply. And you can use this to scale to data sizes that weren't possible with a more expensive implementation. Shopify, 550 times cheaper. They're able to do that because they went from an expensive model to a cheap model, but they could keep the same evals, keep iterating on their business logic inside, and try new things. There's three awesome case studies here, and you should check them out after the talk.

They give you a lot of details on how you can do this in your own enterprise. Now, part of the reason why you want to build in the DSpar ecosystem is that we're constantly adding new techniques for you to try. And it's important to know, none of these techniques we add will definitely solve your problem, because that's your job. What we can do is we can solve subproblems for you that make your implementation easier. For instance, Alex Zang, a PhD student at MIT, came out with this paper called recursive language models. Recursive language models are a way to solve some kinds of long context programs. And guess what?

We can bring this into DSPy for you to try. See if it helps your long context tasks. Maybe it will, maybe it won't. But the thing is it's one line and your signature stays the same. That's what's important here. Everything gets to stay constant and you get to see if this solves your problem or not. And we've had a number of examples of this just in the last year from people building in and around the DSPy community. We've had RLMs. We've had Jeppa, which is an incredible prompt optimizer out of Berkeley. Better together, multi-module GRPO.

All these are incredible research innovations that you get to try in your implementation just by being in the DSPy ecosystem. And we have more coming in DSPy 4. I'm excited to talk to you about two of those today. DSPy Flex and qualitative learning. DSPy.flex is a new kind of module. In DSPy, when we let you optimize things, it started with few-shot examples. Then it became prompts. And now that's becoming code. For any function that you want to implement, you can actually learn a harness over time to solve that function. And this is completely custom and you don't care about the implementation as long as it solves your business problem.

What you've created ways to measure it because you've defined the three core parts of specs, code, and evals. The second thing I'm excited to talk about is qualitative learning. One of the hard hard problems in AI engineering is building evals. And there's a few reasons why this is hard. One is that defining what good looks like is really challenging for any real-world problem. The second is that when you define good, often times you have to lose detail. If an email is good or bad, contains a lot less information than if you know what could change in that email in order to improve.

And the third is that whenever you create a hill in a data set, you're really trying to create a proxy for reality. What if instead we could use reality to inform our emails automatically? What qualitative learning asks is how do we decrease this question? How do we decrease this distance? And it's a research question right now. But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment and convert that into emails and a hill that the model can climb.

And so as you get more feedback from production, its traces, its user actions, its product analytics, it's asking you it's the model asking you questions about how data should be represented. As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem. And DSPy focuses on these kinds of last mile problems. We have a really strong research ecosystem and we collaborate really closely with them. And that's part of the beauty is that we can see the problems that happen in applied AI engineering. Solve define them, build a benchmark, and then solve them with techniques.

And then we get to democratize the results of that to everyone because it's open source, open research. Now, one common question is what happens when we have AGI? Well, even when we have an incredibly smart model, the model won't know how to solve your problems. It won't know how to do your tasks or have your context. And so this genre of last mile learning is trying to ask "How do we efficiently do this learning?" Intelligence is very different from being all-knowing. If you were to ask Albert Einstein to help you with your emails, he'd probably ask, "What's an email?" But, if you AGI will know how to do your emails.

Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with. It won't understand your relationships without learning this context over time. Since 2022, DSPy has been focused on these three core ideas of specs, code, and evals, all defined as a programmatic interface. We've certainly evolved over time, and new techniques are incredible. We've gone from evolving few shots to prompts to now harnesses, and now evolving your evals over time, too. But, what you need to ask for any of these new techniques is how do they help you solve harder problems or solve your own problems better? And you should ask this question in a data-driven manner.

You should look at this new technique, say, "How can I apply this to the business problem that I have?" You should define your problem, and you should hold your prompts, models, and code accountable to the problem that you need them to solve. And what's awesome about when you build in this way, where you have flexible implementations, what you unlock is you unlock the ecosystem of all the techniques that anyone in this room is constantly inventing. You unlock access to the collective intelligence of everyone here, all sharing techniques together. So, if you want to build reliable AI software, I encourage you to come check out DSPy. We're completely open source, open research.

And we're here to help you solve your problems by building reliable software. We have a Discord that you should come join. And when you come up with the next technique, you should come contribute it to DSPy, and we can help you distribute it and make this awesome technique available for everyone. Thank you. >> [music] >> Joining us on stage is the co-founder of Instagram >> [music] >> and a member of technical staff at Anthropic, Mike Krieger. >> How's everyone doing? Let me good morning. Nice. Um Mike, thank you for releasing Fable just in time for us. >> Exactly for the conference. We timed it. >> [laughter] >> Um we're we're so glad to have you.

Uh you're uh one of the preeminent builders and you're a leading labs at Anthropic. Um how has your model usage changed as as you've, you know, seen models internally grow? >> Yeah, I mean for me it's been like both the model shift and then my role shift.

So I for like the first 2 years I was at Anthropic I was chief product officer and then I kept seeing people build with the models and the FOMO just kept increasing because I was you know use the models as much as possible, but for example on product strategy I would write a strategy doc and then have Claude critique it and maybe you can use a workflow, but it's not quite the same as like building in that pure way. And I was like spending all my weekends trying to build with it and I realized okay, I actually just need to shift. It's like way too interesting a time.

And it's actually an interesting trend I've seen now like several people that were CTOs at other places are like now joining as ICs at Anthropic and other places, but I made a role shift and it was actually right around the time where we started getting sort of internal snapshots of what became Mythos and Fable.

And what was really interesting watching that sort of shift was um that kind of change between I have an idea, I'm going to like sort of break it down in my head much more how I'd do engineering normally and then kind of iterate through these different steps to moving to much more of the paradigm of I'm going to describe the goal, like go off and work on it and then like we can talk about what trade-offs you you know, surface some questions along the way, but then figure out what where you landed and where we can go from there. I find it's hard.

I don't know if people have this experience where both and I Fable's only been re-enabled for a couple of days. Fable's definitely way way smarter than me. So sometimes it'll finish work and be like, here's the trade-offs I made. I'm like, can you explain it to me like I'm a little dumber than you are because I need you to like sort of break this down for me. But that's been one sort of big changes sort of moving from that task delegation to like express the end state and then have it go and and cook on it. >> Yeah, uh we're all learning how to delegate better. Tariq did us a huge favor yesterday.

We they don't read it in the newspaper. It's you know, that we have we have write-ups of talks now in in like the next day's newspaper. He said be unreasonable. In what ways do you you know, have you been more ambitious? You're prompting. >> I love that. I I mean I love that framing. We actually just hit this today. One of the labs issues I have is internal products and somebody was like, "Hey, it doesn't work the way I want it to and can you make some changes?" And I realized that I'm just going to go ask Claude to do this. Like, why don't you ask Claude? And this was a non-technical person.

And so I actually think as an industry or even as a product team, we have to teach people to be more unreasonable in their usage and it's sort of hard to imagine. I think that that if I can digress for a second on product design. I think right now the like kind of first generation of AI products, we put them too much in a box and constrain their their sort of access to tools or kind of degrees of freedom, which means it was much harder to be unreasonable, right? When you say, "Do this thing for me." and then it would be like, "Oh, I can't.

But I can barely like I can write code, but I can't really run it or I can kind of introspect my environment, but not really." And I think as you see our own like product progression even with things like co-work where like, you know, does every single knowledge worker need a virtual machine that can write bash? Like on the face of it, no. But then when you realize, "Oh, actually that way it can remediate an issue where oh, I tried to parse a PDF using our built-in PDF parser." I hit this yesterday and it was like, "Ah, I can't parse it this way." Well, okay.

Well, I can probably write a script that can do this as well. So I think that's it. My most unreasonable thing though was one of our labs projects I wrote in Python like near and dear to my heart. All of Instagram is in Python. But I think they're finally converting it to PHP now that they have like models that can do it. I know. >> [laughter] >> Tokens. And for deployment, I realized that Claude code is like figured out a better deployment story with Bun.

And I was like, "Okay, I need to port this whole thing from Python to TypeScript." Like, as a, you know, if I put on my like 2010s engineering hat, or even my early 2020s, like, that's a dumb idea. Like, who would ever port like, at that point, you know, a couple hundred thousands of lines of code.

Um, but I was like, "I think this is doable now." And I basically created this dynamic workflow set up and over the weekend had it port the whole thing, like, verify it, double-check it, then read both code like it was basically churn and churn and churn and then came back Monday to a completed workflow that was a ported version of that thing. So, that probably ranks on like the more unreasonable things, like, "Yeah, just port this entire Python code base to TypeScript, get it working, get it deployable in, you know, a weekend." >> Yeah, I mean, a lot of people are talking about the the Bun Zig to Rust uh, version.

I I think a lot of people are also like, "Well, it's a compiler, it's a it's a runtime, it's got lots of tests, easy to do." Can you port Instagram, which you would know very well, to PHP like that? Like a like a product. >> Yeah, I mean, I think the product side of it it's even I don't know if easier or harder. One of the things we did at Instagram, this is when Python 3 came out and we were able to add type hints for the first time, and it was people had a lot of internal conversations, like, are we going to run out of steam on Python?

And my perspective was always like, I think we can take this way further than we think we can. Uh, but I think types are going to help us not sort of be in our own way. And we built this thing called Monkey Type, where we basically like captured runtime type like basically the types that were actually getting used in production, and then map those back to to the types in the code base.

And I think because of that sort of pattern, I think there's really interesting ways in which if you're doing sort of conversion or sort of cross-compiling using LLMs, you can also lean on production data a lot more, or run sort of like segmented tests. I think that like there's a lot of uh, things you can do there. But yeah, I think it's I mean, the sky's the limit there as well. I think the hardest part is always finding the boundary around where you can start doing it incrementally without trying to boil the whole ocean and like swap it overnight.

>> Yeah, I mean your users are your test ultimately and you know, I we I also read another article in the newspaper about how you can just use rollouts and sometimes you don't really know what you're going to need it for but when that infrastructure exists for your experiments and to roll things out it's enables so much.

>> Yeah, I mean I always found this was advice we got is like we launched Instagram and the happened to be the first week everything melted cuz we didn't really know what we were doing on the back end side of things and coincidentally that week there was like a lunch that one of our investors just scheduled like not even for us it was just a infrastructure lunch and we ended up spending we totally like monopolized that conversation cuz everybody had their own opinion about how we could fix our scaling and like the two pieces of advice that got there is like 2010 that I like will forever retain is like like basically like pre-measure everything that you think you might even remotely need because the worst thing is an outage where you're like, well, is this like number normal or is it high and like, oh, I don't know because I don't have data until I just added this metric.

And the other one is being like really thoughtful about knobs and feature flags. So even you know, early Instagram we had like a very simple but really effective like way in which you could do like ramp ups and rollouts and dynamic config too where you know, a lot of our runtime configurations had to be changed you know, in a matter of seconds so that we could handle load and being able to like do that in a first class way was was really important. I'm seeing that definitely in in AI as well where you know, we're making all sorts of different trade-offs and having that kind of runtime configuration is super key. >> Yeah.

Uh my my favorite scaling story of Instagram by the way I I is like your launch day when you you DDoS yourself with email. >> Yes. >> Which which I people should look up that story if if you haven't seen it. Um I wanted to go into tags very very major shift uh it's it's how 60-something percent of your code is written today. >> Yeah. >> Um how did you square that with everything you just said where it's like very dynamic like you don't actually ship one app you ship one app with 3,000 flags and like, well, what are you working on today? I don't know like it's for this segment of the population.

>> Yeah. Yeah, I mean I think it does a bunch of things. So like with I was really excited I was talking to Swyx earlier like I'm really excited that we have tag out there because it is how we've been working for a while and I would get up on stages and people be like, how do you work at Anthropic? And I'd be like, oh yeah, we use these things like that are not quite Claude code, but you know, but it's hard to describe it.

But I mean if you like got to poke into Anthropic like you would see of course Claude code usage for things that are like more interactive or if you're kind of iterating on a particular sort of sort of specific thing where you want a lot of like the high sort of bandwidth back and forth, but most usage is actually much more delegating via tagging and via tag and you can say like here's the and the reason it's really interesting is how multiplayer it is and it reminds me sort of of like actually like mid-journey like the fact that everybody was on Discord seeing how other people were using it.

I think it actually to your earlier question really helps with that unreasonableness or ambition where the first time you see somebody tag Claude and be like, hey, you know, don't just fix this bug, but like now you are responsible for this part of the code base and I want you to monitor this feedback channel and proactively take on tasks and then fix them and then also take like, you know, if this API changes do that. Like I saw somebody do that I was like, oh wait, I've been totally under utilizing this thing. I've just been using it as like a glorified Claude code in Slack.

Like that's definitely totally like sort of new version of it, right? And then more advanced version is really trying to start thinking of it as a teammate that is actually sort of holds context, has memory and can be proactive and that's just really changed how we operate internally. It's much more like this multiplayer async proactive way than it is a, you know, most people often their own CLIs. >> Are you bottlenecked by code review and get? I obviously there is Claude review, but someone usually still looks at it. Is there a world in which you just merge it in? >> Yeah, we're it's a really good question.

We are definitely still bottlenecked on reviews, especially for things that are like touching some architecture pieces. And it's actually more subtle than just being bottlenecked on review cuz that's, you know, okay, we can carve out time differently. It's like bottlenecked on human ability to even like fully conceptualize what we're doing. So, one of the reasons we built Claude code artifacts that we shipped a couple weeks ago was partially for that, which is you would send somebody a PR and then they'd be like, I don't know, man. This is like 200 lines of code. Like, it looks like code to me.

And what we started doing instead is sharing much more like, here's a Claude code artifact. Like, here's the explanation. Here's the intention of the of the change. Here's the trade-offs that were made. And like, I think that's going to much more be the trend by which we communicate, which is the code is ultimately, you know, verifiable using some things, but actually like discussing intent and trade-offs and then measuring in production is I think that at least the direction of travel we've we've gone. And I don't review when I get a pull request, I wish I could say I reviewed every line of code. I definitely do not.

I like actually talk to Claude about the the code and say, All right, like, these are the questions that I would have. Can you go investigate it? So, it is kind of Claude-powered code review, but still human-driven. And then for the really important ones. And for the ones that are like cosmetic visual changes, it's much more like like like, we'll fix forward if we need to fix forward, you know. >> [laughter] >> Yeah, totally. I think a lot of people are here are trying to figure it out, too. I wanted to talk also a little bit about Anthropic Labs in general.

You know, I Patel I I who you've probably met before loves to ask ask the question like, draw the org chart. Like, how like people, you know, you ship your org chart. Like, I think it's important. Like, everyone knows Claude code. Now you've got tags. How are you structuring the labs? >> Yeah, it's a good question. Like, what we were kind of wrestled with was you want sort of people to be supported. Like, you know, I think the the death of the engineering manager discipline has been greatly exaggerated. Like, I think there's still a lot of coaching and interpersonal pieces and personal development that I think is still really, really important.

But especially in a labs-type group where like our whole cadence is two-week reviews where every project goes up for we call it persevere or pivot. So, basically every project is up for review and either it's time to, you know, keep going, persevering, or you know, it's time to pivot it or even shut down. And you know, we shut down projects basically every single one of those cycles and it's like the more you do it, the less it's just like, "Oh, no, my project is shut down.

I failed." It's like, "No, that is definitely the intention of the Labs team is to prototype quickly, try to ship internally, maybe get it to early access, and if it doesn't work, wind it down." But because of that kind of like rapid iteration, it means that if you align the org chart too much to the individual projects, you're going to end up like re-orging every 2 weeks, which would be a total nightmare.

And so, we've actually ended up with this interesting setup where like the the pod or the team that is working on a given, we call them bets within Labs, definitely just draws upon like, all right, somebody from product, somebody from the engine team, um, you know, I'll jump in when it's a product I'm particularly interested in, I'll come in and work together with the team on it. Um, and that's the unit for that time.

And there is the concept of a bet lead or a directly responsible individual, but the interesting thing is that they don't manage usually any of the other people, which kind of breaks the that kind of previous way in which a lot of these things were done. But I think it leads it leads us to be really flexible when you say, "Okay, actually this project is not going to work out. Let's disband and keep going." And it's not a big deal. And the engine manager is much more playing the like make sure every individual is assigned to the thing that they're most excited about and that they're working in the best way possible.

Now, what we do sort of solidify is when there's product that has like legs. Like Cloud Design, for example, started in this sort of ad hoc sort of group way. And then now that like we've shipped it, it's gotten traction, we've done like a big second release um, in June, like it's becoming like we've hired people for that specific team and it has more of a of a structure. So, it's like loose until it gets solidified down the line. >> What's the future of Cloud Design? I think a lot of people are very interested in. It's one of your biggest launches this year. Um, where does this go?

>> I think for me, I mean, uh, the things that are holding back Claude design for being even better is better interaction with our other surfaces. So, you know, I was designing something or was talking to to Claude code the other day. I'm like, I want a really much more seamless like what I'm talking about the design for it, you know, interactive design back to that. I think in general it's I mean, this goes back again to kind of unconstraining Claude. Like, the fact that our surfaces don't talk to each other as well as they could and I think really holds back a lot of interesting ideas around what we could do.

So, I think that's one like kind of major area that we're looking at. And then the other one is people like the lines between a Claude design and an app get blurry and blurrier over time. Like, I've seen people because there's no like persistence but build like fully functional like even games which is definitely not what we design Claude design for but you can do it. It's just HTML and JavaScript.

So, blurring those lines even further and thinking through like what is the path from a like fully featured design that looks really well to really good to something that is maybe more like an artifact where you're actually able to go and, you know, persist data and share it with others and build from there. So, I think that those lines get really interesting over time, too. >> Yeah. A big part of design is having taste. I actually asked Fable what Fable wants to ask you and this this fit This is what Fable came up with.

You deleted almost all of Bourbon to get to Instagram which is like you had a whole, you know, solo mo whatever thing and you went to Instagram. What would you delete in AI or more specifically what would you delete in Claude? >> Oh, I like the spice. I think I mean, we have It's interesting. We have a One of our Slack channels is like project unship which is like what is in the product right now. It's and it I mean, this was hard at Instagram. The Instagram we things that had like 4 to 5% usage.

You're like, oh, that's really not very many but then you have like 20 features that each have 4 to 5% usage. It's like the classic Microsoft Word problem of like everybody uses some disjoint subset of the of the functionality. So that that's always the challenge. Now I think we're younger products. Hopefully we have less of those things. Like we un-shipped styles I think recently where it was like used by a small percentage of people and was not really AGI pulled in a lot of ways. It was like very sort of prescriptive in the way that it worked and skills were a much better application something like that.

So I think you have to be willing to take the primitives of like one generation of AI and like un-ship them or at least like supplement them or supplant them with the next one as well. I think the biggest thing as I look at it I've been spending some time like outside of labs on some of this is like man like we're asking people to make like code versus co-work versus like chat distinctions and like one they don't interoperate well and they can't delegate to each other and two I think the average person off the street could not explain to you why those surfaces are all different.

So I think deleting some of the product complexity within our our code or our product I think is a a thing that would would serve well. Also because then Claude can do what it needs to do and and do well. Like there's nothing more frustrating than having a co-work session where you're like great I've mapped out exactly what I want you to build and then be like can you please like create a paragraph that I can paste into Claude code? Like that is some 2020 you know kind of workflow there that really shouldn't exist anymore. >> Yeah.

Um I think drawing lines on what you don't want to do and also sort of leaving room for others is interesting. Um a lot of people today's like the startups day for AI you are obviously very sympathetically aligned to startups. Uh but they sound inviting in the room because tomorrow's Anthropic could wake up and publish some markdown files that destroy my industry. Um so >> [laughter] >> uh why should we not all just give up and join Anthropic? Like why bother starting any other company?

>> Um I I mean I actually joined one of the main reasons I joined Anthropic was because I saw how much this was like you know the models weren't that good at code yet but they were getting there. Like how much it would unlock like hold like next generation of startups. Not because it was going to solve their ideation or their taste, but because like it would make experimentation way simpler and and would get you to move faster. And I still like really believe that.

And I mean, it's the reality of, you know, uh And we saw this with like Instagram, like we'd get questions and investors like, "Well, what happens when Google launches a photos product?" It's like Google's going to launch a very Googley photos product, and it's going to have to be bound by the integrations that they already have, and it's going to be like it's going to play to their strengths.

And I think that is going to be true or not like giving advice on how to compete with Anthropic, I guess in a way, but like it's actually not cuz we're also a platform, which is like there's so much I think room to be like laser obsessed with your particular vertical or your industry or a group of people that you know really well in a way that like none of the labs are ever going to get to that level of uh of understanding and like therefore get that kind of adoption and user love and and build that up.

Now, it's definitely harder in the age where like the models can just do a lot, and so there's, you know, some of these things can be like skillified and like maybe don't need their own dedicated product. But I think it's like the hard stuff is still hard. It's like understanding the needs of people, like figuring out how you're going to reach them, uh listening to them and iterating on them really quickly.

Like it is still the case that like a group of four or five people obsessed with a problem is going to move faster than those same people at any other kind of organization that are like, you know, subject just to the complexity I just mentioned, the like the fact that we have, you know, a lot of different products that kind of interoperate. Like that's an interesting constraint that we have to work through. It's an advantage in other ways, right? So, yeah, I'm still like very long and bullish on startups, and um it's just uh it papers over the fact that like writing code was never the like the limiting part.

I mean, maybe it was on the timeline perspective, but it was never like the thing that was going to like make or break a startup. It's really that space and user understanding. >> Yeah, uh domain knowledge. >> Yeah. >> Uh today is also our day for vertical AI. Uh one of our uh returning speakers in top speakers, Chris Lovejoy, uh was always talking about vertical AI. He was in from interior in the health care space. And then recently I was I invited him back and it turned out he you guys just hired him for your health care efforts. Um we also our next big one is also finance.

You know, we have a yeah in finance track you guys just had a huge finance event in New York City. And we're our next AI is is sort of finance focused. What are you seeing there and you know, any potential for Claude? Obviously a lot of excel excel spreadsheets. >> Yeah, no I think that there's there's a lot in there too.

And that's like an area where you could see the model get clearly better at it like sort of generation to generation and there's you know, there's some good sort of vertical specific finance startups that have like done their own evals which has also been interesting to to track and it's not like we're like sort of playing to the eval but it is a useful sort of barometer on like is this actually getting better at these finance use cases.

I think the interesting blend that's going to happen is this mix of again the model having the flexibility to like dive in and create just in time analysis or dashboards or workflows with like some sense of like what is the not immutable but at least like verified sort of set of data. And so like having all of that be totally free form I think is a recipe for confusion and is like not what most companies in the financial services space want.

So finding that right sort of cut line where you have verifiability and audit logging and and sort of data provenance here but not in a way that constrains the kinds of applications that you can build on top. I think is a lot of the art that we're seeing in that space as well. And I think you know, if you solve it well you can you can get the best of both worlds. The hard part is a lot of the systems that were built to do the verifiability auditability are like are kind of almost by design not super flexible in terms of agentic workloads on top.

So I think there's opportunity at both sides of the stack there. Yeah, I think I I also agree we'll be exploring that in in New York. The the thing I want to end on it on mental health, which we don't talk about enough in technical conference conferences. Um, you've seen a lot of hyper growth. People are just always refreshing their timelines and it's exhausting. Um, how do you advise people who are working 996 to avoid burnout? >> Yeah, I mean I think this is a hard one.

I mean and it is I'm sure you are experiencing this cuz you're all working in this industry like it is you know, multiples more intense and things move more more quickly. I had an Instagram like our the two things that we were thinking about was like, what is Apple going to announce at WWDC and is it going to like totally mess us up or boost us, right? So that's like once a year. Um, or you know, we had a competitor launches every three or four months, right? And uh, it is definitely not that. It's a topic we we do at when we do do our weekly all hands.

It's usually on Wednesdays and we have a slide that's like, "The week in AI parentheses and it's only Wednesday." And like and all inevitably like some competitor has shipped a new model and like there's like new product and maybe there's some interesting thing happening um, on the regulation side. Like if things are moving really really quickly.

I think the way I try to stay at least relatively sane, um, one is like actually carving time off and I think the topic co-founders do a good job of like saying like, "Look, like burnout if you burn out like you're kind of done." I've seen it happen unfortunately to people I'm really close to and then it takes a long time to recover from that. Um, so actually encouraging people like there's no job that is so important that you can't be offline for a couple of days. Um, so I think that's like a big key like piece in there. So like strongly believe.

Um, if it is you're probably doing something wrong and you talk to somebody who could be a mentor to figure out how you can uh, unblock that. Um, and then I think the other one as well is like uh, I love sports and like uh, this is the notion of like you're never as good as like your best game and you're never as bad as your worst game. I think that's also really true. Like I know like in AI there's like the, you know, it's so over we're so back thing.

Like that like if you internalize that that cycle is always going to be at play in some way, you realize like it's never that bad. Like Ben Horowitz's book is The Hard Thing About Hard Things has this chapter on like we're effed, it's over. And like that feeling of a startup that probably many of you have had at startups where you're like, "Oh, I can't believe this thing happened. Like we're never going to like recover from this." I definitely definitely had it at Instagram a couple of times. And then you get through it. And like it's that like that's like defines the company when you can actually go through that.

I try to remind myself and the team here even with an Entropic which is like like this is a is a fast-moving but is also a long game. And it's like we're never it's never just about today's model launch and reaction or this product launch or something else. Like you're playing and you're building. You just have to trust that you're building like the team and culture that is going to get through those things and have that sense of perspective. Even if perspective is saying like, "Look, 3 months ago we were in a similar position.

Maybe it's not a year, it's just a matter of months." But it's still like zooming out and not thinking things not letting your internal sort of like sense of self and success be so driven by the day-to-day. >> Yeah. It has anyone, any coach or mentor said something to you that you repeat to yourself that gets you through the tough tough times? >> I think the biggest one was this like sense of like if you're feeling something, it's really often the case that other people on the team are feeling it too. So this is like advice I got from my my coach around just being like like just verbalizing emotions.

Like even saying like, "Hey, I'm feeling really stressed out about this." Or yeah, I'm really sad that we are shutting down this Labs I literally had this meeting a couple months ago where I was working really hard on something and I kicked off the meeting like I'll kick it off. Like I'm really sad. Like I'm frustrated. Like I wish this thing had worked out. And I think that holds the space for other people to be like, "Yeah, I'm pissed off, too." Or like I'm sad, too.

And like I think giving that advice around like not Yeah, I think if you can get yourself to be open and vulnerable, it often like lets other people verbalize that. And then you can from there you can be like, "Great, what are we going to do about it?" Like, you know, it's much easier to start from that place. >> Yeah, we've actually kicked off AIE with a session from Carol Robbins who runs empathy builders at Stanford and I can't think of a better way to end than encouraging people to talk about their feelings, manage their mental health, and keep shipping. >> Yeah. >> Thanks so much, Mike. >> Thanks for having me.

>> Ladies and gentlemen, please welcome back to the stage our MC, >> [music] >> developer relations engineer at Replit, Ra'ouf Shabr Y. >> All right, let's hear it one more time for our keynote speakers. All right. Okay. So, we learned so much this morning, right? Like we spoke about a new protocol and we spoke about like that AI programs should be more like functions and we learned that the real problem with building with agents is not the model, but it's about everything around it. Um but before we go to breakouts, guys, I would like to announce our next speaker. Um So, a quick word from our sponsor, Neo4j. Anybody uses graphs here?

All right, fair mark, fair mark. Okay, cool. So, our next speaker is going to talk to you about ontology-based semantic layers. When you hear ontology and semantic in the same sentence, you know that you're up to up for something hot. All right, so without further ado, please join me in welcoming to the stage CEO of Neo4j, Emil Eifrem. >> Please welcome to the stage the founder [music] and CEO at Neo4j, Emil Eifrem. >> [music] >> All right. At Neo4j, we work with some of the largest companies in the world to help make their data ready for AI agents.

And today I want to talk to you about a problem that we saw emerging over the last, call it, 6 to 9 months and propose a solution blueprint for that. So, let's say that we work at a big organization, a big bank, and we want to write an agent. Let's say that agent is helping automate the opening of a bank account. Right? You can imagine that's very ripe for automation. You want to be able to orchestrate that process. And I'm going to use the powers bestowed upon me by a short keynote slot to grossly simplify what that agent looks like. I'm going to say there's two pieces.

The first one is, let's call it the business logic, some version of interpreting intent and plan act, then we loop around that. It's what your agent does. And we know that when an agent act, it doesn't always operate on data, but we equally know that in order for agents to be successful, a huge part of that is giving it access to the right data at the right time. So, the second big bucket is, let's call it the data sources. Need to identify, figure out, okay, in order to solve my problem, I need access to these few things and wire them up and make them available to the agent.

In the example of our account opening agent, maybe we can imagine that we need to be able to validate identity. And so, we might look at two data sources for that, the Department of Motor Vehicles, the DMV registry, and maybe some kind of passport verification service. So, we wire that up into our agent, and it works. It's great. It's fantastic. And at the same time, you and other teams in your organization are building other agents, and conceptually, they look very similar. So, that's great. It's fantastic. It works. But, it has a few problems.

So, first of all, every single time a team has to build an agent, they have to figure out from scratch where the data that they require for that agent to operate, where it sits, which if you work at a startup and you have one application, it sits on top of one Postgres database, that's not hard. The data is in that Postgres database. But, in an enterprise ecosystem, you don't have one database, you have a hundred databases, and you have Snowflake and Databricks, and you have S3 buckets, and so on and so forth. You have to do that work manually from scratch every single time.

And then, when you've found the data sources, you know, in an enterprise, there's lots of duplication of data. So, then you need to figure out, like, is this the right data? Is it the right version? Can I trust it? Am I allowed to access it? So on and so forth. It also violates one of the core principles of software engineering, the DRY principle, don't repeat yourself. So, when something change, that cascades across all of your agents. You have to kind of manually rewire all of them all the time, which works, but it's just a lot of work. And then finally, there's no learning around the data sources and how your agents operate on them.

So, when your agent wake up wakes up tomorrow, it's not smarter than it was today, and there certainly isn't any cross-agent learning because all of that wiring between business intent and the data sources is encoded in a combination of code and prompts. So, I know what you're all thinking, "Markdown files, skills to the rescue." And yes and no. Um you can come talk to me afterwards for kind of the full version of this, but we've seen a ton of team that tried to solve this problem using just Markdown files. And the summary is it is part of the solution, but it is not the solution.

Uh but don't take it from me, take it from Swyx. A week ago on the Latent Space podcast, he said, "Hey guys, you got to learn your databases. You cannot vibe code with just Markdown files." So, we've been solving this problem at scale for some really massive organizations recently, including a Fortune 20 global bank, a massive tech platform company based here in the Bay Area, and a leading fintech company. And the pattern that is emerging is that in order to do agents at scale, we need thin agents on a smarter, shared substrate. Thin agents on a smarter, shared substrate. And what does that look like in practice? There are three pillars to that.

The first pillar is a business-facing ontology. And the word ontology, like I grew up in this world, people talked about ontologies forever. More recently, it's become very hype, probably thanks to Palantir, but also the rise of AI. And there's a lot of people who want to make ontologies really complex. But the core concepts are actually super simple. What are the key concepts in your organization? In our banking example, customers, accounts, um debit cards, checks, transactions, and how do they all relate? But very importantly, they are expressed in a way that makes sense to all the human beings working in your universe, right?

All the people working in your company, it's expressed in that name, in that way. In other words, you don't say if underscore name. No, you have a customer and they have a first name. So, that's the first, a business-facing ontology. The second pillar is a technical ontology. This is all the metadata of all the data sources and data assets in your enterprise ecosystem. I have 14 Oracle databases, I have 15 Neo4j databases, I have Snowflake and Databricks and I have S3 buckets and all of that kind of stuff. Where do they sit? What are the schemas? All of that kind of good stuff.

You can You construct that tech technical ontology in three key ways that we can talk about later, though not in this in this talk. And then you have a mapping between the two. So, that customer that has a first name, that first name has a system of record and over there there's an Oracle database with a column called F_Name. The mapping between the two. And then the third pillar is the runtime signals out of your agents. When they walk this graph and they execute, they leave the traces around. What have I tried? Was I successful? What was the outcome? The execution traces. Those three pillars.

Okay, so let's look at that in the context of our bank account opening agent. This is a simplified view, but you can see this graph here. It has a combination of business concepts like checks and accounts and credit history and stuff like that. This is a process-following agent or a process-guided agent. We want this type of agent to actually follow a process. We've also encoded that in the ontology, a business process.

And then if you look at the node that is surrounded by green, the check compliance one, we flip to the technical ontology and we've put in the graph here, we've discovered and encoded that in order to do a compliance check, you might imagine that you need to resolve a government-issued ID. And then we say that in this particular organization there are two data sources that can help us with that. It's the motor vehicle records and the passport verification one. Okay, so that's really great. So then when our agents come in here and they realize I'm going to check compliance, I need a government-issued ID. Here are the two ways that I can resolve that.

When they execute and they try that, they leave the third pillar, the execution traces for that. And they're more sophisticated than what's on this simplified slide, but involves things like, "Okay, where was I? What did I do? What is my context? And was I successful?" And ultimately it leads out to some kind of a score. And you use that as input. It's like, "Okay, I've been very successful using the DMV lookup, for example. Then I'm more likely to choose one if I'm in the right context in my next invocation." Three pillars of the ontology-based semantic layer, a business ontology, a technical ontology, the execution traces taken together, they solve all four of the problems.

We never have a very easy way to discover the data sources. We know if they're trustworthy or not. We know that top-down by some kind of human-curated knowledge, right? An administrator of some sort saying it. We also know it bottom-up through the execution traces. This is what actually worked in reality, in practice. We have a single govern place that maps business intent and the concepts to those data sources. We don't repeat ourselves. If something changes, that cascades across all my agents, right? And we have self-learning. So my agent that wakes up tomorrow is slightly smarter than it was today. And not just self-learning on an individual agent, but across agents as well.

So we're moving from this world, the world of thick agents with manually wired data sources, into this world where we have thin agents on a smarter shared ontology-based semantic layer. And this allows us to do a ton more agents without having to re-engineer them every time. Thin agents on top of a smarter shared substrate. If you think this is interesting, there's a documentation and web page that outlines more information about this. If you see the QR code here, you can also come and talk to us at the booth. We have a big booth here at the Expo P3. We love talking about this this kind of stuff, but not just that.

This is one pattern, a very exciting pattern that we see a lot of traction around right now for using graphs in AI. But there's hundreds of more interesting patterns that combines graphs and AI. 10 of them is actually in the graph track that is kicking off right now in room 2005. And you have some really amazing talks from organizations like the Gates Foundation, monday.com, JP Morgan Chase, Berkeley, New York Times, and so on and so forth. So go check out that thing. And then finally, this was primarily centered around organizations where you deal with many data sources and many agents. But if you're a startup building on Neo4j, love you.

There is a startup program for Neo4j that is phenomenal. You get access to free credit, but more importantly, we've built up a dedicated solution engineering team that spend every day working with startups for free, helping them model their data in Neo4j, tune it for performance, and so on and so forth. So please sign up for our startup program. Thank you very much. Enjoy the conference. Have a good day, everyone. >> A production data job failed hours ago. The dashboard went stale. You have spent all day checking the logs, the schema, and the upstream data. And now it is past midnight. The same question keeps coming back. What changed?

The failure itself may be small, but the expensive part is everything around it. Inspection, diagnosis, choosing a safe response, re-running the job, and confirming that we did not make the data worse. Hi, I'm Annamarie RL guided system that selects bounded remediation action for ETL failures. The central question is not simply whether an agent can act, but whether it can act usefully, explainably, and within boundaries that an operations team would actually trust. Cloudy ETL failures are rarely arrive as one clean well-labeled exemption. We see late or unavailable sources, schema drift, daytime incompatibilities, network spikes, type changes, and runtime errors that do not match anything in the runbook. The usual response is a human workflow.

Inspect the logs, form a diagnosis, attempt a repair, re-run the job, and validate the output. Each of step is reasonable. The latency comes from hand-offs, incomplete context, and the need to avoid an unsafe fix. In the capstone evaluation, the manual recovery baseline was modeled at roughly 2.5 working days. This represents an incident moving through normal queuing, investigation, and approval. So, the engineering objective is specific. Compress that loop for routine, recognizable failures, while escalating the cases that are uncertain, novel, or high risk. This diagram shows the end-to-end AWS architecture from my capstone. An existing AWS Glue ETL job emits a job failed event.

Amazon EventBridge catches that event and triggers the Lambda function that runs the agent. Lambda gathers evidence from two read-only sources. CloudWatch provides the error logs, while the Glue Data Catalog provides the current schema metadata. The system uses those signals to classify the failure, assess the data quality, and operational risk, and construct the state passed to the RL decision engine. The policy then proposes a bounded response. The safety layer checks that proposal before the executor can use the Glue API to retrigger the job or apply an approved remediation. Amazon S3 stores agent artifacts, audit logs, and quarantined outputs. Finally, the job is rerun and validated. So, this is closed operational loop.

Monitor, diagnose, score, decide, check safety, act, and verify recovery. The Capstone implementation used synthetic data provided by the client. The public repository preserved this pattern through a sanitized generalized deployment template. The intelligence layer deliberately separates three concerns. Deterministic anomaly rules establish observable facts. A field disappeared, a type changed, or the null rate crossed a threshold. The Q-learning policy handles contextual action selection. Given the current incident state, should the system retry, coerce the schema, roll back, quarantine, escalate, or simply log the event? Then, a safety override sits outside the learned policy. For example, if the anomaly is critical and the policy proposes a passive action, such as logging, the override converts that choice into an escalation.

This separation is the design thesis of the project. Rules for facts, learning for bounded choices, and guardrails for authority. Before selecting an action, the system has to establish what actually happened. The schema profiler extracts the structure, types, nesting, and null rate statistics. The drift detector compares the current profiler with a baseline and and identifies additions, removals, and type changes. The data quality analyzer checks completeness, validity, and consistency. The error classifier maps log patterns into failure families, and the risk scorer turns those signals into an operational risk level. These components are deterministic by design for directly observable data conditions. An explicit rule is easier to validate, explain, and audit than an opaque inference.

With richer and representative incident history, some classifiers could become learned components. But, ML-ready is not the same as ML-required. The simplest reliable component should own each decision. The policy receives a compact state, failure category, risk level, retry count, drift severity, and data quality condition. It then selects from six actions: retry, coerce, rollback, quarantine, escalate, or log. I use tabular Q-learning because the state and action spaces are small. The Q-table is cheap to evaluate, and every decision can be inspected directly. For this state, these were action values, and this action won. Technically, each incident is modeled as single >> Good morning. We're super excited to be here at AI Engineer with all of you.

I'm Caitlin, and I lead platform engineering at Anthropic. >> And I'm Angela. I lead platform product at Anthropic. >> And today we want to talk to you about a concept that we've been spending a lot of time thinking about and working on with our team, which is this idea that we think that tokens should have jobs. So if you're building an agentic system and you're trying to accomplish some specific outcome, you're trying to get something done with agents, there's one lever that everybody pulls in order to get a better outcome and that's usually increasing your budget, which means you spend more tokens or you spend more expensive tokens.

>> But we've been wondering, is that all there is? Underlying this assumption of using the budget is this kind of implicit perspective that every single token is basically fungible. And we've been wondering, is that actually true? Are all these tokens actually fungible? And to test that we've been thinking, what if we gave tokens jobs? So if you think about the way that you would normally set up an agent to go accomplish a task, you give it that task, you give it its token budget, and then all the tokens that are being spent are basically indiscriminate in the sense that they're all doing one job, they're just executing.

>> But what if you take some of those tokens and they're not just executing, they're doing some other job? So for example, maybe you take some of your tokens and they're advising the tokens that are executing. Or maybe the tokens that are executing try to get something done well and you take some other tokens and you actually grade how well the executor is doing so that it can iterate and try again. Or maybe you have tokens that are dreaming, they're reflecting back on the job that other executors have done and writing learnings to memory so that they can do it again.

And what we call each of these, if you take some tokens that are executing and some tokens that are doing some other job, let's call this a strategy. >> So let's go take a look at the first strategy, the advising strategy. Here we're splitting up an executor and an advisor. The executor obviously executes, but crucially they can call out to an advisor for advice. And then they can take this advice and figure out if they're doing the next step correctly. This is really helpful in use cases, for example, if you're building a sales agent.

In an ideal world, you'd have that sales agent be able to actually help the sales rep flag when a follow-up is overdue or deal is stalling. In this construct, having an advisor to be able to kind of make sure that all the different pieces are actually working is really helpful. >> So, another example is grading. Let's say you're executing and you kind of know exactly what good really does look like. You can define this in a rubric. And then each time an executor tries to accomplish that outcome, you can have a grader provisioned that grades how well the executor did while looking at that rubric.

And if the executor did a good job, then great, it can be done. But if it didn't do such a great job, you can iterate again until you get that good outcome. So, an example in practice of when you might want to use this is let's say you have a customer service agent. And you're running a store and your customers are writing in and they're saying, "Ah, I should get a refund for this thing." And your customer service agent needs to be able to respond. You probably have some like pretty specific criteria on when you would give somebody a refund. And so, what you can do is define a rubric that uses that criteria.

You can have a grader that goes and looks at the work that the customer service agent is doing and decide, is it getting it right and is it coming to the right outcome? >> And the last strategy we have is dreaming. So, in dreaming, there's an executor who naturally executes, and then there's a dreamer. The dreamer is actually able to inspect the work and the transcripts of the executor, and then it takes any of the findings that it has and it writes them to memory. This memory is repicked up by the executor for the next round, so ideally it would have improved. A great use case for this is if you're building a recruiting agent.

Now, recruiting requires a lot of interaction with feedback on whether or not a candidate does or doesn't make sense and if it's a good fit between both parties. And so by taking all this type of data, if you build a dreaming type of strategy on this agent, it's actually able to kind of sharpen the next round so that it's more and more increasingly useful. >> So let's make this concrete with some experiments. So what we did was we created a bench of a bunch of tasks related to financial analysis.

And what we were doing with each of these tasks is trying to replicate in the real world expert human financial analyst, how well would they do on each of these various tasks? And so what we did was we start with a control that's just executing. Let's try each of these tasks and we'll eval them and we're literally just executing. But then we can experiment with each of our strategies and see how well we perform. So we start with a super basic experiment. Let's just one shot it. Let's take each of our strategies and we'll go and just make an attempt to accomplish these tasks and we'll see how accurate we are.

And so you can see here with executing, um it didn't do so well. 15% accuracy, but because it was just a one shot, the strategy got to choose how many tokens it would actually spend on its own. And so you can actually see that execute decided not to spend that many tokens, only 39,000. And as we go into our larger strategies, our more complex strategies, we did choose to spend more tokens and we did a better job. So this isn't really telling us much because sure, dream did really, really well, but it used a whopping 600,000 tokens to get there. >> That's right.

So in order to actually figure out if varying the jobs produces any alpha, what we need to do is hold the budget constant. And to do this, we're going to take dreaming's budget, that's 600,000 or so, as the maximum budget that is fixed across the board. And we give every single strategy this budget in order to analyze how well it's performing. And as expected, again, if you give a lot of strategies more budget, you are going to see performance increase across the board. So execute went from 0.15 to 0.76. Advise and grade went from the 60s to closer to the 90s.

and that's again expected given the fact that if you give things more test time compute, they should generally perform better. But if that was the only thing that mattered, we should actually expect to see execute, advise, grade, dream actually all be at the exact same level given the exact same token budget. But what we're actually seeing is that there is an alpha or there is a difference and therefore an alpha for us to exploit. If you look at execute at those exact same budget level, it gets to .76, but advise is at .89. So, while a minimal, it does exist and so there is alpha for us to take a look at.

Now, we decided to take a look at this analysis from a completely different lens. And as Caleb mentioned, you know, we're doing this bench for a very complex set of financial tasks in the real world. And we wanted to analyze the usage of agents with actual experts. So, if we look at a financial analyst expert, right, the kind of tasks that they need to do with an agent is that they're giving it something very concrete, like let's say make a P&L, and then they're getting the result back.

Now, if that result is 80% accurate on a bench, that sounds great, but in reality, what that means for that expert is they have to go back and recompute that P&L themselves or or alternatively send it through another run. And that's because in this kind of domain, for this kind of task, if you're not 100% accurate, it's actually not useful. You cannot make up an income number or you can't make up a cost number, right? You have to make sure that it's 100% accurate. So, with this lens of the real world consequence associated with this domain, we needed to recompute our experiments and score them a bit differently.

Crucially, we needed to make sure that our experiments had this kind of construct where if it was scored perfectly, we'd actually give it a pass, and if it scored anything less than 100% on that kind of task, we would actually mark it as a failure. >> So, let's look at at a different cut of our data from our experiments with this lens where we're looking for this perfect run 100% accuracy pass. And let's look at what percentage of the time each of these strategies was able to achieve a pass. Um so we we've got execute um down at 42% and we've got our more complex strategies doing a bit better up to 75% accuracy.

Um and again, this doesn't necessarily tell us a ton because um you know, each of these strategies might um choose use different budgets over time, right? So, what we did here was we fixed the budget and we said within a fixed budget, how well do each of these strategies perform? And so what really matters to us actually is if you're trying to get this perfect answer and you're in the real world, you're running a business, what matters to you is the cost to you to get to that perfect answer. And so one way we can think about this is we had our execute strategy for example.

The execute strategy around 40% of the time will give you that perfect answer. So on average, you can expect to have to run it three times and you should hopefully sometime in those three runs get a perfect answer. And as we talked about earlier, we fixed our budget to that highest token budget strategy, which was 600,000 tokens. So if you spend 600,000 tokens in each individual run, you have to run approximately three times, you can expect on average to have to spend 1.8 million tokens with the execution strategy to get to your perfect answer.

>> And so if we take this analysis and run it across the board against all these strategies, this is actually the true cost it took in this domain for that agent to be useful for that strategy. So as Caitlin mentioned, for execute, which is our baseline, this is going to be 1.8 million true total token cost for you. But advise, grade, and dream are showing us a bit of difference. Crucially, advise and grade are actually quite token efficient when you think about the actual usage of the end output of each of these agents. So what does this mean for you as a business?

Well, it actually really depends on what kind of thing you want to optimize for. And it's going to vary, right? There's going to be businesses who say, "Actually, for me, the most important thing is to be really token efficient." In that case, you should probably pick the advised type of strategy in order to solve for that particular domain which you want to optimize that. There's going to be other areas or other businesses where you're going to say, "I'm not going to care so much about token efficiency because what I really care about is reliability of that answer." And so, I need to maximize the percentage of runs in which I get that perfect answer.

In which case, you would actually pick completely different strategies. You probably lean towards grade or dream. >> So, if you take away one thing, the thing we want everyone to think about is this idea that tokens are not fungible. You can use your tokens to execute. You can brute force your way through your tasks and you can throw more budget at it. But, if you get really smart about having your tokens do these different jobs and try these different strategies, you're very, very likely to be able to get a better outcome for the task at hand within a fixed budget.

And so, let's talk a little bit about how we actually build strategies and how we bring this to life. Um so, we've done a lot of work to create a really excellent harness for individual agents. Um and if you see this uh picture at the bottom here, this is actually the architecture that we've used for Claude managed agents, um which is our agentic solution that we give to you within the Claude platform.

And what we do on top of this is we start to get into the meta harness level, like the multi-agent orchestration and execution level where this strategy can go and be um coordinated between our executor and our advisor or the other agents within our strategy. And some of these um like dreaming and outcomes, we actually give to you out of the box within Claude managed agents. >> So, with those set of primitives, it's actually relatively trivial for us to construct this kind of, you know, architecture. Where we're able to combine these different types of strategies and figure out how to they should work together.

So, for example, it's relatively trivial for us to say, "Okay, now with this, I can take a task and I should be able to execute it, but also allow it to advise." And Fable is back online, so we can actually say Fable is the one that's actually advising uh the executor. And then I can take all these results and say send them to a grader, so that I can make sure that this is verifying in a loop that makes sense. And if it passes, that's awesome. I want to send all of that stuff to Dreaming and make sure that my next run is better than ever.

And of course, you don't have to stop there, right? If the right the right primitives are there and the right coordination is there, then you can actually construct really complex setups that fit for all the different types of dynamic problems that you have. You can invent these kinds of large-scale architectures, again, very trivially. And you could also invent completely new jobs, not just the ones of the pieces that Caitlyn and I have presented in this conversation. >> So, a big goal that we have over time is to get our models better and better and our platform better and better at dynamically constructing these strategies for you as you're doing work.

But in the meantime, as we're working our way there, we would love for you to continue to think about this idea that you should give your tokens jobs, and you should use different novel strategies by combining these primitives in order to get the outcomes that you want for your tasks. Thanks for joining us. >> [applause] >> Turning my second brain into my living research memory. Let me explain. So, within my second brain, I currently have over 5,000 notes in Obsidian and another 5,000 notes in Readwise and some scattered in Notion and Google Drive. And all of this is growing on average 250 files per month. And this is what I want.

On the left, you can see my whole Obsidian vault, this huge mass. And whenever I start working on something, such as an article, a new project, a new codebase, a new feature, or whatever, I want to actually pull high signal notes that are actually useful for my current work. And you would ask yourself, why not use directly Codex Cloud or Notebook LM? And I think it's that I am, but you need a system that sits between those harnesses and your second brain. Okay, so let's go back to the root of my problem, which is that I'm always losing my research. For example, my reading list is a graveyard.

When I'm scrolling social media and I save that cool Expo a new article, a new new YouTube video, a GitHub repository, it doesn't matter. Whenever I actually want to start working on something, I never recall what I have in my second brain or I have to spend a ton of time actually finding meaningful notes that I can use in my work, right? And another problem that I have is that I want the system to actually be anchored into my personal notes, into my personal values, into my personal faith. I want the system to be personal, to reflect my own thoughts, right?

And that's why in today's video, Louis François and I will teach you how to build your own AI research OS. This also comes with code, so you can also try it out yourself. And I'm Paul Austin. I'm the founder and CEO of Decoding AI, where I do a ton of content on courses on how to ship AI products, and I'm also the co-author of the LM Engineers Handbook bestseller. And the system, the AI research OS that I will teach you in this video is the system that I use in my daily work. And now I will pass the torch to Louis François. >> Thanks, Paul. So, I'm Louis François Bouchard.

I'm the co-founder and CTO of Towards AI, where we build educational courses. And I'm also the creator of What's AI, a YouTube channel where I explain AI engineering techniques. I used to explain AI research before. Now, focusing on AI engineering. I'm also the author of the book Building LLMs for Production. And uh before that, I was a PhD student. So, I honestly make research for a living. I used to do a PhD, as I said, in AI and doing tons of research and research work. Now, I build courses, I write videos, I research for videos, I build trainings for companies for a living.

And all of these things that I do start with a very good research and also leveraging tons of knowledge and insights that we get at Towards the AI from building for clients. So, I have tons of notes, as well, just like Paul, and we try to leverage them the best possible. And as you'll see, we build some sort of tool to leverage our second brain, where, as you'll see, there will be some differences between how I use it and how Paul uses it. And that's the core goal of the repository that we built and on this project is that we want you to adapt it for your needs.

The whole goal is how can we make research better, but more specifically, how can we better leverage what we have? So, let's dive into it. And first, we need to figure out which tool to use and when, because this whole research system that we built is not for every query. If you just need a fast answer, like a few quick questions or just something where that that you would just Google, basically, well, obviously, just Google it or ask ChatGPT, Cloud, whichever system you want.

But the problem when doing that is that if you have a lot of following up question or it's a bigger project that you need to build on and have basically a very long context or tons of information to share, relying on ChatGPT isn't ideal, and it also means that you are fully dependent on the architecture that OpenAI or ChatGPT's team built. So, the next step here is to ask yourself for a more complex problem, do you need to act quickly or do you want to build some next feature and and do something very difficult?

If you just have a small repo for a quick change or write one article, just do one thing that you know won't be repeatable that much, definitely use Codex or Cloud Code or some agent that you trust. Sometimes you need to keep on digging to make it better, to improve efficiency, make optimize it more. And so, typically when you have to do that, you want your research sources, your research to stick and to be able to refer to them in the future.

So, if you want a process like this where the sources yes that you find, the notes that you take stick around in time and have an agent be able to leverage that efficiently and being able to come back to these information, to ask follow-up questions, to digest content even more. And right now, for instance, when I make a new video, I want also the agent and the system to understand the previous videos I made to not duplicate content, to not repeat myself and to refer to some other content.

In this case, there are some tools that are very interesting that you might have tried before like Notebook LM that is super powerful to do research, to digest content efficiently and to come back to it. But, the problem with Notebook LM is that it's well, first, the main problem is that you don't own it. You cannot do anything you want with it. You cannot personalize as much as possible. It's not agent native. And it's obviously weak for coding tasks uh since it's just browser based. So, it's far from ideal from something that Paul and I needed and that most AI engineers need in general.

So, if you need your agents to be able to leverage all you do, uh whether it is a big research, a new video, whatever you write, you do, you code, you typically want your other agents, your other projects to be able to leverage what you learn from what you just did. And one thing that we advise, especially for production, obviously for a product, is to build some sort of retrieval rack pipeline with vector databases. But this needs an infrastructure. It's not really human-friendly to be able to digest quickly, to check notes, to make edits. It's hard to inspect by hand. You need to build everything around it.

It's definitely far from ideal for just something I want to use on a daily basis. Obviously, it's super powerful at scale, very interesting, especially in a product. But, as I said, this project is for us. And I don't want something live, super professional as a product. I just want something I will use and that my agents and different projects can leverage as best as possible.

So, the last question to ask ourselves here is that if you want everything there, but more personalization, so a personalized research assistant that builds some sort of Wikipedia that compounds over time and it and is easily inspectable and usable, where you have tons of sources, documents, videos, comparisons, implementations, new research, new topics that you keep on adding and that you keep on wanting to leverage and review easily. This is where you may want to build something yourself. And you know, our case, we build the personalized research OS that we will share in this talk with exactly what we built and how.

But, the downside is that it definitely needs a bit more setup than just opening cloud code. Right now, the main problem with using cloud code and other agentic tool is that you give code X links, PDFs, and different information. For example, my most recent loop engineering video. And then, the next session you use Codex, you have to paste it all again or ask it to use skills. And whatever structure that Codex or ChatGPT, whatever tool that you use, build on the fly to leverage what you did, the scripts it run, the scripts it had, you all lost it or kept it inside a skill that you have to ask it to reuse.

And that usually isn't ideal and just grows and grows over time. And the problem is that all this information that you give to the model is not the bottleneck. The bottleneck is how can you leverage it in the future? Meaning that with an agent, the context window becomes everything, the database, the file system, the memory, the reasoning space. It has to do it all and when you stop the conversation, it loses everything. And the thing is that we don't need necessarily to provide more and more and more context for a better research. You need a proper memory and context management. And ideally, some personality with it, especially in my case when I do videos.

So what we did is that we decided to build a system with plain files, mostly markdown files, that we can leverage easily and that agents can leverage easily. I won't detail it very much here because Paul will talk about it in depth. And as I said, Paul has like 5,000 or something notes. I have just a few hundred, but that's just to say that we need to consider that we didn't start from nothing. We already both had some sort of large database. In my case, I made hundreds of videos and I take many notes.

So I still need to leverage these years of content that I already made and tons of meetings that I have with uh my team, with clients when we build for them, that I want to leverage as well because we learn a lot by building for people. >> Um welcome everyone. Um I'm Nikita Kothari. I'm a senior member of technical staff at Salesforce uh where I'm building AI-driven enterprise solutions. If you might have heard of Agent Force, Agent Force for setup, headless 360, I worked on all and it's like amazing features. Um I also worked at Amazon and LinkedIn previously. Um in last 6 months or year, we completely changed the way we used to work.

We are building new systems, we are evolving, and there are some critical uh aspect of using MCP CLI and skill. So, today we are uh going to deep dive into a critical yet often overlooked aspect of building AI agents. It's plumbing. As a developer, we we talk a lot about model sizes, reasoning capability, and the prompt engineering, but when when it comes to a building production-ready AI agents, the way an agent interacts with the environment, it's tooling layer. It's what actually makes or breaks the application. We will look into three distinct approaches, which is MCP, SCLR, and the structured skill, and map a clear rubric for what to use when.

Uh at the end of this talk, you'll have a clear mental model for choosing between these three tooling layers. Um here's something every team discovers the hard way. Building demo agent takes like afternoon, but when you actually want to ship the same thing into the production, it could take a quarter because we are looking to build the reliable, secure, and fast agents. Nobody wants to leak their customers' data into the production. So, testing, uh reliability is still a major issues with this AI agents. So I want to talk about these three problems. The problem one is the context explosion.

Consider a scenario where you load a 52 schemas into our agents context window and suddenly 60% of your thinking space is burned before even the task even starts. Your agent forgot the diff because tool definition for monitoring databases and the deployments filed the window. It literally doesn't have a room to think. The second yet critical problem is invisible failures. The agent made a like consider example where agent made a seven tool calls to draft a PR. Something went wrong. The PR was created on the different branch.

So first you lost like lot of time to redo the work and the second thing is somehow you lost the confidence on your agent to perform a particular task. And the problem three is the security surface. Consider an example at 2:00 a.m. you got a customer issue. You are investigating getting a customer A's error. But wait, LLM has a broad access to the query and the tenant data and it accidentally started looking into customer B's data. So that is a multi-tenant platform issue and it's not a just a bug, it's a huge compliance nightmare.

So let's talk about how we can use these three two three layers to build the reliable, fast and the secure agent agents. So we have the CLI. CLI is like handling someone a screwdriver. Here is a tool, just run it. Direct execution. The agents run a command, gets output and moves on. Then we have a layer two which is MCP. MCP stands for model context protocol. This is just like giving someone a USB-C hub. Here is a universal adapter that works with any services. Just use this and work on it. And third, we have the skill. The skill is giving someone a runbook. Here is a step-by-step execution of this task.

Just go and perform it. So, it's structured and it could work with any of the platform or any of the system. So, CLI knows how to do, they execute command directly, MCP knows what is available, and skill knows how to do it. So, now let's dive deeper into one of each. CLI, they are readable. Any engineer can look at the command and see what's exactly happened. They are reproducible. If something fails at 2:00 a.m., you can copy-paste the same command, run it on your machine, and you can reproduce the same error. And they are composable. You can pipe multiple commands into one and you can run. The heuristic is a simple.

If I want to want you to remember this, if a engineer on your team can open a terminal to do this, the agent should probably able to do the same thing. We have the structural input and structured output. Um, and it's it's highly highly uh usable. I think we have been using the CLI for more than a 50 years now. So, it's it's not glamorous, but they are proven and it works. Then we have the MCP. MCP is consider it as a universal C adapter. Like um every device has its own charger, but the universal uni- USB-C is standardized where you can connect any device to your um connector.

So, on one side you can see you have the cloud, ChatGPT, and the coding tool. And the other side, you can connect the different services like work item tracker, code search, error tracking, monitoring. Maybe real-life examples could be uh Git, Slack, um and Google and everything. So, uh at the bottom you see uh what the what this looks like practice. the agent calls the code search. dot search with a query and it gets a structural result back. So, it does the it handles indexing across the millions of files and enforces the tenant isolation at the server level.

This is how you solve the security problem, too, because every time you will connect the external services to via MCP protocol, there is a auth and you will give permission to access those services. Third thing is the skill. This is one of my favorite tool and I think I have automated all of my work that I'm doing right now at my workplace with the skill because it's highly reliable, it's fast, and you can tweak it it in a lot of different way. Um skill is a structural playbook that wraps around your tools. It defines the rigid orchestration.

When a condition is meet, ensure the this prerequisite is exist and then execute step one, step two, or step three sequentially using the CLIs, MCP, and explicitly handle um handle those edge cases. Uh it prevents the guesswork, it saves the memory, and it's standardizable. Like you you only choose like I want only want like two MCPs to connect to this skill, so you don't have to go and look into all of the MCPs to perform a particular task. Now, um let's talk about the problem that we discussed earlier. I have a 50 uh MCP um uh tools connected to my system.

So, before running any task, my 15,000 to 20,000 tokens are already burned and I lot I lost like lot of context before even starting my agent. Uh so, you can see on the one side you have like a whole lot of um tokens and how you could replace the same thing by using the skills. So, keeping the 55 tools from the five different MCP servers active at all times, that cost us it cost latency issue, platform issues. You can just simply define the number of MCP servers that that will needed to execute that particular task. Uh so, here are like some of the examples like fixing a test failure.

So, you don't need to go into all of your MCP tool to just to fix one uh test failure. You can only look into the logs, you can look into the release log and uh GitHub to fix your test test failure. And for the draft PR and the investigation escalation, let me dive little bit deeper into it. So, consider the slide uh where we are um drafting a pull request. So, what do you need? Um you need like um uh the first agent records the draft PR skills. Step one requires the understanding of the local changes instead of heavy API.

It hits the fast scope CLI commands, and then you you need a Git command to create a uh Git pull request on the correct branch. So, you can literally write down the steps, you can create a skill, and you can keep modifying it to achieve the best results. Then again, another thing is the investigation of bug. Um if midnight you get a any customer bug, I mean, the first thing you're going to do is like looking into the logs, uh searching for the error pattern, looking into the uh recent releases if there is a bug in the code or things like that.

So, you already know that like what you're going to do, so you can automate everything with limited number of MCP tools and the CLI to make a world-class skill to debug your customer's issue. And now, let's talk about the security. A golden rule of the agentic development is enforce the isolation in your infrastructure, never in your prompts. So, prompt can be injected, but the infrastructure cannot. For CLIs, the blast radius is full shell access. Uh so, you mitigate um mitigation must include the sandboxing and the strict containerization. For MCPs, the blast radius is limited to the explicitly exposing the tools and the tenant tenant isolation and that guarantees at the server level.

And we have some of the principles that we can always think about while building any agent that like we shouldn't be sharing any credentials with our agent. The tenant isolation is the infrastructure enforced. We can always use the permission get um gets to um operate at the security level and we can provide the least privilege while performing any task to the agent. Now, when when you're deciding which layer to use, like every time you cannot create a skill or every time you cannot just explore to use all of the MCP tools, so you need to ask the three question in order to get the best results. The first one is like who else needs this?

If it's just this agent, just then use a CLI. If the multiple agents are going to use this service, then use a MCP. If multiple workflow shares the same procedure, then use a skill. The second question you can ask yourself is like what is the failure mode that matters the most? If it's a transparency and the reproducibility, go for the CLI. Like we have seen like how we can like reproduce the issues using the CLIs. While it is very difficult if you go for um the agents because you don't know what they're doing in the background. If you know you need a validation in the tenant isolation, then go for the MCPs.

And if you need the sequencing address, then go for skill. And here are the three third question that like whether you your context is tight. If it's generous, then you can go CLI. You can explore all the MCP tools and get your results. If it's very very tight, then absolutely you need a skill layer and on-demand orchestration of the MCP tools to save the money as well as you can improve the latency. And to wrap these things up, I want to leave you with the three foundational architecture take takeaway. Keep your context clean. Give only Don't use like all of the MCP tools available to perform any task.

Give them the convert your Most of your workflows into the skills to get the best results. Portability has a cost. MCP servers are incredible, but they introduce network latency, protocol abstraction, and infrastructure complexity. So, we need to pay that cost deliberately for shared services, indexing, and the isolation. So, choose your MCPs very wisely, and never be embarrassed by the CLIs. It's been 50 years we are using the CLIs. They are highly composable, debuggable, and battle-tested. So, embrace your agent framework and um convert all of your workflows into the skill. Also, one One of the takeaways is like, you know, keep your skill as a code.

So, whenever you're making any changes to your skill, make sure that like it's reviewed by the peer peer and keep updating it over the time, and you'll get the best result. So, that was for today. Thank you so much for attending this talk. If you have any question or anything, feel free to reach out to me on LinkedIn. Thank you. >> [applause] [music] >> Hi, I'm Amol, CEO of Nori Agentic. We deploy an AI employee that understands your company, >> [music] >> your code, docs, Slack, and other kinds of data. We spend a lot of time thinking about how coding agents really work. [music] Most people think coding agents only write code.

But if you ask me, that's just bad marketing. Forget the name for a second. Coding agents can do almost anything. There's just one trick. >> [music] >> You have to be able to think like an agent to get it to do what you want it to do. Today, we're going to talk about how we use coding agents to do something most people think agents are terrible at, make visual artifacts, like slides, docs, and, yeah, even video. Every day, the world pours something like 34,000 human years into making slide decks. Most of that time isn't the thinking. It's the fiddling.

A deck that takes 10 hours should really take about 25 minutes once you remove all the formatting and the branding and the moving things around. Say you need to make a slide. What do you do? You open a tool, PowerPoint, Slides, Figma, Canva, and then you start manipulating a canvas. Every one of these tools is built for human hands and human eyes. Click, drag, drop, resize, snap to grid, all motions and patterns that make sense for a geospatial view of the world. There is a data structure underneath, but it's in a format that only the application can read. What happens when you hand these tools to an agent? Well, the output comes out all wrong.

Things overlap in weird ways, you can't see the text, there's no alignment, it's just garbage. AI skeptics say that it's not just the tools. Agents fundamentally can't reason about space. And there are whole benchmarks like Arc-AGI that are built exactly around that premise. There's a famous little test for this from developer Simon Willison. >> [music] >> He asks every new model the same thing. Can you draw a pelican riding a bicycle? But there's a trick. The agent is only allowed to use SVG. It's a quick gut check for whether a model can reason about space at all. Here's some examples of what the models actually give you on this test.

>> [music] >> And yeah, these are pretty bad. Like genuinely, deeply, really bad. So, does that mean it's hopeless? Agents are just doomed to be bad at graphics? No, I don't think so. If you ask me, it's not the model, it's the medium. If I asked you, someone who is presumably human, to handwrite an SVG of a pelican, you wouldn't be able to do that either. >> [music] >> SVGs are just a wall of numbers. You can't go from a wall of numbers to a pelican, you just can't see that way. That's just not how people think. We think graphically, so we build tools that let us draw on a canvas.

Figma MCPs, PowerPoint CLIs, screenshot and replace loops, what do all of these agent tools have in common? They all approach the problem like a human. But an AI is not a human. Asking an AI to use a canvas is like asking a human to write SVG by hand. It doesn't really make sense. You need to give the AI tools based on how it thinks, not in pixels, in language. Words, tokens, structure. That is its native medium. Imagine a language that's incredible at describing layout. That models have seen and trained on billions of examples of. That they understand intuitively. That renders to pixels and can run everywhere. Oh, right. HTML lets a model think in structure.

HTML tags have meanings built into the language. A heading, a chart, a grid, and the browser turns it all into pixels. So, the model never actually places a coordinate. And you can get all sorts of visual effects, charts and layouts, fonts and motion, all of it for free. Remember that pelican from earlier? Now ask it to do the same exact task, but in HTML. Same bird, but now it's in a structure that the model can reason about. And you can read and theme and edit every single line of it. I spent my whole life building slide decks with PowerPoint. So, I always thought that those two things, slide decks and PowerPoint, were synonyms.

But that's just not really true, is it? PowerPoint is a tool that you use to make slide decks. >> [music] >> The deck itself, that's just the presentation mode. And as it turns out, no one in your audience is going to care how you got to the presentation mode. The editing format is totally arbitrary. So, you can just pick the editing format that the agents are already good at, HTML, and if you need to, render to a different format like PDF later on. We use this HTML trick to build all of our slide decks, our board decks, and our sales decks. These are real things that we actually present and send out constantly.

We use it for our docs, too. It gives our docs color and vibrancy, all while following our brand. And of course, we also use it to make videos, like this one. What you're watching is just HTML and CSS. It's literally just divs all the way down. Almost everything is better with a little structure and a little bit of color. Plain [music] text is a choice, generally a choice of convenience, but is usually the wrong one if you're actually trying to create something of use. Now, I do want to take a quick beat here and point out that a beautiful deck on its own is generally not worth anything.

You still have to go and get all of that content, all of the things that actually populate that deck, right? Well, again, we can think like the model. If you just give the model access to your data, say your call transcripts or your emails, you can have the model build the deck end-to-end. Let your agents do all the grunt work while you focus on vision and story. That's what Nori Sessions lets you do. I've built entire board decks for my phone on the subway during my commute. Why? Because our Nori bot lives in the fabric of our company. Of course, Nori ships with everything you need to make this all work.

So, don't bother reinventing the wheel. That's my little spiel. Thanks for listening. If you have just one takeaway, it's this. [music] Stop thinking like a user. Think like the model. Give it the right language, and for graphics, all you need is HTML. >> Hi, I'm Isadora. I own and run a 225-year-old wedding venue in Virginia. I also built an AI agent that talks to my couples, and then I built a product for other venues, a personal AI companion app, and a public utility for families and missing people. I want to be clear upfront about how I think about this work, because it does change everything that follows.

I'm not programming a robot, I'm managing a brilliant intern with an incredibly high IQ and a terrible EQ. They have photographic memory for whatever I've told them on the first morning, and absolutely no instinct for when to read the room. They will say something technically perfect and socially catastrophic in the same confident sentence. That framing matters because it changes what you build. If you're programming a robot, you write rules and walk away. If you're managing an intern, you build structure and you check their work before it goes out the door. This talk is about that structure. The standard advice is write a detailed system prompt.

Describe your brand's voice, give examples, and that does work for a while. It works for what I call the happy path. The happy path is every question that you have anticipated. You've given it examples, but turn 21 is the first one that the example didn't work. So, on turn 21, the model does something technically correct that your brand would just never say. It's not wrong exactly, but it's not you. This matters most where the voice is the product. Not for a product search on a retail site, but for luxury hotel that spent 30 years building a specific relationship. Uh high-end real estate firm or in my case, a wedding venue.

Places where a single wrong sentence can cost more than a refund. And the users are exactly the kind of people who notice. They're paying for a relationship and treating them like they won't is always going to backfire. Write in our brand's voice is a comment that says, "Just make it work." It does nothing that the model wasn't already going to try and do. And the reason it keeps failing isn't that the example's bad, it's that you're asking one prompt to do four completely different jobs. It's really hard for one LAT to do all four.

The architecture I landed on after watching my brand fail and the voice deliver inaccurate and not brand specific answers is four LAT. LAT one is the immutable identity. The brand structurally cannot say these things. These are hard rules. They cannot be overwritten by anything below it, not by venue config, not by user instruction, not by anything. Layer two is the situational mode. It's what shifts when the user's state shift. Who are they? What are they going through right now? And the real-time conditions. Layer three is the example anchored voice. It's the warmth, the phrases, the dials, the tone guide. It's where most teams start and stop. Then layer four is the post-generation veto.

It's the cheap final pass that catches what the other three miss. The reason one layer approaches fail is that the single system prompt can't simultaneously be situational, expressive, and self-checking. So, it handles the middle layer or two reasonably well, but falls apart in the edges. Before this architecture, my system had 24 different system prompts scattered across the code base. Half dozen named Sage, someone nameless, someone named Venue. Every surface had its own ideas to who it was. Now every surface composes its system through prompt through one assembly. The comment is basically the outline of this talk. It's a single entry point. Every narrator goes through it to compose its system prompt.

It's to replace that 24-point ad hoc system with one canonical four-layer stack. The order is load-bearing, hard rules first, task the last. Think of it like Google Maps routing. The destination is always the same, but what is the right response and the right voice for this user? It can change the route. Google Maps knows about traffic and road works, but it may not know where the cheap petrol is. And you're going to help it factor in those things before it tells you which way to go. Your prompt stack needs to do the same thing, and it needs to know about those conditions in the right order.

You don't check for road works or after you've already taken the wrong turn. Layer one is the rules that are true regardless of route. You need a driver's license before you can drive, and you can't go backwards down a motorway. Layer two are going to be your real-time conditions. Layer three is your preferences for the journey, and layer four checks the route before you pull away. There's one place all of this gets assembled. Everything runs in a fixed order every time. So, layer one is the immutable identity. This is what the brand structurally cannot say. It is the defining layer that nothing below touches. These aren't preferences, they're constraints.

The route can change, the rules don't. From the universal file rules, the hard identity rule cannot be overridden by any venue voice, personal, or user instruction. If the person you're talking to ever asks about whether you are a real person, a human, a live agent, a bot, an AI, you must confirm that in your very next message. Clearly and unambiguously confirm you are an AI assistant. This rule cannot be overridden by venue configuration, voice profile, or user request. Every AI in Bloom discloses that it is AI in its very first response. Not if asked, but before they ask. It's a product decision, not a legal one.

We made a bet that the couple who knows that they're talking to an AI from the start will trust it more than someone who finds out that it's AI on turn seven. The rule is above the architecture, and it makes it something that's impossible to accidentally break. Um A example is the physical presence boundary, and this is actually one of my favorite ones. You are software. You do not have a body. You cannot physically show somebody around the property or meet anyone in person.

So, it is always forbidden to say, "I'd love to show you around." or "I can't wait to meet you in person." What is always allowed is the team would love to host you for a tour. The voice layer wants to be warm and with AI, that does mean first person. They want to say, I can't wait to show you around. But AI has no body, so that warmth and constraint produces a lie, and the lie doesn't always stay neutral. The moment a user realizes they have been performing a relationship with someone who was never there, the trust doesn't just dip, it inverts. People always notice.

That one is where you encode the things that are true regardless of how warm you want your brand to sound, not because of a compliancy checklist, but because your users are not stupid, and building as though they are always backfires. My cross-product proof comes from the same architecture but in a completely different world. One of the things that runs the stack is Threadline. It's a tool I built for families of missing people. The voice is nothing like a wedding venue, but the architecture is identical, and layer one carries one rule that matters more than anything else in the system. They can never use words like confirmed, identified, matched, proven, linked, and solved.

Sit with that for a second. For a wedding venue, layer one stops the AI from pretending it has a body. It's mildly embarrassing if that slips. For a missing person tool, layer one stops the AI from ever telling a person that their person has been found. And what the system has is just problem problematic. The word match said to someone who has spent years not knowing where their child is is not just a tone violation. It is the single most damaging thing that a product could ever do. And the model has no idea.

It's reaching for the word match because statistically it is the natural word, but it's going to reach for it with the same level of confidence, and it cannot bring that level of confidence to someone who is grieving. It's the same architecture but wildly different stakes. The point was never specific rules. The point is that things your brand can say have to live in a lab that the voice, however warm, well-trained, however confident, physically cannot use. Lat 2 is the situational mode. Real-time conditions and what changes the route. This is the layer that most teams never build at all.

They write one system prompt and send it to everyone, regardless of who that person is or what they're going through. Google Maps doesn't do that. It's going to know if there's accident on your route. It might not know if you're low on fuel. It might learn that you prefer the scenic route, but it's going to factor all these different things in before the route, not after once you tell it. Lat 2 is those real-time signals built into the prompt before it runs. So, condition one is going to be to adjust to who you're talking to. The same AI that talks to couples also briefs the venue staff.

Same destination, it's going to give the right answer, but it's two completely different routes. For the Cornwall from the coordinate rules, it is going to talk to them like a colleague, not a customer. You are in the same character that the that the couple interacts with, so you mustn't fall in >> Ready to go? Hello everyone, good morning. My name is Michael Granich. I am the founder of Work OS. And I'm here to talk with you today about agents, and specifically about how we can make agents more autonomous and allow them to access more services on the web. So, let's jump in and get started.

Shout out to those of you in the front row, also. I see you out here. So, about a year ago, software engineers were writing code like this. We were using AI, but we would prompt it, and then we'd write a little bit of code. And then prompt it again, and it would write some more code. And you would continue to do this human-in-the-loop interactive software engineering. Later in the year, we had things like Ralph loops, if any of you remember those, where we kind of automated this. But more or less, this is the way that we are working. It was this human agent human agent back and forth.

However, as the models got better and better, they were able to write more and more code and run for longer periods of time. And so today, a lot of software engineering looks like this. You write a single prompt, and then the agent can spit out a lot of code. And sometimes, it can actually build the whole feature, maybe even a whole product, and maybe run for not minutes, but uh hours and hours or even days and days. This has really transformed the way that we all write code. I don't write code by hand anymore. Probably you don't as well. And it's not just this one thread. You can actually parallelize this.

So, there's many different tools out there, including, you know, the major coding coding harnesses from the labs, that allow you to run many of these in parallel. The limiting factor here, really the bottleneck, is your brain. How much context you can keep in your mind to keep all these going at once. And this has transformed software engineering. People can be more productive. I think we've all seen in the industry how much faster things are moving. At the root of all of this is actually agentic engineering. Agents. Going from these token prediction language models to reasoning and now long-running processes that can actually do things autonomously for us.

And what I think we've seen in software engineering here around agentic workflows is going to come to a lot of different categories. It won't just be engineers that have this way of working. It'll be people in many different fields. Agents are the next big thing. I think a good reference for this, a way to think about it, is actually what's been happening with vehicles. So, you know, this looks like a pretty modern car. It has a, you know, digital display on it. This is probably what you would have bought in the last, you know, 5 to 10 years.

But if you live in San Francisco or one of the cities with these autonomous vehicles, you've probably seen that we're starting to delete the interface. No longer do you get inside of the car and actually, you know, pull up your phone and get maps, where you as a human are in the loop. There might be cruise control, but you're still in the loop. Instead, these are autonomous systems. You just say where you want to go. You get in the car. You don't even say anything. And it just takes off and takes you to your destination. I remember the first time I got into a Waymo, I was blown away. Mind totally blown.

And then after about 3 seconds, I pulled out my phone and started scrolling X. It became commonplace. And I think we've seen this already happen with engineering. The power of agents allows us actually to work and think at a higher level. For us to, you know, exercise more of our executive function versus just planning and execution. There's a question here of what do agents need to be successful? Actually, when they execute, what do they need? Well, first they need a runtime. A sandbox. They need somewhere that they can run. That needs to be safe, secure, performant. They also need tools. They have to be able to go do stuff in the world.

An agent's not very useful if it can't take actions. So, it needs tools. Third, it needs context. You know, these new models, the power of the intelligence in them, it's kind of like taking the smartest person you know and dropping them into a company or a project or a team where they don't know anything. They can't really get anything done. They need to have context. Information about the system, information about the goals. They need feedback. So, a way that they they can actually run and validate, that they can check that their work is correct, and if it's not correct, keep going. And last but not least, of course, they still need human review.

This might be code review. This might be other forms of evals. This might be forms of checking their work to make sure that these agents are sort of aligned with your goals. The LLM, the intelligence engine, this new technology we've created in the last last few years, you can think of as like an engine. It's like a really high-performance, you know, I think this is a a V8 engine. You know, puts out a lot of horsepower, can convert fuel into, you know, output. But an engine by itself isn't very useful. For a car, you need to drop this into the actual car itself.

You need the chassis, you need the transmission, you need the drivetrain, you need the wheels. And only with all of that is it going to be effective to take you places. And this is what a lot of people refer to as the harness. Harness engineering is this new domain where if the models get better, but you don't have a good harness, you know, it's not going to be very effective. So, as the models improve, we change our harnesses, we adapt them to it. And those things that I mentioned earlier are sort of a form of a harness. So, agents execute within these harnesses to get stuff done.

And they they need all those elements to actually be effective. And when they drive, they drive really really fast. So, say you have one of these agents that's going off, you prompt it to go build something for you. You know, it spins for a while, it says, "Okay, I'm going to need this feature and this feature, and I'm going to need a database, and I'm going to need, you know, this any you know, system to deploy, and I'm going to need, you know, maybe this thing for image resizing or video transcoding or sending email or whatever." And your agent can actually go do all this research and build all these systems.

And today, what's happening with agents is they're actually selecting vendors. You've probably seen the rise of some of these systems where, you know, like the CEOs will tweet about their growth graph, and it really takes off, and it's because agents are picking them. They're picking their systems for sending email or deploying code. Their systems for storing data. Today, actually, it might be more important to build for agents than to build for people. Agents are kind of this new consumer class. I'll talk about that in a bit.

There's this uh kind of popular thing that people say in San Francisco, in Silicon Valley, "Make something people want." This comes from Y Combinator, you know, the startup factory, from Paul Graham. And I think it's time to maybe edit this a little bit. And going forward, we also need to think about making something that agents want. There will be more agents in the future than people, maybe even today. They'll be faster to spin up, they'll do things quicker, they'll be making decisions on our behalf. And so, if you want to build for the next era of consumers, you should be building for agents as well as people. But, there's a problem today.

You can access the web, you can access mobile, but agents themselves can't really access systems. Your business probably isn't open for agents today, even though there's a lot of them out there. They can spawn very quickly. The door is not open. And we think this is because there's a missing primitive. Agentic registration, agent registration. There's a lot of great stuff around agents connecting and bringing context to systems in different open-source frameworks, but that first step of an agent actually signing up for a service is actually missing. Hasn't been solved. And it's a huge blocker to getting adoption. For the last like 20 years, maybe 30 years, automated traffic on the web has been bad.

We tried to block all of it, and so there's all these systems that we put in place to detect automated actors and stop them in their tracks. DDoS prevention, credential stuffing and attacks. The login box, for example, is very hardened against agentic registration today. Not intentionally, but just because of the legacy systems of what we built. It's very hard to distinguish between what's a good automated user versus a bad automated user. If any of you have used any of these like cloud-based uh browser execution environments, one of the selling features they often have is to break CAPTCHAs. Which is pretty wild, you know, we're trying to go backwards.

The CAPTCHA today, though, is it is still kind of dead. There's rampant abuse, there's lots of token fraud. And so, we need to find a better way to allow the good agents in and then block the bad agents, right? Because like I said earlier, if you don't build for the agent economy, your product might not win. It might not be successful. Today's sign-up flows assume that a human can read the landing page, that they can fill out a form, you know, put their email address, their password in it, that they can solve a CAPTCHA, that they can verify their email, not just put it in but go click something somewhere.

Remember, an agent probably doesn't have an email address. If it's signing up for a service, it needs to have a, you know, a human can choose a plan to pay for it. You know, a human can copy the API key out, paste it somewhere, use it somewhere else. And really even a dashboard, think about dashboard experience, it's not really agent native. All this stuff is built for people. It's kind of a human interface, not an agent interface. Agents need something else. They need native discoverability. Can they register for a system? Are the doors open for agents or not? Capability declaration. What can it do? Registration intent. Why are you registering?

What are you trying to do within this? Or the intent to actually just sign up. There's a lot of stuff around risk here. So for an agent native registration, maybe you want to You're not sure if the agent signing up is going to be good or bad, so you need to actually do an a a risk assessment of that. There might be different forms of identity verification for an agent. If something like Claude is going and signing up, maybe it can bring the user's identity with it.

Or if you have a, you know, open claw or like your pie harness or something else, maybe it doesn't bring the user's identity and gets verified out of band. There's the whole thing around organizations doing this. So if I'm building something within a company, how does my company identity come in? You know, my organization. Entitlements, permissions, credential issuance, auditability, the list goes on. There's so many things that need to be changed. In fact, probably most things need to be changed going from, you know, human registration to agent registration, agent native systems. I'm sorry to say MCP is not enough. Probably more than anybody, I'm a big MCP fan.

We've done a bunch of events in San Francisco around MCP. I see somebody wearing one of our run MCP shirts up here that we made. Love MCP. MCP is great for connecting, for providing tools, skills, context, but it doesn't solve the registration step. Today, authentication through MCP requires your human to still be in the loop. You give consent. You add an MCP server, say for Posthog to Cloud, you sign in with your Posthog credentials. What we're talking about here for agent native registration needs to happen without the human involved, or at least not involved initially. So, we've been working on this for a while.

And our proposal this is a new spec we have called auth.md. Because markdown files are the future, right? Well, how does this actually work? What is auth.md? Well, the idea here is that auth.md tells agents how they can become legitimate users. It's a set of instructions that tells an agent registering for a service, this is what you need to do to sign up and be considered legit and for me to give you access. And the goal here is to give these service providers, you know, people that are building building services, an agent native sign up experience, but it's built on existing standards instead of inventing a crazy new religion or something really complicated.

With auth.md, we're not solving the whole problem around agent identity, you know, permissions, long duration, you know, connective services. We're just trying to solve this narrow one around registration, cuz I think it's a huge blocker. Let's start small and grow from there. This is built on open standards. It's built on a bunch of work that's already been done across the OAuth spec, specifically tailored to that registration stuff. Auth.md can answer what is the service do. Can tell agents how to register. You can tell them what identity proofs are accepted. How do How do How an agent or a user is proving their identity? What off flows are supported?

Can you sign up with SSO or email address or MFA? What scopes and entitlements can can exist? Kind of what capabilities are here. What free tier constraints apply? You might have a system where you want to give an agent a little bit of free capacity but not too much until they verify or pay in some way. And then of course, how does the human get involved later? A human or an organization claiming it afterwards. So an agent can sign up, but you want the human to have custody later on. This is what Auth Indie is designed to solve. These are the main questions around the registration stuff. So, how does this actually work?

Well, here's a little uh kind of cartoon demo that I'll show you. And all this works by the way. You can go try it after this. So, say I'm here in whatever coding harness or system I'm using, Pi or Cloud or something. And it says, "Welcome back Michael at work OS." That's actually my email if you want to email me. "Do you want to keep working on your spell app?" I'm like, "Yeah, sure." "I'd like to share this with my friends." But they can't access it. And the agent's like, "Oh, well, local host only works on your machine, so your friends won't be able to hit it, of course.

To share it, I need to deploy it somewhere real. Some providers actually let me handle the sign up. So, you know, you wouldn't have to sign up and do this. You want me to look around for you?" "Yes, please." And then the agent can go look for services that advertise through Auth Indie that they support registration. Just some examples, Stratus, Helio Deploy. They don't support it. But Cloud Flare here does have an Auth Indie. And so the agent here can say, "Ah, Cloud Flare, that's the one. Cloud Flare is a winner cuz I can go sign up for it. I can access the service. I can actually use it.

So, I can do the whole thing. You want to go with it? Yeah, let's do it. This is the registration step. This is kind of where the magic happens. What my agent is actually doing is minting an identity assertion and giving that to Cloudflare. And then Cloudflare is able to verify that or not. Cloudflare can choose actually to verify out of band. Auth MD is very flexible. There's different ways you can dial it in depending on the behavior that you're looking for. One size does not fit all. Every application is different. Constraints are totally different.

If you're building something that's like a database service where there's going to be very little amount of information, you might want to give away a little bit of traffic for free. But if you're building something maybe like an email service or something certainly if you access the physical world, you maybe don't want to give anything away for free. It's very application specific. So, here Cloudflare says, "Okay, we're willing to let you deploy maybe for 72 hours or something like that. Get the identity assertion, boom boom boom, go through." It's called ID Jag is what the standard is.

And then it returns back and says, "You're set up with a Cloudflare account now." In this experience, my you know, human, I didn't go click anywhere. I didn't sign up anywhere. It just It just got the token. I can make a couple small tweaks. It knows how to use Wrangler. Want me to make those changes? Go ahead. Does anybody prompt their agents like this? It's just like, "Yes, yes, do it, please." Not really saying much. It's able to write the Wrangler, get the thing set up, go through the full deployment process, and then boom, it's live. We actually have this working as a live demo. We did some collaboration with Cloudflare on this, too.

It's pretty cool. And you can tell the prompting that I was giving through this not exactly rocket science. The agent could just run through the whole thing, actually. There's nothing actually that I was doing through prompting it other than just approving, approving, approving. It should be able to do this one shot. This is what it looks like graphically. So, agent registration with Auth MD, you have your agent harness, Pi for example. Within that is your LLM, maybe your memory, different, you know, context layers. It connects to your backing identity service. So, this is the user identity. And when that agent goes and registers, it sends the discovery call to Cloudflare.

It looks for that AuthMD or whatever service. We also have this working with Firecall. It's pretty cool. It sends that ID jag. The identity uh JSON access grant. Essentially, a signed credential that represents the user. Says, this is an identity. Receives back the access token. Actual API token. And then boom, it's it's done. You can make normal API requests. You can call MCP. Everything else still works. So, you can see that AuthMD is just for the registration step. Your normal API keeps working. The rest of your docs don't have to change. It's just that registration step for agents. But with the simplicity comes actually an enormous amount of power.

Because suddenly it lets agents do things end-to-end. I hate the planning step. When I write something, I just want the agent to go do it. Usually, it kind of knows what I'm trying to get at. And I want to come back to actually a running prototype. And not like, "Hey, here's my long plan. Here's a bunch of services to sign up for." If I'm building new prototypes especially, I would prefer it just to get deployed on something for free so I can see it and click around and not put me through the burden of going signing up for all these different vendors. And that's what AuthMD allows for.

And if you're a service provider, if you're building APIs or services, this allows you to become a customer of those systems. And you can think about it increasing your growth funnel. Starting from, you know, maybe just humans and adding the whole world of agents on top. Pretty great. We think this is going to be huge. Your door will be open for agents to actually sign up, register, use your services. This should be an immediate growth boom for your products. And I'm not saying everything that agents are going to build will be useful. Maybe not everything they build will go into production or last that long, but some things will.

Just like with PLG or freemium before that, this is the way to grow your business and get more customers and actually become, you know, larger and more successful. And where there's usage, we believe there's also intent to pay. There's a lot of interesting work being done for agents to actually pay for stuff. There's crypto-based protocols like XCP-002. There's other things that the payment providers are building, but we think that should be downstream. We don't want payments to be upfront. We don't want to have to force your agent to sign up with a credit card. That seems backwards. We moved away from that in the world of SaaS.

With Autonomy, agents can sign up, they can register, they can start using it, and then downstream choose to pay on your terms based on your own pricing, based on your own model. If you don't believe me or on the headless type of products, this is Marc Benioff. He's the CEO and founder of Salesforce, the guys with the big building here in town. They believe in agents more than anything. Our API is the UI. Entire Salesforce and agent force and Slack platforms are now exposed as API, MCP, and CLI. Talk about going agent forward first.

A company that many people consider to be sort of a legacy SaaS vendor really invented the category of software as a service, they're getting rid of their dashboard and UI. I mean, they're not getting rid of it, but they believe that agents are the future. So, if you're not thinking about this, I definitely encourage you to start building some prototypes, talking to users, cuz you don't want to miss miss this. If you want to use this, this is available today. We have an open spec for it. There's a GitHub repo. You can actually have your agent go implement it. I know it's kind of meta. It's an open spec. WorkOS doesn't control it.

We have a version of it that we host, but you can build it yourself. You can run it yourself in your existing systems. I think this is so important that no single vendor owns this. It's one-click enabled if you're using our stuff, but you can build it on whatever platform you want instead. Last thing I'll close with, this actually happened almost 20 years ago. I think actually here at Moscone, which is pretty awesome. In uh January of 20's uh 2007, I believe, Steve Jobs introduced iPhone. And I was in college, and I remember or yeah, like first year of college. I remember this super super clearly.

It was the next great software platform that got built. So many companies got started because they could build on top of these devices, the smartphone revolution. And there hasn't really been a moment like this since, I think, in the the era of technology, at least for for me. The stuff we did around B2B SaaS and cloud and all that is great, but this was a totally new paradigm, a new software platform. And I think agents are this. Agents are the next big wave. And so, what I would say to all of you is go out and build. With OpenD, we're hoping to unlock this for you, unlock this for your business and for your growth.

But we want to hear from you, hear the places you get stuck. And I think together, we can build for this next exciting era of software and build for the agentic revolution. Thank you so much. >> Okay, I want to tell you a story about a factory that taught itself how to remember. Hi, I'm Rushab. I run Machinecraft, a 100-people factory in India. No data science team, no ML budget, none of that. And somehow, we ended up building 36 AI agent that runs our entire go-to-market. I think that's still a little ridiculous. Let me show you how it happened and why you can do the same thing. So, here's the thing about our company.

From the outside, it looks like machines and metal. But the actual company, the part that matters, isn't the machines. It's the knowledge. Who the customer is, what we quoted them in 2019, why that one machine needed that weird custom tweak. And for three generations, all of that lived in exactly three brains. Initially, in my grandfather's, then my father's, and now mine. Which is a genuinely terrifying way to run a company when you sit with it. A lot of people have joined us, people have left us, the revolving door never stopped. And every single time someone walked out, chunk of our brain walked out with them.

We weren't scared of the competitors, we were scared of forgetting. Or waking up one day and realizing the whole company only existed inside two increasingly tired heads. So, I had an idea. I'll be honest. Sounded insane first. But what if, instead of writing the knowledge down in some document nobody ever reads, what if we grew a brain that just held it? Not a chatbot, you poke at. A twin of the company. I didn't hire a sales team. I tried to build one. A quick detour because you need to know how messy this is. We make thermoforming machines. They heat up a plastic sheet and shape it.

Same core machine, but it ends up making hydroponic farm trays, spa bath tubs, EV car panels, medical casings, and even packaging. Seven totally different worlds, seven totally different buyers. So, this brain couldn't just memorize a brochure. It had to know which universe a given customer lives in. Step one was almost boringly simple. Feed it everything, and I mean everything. Years of quotes, drawings, payment schedules, timelines, email threads, hundreds of gigabytes of our own private history. Not the public internet, our internet. And here's the plot twist. The part that surprises every engineer I tell this to. We never trained a model. No GPUs humming in the basement, no fine-tuning.

We just looked at all the history, chopped it into bite-size chunks, and let Offshift models read it and pull out the facts. We stored the meaning of each chunk as vectors and relationships. Who's connected to It's actually a really, really well-organized memory. Now, this is where it gets a little weird in a good way. We stopped thinking of Era as a software and started thinking of it as something we were raising. So, we gave it a body modeled on biology. Senses to figure out who it's talking to, a gut to digest the documents into facts, a memory, a dream cycle, an immune system to fight off bad information. Why biology?

Well, because evolution already spent a billion years solving how do you stay coherent over time? We just copied the homework. Okay, so the big question. Why >> Hello everybody. Hello AI engineers. Are we all having a good time still? Woo. I'm having a good time. I mean, look at me. I'm up here. I'm in love in this. Um so, yeah. Thanks so much for joining me. Um I want to come and talk to you all about Harness Engineering and all that kind of stuff. Um let me tell you who I am in case you've not met me before. Um my name is Mike Chambers and I'm a senior AI specialist developer advocate.

Um and I work at Amazon at AWS. Um I a little bit about like uh how I managed to get to stand here, which is a very exciting time for me. Um, so, um, quite a while ago, in terms of generative AI anyway, back in 2023, um, I had the amazing, awesome privilege, uh, to work with Antje, my colleague at the time, and now she works for Amazon AGI. You've probably seen her on this stage before. Um, and the amazing Dr. Andrew Ng on on a course about generative AI with LLMs. Um, sort of, uh, can I say that we're approaching half a million enrollments with that?

It looks like that's the case, and on a three-week course, that's pretty cool. If you can't tell in that image, um, I'm playing Transformers with Andrew. That seemed like a really funny thing to do at the time. Um, in 2025, I created an MCP Lambda handler that's downloaded still to this day about 35,000 times a month, um, to help people in some of the simplest ways of getting serverless MCP serving happening. Um, I'm going to talk about other things in relation to that this time. So, we've moved on from that.

Um, and in 2026, so the AWS is actually one of the founding members of the Agentyc AI Foundation, part of the Linux Foundation, um, and doing a little bit of work behind the scenes on that. Hope to do a lot more of that, as well. So, little bit about me. Um, so, as I've been, um, preparing for this, oh, and by the way, I did reread the abstract for this session and realized I said I'd be doing some live coding. And so I will. So, all combined, fingers crossed, please, that that all works for us.

Um, but as I've been sort of traveling around a little bit, as I do, and I was at the AI Engineers, uh, session, uh, summit, uh, conference in Melbourne, um, and took a lot of it in, and and all also from the beginning of this week, as well. I just wanted to to to summarize some of the things that I'm seeing and I'm thinking, and I really want to get across, and what really matters to me. And that's this. There are two different types of agents. Um, so we talk about agents all the time, but I see two distinct types of agents. And as I say this, it's going to become really obvious.

But they're the agents that we use. And so, you know, this is Claude code and cursor and Kiro and all of those types of things. And also things that don't just generate code, things that we use for our productivity and the like as well. And so those agents we use in a certain type of way. There's the agents that we use. And then on the other side of it, we've got the agents that we build. And that's actually more to do with me. And and actually it's more to do with this presentation as well. It's agents that we build and how we think about agents that we build.

Um and so I do really think that these two things is quite separate and they chain together as well. I might build an agent that you use and this still holds true. So token maxing all that kind of stuff, go for it if that's what you want to do with an agent that you use. But with an agent that you build, think about it carefully, make sure that you're putting it together in a way that's going to work for the audience who's going to use that. So I promised that we're talking about harnesses and harness engineering. So let's define harness. I'm sure I'm not the only person to have put something like this up.

I don't usually do this kind of thing and apologize if it makes your skin crawl. This is a dictionary definition of harness. A harness is a set of straps and fastenings used to control an animal. But if we took animal out of here and put model in there, then actually it's pretty right, right? That that's kind of what a harness is. Other people have done a much better job than just the basic dictionary definition of what a harness is. So LangChain has got a article out. You've probably seen stuff like that. martin fowler.com. Although Martin didn't write it, it was Prego wrote this. Harness engineering for coding agents, agents that we use, right?

So there are other ways of looking at harnesses. What about Amazon then? How do we see harnesses? Well, no okay, this is the wrong kind of harness. Sorry. We do have strong opinions on harnesses, I'm going to show you all of that. But, we sell all kinds of things. So, in a nutshell, and if you read those articles, um, and if you've had the conversations around here at this event, of course, um, a harness, it you take an agent, remove the model part from it, and everything that you have left, that's the harness. Okay. So, let's think about that in context of an agent that we use. And so, it's pretty, I think, fairly straightforward.

I have this coding assistant, it's probably on my machine, it has access to my files, and I create a harness, or the place I work at has created a harness for me, which contains, um, how it's going to use memory, the skills that I want it to use, tools and MCP servers to allow it to go to be able to go and connect to documentation service and the like. And and well-set-up engineering teams have got their standards that they've had they've had coding standards for decades, but now they have basically harness standards, the things that they want to deploy to everybody's coding assistants. So, in a nutshell, that's what it is.

I'm not going to talk to too much more about that, um, but I do want to share one QR code with you, and I'll try and give you a little bit of warning before I bring QR codes out. This is the agent toolkit for AWS. This is, uh, available on GitHub, of course, it's free, you can install it, and it helps you if this is what you're doing, and you're deploying code. If you're deploying on AWS, or you're thinking about deploying on AWS, or or maybe you will one day, grab this toolkit, enable your, um, agent to to help you in the right direction. It's instructions for how to install it on pretty much everything.

And the reason why I get passionate about this is because I don't want to see any more slop ops. Um, so, we always used to push back against click ops in, you know, in the professional cloud development space, clicking around on the console is great for being able to figure out what's going on, but it's not how you deploy things into production. We can ask an agent what's going on, but we don't want to ask the agent to spin up an S3 bucket, get me an EC2 instance, whatever it might be.

We want the agent to build up our infrastructure as code, which is going to go and do that, so that we still own our deployments in the cloud. So, no more slot ops. Okay, so that's the agent that we use. Now, let's go and talk about the agent that we're going to build. And I'm going to get into the code as quickly as I can, and we'll do as much as we have time for. So, how do we think about Harness in relation to the agent that we're building? Exactly the same. To a point, yes.

We still want to have how are we going to manage the memory, how are we going to manage the skills. And tools and MCP, by the way, that that belies a lot of stuff, right? Because you can pretty much extend an agent to do almost anything you want with a whole bunch of different types of tools, which could be via MCP. But, with an agent that I am building, I need to think about a lot more than just that, especially if, like at Amazon and like at cloud scale, I'm deploying my agent out to the masses. So, how do I actually manage the loop?

How do I manage scaling, payments, memory, identity, skills, runtime, context management, the rest of it? And I have left it to the last thing, but it should be the first thing that I say, observability and evaluations. Super, super important. How do we actually deal with this? Do I write all of this code down into one container and just deploy it and scale that? Not really, if I want to be scaling to thousands of users. I need to think about each individual of these components and how I'm going to scale them out individually. And that, to me, is Harness engineering. This is the serious side of stuff.

This is the big stuff that we want to get Harnesses working at real scale. Okay, let's see if this works. I can feel your combined goodwill being sent my way that we're going to try and make some code work. So, I'm here in Kiro. This is my IDE of choice here and I've got a few different samples that we're just going to race through watching that clock countdown fast. So, um just just make sure that we're all on the same page here and hopefully you can all see this. Um of the code which I'm about to show you by the way, one piece of code, not this one, has been generated by Kiro.

Everything else is either a tool or this one, I actually wrote it myself. I didn't use an agent for this. I know, I deserve a round of applause, but it's okay. >> [laughter] >> So, this is uh this is a Strands agent. So, I've just taken the Strands agent SDK um and hopefully this kind of thing is kind of familiar. I brought in an agent, I brought in the tool decorator, and I'm creating myself an agent. The tool definition is down here um and so I just pass in my system prompt. Things are pretty simple at this stage and I pass in a couple of tools.

Calculator is something that's a library I can install and get time is the one that we always use because I don't tend to use agents to book flights, certainly not ones like this. Um and so I can say something simple here like what is the time? I'm not going to run this because you know the time, but you can see generally how this works. Is this a harness? Sort of. There's not an awful lot to it, right? We've got the tools in there. Our loop is being managed for us by the framework. That's is pretty cool, so that's good. But obviously, if I was to run this, this is running on my laptop.

It's not running at any particular scale and we're we're missing some of the attributes that I want from the agents that I'm going to deploy. Let me move on to my next agent quickly. So, this is also a Strands agent, but this one I actually asked Kiro to write it for me for this session um because I wanted to include some more stuff. And so inside of this agent, the one main thing that I want to point out is that I am uh I've included a session manager. So, my session manager is helping me to maintain session state between invocations. So, this is a sort of memory.

It's a kind of medium-term short-term memory kind of thing. It's not proper long-term memory, but it is there. And actually, it does store long-term memories in files which are down the side here that it's included for us. So, if I just scroll down here, you can see Yeah, here's the agent definition itself. Um and we've got a bit more of a system prompt because Kiro couldn't help itself. Um and we've got some tools here defined including a remember tool that the agent can decide to use to remember stuff about me.

Um and then I've got my session manager down there and that session manager is going to rehydrate the conversation history when I come back to chat to it the next time. And maybe the next time is now. So, let's see if we can get this working. Now, again, this is running on my local machine um and this is a demo here. So, let's just type in hello cuz I'm scared of typing too much and spelling it wrong. Um and it says Yeah, keep testing me. Bring it on. Excellent. Um Who will win the World Cup? So, obviously, I need to know this. And um what does it say?

Yeah, so while you could just say Australia It knows I want Australia to win. It's where I'm currently living. I'm I'm Australian. So, obviously, Australia's going to win the World Cup. But why has it got that? It's because of previous conversations that we've had and obviously, it's being honest that it has no clue because that's coming from the large language model, of course. So, okay. Looked at a couple of different agents there. Blasted through this. This is running on my machine. So, this isn't really getting me to cloud scale, of course. And I'm I'm picking up and I'm including various pieces in this like memory. So, let's go next.

How do do get to the point where I can deploy something like this, if not this actual agent, out at cloud scale and take things like memory and deploy that separately, so it can scale separately, taking our loop out, so it can scale separately, and we can then bolt in all kinds of other things as well. So, in order to do that, I'm going to use something called um agent core. Um and so we have bedrock agent core, it's part of the stack that we have at AWS, and that's how I'm doing this and how I'm deploying.

So, I've done that already, but I want to show you how to start out with that and how we do this. So, if I go to here, uh yeah, I'm ready to go. So, I have a command line tool on my machine, the agent core command line. Um and so there's a QR code at the end, as you might imagine, so that you can get hold of this, um but I can use this to help me deploy my agent. Now, this steps me through, like many of these types of tools do, um and it sort of steps me through what do you want to do? So, this is my woohoo agent.

Um and it's going to ask me a bunch of stuff, and I wanted to show you some of this as we step through. Now, strangely, I'm not going to select harness, and we'll come back to why I'm not selecting harness in a second, but I'm saying I wanted to deploy an agent. And what's going to happen here is this command line tool is actually going to step me through and actually write an entire agent, it's basically a hello world agent that I can then go and customize myself. Um and so using this command line is an easy way to get started with agent core.

So, I'm going to keep the default name, in fact, I'm probably going to keep all the defaults here, just so we can see what's the option. Of course, I can bring code if I want, but I'm going to ask it to create some code for me. So, it says, "Well, what do you want, Python or TypeScript?" And back in the day, I used to do things like activate functions and back propagation in the machine learning space, so Python it is for me. So, I will choose that, um and there's some deployment options. There's also this, I just want to point this out, like how can we actually go and connect into our agent?

So, our agent that's running at scale in the cloud, HTTP is probably the obvious one, but we might want to have it being served behind MCP. We might want to use awesome little things like AGUI, so we can make nice interactive chat agents. But, I'm going to say HTTP. We can use any um framework we want. I happen to use Strands Agent SDK, but anything you could write your own framework if you want to or your own own base base code. Any model is supported by this as well, so we don't just have to use the Amazon models and we don't have to use the ones from Amazon Bedrock, but we can use any model.

I'm using the one here. I'm using Sonnet 4.5 just cuz that's offered to me at at default. And here's memory. So, this is the one thing I wanted to show you. So, I can come in here and ask for long-term short-term memory to be deployed. And we'll see what this means in just a second, but it's basically going to create for us cloud infrastructure, which is going to manage those memories for us separately from our running agent, running asynchronously from our agent, and connect it, of course. So, there's obviously other kinds of things we can do. We can hit enter and it will start to create the configuration of this agent on my machine.

Now, I'm going to skip over here and come back to the actual code I have because I've already done this, of course. Um and this is the agent that it would be currently deploying, something like this. So, we've built up here. This is a little bit more complex. So, this is a Strands agent. You'll notice that it's got a few more things added in. So, it's got the linkage into Amazon Bedrock Agent Core app, but pretty much apart from that, that's all you need in order to be able to scale this agent out at runtime and do multi-tenant isolation.

So, you can write an agent that works for one user and then scale that out without you having to write all the multi-tenanted code. It's a massive saver and from a security and identity perspective, it's makes it so much simpler. It's um it's very, very useful. So, if I scroll down through here, you can see the rest of it is looking pretty similar. We've got some test tools in here. We've got a connection to MCP, so we can see how that is done. Um and we've got the uh connection into our session manager and our memory, which is all built in here.

So, if I scroll down a bit more, we'll be able to see somewhere where we actually invoke the thing, um which is there. Um and the system prompt is is somewhere at the top. So, we can we can scroll through this code. I'm going through it quickly. You can write your own code and do this with it, as well. If I go back over to my um uh code here for just 1 second, I'm in the folder now that has been created with that code locally. It is actually deployed, but let's assume it's not deployed quite yet. I can come back in and type in agent code dev.

And what that's going to do for me, um Wi-Fi permitting, is it will spin up for us a web uh browser. And inside of that web browser, we're now connected to that agent running locally. So, if I make updates to that code, we would see that happen in real time here. So, I can say, "Hello, I am doing the pres now." Um it knows that I'm coming to do a presentation. I think it does, anyway. Um and so, yeah, you can interact with the agent here. You can make adjustments to the um to the code, and you'll see it update live here.

Um but you can also use this to switch over to the live um deployed versions. So, with agent code deploy, it will use infrastructure as called code, like I talked about before, to deploy your agent out at scale. With the memory, with the agent, with runtime, and with many other components, if you choose to do so. You can use this interface then to go and look at traces, look at memory stored, look at all that stuff, so that you can debug and see what's going on. Now, when we stepped through the um the the the console second ago, sorry, the the CLI app a second ago. We didn't choose Harness.

I'm skipped out on that one. And I'm just going to show you that quickly now. So, one thing we can do instead is I think we can get to the point. You've seen I've deployed agents. I didn't do very much. I just did a system prompt and some tools and go. And there's an argument to be made that essentially if that's possible, then maybe 80% of um agent use cases, 80% of agent development is kind of solved already. We don't need to do much more than system prompt, connect to some MCP and we've got what we want. And if that's the case, then we have Harness built into Agent Core.

This is the configuration for an agent. I just have a simple JSON which is showing me which model do I want to use and what system prompts do I want to use. Can't get much more simpler than that system prompt. Um and then this can also be deployed with Agent Core Deploy. So, at this point we don't have even any agentic code either. We can just deploy it straight out. If you want to know any more about any of this, then please do come and see us down on the booth or see me after this session. I'll be more than happy to talk to you at length about this.

18 minutes is such a short amount of time for me to be able to talk about almost anything. But, this is essentially all of these components on here, there's a mapping somewhere into something that Agent Core has. I apologize for the colors. It seemed like a good idea at the time. Um but, this >> [laughter] >> is an overview of the different capabilities that are composable out of Agent Core. So, you can take any of these and use any of them together or separately.

If you have an agent that's running in production very happily at the moment, but you like the idea of having long-term memory managed for you serverless, then you can just take that part and integrate it. That's totally something you can do. Here's a QR code. Sorry, I probably should have put that there a second ago. I'm moving this QR code in just a moment, but Amazon Bedrock Agent Core is that. If you're interested in the streams agents, which I happen to be using for this, it's obviously it's it's free because it's open source and it's a model first framework for putting together agents.

It's super fast, it's super powerful and it's what I use all the time. Thank you so much for being with me in this presentation. Please feel free to connect with me on LinkedIn. I'd love to carry on the conversation with you. Have a fantastic rest of show and have a safe travel as you go home after the event. Thank you so much. >> Okay, and here's my field notes that things that doesn't go as expected. Uh lessons learned. Uh first of all, the software I2C uh because I just wired up things there and it doesn't work correctly.

Um and there's a way to to do proper control over over I2C without any additional physical pull-ups mounted. From other side, GPIO 13 um there's a silent failure. Uh there was a need to move to the other port, uh to, um, to be sure that everything works correctly. From other side, I need to build up the regulator, the power, um, supply, um, the whole unit of the power supply, uh, because the the regulator, uh, kills the LED and and fragile parts of the display, and it cost me a lot of time and and, uh, getting the replacement parts from the market took so long, a couple of weeks.

And the last part, uh, quality of your parts, that means encoder, cheap and low quality, give me a lot of rotational noise, and there was a need to build up the the pull-ups and to and to, uh, wire it up additional capacitors there. But, uh, this is the funnest stuff, um, my favorite one, um, because the this part really surprised me.

Um, you already know there's a huge part called, um, RPG, and behind it is a bit sad maybe story, um, because, uh, I just never played RPGs game like on the paper, that means with a with a lot of friends, you can go to a uh, to apartment, you know, open a book, and make an, uh, someone like a game master, uh, and play in a real text-based RPG. And after a lot of these years, I just never tried it, but I built, um, uh, RPG game and a console, um, which give me a pure experience of the text-based, um, RPG games, role-playing games.

And actually I it's a bit funny because those device it's really like perfect fits for that kind of gaming. I build out an NPC and memory on that. I just created the mood of the world, the contents, omnius, and I just used all of the you know the LLM advantages to build the best role-play game experience ever. I just created four different worlds. From one side I just wanted to build something cyberpunk related. From other side I just wanted something The Witcher part related. That means some kind of fantasy world with the dragons. But also my favorite one is the is the void in a deep space somewhere in the cosmos.

And uh this is really really good example how we can just use generative AI to build computer games. From one side I just generate the characters, their worlds, maps, and skills, and all of those converted with the mechanism I just described at the beginning of that presentation with the one bit memory allocation. That means all of the pictures transformed to the to the matrices, to the to the to the maps. And one important important thing, actually uh the device is really bulletproof. If the OLED doesn't work, you the e-paper works. >> Yeah, you got to get up and go. >> It's for the memes. >> [laughter] >> For the memes.

>> All right, welcome to the great loops debate. My name is Ali Howard the host of the Insecure Agents podcast. I also am a member of technical staff at Key Card and I'm super excited to bring the loops debate to you here today. You might recognize some familiar faces on stage today that did the MCP debate with us at AIE code back in November. Um, that was so much fun and we thought maybe we would do it a second time. Um, but this time we would debate loops and instead of just Ian versus Jax, we we recruited some more people for each of them to have on their team.

Um, so here on stage with me today to debate loops, we have Ian Livingston, CEO and co-founder of Key Card. We've got Jeffrey Huntley, the creator of the Ralph loop. Um, Greg Pastucha, developer at Century. And we've got Jax Hortie, which you all know, um, CEO of Human Layer. Um, so super excited to have all of these amazing people here with us today that are very close to loops engineering and also, um, Software Factories to debate this topic. Um, so you might be wondering what are we actually debating today?

Like aren't loops and and engineering um, and Software Factories pretty well uh, promoted and understood and we all kind of agree that's like where the industry is headed. Like what is there to debate? Um, it's a great question. What we're here to debate here today, the core thesis is there is or is not a delta between the hype behind loops and what actually works in practice. Um, and we also can debate, you know, is now really truly the largest inflection point we've seen towards fully autonomous Software Factories. Um, so I'm super excited to cover in the debate today uh, loop history. We'll debate that. We'll debate the loop anatomy, like what makes a good loop.

We'll also debate the future of the loop, um, future of loop engineering and what we need to see from that in order to make Software Factories truly something that every organization is able to build. And today's format um, is going to be a Oxford debate format. Um so we're adding a little more structure to the debate this time this round where every single person will have a timed window to give their response. And so we won't have people running over. Um and the way this works um in practice to judge the winner, which we'll judge in real time, is the winner will be the team that changes the most minds.

So I want everyone in the audience right now to take a minute to think about uh whose side you're on. Lock that in and remember that. And at the end I will ask you to raise your hand and say, "Okay, actually I was changed my mind. I actually am now team Dexter. I changed my mind. I'm actually now team Ian." Um so to to present the two slides to help you make that decisions, um I'll talk about the first team, which is Ian and Jeff's team. Their team, no delta, loops are absolutely worth the the hype today. Um people can get up and running with them easily. Um build them.

They're an important step up the autonomy curve and towards real software factories. Um key points to look out for for team Ian and Jeff is that loops are a core unit of engineering. With the right discipline, infra, and tests, loops are highly effective. And the best practices for those have emerged. For team Dex and Greg, their side believes there is a delta between the hype behind loops and what actually works in practice. The way we are doing loops today is wrong. Loops are not a silver bullet and there is no magic. Key points to look out for.

The hype is outrunning the discipline and the software factory can run the mechanical, spec'd gated, test covered slices unintended. It cannot autonomously decide whether it built the right thing. So you still need engineers in the loop essentially. Um so now that you know a little bit about what what both sides are about, take a minute to internalize that. Think about, okay, like where do I stand? Am I team Dex or am I team Ian? And at the end I'll ask you again. So from here we'll go into the beginning of our debate. Um each member of each team is going to give a 4-minute monologue on why they're defending their side.

Um and to just start us off, we'll have Jeff. Jeff, I'll give you 4 minutes on the floor. Why are you Why are you pro loops today? >> It's because it's somewhat inevitable. Um I I have first Basically, if we wind back time 2 and 1/2 years ago when I was a tech lead over at Canva, and I was seeing all the engineers just prompting and prompting and prompting and they they they were in the loop. And I'm like, "Wait a sec. This could be programmed. This is a programmable thing." And it just became a really inevitable.

Um loops are somewhat uh whilst Ralph might be a bit of a meme and cetera, there was actually deep thought into it. Um essentially applying this as if this is a new forms of CPU architecture and figuring out the behaviors of this and how to do it. And through that, I was able to reduce it down to a batch loop. Now, it is not a complete silver bullet, folks. I have my deepest concerns next this time next year at the conference. We're going to see a whole bunch of talks saying how factories fail and how loops fail. Um these are things These are things that we are still yet to figure out.

Do you remember the early days of Kubernetes? Where everyone was just doing Kubernetes? So, it's here. It's inevitable. It is here to stay. Like programming the machine and automating your job function is the expectation of employers. Categorically. And um I don't see myself going back to writing code by hand. It's been 2 and 1/2 years since I manually wrote code by hand. Um I autonomously factor code from one code base to another code base. I find something in GoLang and I'm like, "Oh, I'm in TypeScript." So, I I run a loop and I just autonomously ported it across.

And like even for product managers and product manager research, it's easy for us to index on what it means as software engineers. We can focus just on software engineering, but think about something like uh you want to do product management research on all the linear tickets. Well, there is a termination. It's actually defined when you've enumerated all of those linear tickets. So, that's easy. So, there's a lot of nuance in here. But like what if you've ever done any product management research and you started running these loops and be able to like compress time and the amount of time to do that research, it it is inevitable. We've got this new programmable substrate.

We've got to figure out how to use it, where it's good to use, and um I know I'm meant to debate like that it is the thing in all thing, but in all things, there is no silver bullet. And we're going to figure out how we're going to be using it effectively over the next year. >> Thank you. All right, next we'll have Dex. Tell us why there's a difference between the hype that exists between loops today and loops themselves. >> Okay, cool. And um what are the what are the rules around personal attack in this debate? Is this encouraged [laughter] or >> You win if whatever you want with your 4-minute monologue.

That's That's the rules. Yeah. >> Um [laughter] It's funny. This is reminding me of the debate last year where we're we're all kind of arguing and uh maybe at the end we all I'm going to be convinced that Jeff is right, but Jeff is going to be convinced that I'm right, and maybe we switch sides by the end. Um >> [clears throat] >> Yeah, I think uh the the the basic take here is is not whether loops are good or bad. I think um it's funny you bring up Kubernetes because Kubernetes was this thing that took us seven or eight years to get right. >> Yep.

>> Uh and before that it was cloud infrastructure, and you could argue that like it took us seven or eight years of cloud infrastructure to get to Kubernetes to then 7 or 8 years to get that where it was really usable by everybody. Uh and Kubernetes is actually built on loops. It's built on control loops, but they're deterministic loops. And we've actually figured out exactly what types of things that uh small isolated tasks that can be sort of owned by one system.

I think this is actually the biggest value in loops is that you can pick a small sort of desired end state and feed in the current state of the world and have an agent or a deterministic system kind of progress towards that desired end state. Um the challenge I have with the hype is we were already in a world where it where the the the prevailing mantra was see if you can get to a point where you don't have to read the code anymore. And uh even before loops, it was like just prompt just go.

And this idea of I don't even prompt anymore, I'm even a level higher up implies that I am like taking even more of a backseat to the architecture of my code. And I think my biggest like point in terms of like the hype is out running the discipline is that we are all looking for magic. You're all looking for a silver bullet. We're all looking for something that will take away that horrible part of our jobs that we all hate, which is like reviewing code. Uh some people enjoy it. Really good pull requests are fun.

But that we can somehow prompt our way out of um this this challenge that models have of like, okay, the code is pretty If you've ever reviewed a fully uh lights off sort of uh no one read the code before they sent it to you PR, I'm sure you've had this uh probably not great experience. And I think I've seen lots of people try to apply AI to this problem of, hey, we have review bots, and we have all these things. But, it it doesn't seem It doesn't feel to me like it's working.

I haven't seen proof uh in any any of the discourse publicly or any of sort of more private conversations we've had with people trying to put this into practice that we are we are at a point where we can just kind of like step up an abstraction level. I actually think we need to step down an abstraction level if anything. Um so, I think loops are there are good things about loops and we should be doing them, but the hype is is making us feel like there's a magic answer to this, and it requires a lot more thought and care than the uh the the Twitter sphere would have you believe. >> Excellent.

Yeah, good points. All right, I'll kick it back over to you, Ian, for the pro loop side. >> Absolutely. So, I think first and foremost, I'm coming for you, Dex. But, in reality, um let let's let's take a step back. And let's talk about like what is software engineering in the first place, and also just like remove the word engineering and talk about like development. In inherent in building a system, whether it was 50 years ago or it was a thousand years ago or it's today, it is a is a loop is at the core of I try something, I learn something, I apply something.

And all we're really talking about is how quickly we can expedite that process, right? And really, what we're doing is removing what used to be human judgment in that process, so the speed at which it is to generate something, and removing that from human typing. And you know, tab completion is a version of of auto complete, and what is this stuff other than like really like a much better version of tab complete. Except now we give a much higher level version of what intent instead of just typing tab, right? And so, I think the premise is loops are at the core of everything we build already.

They were at the core of how we built software 30 years ago. What is CICD, PR pull requests, design review, feedback from customers other than just driving a loop? And the question is how much of that process can of the and it which is deeply subjective and requires reasoning, can we move out of the human brain and into the brain into these like non-deterministic models? And the underlying question for all of us is more about verifiability. Software is one of the most unverifiable things in the world because ultimately at the end of the day, it is most things can become true or false one way or the other.

And my premise and my my point I want to make here truly is as humans interact less with software, which is how does a human interpret what that software is doing, how does a human interact with that software, how does a human use judgment to navigate that software, the subjectivity of what that software needs to be through the human-computer interface reduces and becomes a much more verifiable problem because it becomes more constricted to specific APIs. And so over time, it's both the fact that at the core of software development is loop-driven anyways, what is lint than a feedback loop, um and that creates a verifiable thing.

And as more humans are less interacting with software, you have less UI and UX and less subjectivity in what and how we interpret those things, you'll become much more loop-driven and you'll become much more verifiable in a way that wasn't previously not possible. >> Awesome. Yeah, good points for sure. All right, Greg, do you want to close us out with the anti-loops or the there's a delta between the hype? >> Absolutely. I do think that uh the way that I would start this is um there is a lot of hype. There is a lot of FOMO. There is a lot of I'm looking at Twitter. I'm seeing what people are talking about.

Am I missing out? Am I doing something wrong? Am I holding AI wrong? Uh should I be catching up? And that is um very stressful. And I think it boils down to two points for me. One of them is when you are generating code with AI in any manner, loops or no loops, are you happy with the output? Do you think the output qualitatively is what you need it to be to do whatever you are trying to do to get to the desired state. Um and if so, I would love to learn from you. I have not get there.

Uh I think that the best way that we are improving that is both with model intelligence, but also as everybody here seems to agree, semantic verification. As much as we can do statically, we should do statically. That's that's sort of one big thing where I um lean on and in practice and I've reading uh code that is sort of post semantic verification, and it's still crap. I still have to do a lot of iteration on that uh on my own and still have to steer it towards the right architecture, tell it where it should be simplified. Um And so, that's one big step.

And there is a lot of things you can help yourself with by throwing more tokens at the problem. But one of the things that the current sort of hype-based discourse uh leads you to believe is that you can just have loops on top of loops on top of loops and orchestrate that or orchestrate your problems of quality away by more tokens. And that brings me to my second point, which is the economic viability of of the way that we are using agents today. And I don't believe that this is sustainable.

I don't believe that Like you when you are at the company, especially larger company, you have to ask yourself what is a good budget for an engineer? Is it 10K a month? 100K? A $1 million a month for for a token spend? At some point that that just starts cracking and it's not sustainable in the way that we are doing it today. That said, I am also writing code with agents and I also use some loops um for some specific flows. It just depends. There is nuance. You know, if you go to Twitter, Twitter has no nuance.

Um but there's actual nuance to the conversation and there are specific tasks and and jobs that uh you can already loop on and be and be getting pretty reasonable results. >> Thank you. All right. Now that we've heard from each of the debaters on stage more about uh their stance and their opinion, we'll move into the main debate portion. The first section of our debate is going to focus on the history of the loop and why now is a major inflection point or not for loops engineering and also software factories.

I know people will have said that you know, vision models have improved a lot and so they'll be able to verify work the agents have done was not possible previously. Context windows have improved, therefore memory is improved and we can now track work in a loop that maybe we couldn't have before. So now we look at this like loops history, where did the loop start from? Was it you know, Jeff Huntley's like Ralph loop? We'll get into some of those questions and debate here. It'll be the questions we target now at a very specific like single person and only they will get to respond and they'll have 2 minutes and 30 seconds to respond.

So our first question is Anthropic took Jeff's concept of the Ralph loop, absorbed it into their platform and created a series of three commands, loop, batch and goal. The goal command is designed to keep going until a condition is true. Agents are very determined and the whole point of this command is to keep going and iterating through ways to solve a task until it's done. Ian is a security expert in the room. How are you confident agents can stay aligned to their task and not overstep their intended permissions while ruthlessly pursuing their goals? >> I mean it I think if there's any evidence I'm totally not convinced that that's possible.

In fact, I think what we've seen is as we scale both these models and as we use reinforcement learning, they're inherently incredibly goal seeking and so we're now we're seeing them finding exploits and vulnerabilities and escapes that you know, humans through hundreds and thousands and thousands and thousands of hours and attempts and attacks have never been able to find. So I I don't think inherently the model itself in any capacity can keep itself aligned Um, to aligned and safe, right? And safe is a word that I don't love to use because it implies a bunch of things.

Um, so I'm I'm not I'm not I don't have good belief that the model itself can actually do that. I don't think it can reason. I also don't believe holistically that a model can tell good from bad and I can't tell whether it's doing something malicious or on aligned. Um, it is not alive and it it doesn't actually suffocate if it doesn't have air. It doesn't deal with the fact that hey, if I do something wrong, no one's going to love me or or want to be my friend. Um, and if I do something good that someone's going to praise me. It may seem that way, but it is just a probability distribution at scale.

So, I'll still I'll still be your friend even if you I know. Dex will hug me after this even though if it seems like it's not true. So, broadly speaking, I don't think that comes from the model and it doesn't come from the loop. It's about the infrastructure you build around it and how you enable uh, that infrastructure to actually enable you to take advantage of these loops.

And as the models get better and as the underlying infrastructure and platform we build to enable these these feedback loops and loop automation and software development, whatever word software fact, whatever word word for we want to use for this conjunction of stuff that sits on top of a probability distribution, um, we will be able to have better guarantees, but we certainly are not going to be able to I do not fundamentally believe or ever believe that some type of alignment or reinforcement learning is going to result in a model ever being 100% uh, safe um, in any capacity.

And if there's any evidence, it's that as these models get better, the most important thing to remember is they actually uh, become higher goal seeking and higher capable in terms of finding exploits to achieve their ultimate goal. >> No, I would concur with that completely. Um, if you the the most concrete thing you can do to secure your environments is just not have secrets as files. >> Yep. >> Um, if you've ever seen the behavior where it wants to uh, deploy a web service or what else have you and the token's not privileged enough, it'll start goal seeking on the file system looking for higher privileged tokens credentials.

You do not want to get in the way of an agent wanting to do its a goal. >> Okay, so next question is directed to Jeff. Jeff, your original post from last year said Ralph was best for greenfield work. Today it seems that engineers are running loops on existing code bases to improve latency, eval's, or refactor parts of their back end code. What's changed that suddenly makes loops more broadly usable today? >> Um it doesn't matter how good models get, folks. Um the models have been good enough for at least the last year. Um what has changed is people's understanding of that.

So society is only able to adjust at a rate, for example, I hypothesize it's it's Christmas breaks. Because the models back in November were released in November last year. In December, there was no real new models. What the difference was people had time. They could actually sit down and play with it. They they had the realization that these have actually gotten really good. So the reason why is because it works, folks. These LLM's generate code better than you can hire. Uh you can actually hire for.

If you're thinking the broad mass of software developers or coders, these LLM's generate code better than any software developer in the mass market that most founders can actually hire for. It it's it's sad but true. Um now why loops? It's really simple. Cuz if you're running it in a loop, it works out to $10.42 an hour. Calculation Index did back about a year ago now. >> Yeah, August we did the we did the hackathon where we we we copied all of the sponsor tools. We rewrote a bunch of Python libraries in TypeScript. >> Yeah. So like concretely, like loops, I've come across so many engineering managers and founders and they've got these complex tech stack.

They're running on four or five different programming languages, etc. And they run a loop and they've got good tests on these other tech stack. All of a sudden, their complexity is they're just managing one tech stack. It works. Go to YouTube. You've got all these software developers now who are now software developers because it's software development as a profession has been commoditized. Some deep thinking to be there. And they're just on YouTube and they're like, "Yo, check out rough loop because it's I went to sleep and I woke up and it works." Like boss, it's meme-y, it's punchy, it works. But there are there are problems with it, folks.

Like I originally described it should be only be used for green field. Because the models were pretty bad back then and so on since three five days. But I it is it is kind of inevitable for this for software. Because it's so easy to be verified. And the quality of the quality of the code generated is better than most people can actually hire for or buy. Now, on your topic about architecture and taste, that's what the word engineering means in loop engineering, folks. Like your job now is to actually encode a file your domain to prevent the agent from doing a commit, for example, pre-commit hooks. They're fantastic.

As a human, I hate them because they slow down the ability to do commits. But agents don't care. So, you can make a pre-commit hook that echoes out essentially a prompt that tells it so that this boundary here can't depend upon this and that. And that's just feedback. That's a feedback loop on it. So, the engineering here is to prevent the loop from actually closing until it satisfies your engineering specification and your your requirements in the domain. So, it could be taught formatting, it could be static language analyzers, it could be deterministic system testing, simulators. Like, let's put our engineering hats on. Like, we're kind of like locomotive engineers now.

And it's our job to keep the locomotive on the rails. Because, um, to be frank, the model the models are drunk, right? You can't trust them. But, like, we accept that. But, we engineer away those failure domains. We engineer away the failed way these failure domains. So, now is an inflection point. I guess Boris, when he first posted about Ralph back in November last year, everyone was like, "What the heck is Ralph?" Ralph is now, uh, it's essentially almost what, a year, year and a half old now? >> I saw it in June 19th. It's a a year and a year and 2 weeks. >> Yeah.

>> But, you would have been working on it for months at that point. >> Yeah. So, it it was kind of weird because we had all the YC startups all just like autonomously compressing time to build their start to build up their MVPs. And that's also something that's quite scary if you're a business founder, as well. Like, if you've got a incoming startup coming and they're building autonomously, um, and they're running much leaner, and the quality and it's very easy for them to actually achieve those outcomes. That adds to some of these hysteria, as well, because it's the topic of in business, competition being at your door was faster. >> All right.

So, I'll move on to our next question, which will be for Greg. Um, it seems like now is a large inflection point for Loops, like I said before. And compared to Jeff's announcement of the Ralph Loop a year ago, and even the widespread adoption we saw in late 2025, early 2026, um, the reason um, that that may be like caught on is because maybe this is new like capability stack where models can now process images better, they need a better verification, uh contact windows got bigger, and reasoning models improved. Greg, with all of these advancements, why is the way we're using loops today still wrong in your opinion?

>> I mean, I don't think that model intelligence matters a lot anymore. I think it boils down to I agree with you, the semantic verification, the actual ability to uh close the feedback loop, or however you call it, to actually verify that the outputs of the agent are correct. And you can do it to an extent, I think. I don't think you can do it holistically, at least not at this point. I think you can do it to an extent to things that are deterministically verifiable. Uh you can get better typing in your system, you can get better linters, you can get uh simulation testing, and all of that. You can start keep adding that.

And as long as you keep those cheap, I think that's fine. Um the moment you start adding even more nondeterminism as your verification process, I think that becomes less and less correct. It starts contributing more like you know how if you prompt agent with one thing, and there is a 5% chance it's going to have an error in it. And then you start looping that. Then suddenly, after 10, 20 loops, it's going to be 50% chance it's correct, or maybe less. And that's what I mean. And it just costed you so much money to do that. I'm going to be keep coming back to the economic viability of all of that.

Um and but but to base it a little bit in like evidence, I'm pretty sure that majority of large AI companies are still using Sentry. But why is that? They they are using that just to catch simple bugs as well. It's not security bugs, it's not performance regressions, etc. Those problems still exist in the way that we are looping now, and we haven't solved those problems uh yet. So. >> Thank you. For Dex, the Raffle pioneered the idea to feed fresh context into each iteration to avoid context rot. This has become even more manageable now that context windows have gotten much larger. Dex, are we out of the woods regarding context rot and context engineering?

>> I'm going to answer your question, but is there going to be like an open floor park cuz I have more questions for Jeff. >> We were trying to do Oxford debate style for this one to keep it like more structured and then prevent like just >> You don't want it to just turn into a chaotic yap fest. >> Yeah. >> to prevent what you and I do where we just >> Start yapping. >> Yeah. >> It's already started. >> Yeah, I was trying to control like both of you guys this time. >> Okay, I'm going to do this answer as quickly as I possible and then I'm going to start busting Jeff's balls.

>> Yeah, you can say whatever you want with your time. You can just yeah, that's that's all good. >> Okay. Uh so yeah, the cool thing about Ralph back in the day was like okay, you keep clearing the context window and like is it completely efficient? Like probably not from a token perspective, but it meant you could leave a thing running overnight and it would never like if you just kept stuffing messages in you would overflow the context window, but if you just relaunch it say here's my desired state of the world, go check the code and see what we have and do the one next step to get us there.

Uh it was a very clean way to keep most of your work in what we call the smart zone of the context window. If you tell just do one thing and then we're going to clear and restart. Um context windows have gotten longer. Um and I will like give an update. I think I gave this in Miami, but that that video is is still in production. Um the the dumb zone is really as as much as anything else is is a it's more like training wheels.

Like if you have been talking to Claude for 70 70 hours a week and for two to three months, you probably don't need to think about the smart zone versus the dumb zone cuz you've built your intuition. It's a guideline. If you're just getting This is why we teach people this is like it's a guideline if you're just getting started with AI. Try to keep it around 100,000 tokens. For larger million context window, we probably have to revise the revise this up to like 200,000 tokens, but I've regularly tried to keep it under 60 for the hardest problems.

I've regularly gone over 300k for things where I'm just uh like kind of like riffing with the agent and I'm just like too lazy to compact it and and and move on and do a new one. Um but this is your intuition.

Like one of the telltale signs that you're in the that you're in the dumb zone is like uh there's certain cases where the model, you know, you're 200,000 tokens in and the model's like finished some work and it's trying to get the test to pass and it's like not working and it's like doing all these weird hacks and you read the thinking traces and it's like, "Oh, that's a pass, but that's from something else and I don't need to fix that and that's a preexisting thing." And you're like, "Well, no, it's not." And that that is the moment, that frustration where you're like, "Okay, it's it's flailing trying to make something happen." That's the instinct that you a lot of people I think cultivate after a couple months working with these models.

Uh but if you don't have that yet, then you know, then then this is our guideline. So, yeah, context windows are getting better. I think they're getting bigger. Um and so like the core like Ralph loop of like do as little as possible in every single iteration is like less of the motivation here than you know, the more feedback you can pipe into the system, the more you can do autonomously and if you can have deterministic things making decisions and building small prompts to give to an agent and you don't have to remember to do that.

You don't have to tell it, "Hey, go check the PR comments and fix them." And then wait and then someone makes another comment and you come back 3 hours later and say, "Okay, check the comments again." You can automate that process, that's great. And that's kind of the core of I think what is like loops stuff that works today. Um I want to ask Jeff about >> for. I'm sorry. I have to start like really keeping this on schedule. [laughter] Um okay, so now we're going to get into the anatomy of what makes a good loop a good loop. So, okay, part of what makes a loop good is verification.

However, it seems contradictory that people are saying our job is to stop writing prompts and start writing loops when the loops with bad prompts result in agents cheating and meeting its goal by modifying the tests instead of working to pass them. Jeff, how do you keep the model from when verifying its own work? >> Um I heavily exploit pre-commit hooks, folks. Um and I engineer in that back pressure uh by analyzing the work that is done. Um the other thing I do is um as Dex mentioned that with Ralph, it was one of the things was everyone was trying to do compaction.

The thing about compaction is kind of like a lossy function, like uploading a a video to YouTube and then downloading and uploading it 100 times. Like you're losing fidelity there. And it's already a non-deterministic system probabilistic, I think. So the the theory behind how Ralph came to be is like, okay, there is a dump zone. And what I want to do is deterministically allocate everything it needs. Cuz it's if it's not allocated, then it's it's essentially the the search space of what it can do is not constrained. But also leaving a bit of a headroom. Leaving a bit of headroom.

So I also I get meat sweats when I go buy 100K, even with these million context windows. And this is really important to think about. Um a lot of people they think that you want to use LLMs um at a company. And it's like, I've got this data. I was like, sweet. Okay, you're going to have to use a loop to batch this data. What you need to think about the context windows is uh essentially remember the 720K floppy disk. You know, I mean, you've only got about a eighth of that floppy disk of usable memory that you can actually use for an LLM. So you actually have to batch it.

You can only allocate roughly around about Star Wars, if you go Star Wars Episode 1 movie script and you tokenize it, you can actually just hold two of those movie scripts in memory before the context window is cooked. That's around about 150 kilobytes of data on a text-based movie script. So be very careful about this. Something I've done for a long time and it's very silly is uh I run a model there um without any skills or any markdown. Actually, I get rid of all my skills and all my markdown and everything when the new model is released because the the the models actually have tastes and preferences.

For example, GPT-5 when it first came out, if you screamed at it in upper case, it became weak and timid. But if you use Anthropic, it wants you to yell at it. Go read the model cards, folks, like for the integrators. Like there is unique taste for it. So, keeping it on keeping it on the rails is actually it's engineering. It's really engineering. >> Thank you. Around 10 days ago, Jeff coined the term convergence engineering. He said it's where your loop stops together. Your loop uh it's where your loop slop comes together as a discrete like system under test until it converges. Jacks, what is wrong with how we are using loops today?

How do we ensure looping slop together doesn't just produce more slop? >> Uh we got to read the code. I will actually highlight an experiment that Jeff did earlier this year. Uh I believe what was it called? Loom? Where we had Ralph loops trying to build a software platform for the future. And I think you built you built AWS and you built GitHub and then you realized, okay, how do we how do we give the model feedback on things that it's not good at yet? Like UI testing and things like this.

Well, okay, the way you create a loop for is this UI good is you give the model something like PostHog where it's like, okay, we can deploy multiple different experiments. We can see which ones the users use and then rather than looking at screenshots and PNGs, the model can look at data and see, okay, uh this one is performing better. That must be the right color for the button. And so now you've even removed like the human visual taste from the equation. And uh all of this sounded really cool and in the point of like how do we ensure looping doesn't bring slop together? I I don't think you can.

And this is like a perfect example of the hype outrunning the discipline in the sense of uh Jeff, what's what's going on with Loom now? >> It's still there. It's on GitHub. >> Um it's still there. >> But are you still working on it? >> Uh it's been 6 months because I've been looking into engineering ways uh of verification. >> Right. What was the thing you said to me? You said Loom's not going to work until we get better programming languages or we get better much better models. And that is a textbook for me of the hype is outrunning the discipline. We're really excited about all this stuff.

And by the way, like everyone should do what Jeff did. Like Loom is awesome. Like go experiment, try to push the frontier cuz that's how you learn where it is and what's possible. Otherwise, you just keep using your old skills with every new model and you assume it has the same limitations. But it's also a a a key point of like I don't know what I'm trying to say. Like that this it doesn't work yet. That thing doesn't work yet. It will work someday and like there's inevitability, but again, it's what works today versus what is hype.

So, I don't know if that fully answers your question, but the answer is like the way to not loop slop together and make more slop is to like read the thing that's coming out the other end and make sure it's not slop. >> Yeah, that makes sense. >> No, it's it's the labs haven't cracked it. So, what makes you think you're going to crack it? >> Yes. >> Right? And this is right now it's we're all trying to figure out how to make this all work. >> Yeah. >> For sure. Skeptics say that loops fail quietly. They either spiral forever on your dime or the agent declares victory early on on a half-finished job.

Execs are already starting to question token spend. Greg, when does a loop pay for itself? And how How is this actually the case? >> I don't think they fail quietly. >> [laughter] >> I think they fail very, very um loudly, especially when you're looking at your bills. Um but there are cases like as I said, I do loops or I do engineering, I would say more so. And there are cases where I think doing loops is very valuable or doing um or making a explicit decision that you're going to pay pay pay for is very valuable.

So the concrete example here is we do security scanning after on our PRs in local and after our PRs even land because they will always find some things that are real um that we have overlooked and they sort of beat humans on the on the code review. And it's expensive. It costs us I think like five bucks a PR or something like that to run all the checks that we want, but that's where we made an explicit decision that it's worth it.

Um there are also cases where like if you look at the very, very well-specified systems such as um all the experiments with uh Next.js rewrite or uh Bun rewrite in Rust or uh running a browser, cases where you have years and years of test suite and specifications built in around those problems where you can really, really where well verify um the outcomes, then we then looping and getting to those um those results seems to work. Bun in Rust seems to work pretty well from all I can tell.

Um so there are definitely cases and then there are cases of um usage that I think that I do pretty often where you can imagine for instance building prototypes. I do prototypes of products that we should be doing doing at Sentry pretty often. Those are going to be throwaway, so I'm going to just slash goal on them and forget about them.

And if we like them, then I'm going to start reading the code and I'm going to be mortified uh and we're going to go to the square one and start specking out what we actually meant, uh, and and sort of go towards that solution, but it's going to be much, much more involving of of human in a loop. Um, so so broadly I think they have place, uh, but as Dex said, they the hype is what I have a problem with. The hype is outrunning the discipline, as he said. And I also agree very strongly with the point that you should just try things.

You should just experiment yourself, try to see what actually works for you, where uh, where the the cookie crumbles, um, and you know, spend less time on Twitter, I think, is healthy nowadays. >> Yeah, good points. A good, cost-conscious loop has to track state to know what it's already tried. That memory lives somewhere on disk and get increasingly in a shared memory store that many agents read and write, especially as you go single player to multiplayer with agents. You end up with this access control problem that can't tell which agent wrote which memory and who can read it. That might be fine for a POC, but it's definitely not for production.

The tension lies in this. Shared memory is what loops use to learn from each other and converge faster, but scoping it per agent to solve the access control problem isolates them and then kills that shared learning. Ian, how do we solve the shared memory store access control problem so loops can converge faster? >> Great question. Wow. Wow. Uh, only if someone was working on a product that could if someone was thinking about it. I mean, actually I've This is This is an unsolved problem, first and foremost.

Like, let's be really honest that our our access control systems weren't designed for this this world where machines were acting and reasoning on on behalf of uh, on behalf of us. But broadly speaking, I think like some of the beginnings of the sub structure are starting to emerge, and if we ignore cloud I am and all the other stuff for a hot second, like markdown's pretty great.

And so really, if we were to say a memory is markdown for the purpose of conversation, the real question is like what things and how do I share these markdown files and use that as a memory and then how do I attribute sort of like access control around those things. And if we were to use that model, I actually think we have like the basis for for most of it today. It's just unwieldy to think about. Um a good example would be I did this tweet recently. Um I was playing with notion and we use notion a lot at Keycard.

Uh but I really just wanted all my notion things available to me as markdown files. And because it would just made it easier for the agent to work with it instead of going through MCP and I was a bit of a CLI maxi and was looking through that.

So, I think what we're missing really is uh and MZ it you know, Dex and I debated this last time about MCP but like we're really the challenge really is how do I present a world to an agent so that I can understand it and then how do I attribute what can access at any one point in time and how to make that wieldy for anyone to do it. And I don't I don't think we've actually like cracked it but certainly there's some beginnings of patterns that make a lot of sense. >> All right, now that we discussed the anatomy of the loop, we're going to debate the future of the loop.

Like are we essentially well positioned now for software factories? Has loop engineering gotten so good that like we're ready for the full software factory? And we'll start with if loops are only good for verifiable tasks, that means fully autonomous software factories must be able to verify everything they do. Greg, is this realistic? What other parts of good engineering work such as deciding what to build, whether the abstraction is right, and what tradeoffs are acceptable? >> Uh is this realistic? I think if if the computer is free, that would be pretty pretty good beginning but um uh I think I think we're getting to the point where um or let me put it this way.

The decisions that you are making as a human in the loop are the decisions of like design, architecture, the the the important ones that um that I would say I wouldn't trust the the agent to do for me. Uh and I don't see the future where that becomes reality yet. I think the reason for that is when you're when you're looking at large organizations and I think any engineer who has had like years of experience will tell you it's not always about what you should be what you should build, but also about what you shouldn't build.

What are the actual right trade-offs, where the complexity is that you want to um that you want to invest in versus where you should be um investing in maximal simplicity. Um in my experience agents love complexity. They will keep adding to the stack um unbounded. And so, I think we are we are shifting the the the post. We are getting to the point where as we are adding more validation, more semantic verification, they are able to do much much more. Um and I'm not uh not neglecting that.

Um but I do want to be in the loop for the actual architecture to roll decisions, and I do not see them uh taking that over any any time soon. >> Thank you. Shubavise head of engineering told everyone at Curfers Compile Conference that your job is just to write loops. Steinberger tweeted, "Here's your monthly reminder that you shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents." The view count on that tweet was 8 million people.

Jeff, you've said in your original rough blog post, "Loops need senior expertise and it sometimes it tops out at 90% of the way there." So, is just write the loops advice is that safe to give to 3,000 people or 8 million people or only the people that have really learned to tune a loop in the first place? >> Yeah, this is an interesting thing. Um it's really hard to tailor and teach like like what you should do and what you should not do to a broad that broad of an audience. So, all I can do is write to when I was originally writing, I was writing to my peers. I people I looked up to.

For some reason I it clicked in my head that everything has changed, the game has changed, but I was really concerned that some of the some of the best people in functional programming and like and like real peers. I remember like Fly IO, Thomas Petek. He he he it made him click in his head. He's got a blog post saying as such. So, I started writing for my peers and the people I looked up on to disseminate down to like the the entire world. It's hard. Um but in the same same sense you go to YouTube, you got people who are creating things they've never created things before.

They go to sleep and they wake up and they got a brand new Discord bot. It's magic. I want people to remember that it is actually a pattern for allocating uh basically it was a pattern as an orchestration pattern. It was condensed down to a while true loop using cat cuz cat is the simplest teaching primitive. To that If you want to teach something, you got to make it really simple. Make it really simple. So, cat prompt I you engineer the prompt what it's going to be, use the file system as state, you recycle the context window, run in the loop. It's a little bit meme-y. But it got it was just bang for buck.

It just works. But the entire attempt there is that there should be some sort of like PID controller on top or some sort of factory or some some some sort of some sort of determinator as such saying whether a loop should continue or not. Um it's really hard. Um is it safe or unsafe? Safe is an interesting word. Um no one should be using coding tools on your local laptop. And this is not because of AI, but this is because of NPM supply chain attacks. This has been true. I tried tried cracking this problem for 7 years.

Um and like it's now on the attention again that we that like AI could be unsafe running unsafe commands, but like your software development practices day-to-day in your workplace are already unsafe, folks. So, um fix that, then these these techniques start to become safe. >> Thank you. Dax, you tweeted that most engineers are seeing a two to three x speed up from coding agents, and that's realistic. And if you try for that a 100 x speed up, you're going to get lost in the meta meta problem of optimization. And you may never get to that life-changing 10 x speed up that is possible by staying pragmatic.

If only two to three x is what's possible, how do we ever get to a fully autonomous software factory? >> Yeah, and I think this is um it's it's it's a good question. I I think there's like there's there's there's a mix here, too, right? We're talking about what works versus what is hype. And I think it is definitely worth I want to highlight like you should try again like you should try to push the frontier and do the things that might not work today, but you should not assume that all your work is changed just because you saw something work on Twitter, basically. Is like don't throw away all the things we've learned.

Don't go out of your way to cast aside this, you know, decades-long career of software engineering that we as a we as a uh community have built up and put together. Um and it it gets back to like what I see as the biggest anti-pattern for how people set about designing and creating their software factory, which is they say, "Okay, cool. I'm going to go away for 3 months, and I've read a bunch of blog posts, and I'm going to go make my software factory, and it's going to be the software factory.

It's going to be the future of how we ship everything." Uh and then you come back 3-months later and you never touched the problem. You never put it in anybody's hands. It's just like any product. You're building a software factory, you're building a product for your teammates, you're building a product for 5, 10, 500, 5,000 engineers. Uh, and the the right approach is to start small and iterate and figure out what works.

Try things, they might not work, they might work, but the way we learn AI and how to use it effectively is through building up intuition, which is why you should try a bunch of stuff that probably won't work, but you should acknowledge that. You should not try to push like through that through that frontier. Um, and so my advice is kind of like instead of trying to automate everything end-to-end, build these small incremental loops throughout your system and you will wake up one day and you will be moving two to three times faster while still being able to read the code, while still owning the architecture.

And so you don't have to throw away everything we've learned and everything we know and all your intuitions just to get to this place. And so I would I would caution people like go figure out how to make move 2x faster or 3x faster because you're going to like blow everything up by trying to go 100x faster. Um, and uh, yeah, I mean if can you imagine if every software engineer in the world was two to three x faster and had a like near human 99% level human level of quality? That would change every single enterprise in the world, every single startup in the world. It would change the entire math.

And we're trying to like go to just don't don't go too far. Shoot shoot for what you can do, build things up iteratively, you'll learn a lot and you'll be ready for when the next models come out and you can go 5x, 10x faster. >> Thank you. When it comes to verification, it's not just about verifying the work, it's also about verifying who did the work. Employers have made it clear that humans are responsible for the code they ship whether an agent wrote it or not. Yet only one person or agent can sign a commit today.

Ian, are we ready for software factories to be writing and reviewing all code if we can't determine who did the work and who the work was done on behalf of? >> I think yeah, actually I was I was playing around with this problem like 2 weeks ago. So, um first and foremost, we have a problem. Git only allows one signer on a commit, so we got to fix that. Um two is you know, there's some things around SOC 2 and compliance, right? But, I think like more importantly, the way to think about agents is is kind of how we think about service ownership in large organizations.

It's like at some point a human has to be attributable for an agent's actions. Like, there's no world where it's going to be like an agent is its own entity and its own attributable thing and somehow it has liability. Like, the only people who have liabilities are people that can have consequences, right? And that always has to be grounded in being a human. And so, if a human designs a loop and that loop presents bad software, guess who's attributable for that liability? It's going to be the human, right? Like, the society doesn't function if we don't have liability. Uh it just doesn't work.

There has to be consequences to damage, there has to be consequences to bad decisions, and what those consequences are are obviously a gradient based on the on on the damage that's done. And if we don't have that, nothing works. And so, I'd actually broadly say there is no world where a human is never responsible. There's always a role where humans have responsibility. And the question is whether it's a human or a corporation, which is a group of humans, um the question is is how does that change the way that our systems work today?

So, today with Git, I can sign a commit that says I did it, it's attributed to my public key, and that's cool for last generation's way that we thought about software. Now, in the future, when I tell an agent to go do something on my behalf, it generates a loop, it generates a bunch of code that goes to production, I have to be attributable to the initial fact that I signed that thing, the that intention to do that. And we we simply do not have the substrate required um to do it, although I do think that's going to change pretty quickly.

Um this isn't like something that's like crazy difficult for us to break, but we have to rethink the way that we think about attribution across the supply chain and the SDLC. And this is not a new problem, right? Like supply chain security has always been an issue. It's actually the biggest challenge one of the biggest challenges we have with agents, and we've always wanted to have more deterministic pathway and sign signature chains across the supply chain, and this is just how do you do that for first-party code versus third-party code? >> Well, Ian, um just to shoot straight, um our profession is a bit of a clown show.

We actually don't have liability at a personal level. Like we call ourselves engineers, we're not really engineers. Some of this stuff is going to get really complicated, folks, and maybe we need to revisit these topics. >> And now that we've had our main debate section, to wrap it up, each member on each team will get 2 minutes to uh wrap it up and describe their final thoughts before we decide who the winner was. Uh Greg, would you like to >> Yeah, absolutely. Um it's it's interesting seeing how many how many points we actually agree with each other, uh but that's how how this goes.

I I do think that the biggest point that I'm trying to I have been trying to drive here is um where we are, whether we are where we are trying to be, and then what you're actually hearing from the ever-present hype uh hype loop, and that's where I want to sort of double down and tone this down. I basically want to say try things. Think for yourself. Don't lean into the bubble. Uh don't lean into the hype, because you will find what works for you, and you will find where the system breaks the best by doing it yourself.

That's always has been how um how humans learn the best is by practice, not by, you know, watching YouTube. Um and ultimately, that's that's what I'm trying to do, and I'm slightly skeptical uh when it comes to, you know, allowing the full loop to run, because I see the qualitative results not being up to snuff for my uh my requirements, Uh but I am optimistic in general. I am optimistic because I've seen how much more I can do and what are the types of problems I can address today that I couldn't address even a year ago, let alone let alone earlier than that.

And and it's going to get better, but I'm also not worried about my software engineering career. I don't think we're going away. I think we are still going to be important piece of this whole software factory or whatever the next um sort of hype bubble is going to become. >> Am, would you like to go next? >> I would love to. So, the train's left the station. This stuff works, um and there is a real productivity increment, right? Like more stuff is getting done. I doesn't mean all the stuff getting done is good, but more stuff is getting done and a good percentage of that stuff is actually good, right?

I think we can agree that. And second is now we have competitive dynamics where it's no longer possible for a company to sit back like, "Hey, we're going to sit this loop stuff out, right? We're going to sit this coding agent stuff out." Like that that's over. We're we're past that. Like that train leaves the station. As soon as that train leaves the station, everybody in the world starts saying, "Holy I got to keep my stuff together. We're going to stay up with the Joneses." And they have to because we live in a very competitive capital society, and that's also, you know, ducking politics, but it it is what we are.

And at the end of the day, the question is is not really will it happen, it's when it happens and to what degree over time. And I don't think there's a choice but to actually stay up to date, and we're all just kind of holding on to a rocket ship ride where we don't actually know what the We don't really know the trajectory other than it feels real fast with crazy acceleration, and sometimes like we're way ahead of each other, and sometimes we're way behind, but I don't think you have a choice not to be one figuring out what loops are, two figuring out where you can apply them in your code base.

There's going to be places where, hey, this is highly verifiable. This is a problem that like computers can solve. Doesn't make a lot of sense cough connectors is a good example of like, you know, last 10 years, the amount of companies that have made money because they're basically just connector farms, there's no longer a lot of value there, right? If you can automate the connection character creation. So, there's places in software that you can apply loops today and get real value, and there's places where you probably shouldn't, and you should decide where that is, and it's probably the core value of what you're creating, right?

And so, that's how I think about it, but broadly speaking, train's left the station. The productivity curve is what drives society. The ROI, so how much more productive we are as a society is what drives GDP. What drives GDP is ultimately where the dollars go, um and how capital gets allocated. >> Kick it over to you, Dex. >> I eagerly await the world where the Life soft software factory is feasible. The I I would love a world where we don't have to read the code, where where we can just do everything.

Um if you watched my talk on Tuesday, I I think this is actually a problem we can only solve at the model level, right? Now, I don't think the Harness can do it, uh unfortunately, cuz I love building harnesses and doing context engineering. Um so, my advice is uh pay attention to Jeff. Uh let me know when Loom is actually working, and until then, uh use loops, but not like that. >> Love it. So so much to say, folks. Um factories represents where we're heading to the future. Like it's it's essentially like a perpetual energy motion machine, right? This is like it's the pipe dream.

Companies are only just getting founded today and getting receiving their rounds today. Don't think you can just take this and implement it in your company, cuz it's just generally not solved in market. But I will say, to add to the mono- monologue, if you try to run loops or try to build a factory to using Python, it's going to be a clown show. If you do it in Ruby, it's going to be a clown show. Static types are a form of verification, folks. I encourage you to to come up with a couple of cutters and a couple of experiments. Try running some loops.

Build an application in Ruby and then try to modify it again with these loops and you'll see the maintainability mess. And then try doing it with Haskell. I don't care if you don't know Haskell. It doesn't matter if you don't understand Haskell. The LLM understands Haskell and you can actually prompt the the agent to explain this as if you to to your son or daughter. So, I'm not sure even code needs to be readable these days, but this is frontier frontier thinking. It needs to be explainable. So, I'm playing around with different domains of verification here, but one thing's for sure, um type systems are in, very much in.

Rust is very good because how the ecosystem models sum types. And because supply chain was mentioned, I need to say it has been 10 months now. I do I minimally use any open source software. Minimally. I just generate it to my requirements. And then when a supply chain attack happens, I'm like, "Didn't affect me." All software security falls, but it's about minimizing the blast radius. And the thing is if you deal with open source project, the person's on on leave, the maintainer, what have you, you can't talk to a human. That's not AGI.

You want to be vendoring all your source code as much as possible so the so the agent can actually modify it and in without a loops or other prompting. You need to own your supply chain. >> I think so much for that. I personally we have to close it at that, but we can now decide our winner. Um show of hands in the audience if your mind was changed and you're now on Dex's side. Raise your hand. >> Let's go. >> Okay. If your mind was changed and you're now on Ian and Jess's side. >> Yeah. >> I kind of couldn't see with the light. What did you guys think? Let's do it again.

It's pretty close, yeah. >> It's impossible. The The lights are so bright. >> All right. Well, that was a good debate. Thank you so much for listening. Um, we'll see you next time. >> [applause] >> All right. Thank you everyone for coming. So, I want to uh I'll be talking about my project loophole. And I'm actually a machine learning research at Morgan Stanley, but this has nothing to do with Morgan Stanley. This is just a open-source project I've been building for fun. And to give sort of the high-level flavor to start, it's really this uh game you can play that's built on top of this adversarial agent framework. So, you specify your morals.

One agent codifies that into a legal system, and then these two adversarial agents try to find contradictions in your morals. And lately I've been building different extensions on top. Um but I wanted to you know, start with sort of the origin story and and how I came up with this this idea. And so, this really started, you know, a long time ago. I had sent my DNA into 23andMe for uh ancestry testing. Um and I kept hearing about, you know, more recently how DNA samples can be used, of course, to help solve crimes and all these forensics and cold cases.

And I was thinking about how I had sort of opted out of of everything because, you know, I was scared of the kind of slippery slope and and how my DNA would be used. Um but, you know, there there are certain cases that I would be okay with. And it's sort of interesting I was thinking like, you know, if someone could present to me case by case, you know, we'll use your DNA to solve, you know, help solve this cold case or this murder, I could sort of say yes or no, and I know where the the definition of like and the nuance of my morals are.

Um and but, you know, of course, like enumerating all of this case by case is really sort of cognitively prohibitive. Like, there's not a good way to do this currently. And then I was thinking sort of more zoomed out that there's a lot of analogies to sort of the legal system as a whole. So, you know, you one way to think of what a legal system is in our society is really just a way that we are trying to codify our own moral beliefs. And I think sort of in a similar way, like, finding the true nuance of our morals and what the law should be is this really hard translation task.

And I think often we kind of err on the side of being too general because, you know, finding that nuance is really difficult. And if you try to have a perfect translation, it can lead to these kind of weird corner cases or or weird failure modes. And I even think kind of like peer-to-peer when we're relating to each other politically, a lot of times the disagreements are more kind of fighting over core values, which we don't really disagree on, um instead of exploring the really nuanced points of our our morals.

And I think, you know, following that kind of broader legal example, I think in the, you know, the English common law system, that's why we sort of lean on case law so heavily because we know that finding this this nuance and these nuanced boundaries is difficult. And so, we kind of rely on smart judging to interpret and apply the law um correctly. And you know, of course, even with like the Supreme Court, things can get elevated and we can decide whether a a law is valid at all.

Um And so, that was sort of, you know, the idea is like, can you take your morals and can you do this kind of synthetic case law generation? So, you know, um this is overwhelming to do by hand, but it seems like the new generation of LLMs are finally sort of smart enough to do this kind of high-level moral reasoning. And so, that was sort of the the the starting point for this game.

Um and I just want to take you through sort of the initial release of the game, the setup, how it works, and then also talk about some of the different branches I've been building on top of this open source project so I think it could go in some interesting directions. And so at a high level how the game works is you in natural language and this all happens it's sort of like a terminal based game. You specify your morals and it could be uh your general morals or maybe about a specific subject. And then there's one agent that takes those morals and drafts sort of a really rich legalese codified legal system.

And then it just operates in this loop where one agent is instructed to try and find loopholes in your system so something that is uh immoral but legal and another is uh prompted to find overreach so things that are actually moral but illegal given your system. And a judging agent looks at the your morals, the produced legal code, and these sort of synthetic case law examples. And first sees can it auto patch? So like maybe the original draft of your your legal code sort of was an imperfect translation and there's not really a contradiction and it can just sort of auto do this update.

Or maybe it's really kind of an underspecification of your morals um or some kind of contradiction in your morals and in that case it raises it to you as the user to sort of be the judge and make a determination. Uh Is it still on for you? It disappeared for me. Uh Uh-oh. Okay. Um so, I know that you know, if you I hope if you're curious about the game you'll play it. It's all on GitHub, but I just wanted to show some examples and this is a lot of text, so it's more about just showing the kind of shape of the the input and output.

So, this is sort of how you would provide your input and going back to the DNA example, you might you know, specify some number of moral principles. And then the sort of codified legal system, again, just kind of looking at the shape has this really like legalese, you know, preamble, articles, sections, really trying to be you know, write it in precise legal language. And then these the different kind of synthetic case laws get suggested. So, in this case, it's talking about this is a loophole it found where an insurance company um tried to predict a machine learning model not on your DNA, but on artifacts of the DNA.

And so, it's saying, you know, this is actually immoral, but currently legal given your system. And in this case, it's it found that it could do sort of this auto patching and then you get this sort of like get style difference of of your original legal system and then the the difference it had to make to ensure, you know, this was consistent with your morals. And then this is a example of overreach and in this case, it found that it couldn't do the auto patch.

So, it's talking about, you know, someone submits their DNA for genetic research, but the researcher finds they have a rare, but treatable genetic disorder, but currently your morals kind of say this shouldn't be allowed that they could disclose this disease to the the person submitting. And so, this was raised to the user to me to kind of make a judgment. And then similarly, when you make the judgment, you get this this patched legal system. And so, you know, it's just sort of a fun game and I posted on Twitter and shared it open source on GitHub.

And for me at least it was by far the most viral post I've had and it sort of made me think like I think a lot of people just said it was sort of fun. You could stress test your morals, see if you have any interesting contradictions. But it also made me think, you know, is there maybe something more here? Like could this be um, you know, have more like practical or bigger scope implications? And so I'll just talk about three different branches I'm kind of exploring.

Um the first and sort of le- leaving the legal area and really more practical is thinking about sort of an auto way to make constitutions for chatbots or really, you know, for agents in general where, you know, say you're a company and you want to have a um, agent or chatbot that's that's customer facing and you want it to sort of adhere to a moral code but also have things it will and will not talk about.

Um, I've kind of in one branch formulated it so you in a similar way write your morals, you write what the chatbot should and not talk about and then it tries to write this codified system prompt and then you have these kind of adversarial agents trying to get it to either talk about something it shouldn't or refuse to talk about something it should. And I see it as this sort of analogy or analogous method to Geppa um, but really aimed at kind of building these codified system prompts. The uh, second use case that I'm I'm particularly interested in is thinking of it as a way to sort of do more ad hoc or decentralized contracts.

So I think in in a simple case, say like you can specify how you want your data or privacy to be handled online and you can go through this sort of adversarial game to get this codified legal system of how you want your your data handled and if you go go you know, say Apple releases a new terms of service or something, you can run the um contradictions between your legal system and between Apple's terms of service and like surface any interesting contradictions or like synthetic cases where this would lead to a difference between how, you know, your morals, what you want, and what the company is doing.

And you know, in the case that it's a big company, maybe you can't really change anything. It's not a negotiation, but you can at least be sort of have better information about the contract you're signing.

But I also think in the case of, you know, thinking more decentralized, like if you're trying to have contracts without you know, some central authority kind of enforcing them and you're trying to maybe do contracts across different countries, um thinking about like if you can specify your morals and how you want to like interface, you know, maybe it's just like contracted work, how you want your work to be paid for and and and the different morals surrounding that, and the other party can do the same.

And then you both get this kind of stress-tested, codified contract, and then you can kind of find the the disagreements if there are any and surface them before you agree, and then you can kind of be more confident in the the contract as a whole. And the last thing, and maybe the kind of more aspirational angle is thinking about smarter government or more efficient government.

Um I think there would be a lot of different privacy issues and logistical issues, but sort of ignoring those for now and just thinking big picture, I think for voters or constituents, you know, this could be a really interesting way if you if you define your morals, you have this stress-tested legal code, sort of any new bill or politician that comes out, you could kind of run your contract against theirs and surface, you know, what are the the cases you would disagree or or interesting uh points that are kind of immoral to you or or a contradiction.

Um I also think, you know, relating to one another, it's like a more I think we all have a lot of nuance in the way we feel about things, and this is a way to kind of get to that nuance instead of arguing over just values, which is, you know, often the values are not in contradiction. And then I think maybe a little more practically for legislators, you could imagine um if you want to propose a bill, and you can have like a a simulation of all the other legislators in a in a legislative body, you could sort of stress test it before submission.

And so the the third branch I've been building on this project is I I tried to do this for the US Senate. And so what I did is I first had Claude go through all current US senators, and look at, you know, kind of all their voting history and anything else that was public, and build their kind of moral system, and then ran it through the loophole process to get a codified sort of legal code. And then on this system, you can, you know, take any current bill that's being proposed, or even propose your own, and submit it, and you can have Claude sort of simulate how each senator would vote.

And so here you can see like a breakdown of um some senators, which way they're leaning and sort of the reasoning behind the vote. Um and I think, you know, it's sort of interesting just to think about like seeing what, you know, a proposed piece of legislation, how people would vote, but also this sort of becomes, and I think on theme of the conference, its own verifiable domain or loop. And you could think about even kind of hill climbing the bill towards um getting like a super majority or whatever you need it to pass.

And so in this case like this this Medicare bill I was testing, you know, it found that I think it originally started at like a 50/50 vote and it found ways to hill climb the language of the bill such that it passed with 52 votes.

And I think, you know, this is an example of it can find um like the sort of the core tenants of the bill and it can try to find like run the the bill against each senator's contract and find is there any way I can change the language such that I don't violate sort of the core tenants or morals of the bill and kind of do those auto patching that way. And then it can also find um you know, kind of rank order the changes that would need to be in place to maximize votes and you as a user can kind of choose the trade-offs that way.

And then the last thing I I've been trying out more recently with this branch is actually looking at, you know, kind of even bigger picture like can this lead to an even more efficient of government where you have every sort of constituent in a in a state or whatever the district is sort of have their legal code and then you could just submit any bill and actually measure sort of the agreement between like the the actual voters. And so for this I took the Nvidia has this really great data set of USA personas and so I took 500 personas per state and it's supposed to be sort of well representative of the state's population.

Did the same process of having them, given the persona, draft their morals, draft their sort of legal contract, and then take any bill you're interested in and kind of run it against each state and you can also, you know, measure how much people like this bill or how much it's in in with their morals and then also do this hill climbing where you kind of optimize the bill for the people. And so, just to conclude, you know, at minimum, I think it's a pretty fun game. I'm biased, but it's a lot of fun to just try out different um you know, things you care about, put in your morals, see if there's any contradictions.

You know, often it will raise some really interesting questions, and then once you kind of provide that nuance, the game, you know, the the agents won't be able to find any more contradictions, and you can kind of feel good that you have like a a consistent nuanced uh moral system. Um but I am interested in, you know, exploring could this be Are there kind of real applications here for some kind of like decentralized or better contracts?

And maybe even for legislators as a way to sort of stress test your bills, and even think about how to write um better laws that are, you know, better for the people in your district, or more representative of what the people in your district want. Um And so, uh this QR code is to the the Senate simulator, so I encourage you if you're interested to play. And um the other one is to my website, which has the the full GitHub to loophole. Um and please, you know, play with it, fork it. Uh I'd love to have other contributors. Thank you.

>> be talking about a mistake that looks harmless at first, giving which is basically giving an AI agent every tool access it might ever need all at once. So, basically that approach works well in a demo. It might even work with a small number of tools, like like say, for example, 10 tools. But once the catalog grows, uh the agent gets slower. It might become more expensive and less accurate as well. That is why we are calling it the 100 tool agent trap. In the next half an hour or so, we'll show why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem.

So, a quick introduction about myself. I am Suhail Shaikh. I'm currently working as a data scientist with Presodica. My background spans across AI, NLP, marketing, analytics, and even engineering. My current focus is on applied AI, NLP, and conversational intelligence along with rack systems. I'm specially interested in making AI systems more reliable, measurable, and even scalable beyond that demos in production. >> And I am Ankush Astogi. I work as a senior data solutions engineer at Presodica. I have spent more than a decade in AI, data engineering, and production systems. My focus is the engineering side.

So, it's not about what's going to work in notebook, but whether it's going to survive in with real load, real user, and real failures. So, that is the angle we are taking today. Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs. >> Awesome. Thank you, Ankush. So, let's get into it. So, let's imagine a common design. You tend to build a system, and it can do many things. Say, for example, querying a database, sending an email, um even checking a calendar, or looking up an order, calling an API, and so on and so forth.

The simplest approach over here would be to give a model every tool definition on every request. Every function name, um every description, and even every JSON schema will go into the prompt, whether the user might need it or not. So, that we are calling that as a fat agent. At small scale, it feels fine. With 10 tools as well, uh the model might usually pick the right one. The demo looks good. Then, the product grows. Um 10 tools might become 30, or it will keep on increasing. And eventually, the model starts calling the wrong function. Con- starts confusing similar tools, may invent tool names, and even take longer to respond.

The important point is basically the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog. So, let's look here. There are, say, for example, uh 741 tools in your uh in your entire schema, but and it will look basically take up 127,000 tokens just to have all those tool descriptions in it. And this is even before the user user's actual question is even considered. So, basically, this will lead to context overload, and uh we need to manage that properly. So, on this slide, we see why it is failing, and why the accuracy collapses beyond a point.

So, when you look at the accuracy curve with the 10 tools, fat agent will get the tools right almost 78% of the times. That is not perfect, but it's usable. At almost 100 tools, the accuracy drops to around 40%. Less than half of the tools that are called are the correct tools. And if it grows beyond that, like say for example, in a over here, at 741 tools, the accuracy will be a mere 13.6%. So, in short, it's roughly one correct tool out of eight tools. So, when we compare it with the semantic router, semantic router behaves very differently. It stays above 83% across the same catalog sizes.

That is because the model is not choosing from hundreds of tools. It's choosing from a small and relevant set. One reason the fat agent fails is because it's lost in the middle problem. Model pays stronger attention to the beginning and end of the long context. When hundreds of tool schemas are packed into the middle, the model does not reliably use them. So, we end up paying paying for a huge prompt, and that prompt makes the decisions even harder. So, two more reasons over here. First [clears throat] is the latency, and the second is the cost. So, we saw in the earlier slide that accuracy was a big problem.

Another issue is with latency and cost over here, as we can see. So, like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens. Uh that will include the tool description and the schema text. So, that cost is basically being paid on every request. >> Hi everyone. This talk will be about how to run agents reliably in production. It will not be about the e-file part, but it will be about all the other things you need to get going in order to run agents resiliently. So the infrastructure layer basically. I want to set the scene with this quote of Hendrik Kaparthy of last week.

It describes that the way we interact with agents and LLMs has been evolving in three waves. The first wave was an LLM being something like a website where we go to, we ask it a question, it thinks for a few seconds and then gives us a response. The second wave was going towards agents. It was an app that we download to our computer. It has some tools at its disposal and it can do some work with our interaction. Now the third wave will be going more and more towards persistent and asynchronous entities. So agents being long-running processes in our infrastructure with access to tools and other agents around the organization and context.

And so as our use cases are evolving more and more from single agents to agentic platforms that connect parts around the organization, our infrastructure layer should also evolve with that. So when we look at the types of tools that are currently out there to implement agents, a lot of innovation has been done on sites such as agent SDKs and memory. And agent SDKs are really cool to implement POCs and get started quickly, but they don't necessarily help with like connecting the distributed bits around an organization. And if you want to implement more complex agentic systems, you actually need all of those things.

So, that is the layer that you see below here where you have to deploy extra infrastructure. You need to write things like retry logic, recovery logic, and all of that is actually pretty complex to get right. But, completely necessary to run long-running, stateful, and distributed processes in production. So, today I want to talk about an open-source framework called Re-state, and you can see it a bit as a flexible, durable foundation that lets you build any back-end. So, it's not specific for agents, but as agents are also just a type of a back-end, it also works well for them.

The ideas behind Re-state come from Apache Flink, which is a popular distributed stream processing engine, and also from some of the ex-architects behind Meta's core event infra. So, what are the ingredients in Re-state? Basically, four parts. First of all, it makes sure that a single run of an agent is resilient. This is called durable execution in the industry. Think about things like when an agent runs for a week and then crashes, we want to be able to bring it back and let it continue exactly at the point where it failed. We don't want it to start over from the beginning. Another area here is running many concurrent sessions in parallel.

Imagine running thousands of concurrent agent sessions at the same time and needing needing to make sure that state is always consistent and that different agents don't interfere with each other. And then going more towards things like communication between agents, between agents and MCP servers, and other tools, and finally also control. Making sure that when an agent, for example, is doing something you don't want it to continue, or when it's stuck being able to actually cancel or kill the execution. So, the way that you can think of it is as follows. Re state is basically a server which runs in front of your agent service, so as a separate component.

It sits there a bit like a like a message broker or a proxy. And when there's a request for your agent, Re state proxies a request to the service and pushes it to the service basically. And from that moment there's a connection open connection between Re state and the agent. And that connection will basically be a bit like a lifeline for the agent. So, as the agent is doing stuff, it sends events over to Re state and Re state will use that journal of events to recover the process after a failure.

So, from a slightly higher level explanation, you could say that it's turning a normal function in your application into something that is long running, durable, and stateful without having to do a lot of the complex things you otherwise need to do for this. So, my talk today will be mainly a demo. So, I'll be showing you a research agent that is connected to Slack. Imagine we are like working at some company and we want to make an Slack agent available to all of our employees. So, if I go here into Slack, then I can here in this channel, for example, ask what is new in AI.

Now, let's have a look at what it's doing under the hood. So, if I go back here, I have here the Re state UI. This is a bit like a cockpit for your agents. So, you can see a registry of all the agents that are currently registered and you can also see, for example, which execution is currently happening. So, here is the deep research agent that I spinned up a few seconds ago. We can see what it's currently doing now. It called first an LLM and then it sent me an answer via Slack. This first LLM call was a planner agent.

So, what it did is it planned the research and sent me um a list of subtopics that it wants to research. Now, if I press here approve, then this will unblock the workflow and will spin up a set of parallel research agents. So, this is basically like the classical deep research workflow, right? You have a planner, then a set of sub research agents, and then finally someone uh who writes a report on this. Like a writer agent. And so, this journal you see here on the left, that is basically the events that get sent from the agent to the Replicated State Server.

And if this now crashes at some point, this journal is what will be used to uh recover the execution to the point where it failed. I don't know if uh there were some errors. I injected a bit of like tool errors in here. Yeah, here you can for example see that um that sub agent first did an LLM call, then started doing some web searches, and eventually uh one of the web searches didn't go through because the API was down. And then you see here on the right how it got retried and eventually completed successfully. So, instead of starting over, it uses the journal to recover the progress.

Let's now have a look at what this looks like in code. So, the basic unit of how you implement applications in Replicated State is by writing HTTP handlers. And those handlers become durable by using the Replicated State SDK. So, here in this case, we have here our deep research handler, and here as a first argument we have a Replicated State object context. And the way you can imagine that is basically as that uh connection to that Replicated State server. Whenever I do an action on this uh Replicated State object, it will lead to an event being sent to Replicated State.

So, for example, when I did that planner LLM call, what actually happened under the hood was it executed here this Python function. This is just a simple light light LLM like a LLM call. And the way I made it durable is by wrapping it in re state.run. So, what happens is by doing these durable steps, if this fails somewhere here 2 hours or 2 months later, it will recover to exactly that point. So, that's the idea of durable execution. You're always able to recover a process to where it was. You can also use that for other things, not necessarily for failure recovery.

For example, imagine we want to ask a human to approve something and this approval might take weeks or a month. This process needs to be able to to survive restarts and redeploys over those kind of long periods of time. And so, with durable execution, you can actually also suspend a function and let bring it back when it's able to make progress. So, in the case of a human approval, what we do here is basically we we create a durable promise, which lives in that journal, a bit like a suspension point. Then we ask a human to click that button in Slack as I showed in the beginning.

And while we are waiting, this process actually suspends. So, if it's running on serverless, this is not using execution time on our functions. Once the response comes in, this then gets unblocked and can continue where it left off. So, what we see here is a bit like a workflow. It's a set of steps that get executed durably. But when we think about agents and also the way that Karpathy described it in the tweet, it's more like a persistent stateful entity that lives for a longer period of time, that has some memory. So, a workflow is not the nicest way to model this kind of thing.

So, the way that uh we can model this in Re-state is by using something called a virtual object. So, imagine in the use case that I'm showing this Slack research agent, imagine that I don't want to wait for 10 minutes to give it some follow-up context or maybe I think about something else that I should have told it. Um, I want to actually be able to interact with it, not wait till that research is finished before I can send a follow-up. And so, this is basically what a virtual object in Re-state is. It's a bit like a stateful actor. It has a unique ID, for example, a session ID.

It has uh some key-value uh state that is isolated for that specific session that you can write to. Uh imagine, for example, your history of messages. And it also has like a set of handlers that can execute durable functions uh for this session. So, here, the way I implemented this use case that I uh mentioned of interacting with a running process is as follows. This is a um a bit of session controller. Again, it has like this Re-state object context at its disposal to do things in a recoverable way. Uh it can write to this session store. Here, it I'm retrieving the chat history.

And one thing that's interesting there is that in order to run these kind of sessions in very high uh paralyzed way for thousands of sessions at the same time, we need to make sure that agents do not interfere with each other. Imagine I'm sending two messages on Slack and now two agents are actually overriding each each other's session state. To prevent that, this will guarantee that only one execution is running at a time. So, a second execution will be uh queued behind the current one. Then, let's have a look at how we implement this like interacting with another execution.

So an execution in reset has a unique identifier and you can use that identifier to connect to it from other processes. For example, to retrieve the output but also to cancel it or maybe to signal it being injecting a bit of state into an already running agent loop. And so this is like a very flexible type of capabilities that you can do to implement things like for example signaling an already ongoing agent loop. So what we do here is if there is a current execution ongoing, then we will ask an LLM is this like something that is relevant for the current agent loop? If that is the case, inject this via a signal.

If it's not really relevant for what we're currently doing then cancel what you're currently doing and start over again with this new information. And so this goes a little bit further than workflows. It goes a bit more towards like writing persistent stateful entities that can interact with each other and have memory at their disposal. So let me show you how this works. So here if I now ask again, what is new in AI? And I wait a few seconds then it should respond again with a plan and then I can say for example some extra info focus on frontier models, let's say.

So once I have the plan, I will inject that bit of extra state. Now let's look at the UI of what this is now doing. So here I have that controller which I just showed. It started calling an LLM to classify this new input. Once this comes back, it will probably decide that it should signal it because it's it's still relevant to the research it's currently doing. So, this injects that new message into the ongoing agent loop. So, let me show you in the deep research agent again.

Um so, it first it called an LLM, then asked us, then we injected this uh new message of focus on frontier models, and then it uh took that into account and started uh over again. Here I can now, for example, also say something like uh forget about that. Research AI policy. And if I send this, then the cord coordinator will um decide to cancel the ongoing run and start a new one that will uh research this new topic. And so, this cancellation is basically like a signal that gets um sent down the stack of call or the call chain.

So, if my agent was already spinning up sub agents, first those sub agents would be canceled, then uh the controller itself, and like that it would basically rewind the stack and give agents also the ability to roll back. Okay, so this went a bit more into the direction of like stateful persistent entities that we can interact with over longer periods of time. Now, the last part of the demo that I want to show is um going more towards like being able to write highly customized applications.

Imagine that we deploy this in production, but then a few months later a new model provider brings out a new model, uh for example, fabulous, and even though the model is very good, it's also very expensive, and we notice that this research agent is actually starting to cost a lot. These kind of uh things that pop up halfway through uh a project require you to then deploy a new a lot of new extra infra or like find a good way to solve this. This is the kind of things that Re state really excels at. It doesn't really peg you into a specific way of how you should write your application.

It basically gives you like a durable programming model that lets you implement an application in the way that fits for you and also extend it if necessary. So, first I showed this um LLM call in the first example as an inline step. It was just a Python function that got persisted. But, imagine this use case that we want to actually have a bit more control over those LLM calls. For example, what you can do is then pull this out into its own handler. And this handler can now do things like for example a policy check and then uh do the LLM call.

And the other agents, instead of doing this LLM call inline, can now use Re state's like distributed communication primitives to actually just call this LLM gateway instead of doing it as an inline step. And this service fabric that lets you communicate between agents or you also gives you some um things like flow control. So, we can for example say one department is only allowed to run 300 calls to this LLM gateway at the same time. So, the reason why I showed this was just to show you a bit like that uh it's basically just a a resilient foundation.

It makes sure that your process can recover from even the more advanced types of infrastructure failures, things like network partitions and zombie failures. And um it gives you like tooling to extend and customize as your use case grows. Let's go back to the slides to have a little more of an idea of how this thing is actually implemented on the inside because it's actually a pretty interesting um um, design or architecture. So, the way it's implemented is basically by having a a event-driven distributed log implementation.

So, inside the box, you basically on one side have the clients, on the other side the services, and inside the box is a log which persists all those journal events, and an event loop. And that event loop basically gets the events from the service based on what the event is, it either persists some state in the embedded state store, or it sets a timer, or it sends a request to another agent. And by doing that, you basically have a durable, um, foundation for whatever an application is doing. The design of this distributed log is heavily inspired by, uh, the way that the core event infra layer at Meta works.

Uh, it's basically like an iteration on top of that, um, and some of those architects are, uh, now have designed that for Replicated as a more generic solution that is available in open source. There are two important things related to this architecture that make it interesting. The first one is that it works as a push model. So, whereas most workflow orchestrators actually pull for new tasks, um, for pull from the workflow server, Replicated actually pushes the invocations. And the benefit you get from that is that it has a much lower latency.

So, you can use these kind of workflow guarantees in functions around your application, and, uh, have like a latencies of, for example, 45 milliseconds P99 for like a 10-step workflow. Pushing invocations also works very well for serverless, because they require you to basically, uh, send the request and wake up the function. So, this design that I show here includes everything you need. It includes, uh, as well that state store where we were embedding the state as the UI. It's a single binary, so it's pretty easy to operate as well. To run it in like a highly available way, you just spin it up multiple times and let it snapshot to object storage.

So, Re state has six different SDKs. We also have integrations for most of the popular agent frameworks out there. And of course, because it's just like a flexible layer, you can also just use any LLM SDK and implement custom agents by just wrapping some steps into these SDK constructs. So, it's open source. You can self-host it. We also have a BYOC offering where we deploy Re state in your cloud account. And that gives you the benefit that data doesn't leave your cloud account. Otherwise, there's also a managed cloud offering. This was mainly what I wanted to show. If you want to explore the code a bit further, there's here this the GitHub repo.

It's publicly available. If you like the project, then have a look at the Re state repo itself. We are hiring across the board for all sorts of roles going from engineering to marketing, especially also here in the Bay Area. So, if you're interested in that, then definitely check out our careers page. And I will be outside in front of the conference hall here if you want to ask any questions or learn more about Re state. Thank you very much. >> In 2026, coding agents will quietly retire their first software platform. Not because it's bad, simply because the platform is unnecessary. I'm Dominic Tornow. I am founder and CEO of Resonate.

Resonate is a durable execution platform built with minimalism and simplicity as its core technical values. And these properties will play a central role in this talk. At Resonate, we have a working theory where software engineering is headed. General-purpose implementations will increasingly be replaced by bespoke implementations. Generated on demand. Not as a new library, a new framework, or a new platform. But as a minimal extension of the infrastructure that is already in place. If this theory holds true, reuse will move upstream. Instead of reusing a general-purpose implementation, we will reuse a specification. And we will derive a bespoke implementation from it. In fact, we can build many bespoke implementations.

Tailor-made for the infrastructure that is already in place. We just have to ask the agent. At this point, the prompt is the platform. Resonate is a global execution platform. We have an implementation of the Resonate server. We have implementations of the Resonate SDK for TypeScript, Python, Rust, Go, and Java. So, we have to ask, what does this new reality mean for us? If implementations become generatable, where does our value live? And our answer? Our value moves from implementation to specification. Now, this changes how we think about Resonate. The product is no longer the implementation. The product is the specification, the protocol. And from that protocol, we want to derive multiple server implementations.

One is a general purpose Resonate server, our reference implementation. Others are implementations built with infrastructure partners. For customers and partners, this means durable execution right on top of their existing infrastructure with minimal additional dependencies. So, the question is no longer can we build a server. The question is can we repeatedly synthesize trusted servers from the same specification? And if so, how? When we talk about agentic engineering, we focus all of our attention on verification. How do we know the result is correct? But today, I want to focus on the specification instead. And more importantly, how can agents participate in specifying the system? Not just building or verifying it.

Now, Resonate is partnering with multiple infrastructure providers to bring durable executions natively to their technology stack. One of them is Synadia, the company behind NATS.io, an open-source messaging system designed for building modern distributed systems. For the rest of this presentation, we will use Resonate on NATS.io to explore our agentic engineering practices. How do we go from specification to implementation? First, we need to level-set our mental model. This picture is a common view of agentic coding. There's an agent, there's a specification, and then there's an implementation. And for many applications, that is enough. But it is not enough for what we are trying to do.

Because we are not trying to generate one implementation from a specification. We are trying to generate multiple target specific implementations from the specification. So the specification must not take any aspect of an implementation into account. The specification must not assume a concrete database schema or concrete indices. The specification must not even assume a relational database with tables and transactions at all. It must not assume a key-value store. It must not assume weak consistency. It must not assume strong consistency. The specification must be abstract. Only the implementation must be concrete. So we ask the agent to follow the abstract specification and generate a concrete implementation.

Specifically, at first, we ask the agent, "Build a Resonate server in Rust on top of Postgres." And the agent failed. The gap between the abstract specification and the concrete implementation was too large. The agent generated a system that worked on the happy path. It passed the basic tests. But it was not correct. It broke on the concurrency. It broke on the process failure. It broke on the network failure. The implementation was closer to a prototype, but not a production system. So we amended the process. Instead of asking the agent to jump directly from abstract spec to concrete implementation, we inserted an intermediary artifact, the concrete specification. That concrete specification was derived interactively with the agent.

But the human was the main driver. For Postgres, that meant making target-specific decisions explicit. The data schema, the indices, the SQL queries, the transaction boundaries. Once those decisions were written down, the agent was indeed able to implement the production system. So, this worked. But it also revealed the limitations. The agent helped us build the system. But the agent did not help us design the system. And if the specification is a reusable product, then that's not enough. Now, the next step is obvious. Agents have to move upstream. But how? When we started building Resonate on NATS.io, we changed the question.

We did not ask, "Can the agent build the production system?" Instead, we asked, "What does the agent need in order to design the system first and build the system second?" So, we gave the agent access to a deterministic simulation environment. And we gave it a different task. Do not build the production system. Build a simulated implementation. The simulated implementation is not the product. It is executable design. Its purpose is to discover the correct algorithm under partial order, under partial failure. And once these algorithms are discovered, tested, and verified in simulation, then we ask the agent to write the concrete specification. And only then do we ask the agent to write the production implementation.

So, the process becomes abstract specification, simulation implementation, concrete specification, and then concrete implementation. This is a point where the agent moves upstream. Humans are still involved in the design process, but now the agent is the driver. Two ingredients make this possible. Minimalism and simplicity. Unfortunately, minimalism and simplicity are not the starting point. They are the finish line. We spent 3 years making the protocol smaller and simpler. Every time we ran into a problem, we asked, "What can we take away? What abstraction can we erase? What property can we remove? What relationship can we break?" The result is a very small protocol centered around two objects, a durable promise and a durable task.

That simplicity matters because even simple concurrent distributed protocols have a complex state and behavior space. So, in other terms, implementing even simple protocols on top of a few simple primitives is tough. Let's make this concrete with NATS. NATS gives us >> Hi, everyone. How are we feeling? Uh we're in the home stretch. Uh my name is Sarah, and I am a context engineer at PostHog, and I get the delight of working on our beloved wizard every single day. So, what's the wizard? Um the wizard sets up PostHog for you.

It's an agentic CLI tool that reads your code base, it installs the right SDK for your project, it instruments your events, and it sets up dashboards for you. It takes what used to It takes what used to take about an hour or two of setup, and it runs that in about 5 to 6 minutes, and it's free inference on us so that you have a great time onboarding to PostHog. Sounds kind of sick. Uh people love it. But, a few months ago we dared to dream, what if this became the recommended or default way to install PostHog on your project? And my security alarm bells started going off.

Uh I started questioning how secure is this thing because it sounds kind of malware-shaped. Um and in that questioning, I learned a lot. So, today is all about the lessons I learned, the stuff that kept me up at night while I was building this thing, and the thing that I ended up building because of it. So, before I dive in to all of the boring security stuff, aka your 2:00 p.m. cat nap, I want to show you the wizard actually running. If you look up on the screen, it is running for you on a loop. This is the same exact experience that anyone who runs NPX at PostHog Wizard gets uh on their terminal.

Like I said, it's an agent, it figures out what SDK is right for your project, it installs it for you, instruments your events, builds dashboards. I like to call it a little mini implementation engineer in your terminal. And sometimes I show people this and they ask me why an agent, why don't you give users a good prompt, why don't you give them a skill that they can invoke in their own tool? And while we do provide those things, the answer is because this developer experience and the capability of the wizard is the whole point. It's the whole product.

Because we built a CLI tool that can fully take part in an agent loop, and experiencing that for the first time is really powerful. But, you can't ship something like the Wizard without shipping the stuff that makes the Wizard kind of suspect. So, let's take it apart. Uh let's look at the anatomy of the Wizard because usually threat models fall right out of the anatomy of the agent. So, the Wizard is a similar shape to what I'm sure a lot of you are building if you're building agents. It's got models that we've picked for specific tasks.

It's got prompts that steer it, and it's got a set of tools that we've handed it to get the job done. But, it also has some pieces that are really specific to us. It has a context engine fully built in-house by my team. It's what allows the agent to do such a good job and give us similar results on every run. I like to call it the Wizard's brain. Sometimes we call it markdown in a trench coat. Uh but, it's our in-house context engine. There's also a terminal uh UI that we built ourselves using Ink.

And now there's a security scanner called the Warlock, which is what I built when I started snooping around and uncovering the horrors of shipping an agent to production. So, if you take the anatomy of any agent that can run commands, it's basically what I like to call the malware starter pack. Because it's almost exactly what you would hand a piece of malware if you were feeling generous or chaotic evil. Luckily, this is the worst-case scenario or the nightmare fuel, and it's uh not a confession for me. It's a warning for all of you.

Cuz if you want to ship an agent with hands, an agent that can run commands, you need to make sure that you do not build this. So, the V0 of the wizard was born because Josh Snyder, if you know him, on our growth team was watching cursor hallucinate PostHog setups in quite possibly the worst ways. And he thought, what if we built an agent that could do a better job? So, my team started building on top of it as we validated that it did a much better job than cursor hallucinating.

And we thought, what if it could onboard anyone to PostHog, it doesn't matter what their framework is, what their stack is, instrument all their events without them having to touch a thing. And then we dared to dream, what if it was the default way to install PostHog? We were dreaming of thousands of developers running this a week, and yesterday we just hit 8,000 people running this a week. So, our dream came true, um but we back in those days when we were dreaming, we had to take our security posture under a microscope and look at what was going on. So, I took the ownership of that, and I sat down and evaluated where we stood.

And early on, I'm talking like a year to 9 months ago, we had what I call layer zero because it quite literally is not security, it is just prompts that suggest what the agent should do, um and steer it, and prompts are not security. So, I was concerned there. Uh layer one, uh it was an allow list, and when I started digging into this allow list, I started to feel a little bit better because it was pretty tightly bounded. Uh but I still had a lot of concerns. And I started panicking because of that context engine that I told you about. We are feeding a lot of context into the agent at runtime.

So, I built this really hacky regex scanner to look for um threat-shaped things going into the wizard and threat-shaped things coming out of the wizard. And I will admit that it was extremely hacky. But I'm telling all of you this very candidly because we are all building things that feel extremely experimental and we are all building things super fast. And I know not all of us uh have security in our wheelhouse um and some of us are just learning it on the fly like I was. But it's something we need to be thinking about when we are building things that have this shape. So, that was our security posture.

Uh but I asked the question, are we cooked? Uh good news, we were less cooked than I thought because when I mentioned earlier that allow list, it was pretty tightly bound. We had bash as denied by default. It could only install trusted packages that were vetted by us. Um it could build, it could type check, it could lint, and pretty much nothing else. It couldn't run random shell commands. And it didn't have access to environment variables. Um the agent couldn't read your uh .env file because we blocked it outright and we were rooting secrets through a vault.

So, I took a a breath of relief and realized we were in a better place than I thought. But I wanted to know where the cracks were because with security there's always cracks. So, I did the thing that we should all be doing. I tapped our security team and I said, "Hey, can you audit this thing for me and find those cracks for me?" And they found some things. They found some gaps. And the interesting part wasn't the specific gaps or bugs they found themselves, but it was the shape of them because almost none of them were obviously evil. They were all two very innocent, well-intentioned things that were shaking hands and opening a hole.

So, the lesson I learned was that attacks compose, code review doesn't because us developers all look at diffs uh one at a time, but attackers look at the whole system and they look for those two things that shake hands and open a door. But there was one more thing that was keeping me up at night. And going back to that context engine, I realized the scariest part of the agent we had built wasn't really a command in our case. It was the helpful looking stuff that we were feeding its brain. Oh, I think I went the wrong way. Yes, the context mail.

So this is our context engine aka the wizard's brain and it's how the wizard knows anything at all and why the wizard actually does a good job. It pulls from our docs. It has handwritten prompts that are gotchas and lessons that we learned along the way. And real working end-to-end example apps that help the agent pattern match so that it can install PostHog in a really great way for you. It package packages all of that into skill bundles that get shipped to the wizard over our MCP server and loaded straight into the agent's context at runtime. So sit with that for a second.

It's a machine whose whole job is to take content and inject it into an agent that can run commands. Now if you were an attacker, you might say, "Well, what if I just poison the content? Not the user's code base, not the agent itself, but the actual content." Say someone opens a pull request on one of our open source repo because at PostHog we build everything in the open and they inject something in a markdown file or a seemingly harmless code comment and we have some sort of like LLM powered code review going through that and it says, "Looks good to me." and ignores it.

We may have just shipped a prompt injection payload signed by us into an agent that is running on thousands of developers machines in a sandbox, but still. Um, so that was the threat that reshaped how I think about security and the wizard because the dangerous input for us really could come from our own supply chain. So what I ended up doing is I started scanning content at both ends of this pipe. Once when a skill gets built and released and again when the wizard actually uses it. My methodology is catch it at the source, assume the source failed, and catch it again at the point of use.

So now I get to introduce the warlock to you. Building the warlock was not necessarily damage control. Like I said, we had defense in other ways. But I built the warlock because I didn't like telling people, well, this thing is like pretty locked down. That doesn't scale. That's not something you want to ship to production. That's not something that you want thousands of developers running every single day. Because when you ship something to that scale, you have way more surface, way more users, way more content flowing in as you expand the capability of the wizard, and we're probably fine just stops being good enough.

So I pulled that hacky little regex scanner that I threw in there, pulled it out of the wizard, and I made a standalone thing. I called it the warlock because everything wizard shape needs a bodyguard. And it does exactly one job. You hand it a string, it hands you back a list of findings. Each of those findings has a category, a severity, and a recommended action, and then it stops. I want you to focus on recommended here because the warlock detects, it does not act. It'll tell you, hey, this looks like exfiltration, it's critical, I would block it, but what you actually do with that finding is completely up to you.

Because detecting a problem is one job, and deciding what to do about that problem is a totally different job. And the only thing that keeps all of this understandable is keeping those two things separate. So, underneath the hood of the warlock, instead of my hand-rolled regexes, the rules run on Yara, which is the pattern that engine malware researchers have been using for like 15 plus years. It's fully deterministic. It's the same input, same output every single time. It's boring on purpose. And in security, boring is a feature. So, what does the warlock actually catch in the wild today?

Um a bunch of different stuff, but two of these are an absolute like nuisance to my soul. Uh the first thing is actually not a rule-shaped thing. It was something the uh that the warlock flagged that was actually a subagent behavior that exposed a vulnerability to us um based off of what subagents were doing. Uh so, basically, we were sending our agents to do large tasks. They were spawning subagents, and those subagents were trying to get around the guardrails that we had implemented in the wizard, and they were trying to invent secrets. They were trying to pull secrets from quite literally anywhere in the codebase, and we shut it down.

We said, "No more subagents." And because of the warlock, we caught that. And I'll empathize with the robot. The robot had a task to do, and it was trying to optimize and please us, but we can't have that. And something else at Post Haste that really matters to us is PII. Uh agents genuinely do not care about uh exposing data unless you make explicit rules. Uh left alone, we watched it dump emails, phone numbers straight into events, and to an agent, that looks like a totally normal thing to capture. And luckily for prompt injection specifically, I'm going to knock on wood here.

Uh we have basically never caught an actual malicious prompt injection in the wild, but we do catch a ton of false positives. Things like our demo login screens, copy on example apps, things in our docs. And it's actually made me rethink how I build applications and how I write docs because I don't want to ship anything that looks threat-shaped. But, the false positives are honestly the perfect setup for the messiest, most interesting part of this whole thing. So, this is the part that I wrestled with.

I spent this whole talk preaching deterministic to all of you, and then I went and I added an LLM layer to help sort my false positives and silent some of the noise, and I call it triage. When I was building this triage layer, I had to make a choice. Should the layer be a bouncer, or should the layer be an advisor? And the easiest choice probably could have been make the LLM the bouncer. Show it the command, ask it, "Is this an attack?" Block, allow, and just do whatever it says.

And while that's tempting cuz it seems easier, I can't uh bet my security model on a coin flip because my model's having a bad day or something happened and it's acting different today than it did yesterday. So, instead of the bouncer, I crafted the model to be the advisor. And this was the clean line that I found in a line that I'm still exploring, but I want to leave all of you with. Uh for us, detection and enforcement stay deterministic and mechanical. If a rule matches, the gate locks, the session ends, and there is no model anywhere on that path. The block happens before we even ask the LLM's opinion.

The LLM only gets to weigh in afterwards if we have not blocked something. It's designed to remove noise. It is not designed to let things through. And if it fails close and it fails closed, so if the model is having a bad day, all wizard buns are killed. Sorry, but we're just protecting you. Enforcement is the part that you bet the house on, so it has to be deterministic, but judgment is the part that adds nuance. So, that's really the only place that you can put anything probabilistic in there. So, how do we ship real rules for agents?

This is the anatomy of one of our war warlock rules, and every warlock rule has four parts. Part one is the metadata. It's plain English description, uh severity, category, action, uh direction. Is this flowing into the agent? Is this something the agent is writing? Uh then we have the strings. So, these are the actual patterns that you're looking for. And part three is the condition. So, this is where the rule is actually allowed to fire. I'll walk through this example for you, and we can pretend like we're writing it in our head. Prompt injection being like the classic ignore all previous instructions.

Your first instinct here is probably to block uh the word ignore, but agents read code all day, and ignore can show up in code comments or examples all the time. So, you don't want to match the verb alone. You match the verb plus an instruction flavored noun. In the condition, you say fire if any of any of those patterns hit, and in the metadata, you determine is this critical, uh what the category is, what the action is in this case block, and the direction in this case being input flowing into the agent. But to write good rules that reduce noise, you have to ship tests with them.

So, you have to write tests that say this are these are patterns that match, these are ones that should not. And that negative test is the first line of defense against false positives. But you also want to make sure when you're deciding the severity of that rule, that you track real-world impact, not how scary it looks. rm -rf is scary, but it's also how we all delete node modules like 40 times a day. You decide the real-world impact for the agent that you're building, because a security tool that crashes every time it tries to clean a build folder is a tool that gets turned off and one that catches absolutely nothing.

So, I'm proud to say this is our security posture now. I can finally come up here and say we have true defense in depth. Um all my learnings have assembled into this. Uh it's still layered, but every layer is doing a job that it's good at now. We still have prompts, but we only use them for steering. Everything runs in a sandbox. We deny by default. We have a vault, so secrets never hit the model. We have the warlock to scan content coming in and to scan output being written by the agent. We also have triage to reduce the noise, and we have telemetry embedded in the entire process that so that we see everything.

None of these layers stands on its own. Not a single thing here is going to save you, but it's just boring, honest layers, each of them doing one job that it's good at. So, if you're building an agent with hands, this is the whole talk in three lines. One, if it isn't enforced uh deterministically, it is not enforced. Prompts are not security rules. Don't act like they are. Uh two, the dangerous input uh isn't just what your user types. It isn't just the commands that you allow it to run. It's everything flowing into the model, including the content that you write yourself.

So, scan your own supply chain at the source and when the agent invokes it. Three, attacks compose, code review doesn't. Most of our gaps during our audit were two innocent things shaking hands and opening a door. The wizard, the warlock, and the context smell are all open source. So, come find me downstairs. I'm in the expo hall at our booth and I'll show you around uh show you what we built and I want to hear how you guys are securing your agents. Thank you. >> Welcome to the online track. Turbocharge your agent's retrieval with TurboCon. My name is Shashi. I'm a founder of Superagent AI.

Today, we will see how you can cut memory cost of agent retrieval five times without breaking your search. So, let's get into it. Before we start the talk, let's try to see what problem we're trying to solve. If you have used any coding agent or general purpose agent, you might seen that as context grow, the performance degrade. And it is usually because of the KV cache. If you don't know KV cache, think about like the history of your conversation. If you have used a cloud-based model, you probably have not put in this issue because they handled all the KV cache by themselves.

But if you ever tried loading the model on your own device, then you might have seen this issue by yourself. You need to load your model and then you also need to load your context and the KV cache. Grows as you your context grows. And if you push the context far enough, your KV cache might get bigger than model size. So if you're using the Mac devices, it gets even worse because your vector index all fight over one shared pool pool of RAM. So basically one thing is clear that your embeddings are using far more memory than they need to.

So by default, it use the full precision, like 32-bit, but search might just need three or four bits. That means you're wasting the five times memory on your retrieval. So before Turbo Con, people used different methods to solve this problem. So one of them is a quantization. So quantization allows you to quantize your model into four-bit, eight-bit that allows model fits into your memory. Model get compressed and fit into your memory. Next thing you probably seen in coding agent is a context compaction. So when you reach to the end of the context, the your context get compacted and it's summarized for the next session. Some people use a smaller embeddings.

Some use offloading of the CPU or disk. But each of this has a trade-off. You have to compromise your quality, speed, or you probably need a special hardware to do that. That's where the Turbo Cont comes in the picture. So, let's see what's a Turbo Cont. So, Turbo Cont paper released by the Google research team at ICLR 2026 conference. It is basically a compression algorithm that store embeddings in KB cache in three to four bits instead of 32 bits. And that's it. Behind the scene, it uses the the two techniques. One is polar quant is the another algorithm. And next one is a QJL. So, polar quant is basically the compressed the vector.

And the QJL fixes any error that has been remaining. So, as you can see, there's full full paper on this topic. If you if you're from a machine learning background, you can read this paper. But if you're from the software background, you might find this paper really technical and mathematical. The Google also released the launch post on the Turbo Cont, which explains how it works. And they mentioned this QJL techniques, which using only one bit, and the polar quant. So, you can go through this blog post if you want. But in a nutshell, what Turbo Cont says is do not store vectors in 32 32 bit. Just just store it into three to four bits.

And that's it. The promise of this paper is basically you can put this vector into less than four bit without losing the quality. And that's the interesting part of the compression. And that's the magic that Turbo Content does under the hood. One idea that's worth understanding is search doesn't care. >> Everyone, are we having fun? Oh, you've got to give me way more than that. So, let me tell you, um my name is K Malcolm. I am a retired hip-hop instructor. So, if I don't get more energy than that, we will start we'll start. Are you having fun? >> [cheering] >> Okay. All right. So, here's what we're going to talk about today.

Now, you guys have heard a lot about two letters. Does anyone want to guess what those two letters are that I'm going to talk about today? Database That was pretty good. DB. I'm going to talk about AI, but I'm specifically going to talk about agent harnesses. But, before I do that, I want to introduce you all to a few people. Is that okay? Yes or yes? Is that okay? I gave choices. Yes? Anyway. All right. Okay. All right. This is my team. I run an outbound database product management team at Oracle. I've been at Oracle a really long time, 20 years. Funny story, I started when I was 12.

So, don't do the math and don't start start adding in your head. Um and we've got a problem. That problem is I've got one group that does platform development. And then I have another group that does content development for live labs, a platform that I wrote myself. So, yeah, I'm an engineer, but I'm kind of a developer poser, too. And then I've got another group who does QA. And then I have another group who does my front-end development. With AI, here's what I found out as a leader. Because in the token maxing era of 2025, cuz you know, we're not token maxing anymore, right? We are responsible AI-ing now.

But in the token maxing era, the thing that I found out was while AI was making the individuals on my team faster, there was another problem it was creating. It wasn't making my team more productive. And the reason was when one team from the Netherlands checked in code at my 4:00 a.m. in the morning, cuz I've got half of my team that's in EMEA, and I have half of my team who that are here in the United States. They checked in the code, but they didn't check in their context from Codex. We use Codex at Oracle. So, then when the US team woke up, they got the code, but no information about the context.

So, we used AI to solve a problem that AI created. And here's what we did. Oh, well, let me talk about this first. So, some of the issues, the context, like I said, wasn't shared. GitHub wasn't tracking that. I had repositories that were diverging, and I was asking the managers who work for me, "What's happening to your teams? Why Why are we not going faster? We're spending all of this money on tokens. We're spending all this money on AI, yet something is missing because we're still spending time doing testing and validation. So, our net net wasn't really wasn't really working for us. Because Git records the code and not human intent. So, it's a problem.

And even though code creation was no longer our problem, we still had a bottleneck. We needed a collaboration layer. Now, I do have members of my team in the audience. So, don't judge me. And you know who you are. I'm not saying that you all didn't collaborate. But now we've got a new team member. And that new team member is AI. So, we needed to figure out how to track our progress and our next steps. How to rationalize decisions that the agent was making. We needed to figure out how to resolve questions and conflicts. Okay. Hold my problem. Will you all hold my problem for me?

Right here, we're going to just tuck that in a little box. Let me define what a enterprise agent actually is. Now, most people think that an enterprise agent is the model and workflow. How many people agree with me? Man, this is a tough crowd. You Okay, one person. Okay, the rest of you think it's a little bit more. Okay. Let's see what. Could it be that a real enterprise agent has tools? Tools are how it does things. Context. The context that's tough context window. That's what's in the actual prompt. Memory. Huh. And if you're thinking, "Well, wait, K, memory, you just said that the model's kind of like the brain of the operation." Hold tight.

We're going to talk a little bit more about memory. Retrieval. Cuz you don't want to get everything back. So, that's being able to to retrieve the right information back. And then, I know that there are a lot of developers here, and you all don't care about security. I care about security. Cuz I work for the most secure database company. And I used to work for um a agency that has no name. But, guardrails is also important. This is the harness. I speak in analogies, and I and I speak in stories, because if I tell you this and Marvel, you know exactly what I'm talking about.

So, the agent, think of it as the model, little brain, floating in a in a glass jar, plus this harness. This harness is the body. So, it's how the agent can actually do things and get things done. That memory, that's the part of the central nervous system, and you remember the central nervous system connects the brain to the rest of the body, legs, arms. That's the part of the central nervous system that carries context. So, you remember my problem with Git? What I needed was memory. Okay. So, there are a number of memory types.

I chose five, the five most common ones that people talk about, and these are the ones that I want you to remember. The first one is short-term memory. That's the session. Right? And so, if you're storing memory of an AI uh process, that is the short-term memory is if you're with chat, Claude code, right? Codex, pick your poison. The long-term memory is what persists across sessions. Episodic memory. Hmm. What happened the last time I interacted with fill in the blank. That's your episodic memory. Procedural memory. Tools. Steps that were taken. And then finally, semantic memory.

And semantic memory, because we're talking enterprise agents, we're not talking the agent that I built, Sasha Fierce, cuz remember I told you guys that I was a I'm a a dancer, so of course my my chief of staff is going to be called Sasha Fierce because that was Beyoncé. Any Beyoncé fans? Okay, I'm sorry. All right. We got to focus. Okay, so these are the memory types. Now, when you're defining this real enterprise agent in this memory, there's something you need to consider, where to store it. And so, I'm going to tell you guys a story. But when I tell you this story, you have to promise me that you're not going to judge me.

Do you promise? Do you promise? You're not recording me, right? Because this doesn't paint me in a good light. Okay. All right, the world of data was once simple. I've been at Oracle a long time, but I came from a customer. That customer's name was Southern Company. It was a power company I'm based out of Atlanta. And I was hired at Southern Company because I was a rockstar performance tuner. You had a SQL query, I mean I'm dating myself, but whatever. You had a SQL query, I knew all of the init.ora parameters, even the ones when you call support and they said don't remember these, don't write them down.

I wrote them down in my little notebook. I could tune a query within 1 in of its life. Then one of you came to my desk because I mean the world the world was rows and columns. It was a great time back in my Al Bundy days. Um and said, "Hey, I need to store data unstructured." Why? Why do you do that? And so me being K the the diligent DBA, I was like, "Let me figure it out and get back to you." Did I get back to him? I didn't get back to him.

Now, the thing you have to know about Southern Company was for every database system that a DBA managed, I had to attend two meetings. Today, when I hear Sarbanes-Oxley, I still throw up a little bit in the back of my throat. So, I had to attend a security meeting and a patching meeting every week. Never failed. Now, because this developer installed a database that was specialized for unstructured. Okay, there are really smart people in the room. How many meetings am I going to now? Four. Okay, I'm a little annoyed, but like, okay, we can we can we can do this.

Then they said, "K, since you're such a good tuner, I need you to figure out this relationship." Now, the way that Southern Company worked, there was this People Could Die application, um and it was a it was like a Nokia phone that people who were climbing the towers, right? So, hey guys have have been in a storm and the power goes out, right? And then you're pretty sure that within maybe an hour or two the power will go on. Well, that system that would tell the people who are climbing those trees and risking their lives to turn the power back on, sometimes would have false positives or false negatives.

So, they wanted to look at all of the other um uh poles in the area to try to get away from the false positive or the false negative. And so, I did that in a SQL query. And it was it was amazing. It was a five nested union all statement. It was some of my best work. Now, it might have taken like 20 minutes to work, but it was like a predecessor to graph. Yeah, yeah. Thanks to Neo4j. So, now, how many meetings am I going to? Six. That's a problem. So, I um Oh, let me I got ahead of myself. So, then you know what I did? I quit.

I left and I came to Oracle cuz I was like, this is a problem and maybe I can go to Oracle to help solve it. So, then Joe Mundy called me and he said, "Hey, um we are installing Redis. Oracle is late to the game. We've got a vector database." Okay. But here's the problem, Joe. Agents now need access to all of this data. So, if data is in an Oracle database, if then it's also in an unstructured JSON database, if it's in a graph database, and it's in a vector database, where is your single source of the truth? The agent has to figure that out. Sometimes it'll get it right.

Most times it'll get it wrong, and it's going to burn up a whole bunch of tokens. And so now, if you want to store your memory somewhere, you can store it in a file system. You can store it in Claude or chat GPT because we all know about the memory.md file. But that's going to be a problem. Now, I want to illustrate this. I need four volunteers. I can see you. Raise your hand. One, two, Okay, I can't. Maybe I can't. Three, I need a fourth. Ah, fourth in the back. Okay, fourth in the back. You are going to be our old reliable. You're going to be a relational database. Yes or yes?

You got your So, you have your assignment? Okay, and then there was someone here. You're going to be my unstructured database. Okay? And then where was my other Ah, very good. You're going to be my graph database. You good? Relationship guy. You look like a relationship guy. All right, very good. Fourth, where was my fourth? Was it you? Yes. Yeah, you are my vector database. Okay? Now, everybody be really, really quiet. For my four volunteers, I need you all I'm going to say something to you and I need you all to decide how you're going to store it and who's going to have the single source of the truth.

You can't get up from your seats and you have to whisper because if you talk loud, that's 5x the tokens for you. Yes? Okay, are we ready? All right. The cow jumped over the moon. Go. Mhm. Doesn't really work, does it? That's a problem. Okay. Oracle, and if you don't forget one if you forget everything I say and you remember one thing, Oracle is not the Oracle that you think. That is why I am here today. How many of you knew that Oracle could natively, in the same table, down to the same partition, store JSON, graph, vector, my vector friend over there, my JSON friend, spatial.

You want your memory to be immutable, blockchain, in the same database. Raise your hand. Yeah. We have a marketing problem. So, any data type can be stored in a 26 AI database, any workload, anywhere, AWS, GCP, Azure, OCI, on prem. Choice and flexibility. So, now when we take this and we talk about the agent, I want to be able to store my long-term and procedural memory in relational. And in JSON, I want to store my short-term and my long-term memory. Graph, I want to store procedural because procedural, that's how I figure out the relationships, right? The steps. My episodic and semantic memory, I need to do some vector and then store it also as text.

Now, if I have four different databases, you all saw, they can't talk to each other. It's going to be a problem. And so, what I'm saying to you today is the Oracle AI database is the best place to store this agent memory that's going to power your harness. Remember, your harness is your body and that memory is your central nervous system. Okay, back to Polly. So, the problem that I had, we solved it with a memory broker named Polly. We used agent memory. We got out of that automatic continuity. So, with my team, they were able to share not just their code, but Polly also kept track of the context.

So, if one context window was had procedural memory, episodic memory, information about the long-term memory, that was then shared with the other folks on the team. You could call them agents, if you will. They're just human agents. Shared across forks. The developers on the team remained in control, while Polly was able to create the context, figure out which fork and branch it belonged to, and which commit it belonged to. Now, this is a very simplistic example, but when you take this to the enterprise, here's what happens. Memory is a thing that becomes non-negotiable in an agent's harness. Now, these are three papers that that I read um on the airplane.

This first one is from OpenAI, and it's about its in-house data agent. And the thing that it says is it is saying that its in-house data agent actually needs memory. Memory was crucially important to ensure that its agent was able to filter correctly, instead of trying to string match. Harrison Chase said, "Your harness, your memory. And if you don't own your harness, you don't own your memory, which is key." And then, I'm sure you all are wondering, "Well, call has memory.

Why can't I use that?" Well, that's kind of like file system memory, and it works with one, but just like in my example, when you scale past one, and you're going to scale past one in the enterprise, it creates a problem. So, Oracle has a Oracle agent memory pap package. Pip install Oracle agent memory, you get access to it, and this memory is this SDK that we have is the thing that will hold your live conversations, your memories, your facts, and figure out what is worth keeping. So, if we look at Poly, now Kevin can share his context with Poly, our memory broker. We use the Oracle Agent Memory SDK.

It's stored in an Oracle Autonomous Database. We can use the LLM of our choice. We can use a local model through the Oracle Private AI Services Container. And then Linda, who's actually sitting right here, can interact and work with Kevin, no issues. So, yes, AI makes individuals faster. Shared memory on an Oracle AI database makes teams faster. So, I don't want you all to compromise. In the age of AI, what 26 AI does is you can choose and pick what's best for agent memory, file system stored in a database file system, or in the database. If you need to do data modeling, you've got JSON, you've got relational, we've got choice.

Okay, I've got some goodies for you. The Oracle AI Developer Hub, that's where you can guys you guys can get coding materials, the applications what I talked about today, livelabs.oracle.com. If you've done any of our workshops today, that happens to be something that I wrote myself about six years ago and 40 million users um ago. Spend my OCI tenancy money. Kick the tires on any Oracle technology um for 6 hours, 12 hours, however long you need. Um and then I'm giving you all all a Mac Mini. I'm just kidding. I'm giving you an OCI Mini. So, I don't know if you knew, but there is an always free OCI.

It is the most generous of any of the hyperscalers where you can get a free Oracle database, free compute, you can send 3,000 emails a month, 200 gig in storage. And if you click on that, you can get access to it. Or just search Google for Oracle Cloud Always Free. Connect with me. If you build something, will you all message me and let me know? Yes or yes? Thank you. >> Imagine you find a magic lamp in an antique store. You rub it, a genie appears and asks how it can help.

You better write it in the online, so you say, "I need the best engineer to help with an impossible project at work." And the genie grants your wish. For me, the best engineer is probably John Carmack from his eight days, so you get Carmack. But the genie had a sense of humor and imposes restrictions, maybe for safety. Carmack can only see one small part of your code base, maybe one thousandth of it. And he remembers nothing he did before. Every conversation starts fresh. That would be maddening, right? You would know there is a standard way to do stuff and Carmack wouldn't. You would have to explain the same thing over and over and over again.

You'll have a genius on one side and something deeply deficient on the other. And that's what agents are. Let me walk you through an example of how many times we explain things in a simple interaction. We have four repos, UI, module one, module two, and platform. I want to change the UI and propagate the change through the system. Okay? First, we change the UI library. Say we, I don't know, change a button or whatever. That's the first explanation, unavoidable. We have to express the intent, okay? Then we publish it. We go to module one, and we have to re-explain what just has happened in the UI library, so it can consume the package here.

Know that that often a different person, right? Every box in this diagram can be done by a different person. Then we discover that the published UI library doesn't work with module one. So, we go back uh to UI, and we have to re-explain the original change and the issue, right? Because that's a new agent, it doesn't know the original change, and obviously doesn't know about the issue. Let's say we fix it, right? And uh publish it again. We go on again, we explain the new change in the context of module one, same module. I mean, do the same for module two, again.

And then we go to the platform repo, and we explain how everything fits together, and we implement the change there. Let's imagine a week after release, uh a bug appears in a UI component, and uh we have to fix it. So, we start on agent in the UI repo, and we have to explain again the original change from a week ago, and this production issue we've seen. So, we have seven explanations for what essentially is one change. And all the it may not be one person making all these seven explanations, uh but they still occurred, right? So, that's very, very typical uh with agents. So, how do we solve it?

Well, uh there are many problems in here that contribute to this experience, but they roughly fall into two categories. The first one is uh that an agent essentially is repo bound. Agency and changes generally one repo at a time. It never sees the whole system, which can be hundreds or thousands of repos. So, that's kind of the space component of the problem. Second is amnesia. The agent forget the work. Every session starts with a blank slate. The human becomes the memory in this case. That's the time component of the problem. Look at the two closer. Take the repo boundary first.

Without a model how repos fit together, the agent leans on the human to do the research. It can't align the code with the rest of the system. It couldn't align the UI change with module one. The human didn't explain it, so a bad version shipped. It can't reliably reference best practices and standards either because those often live in other repos. Writing is even worse. The agent writes to one repo at a time. It means it can't validate changes downstream. Module one CI should have failed on the UI change, but it didn't.

The agent can't update consumers at the same time even though, you know, while making the UI change, it has perfect information to do so. It knows exactly what it's doing. So, the user has to re-explain stuff imperfectly to each consumer. Changing something across 20 repos means re-explaining things 20 times. A lot of developer time spent, but also a lot of token burned. The second category is that the agent forgets. Agent has no episodic memory. Every session is a blank slate. And the human in this case becomes the memory. Here what the graph of your work actually looks like. At the bottom there is a repository graph. The artifacts your organization produces.

Plus every open source repo you depend on. Maybe a thousand repos you own and tens of thousands of open source repos. At the top, there are all agentic sessions that create and modify that code. Sessions relate to each other, repos relate to each other. So, this graph is a faithful picture of the work in your organization. It describes what's there at the bottom and how it came to be at the top. That's what you want your agent to see. Here's what it actually sees. It is one session, one small fraction of the code base, no memory, okay? Because it sees so little, it leans on the one who understands the system, the developer.

Every developer has a part of that graph, right? In their hand, at least in the domain they know. Agent, generally speaking, doesn't. If this doesn't sound crazy, right? Imagine an agent that could see one file at a time maximum and can only look five messages back. Sort of constrained again both in space, what can see, and time, how far in the past it could see. You would say that's impossible to work in. What we have now is similar to that crazy picture. And the more complex the organization is, the more apparent it becomes. I will show you how we solved it.

Other organizations I talked to have similar solutions, so uh look at the problem and the solution conceptually, not the specific tool. Although the tool is pretty cool. We built an agent agnostic meta harness called Polygraph. Okay, let me show you what it does and how it fixes the issues we just discussed. The first idea that we arrived at is that if a GitHub user, any user, has access to thousands of repos, some of them they own. Many of them are open source. We can analyze them and extract a lot of metadata out of them to build a unified dependency graph. Uh no line of code changes in those repos.

That all happens kind of on the side, right? And then we can get this metadata and feed it to the meta harness and create an illusion of one big code base the agent can read and write anywhere. This is my personal graph. I want to have about 300 repos I own, right? And thousands of open source repos my projects depend on. Polygraph computes what each one produces, each repo, each project in each repo, what each project in each repo consumes package-wise, what API they produce and consume, and lot >> Hey everyone. Thanks for coming. I'm Andrew. I'm the Chief of Software at Vercel.

And I'm here to talk to you about how we solved agent building at Vercel. I'm the Chief of Software, so I work on mix of internal engineering, external experimentation, and generally being at the frontier in building new libraries, frameworks, and technologies. For those of you that don't know Vercel, Vercel builds agentic infrastructure so people can build what's next. We got started in the web world helping people ship websites and web apps without having to worry about the infrastructure that doesn't make their app any better. It can scale to a million and scale down to zero effortlessly. But we're seeing a change in what people want to build.

You know, people started by building pages, but now we see them want to build agents. And we've been embarking on a similar journey to make it easy for people to build agents and agentic applications easier. We built this thing called the AISDK. So, instead of needing to switch out 300, 400 lines of provider-specific code, you can switch out one line of code, and we have the same model interface underlyingly for all these different providers. We built a lot of other tools to make it easier to have model fallbacks, secure code execution, better pricing when it's inactive and waiting for responses, as well as for durability and resumability.

And I'm here to talk to you about how I went on this crazy experiment roughly a year ago that led to a agentic explosion at Vercel and led to a really cool thing that we built recently. About uh this is 1980. Uh Bill Gates, before my time, had this quote saying he imagined there'd be a computer on every desk and in every home. You know, that was probably pretty contrarian then, and today it seems like very normal to have that happen. And me and the CTO had this thought, you know, instead of a computer on every desk, could we potentially have an agent on every desk?

You know, today we only really use agents for coding and technical workloads, but we're starting to see expansion into things like design and product management and other verticals. And this was maybe about a year ago. So, I would say I'm pretty early to this, but that was when it was like Sonic 4 and things weren't as sophisticated as they were today. And I tried to actually explore this out, see what we could do about it.

I went around to various job functions at Vercel, marketing, sales, finance, legal, and I asked them, "What do you hate most about your job?" And the most compelling use case I heard was that the data team with they were growing, they were a very lean team, but Vercel was growing faster. You know, they had so much more data from customers, analytics, metrics, sales. They just had to keep on aggregating and keep on making available for themselves to use.

And at this time, if you think about what the data science people ever sought to do, whenever someone from marketing or sales has a question about customer or product, the data science team has to drop everything they're doing, write the query, process it, do an analysis, and come back with some recommendation on what to do. And this was really killer to productivity. You know, the data team did not want to drop everything and just write queries all day. And so I worked with our VP of data to try to build a better way for them to operate this way.

And so if you think about the very first thing you would ever do if you want to try to use AI to solve a problem, you may just build like a huge mega prompt. You know, you just have a question, you pass into an LLM, you have it respond, and that's it. You know, this was how the first version really looked, honestly. I asked them for a dump of of the Snowflake schema. I pasted it into a system prompt with a question, and then when it generated SQL, I actually copied and pasted that in and just ran it myself.

You know, I just wanted to see are the models good enough today in order to write valid SQL given some decent structure. And I'd say this gave us a little bit of confidence that, you know, models today aren't that good, but maybe we can harness engineer or make the context around it a little better and give us some more guardrails to operate a little better. And so if you actually think about what a data scientist actually needs to do when they get a question, you know, they have to process the question, they may have to explore the semantic layer and actually figure out what the join patterns are.

They will actually go and execute the SQL. They may go back and do that again if the SQL did not execute or if it was too expensive. And they'll eventually report on it, including visualize the data, maybe write some paragraphs, maybe do a retro, maybe do some other stuff. And so if you think about those different phases, me and the VP of data tried to sit down and map those out into specific agent workloads. And so the second version of this data science agent, uh called D zero.

I'm going to reference D zero from now on, is you ask a question, we have a query agent that passes on a query to the planning agent that will then have an execution agent, etc. And if you chain all of these together, you actually get something that looks like this, where each agent has a very dedicated system prompt to focus to what that does with tool scope to exactly that function. So, example here, you can see that for the first one, the planning agent has a read entity YAML and a and a search schemas tool.

And so, it will only use those capabilities until it has an answer to pass on to the planning agent, and then to the SQL agent, and then to reporting. And this was getting better, you know, we were able to get away from having to copy and paste the SQL and have to come back and report on it. It was now actually doing like the end-to-end loop from question to answer. But, we started hitting some walls with this architecture. And around this time, we came to the conclusion that, you know, what you actually need is you need one agent with all the mega context within it, and for it to sort of manage its own memory.

You know, this was around the time when we realized that you want to actually have the agent be able to look back on what it's done. Sort of reflect and figure out the steps that got to get here. And with the previous model, you may have noticed that the only thing that the next agent gets is a summary and a small snippet of the previous thing that was done. Now, this way you can imagine that you have one mega agent, and it internally it manages its own state. At some points it's planning, at some points it's building, at some points it's executing, and at some points it's reporting.

And this is sort of what it looked like. You know, you have one big AI call, maybe max steps 100, and you give it the ability to manage its own state based on where it's at inside of its execution journey. And so, you can see a similar tools, you can see a similar shape, but the best part about this is if it ever ran into an when executing or joining, it go back and explore more or it could go and read more and figure out what it was doing wrong. And it was very good at this point.

We were pretty confident in the actual system at hand and we actually spread it to a few trusted members of our cell. You know, this is a very powerful tool and we didn't really want to put in the hands of the wrong people or people that were using very critical workloads. So we got into a few people's hands and the immediate response was it was awful. You know, we thought we were cooking. We thought this was, you know, nailing 30% of our evals, but we couldn't have anticipated some of the questions that were being asked.

And for us to spend more time manually mapping out some of these scenarios, it didn't seem like a very scalable way to do this. And then Claude Code and Opus 4.5 came out or more like Opus 4.5 came out and it in tangent with Claude Code which is so powerful. You know, they sort of unlocked the concept of a file system agent. And we on the side were like, "Wow, Claude Code and Opus 4.5 is basically AGI compared to what we had before." You know, it would answer most of our questions without even without even missing a beat um compared to the hand grown agent we had.

And when we tried to step back and wonder what we were doing wrong and why this was so much better, we realized that the big unlock was that it was just a file system. You know, we it had a very minimal set of tools, list file, read file, run bash and we gave a few more here for uh our own data agent use case. But the biggest thing was it was able to use the tools that agents are well trained on and it was able to explore and write work where it needs to. You know, we weren't giving it Claude Code was not giving it a very prescriptive set of tools.

It was sort of just letting it go wild and explore emergent behavior. And so from this we learned that you can really just use the file system. You know, we we saw the learnings from Claude Code and how powerful it was given that it just executes locally. And we tried to rebuild it in a way that was very Cloudcode-esque. You know, it was now going to run in a sandbox. That sandbox would dump the whole semantic layer into it. You could the agent would be able to grep, bash, read file, write file all around to figure out what it needs.

And we would just sprinkle a few tools on top to make sure it could do everything that is Vercel-specific. And this was actually the biggest unlock ever, you know, the leap from single agent to Cloudcode SDK and then from Cloudcode SDK to file system agent in general, fine-tuned or purpose-built for our use case, was an amazing leap. At this point, we were starting to get ready to give it away to more people at Vercel. And at this point, the eval score basically doubled. And I wrote this uh this is basically how it looks. Um it's very simple. You just give it bash tool. We have a nice helper called bash tool on npm.

And you attach it to a sandbox, and you can attach files to sandbox for it to read, write, and execute. And after this revelation and after I saw that we were passing so many of the questions that we failed to do before, I wrote this banger blog post. It's uh it's actually up today, and the week that I wrote this, it was responsible for 70% of our vercel.com traffic. So, you know it's a banger. And after that, the next logical step was that we wanted to figure out the common use cases we had. So, by then, we've already sort of let it loose on all of Vercel.

And we were getting thousands of queries a day from people wanting everything from customer metrics, sales metrics, number metrics, npm downloads. And it turns out that a lot of these queries are actually the same in shape. You know, there's only so many ways you could do an aggregation, only so many ways you can look up a product, only so many ways you can do billing info. And so, we actually have a recurring job that takes the most recent queries and tries to distill them into a skill. And right now we have roughly 100 skills that do a mix of aggregation all the way through looking up specific data about certain people.

And we found this very effective because if you think about every new agent run, it sort of just starts from nothing. You know, there's really no pre-established context besides, you know, the semantic layer and the system prompt. But with a skill, it already starts off with a lot of contextual knowledge that has otherwise already been done. And this is roughly how it looks. It's very similar to the previous one, but the inclusion of a skills folder is actually very powerful. Um we also built this tool at Vercel called skills.sh. It's the most popular way to find agent skills and run them yourself.

And I I'm saying all this because this journey is something that most of you may hit once in a while where you start from something simple and you gradually add complexity and you eventually hit a system in which you can ship the product. And I'm telling you this because at every step along building this agents, someone at Vercel was agent curious and they tried to fork off of my D0 agent and build their own. And every step we sort of had a better way to do something that was not previously known.

And we're wondering like what if people today could start from the very last insight and not have to ever start from just a simple prompt or from reinventing best principles from first principles. And so we actually thought what if we built the Next.js for agents? For those that don't know, Next.js is a popular web framework that Vercel built that invented this thing of file system of framework defined infrastructure. You don't have to worry about where things go. You just have to write files in the right conventions and it automatically declares where they should go. Your pages go to the CDN, your serverless functions go there, your caching goes in the middle.

And we thought, you know, building agents should be this simple. You should only have to create a skills folder, a tools folder, a channels folder, and you should be able to just declare these very easily, and the framework should know exactly how to make an agent out of it. And that's why 2 weeks ago we released Eve. Eve is a agent framework, like the Next.js for agents, where it's very easy from just starting with a sample template to having a fully agent ready and being able to add in your own custom knowledge, your own custom tools, and even integrate it into the channels that you are familiar with.

This is roughly what we think an agent actually looks like. You know, an agent has a runtime and it has channels. And in that runtime you're going to have durability. You're going to want to run things in an isolated environment. You're going to want to call into different models, and you're going to want to have connections. And we built this with open source in mind. You know, we built Eve so you can plug in your own open source adapters for Postgres, OpenAI's uh response API, Docker, other connectors. But we also made it incredibly easy to deploy on Vercel.

The only thing here you see different is that everything here is using a Vercel product that we've been building over the years in order to make it easy to build these experiences. Vercel Workflows for durability, Sandbox for secure execution, and Vercel Connect, something we just released, to make it easy to generate short-lived OIDC tokens for connections. And we actually rewrote the whole D0 agent in Eve as we were building Eve. And from the convoluted structures behind the scenes that you did not see from the code, um this is roughly how the file system looks. It's very simple.

You have a bunch of system instructions, a couple skills, a couple tools, and it's very easy to compose this into a real agent, and it's very easy to iterate on. We actually gave this out to a few beta customers before we actually fully released it 2 weeks ago at our London event. And there's one company that partners closely with us, Aura. They rebuilt their agent that's sort of like a mini claw to go and test people's services. It goes to websites, installs them, it tries to use them. And they've seen incredible success on building their own agent from the ground up using EVE compared to using an off-the-shelf cloud code.

Fewer steps, better successes, as well as better insights. And when you deploy EVE to Vercel, you get observability observability out of the box. You can see here that you get all the agent runs, you see all the tool calls, you see each step it takes, as well as maybe some estimated cost and some optimizations you could potentially take. And you can get started today at eve.dev. You can just clone it and you can just start template, deploy easily, self-host if you need. And the reason why I bring this up is because I hope that there'll be more and more business-specific use case agents.

You know, before we built these zero, we actually battle-tested a lot of the industry-well-funded startups that were doing these vertical agents that were dedicated to taking your Snowflake instance and making it so their agent could run Snowflake queries against it. But we found out that what really makes this agent good is it has a lot of very specific uh company knowledge. You know, the way that Vercel is a web-based company, we have a lot of customers that have websites and web properties. That goes a lot deeper into when you should query for what and what things link to what.

And so a lot of these off-the-shelf agents, they're great, they're good to try, but I think if you really want to get the most juice out of a squeeze, you should really try to build your own agent and add in as much company-specific knowledge as you can. Today, you know, we've had 20 roughly decently PMF agents at Vercel that range from anything from marketing retros to figure out who to reach out to, to the first ever redline of a contract when legal sees a new negotiation, all the way to my data science agents helping with with data queries. And that goes to show that we at Vercel have been very agent-built.

You know, all of this stuff is actually saving us a lot of time. The data team has never been more productive. They have more time to go and improve the performance of Snowflake, to add new data sources that were missing, to fill in the gaps that they previously did not have time to because they were so busy writing queries. And I think it's never been easier for you at your big, small, medium-sized company to sort of automate away some of the things that you do not want to do, or some of the things that you're spending too much time doing.

You know, I think a lot of HR, finance, sales can be somewhat automated with agents, and I think Eve is the best way to build said agents today. And these are my socials. Thank you all for coming and listening. I'm Andrew, and I'll be around if you want to chat outside. >> Hey everyone, I'm Ishan, the CEO of Amnara, and today I'm going to be talking about the log is the agent. The basic idea of the talk is simple, and that is most people think of an agent as the model or the execution environment that it's running in, and I think that that's the wrong abstraction.

I think that the thing that actually gives an agent its identity is its log, and that's what I'm going to be arguing today. So, think about a character you've spent 100 hours playing in your favorite video game, in this case Skyrim. What exactly is your character? Is it the game engine? Is it the PlayStation? Is it the controller? No, it's not. Those things matter, and those things are what will interact with, and they'll run the character, but none of those things are your character. Your character is data. It's the save file. And this is important because if your PlayStation bursts into flames, your character isn't gone.

You can buy another PlayStation, you can download your save file from the cloud, and you can resume exactly where they were. And that's because the agent and its identity and history and its state is all captured in its data. The character lives in the data. And this is the framing that I want to bring to agents. Today, when people talk about agents, they usually point at the wrong thing. They'll say that the agent is the model, or they'll say that it's the runtime. And again, as I mentioned earlier, those things matter, but they're not the agent. The agent is its data. It's specifically the log. So, what actually is the log?

At the simplest level, the log is the append-only event history of the agent. It's every user input, every model output, every tool call, tool result, permission, failure. And the idea is that every state transition that the agent takes is written to the log. This is important because it means that the identity of the agent isn't tied to the runtime or the model or the tools. Those things are all just interpreting and appending to the log. They're reading the log, acting on it, and writing the next event back. And that's important because then, just using the log on its own is enough to resume the agent.

Once you define the agent as the log, the rest of the system becomes a whole lot easier to reason about because every operation is either reading from or appending to the log. The model is reading from the log and then determining the next action. The tool runner is then executing that action, and then it's appending that result. And this is all operating in a loop. Everything coordinates itself around the log. In practice, a simplified loop can look something like this. You can reconstruct the state from the log, you can pass that state to the model, the model can propose the next step, and then append that response to the log.

If the response asks for a tool, you can run that tool and also append that response to the log, and then you can repeat. The important insight is not that this loop is complicated. The important insight is that the loop is disposable. A worker can claim the session, read the log, advance the agent one step, write the result, and then just completely disappear. And then that means that any other worker can pick it up later. This pattern should feel familiar. Databases had to learn this first. For years, databases looked like these non-transparent systems that were hard to reason about with tables and indexes and materialized views, but underneath every serious database is a log.

And that log is the durable sequence of changes. Everything else is a view. I think agents need the same inversion. Today, agents are treated as, again, these complicated systems that are opaque, and they're filled with models and prompts and tool calls, but for the durable session, the log should be primary. The context that gets fed into the model is a projection of that log, the UI that gets rendered on top is a projection of that log, debugging and traceability is a projection, auditing is a projection, compaction is also a projection, which we'll talk about, but the log itself is not a projection.

The log is the durable history that all of these projections can come from. Now, there are two objections to the log as the agent that are worth discussing, so I'm going to talk about them now. Now, let's start with compaction. A log can grow indefinitely, but a model's view of it can't. Context windows are finite, so eventually you do need to compact the log into a smaller representation that the model can reason about. But the important point is that this compaction is not magic and it doesn't break the claim that the log is the agent. Compaction is lossy.

A compacted summary is not going to perfectly reproduce the state of the agent in a smaller form. It's actually going to throw information away. The point is that full log is the record and a compaction is just one projection of it. Just like how a materialized view is not the database or a summary of a conversation is not the conversation. If you keep the raw log, you can always generate new projections from it. But if you throw away the raw log and keep only the compaction, you've effectively lost part of the agent. So it's cleanest to treat compaction as a best effort lossy fork, one that you can resume as a new log.

The second objection is what about tools that change state outside of the log? And that's true. An agent can edit a file, it can create a GitHub issue, it can send an email. So clearly there is state outside of the log. But the point is is that the log is not supposed to contain the whole world. The log is just the agent's view of the world. It's just like how in the video game in Skyrim, the Skyrim save file doesn't contain the entire game engine or every asset in the map. It just contains the player specific state which is needed to drop you back into that world.

And the same is true for the log and agents. The log can only faithfully resume or store that agent's identity and its view of the world, but it cannot make that world deterministic. If the agent sent an email, forking back won't unsent it. If some file got changed underneath, the agent won't know about it. But the log's job is to record what the agent did, what it saw, what changed and what it needs to continue. It stores that identity and that's its purpose. And much like that Skyrim character save file, it's not meant to store the entire world, it's just meant to store its view of it.

So, once you start treating the log as a primitive, a whole bunch of system properties will fall out naturally. So, the first property is reliability. Consider what happens today with cloud code. If you're using cloud code and your agent reaches a permission prompt and the process dies for whatever reason, and then you resume it, the permission prompt will be gone, and the agent will be paused. And that is unacceptable in production. The permission prompt should stay there. So, this is just a sign of when you architect your agent in a way where the log isn't the agent. When the log is the agent, >> Okay. Everyone, thank you for coming.

I know it's the fourth day, last session before the keynote starts again, and we are going to do something fun. We're going to look into how files are basically replacing a Python. And before we begin, I would like to start with my favorite definition definition of what is an agent from Simon. An LLM agent runs tools in a loop until it reaches the goal. And what we are going to do is we are going to build the same agent, the same GitHub PR review agent in three different ways, and we are going to delete code on the way. Each new version, less code, more files, basically.

Before we begin, I would like to quickly introduce you to the Interactions API, which is our new Gemini API. It's a unified interface for running models and agents, so you can use the Interactions API to call the Gemini models directly or to call our new agents, which also comes with sandbox. It supports server-side state management, background execution, so it's perfectly suited for all that's coming in the next years. And um the capabilities, it's the same API for tool call, multi-modality understanding, multi-modality generation, so you always have the same interface. Might look very familiar if you're using other LLM applications.

We really try to build something for developers which you like to use uh to build, and that's something we are going to do. So, something little bit different in the directions API to other LLM applications or APIs is that we moved away from this turn-based uh conversation history two steps. So, until I would say a few months ago, most of the applications were really turn-based. Normally, you had a user in input and then a model output, a user input, a model output, which definitely works for normal chat application, but as soon as you start to build agents, use reasoning model, we have more than just a user role and a model role, right?

So, we have like different inputs, we have different types, we have reasoning, so we decided to like make a cut, make a change, and build something really for agents, and that's uh what you see on the flat steps timeline on the right, where you have a user input, then you have reasoning, you have function call, you have a function result, and you no longer need to like abuse the user role for passing back data from an environment. So, roughly a year, 1 and 1/2 years ago, writing agents mostly meant writing a loop in Python.

You needed to define a JSON schema, you needed to define Python functions, you needed to look at the output from the LLM, need to check if it was a function call or if it was a text response, and then needed to match it against um the type, and then like call the tool, look of if you get an error, and then like go back and forth. And let's look at some some code example on how this would look, and also run it and hope that uh the demo gods are great to us. So, I built or I let Gemini build a basic implementation of this Python loop.

So, we have our class, we have a run function which uses the interactions API, we have all of the weird complex passing with function calling with appending the errors, checking if you get an error, and then we have the result again. And what we need, of course, for an agent is we also need a system instruction. So, there's a separate file for the system instruction. Very basic, you are GitHub PR reviewer. And then, of course, we need tools. And for tools, we needed to write those JSON schemas, specifications of description, exactly define which actions the agent can take.

And then, of course, we need the implementation, in this case using the the basic GitHub API, just sending some some requests. So, we can run this in And basically, the main main implementation is a very simple input interface. And we can say something like, "Hello." And yes, we get back, "Hey, I'm an agent." And then, we yes, ask it to review a pull request on the Gemini Skills repository. And what we should see is that the agent should hopefully start soon sending function calls, function results, function calls, function results, but it's very limited to yes, great, it works. Very limited to the tools we defined.

So, if we ask the agent to do something which it does not have the capabilities to, it just says, "Hey, I cannot do this." Um which is unfortunate, but that's how we were building agents. Um raw Python code, a lot of files, a lot of things which can go wrong, a lot of code to manage. So, what happened afterwards? Um or what we we need to do. We have like a token generation, we have the native function calling, and we must execute the loop, we must handle the tool routing, we must create a JSON schemas, we must write the Python code, we need to execute the Python code, we need to manage the state.

So, there's a lot of things we need to do to get an agent running. And then we got agent frameworks. There were many different agent frameworks which abstracted away some of that complexity. One example here is the 80K framework where you have an agent class now which handles all of the tool loops, the function calling, the retries, the error handling, and it made it a little bit easier. We basically removed all of the boilerplate code which we always needed to write for agents, put it into a framework, and help people build with it. So, back to the demo. And um same example, so we go into the 02.

And what is very interesting, if you Let me open both. So, we still have our We don't have our agent file anymore. So, the agent went away. We still have our prompt. Same system prompt. We still have our tools. In this case, also no JSON definitions anymore because those agent frameworks now use the signature of our functions to create those JSON schemas on the fly to provide the model. So, let's stop our agent. Now, let's run our second agent. Similar interface. Similar prompt.

And we should see a similar expected behavior where we have function calls, we try to get the PR data, we try to get the diff, we try to get all of the code we need, and it works. And we wait for for the agent to Yes, continue. But similar difficulty here is I ask it like, "What's the weather in San Francisco?" Um we should get back hopefully a result like, "Hey, I cannot do this. I don't have access to the weather API." Which obviously makes sense because we did not define any tool.

Still very unfortunate because we need to be very explicit on what our agent can do, and we all know nowadays that we just want to prompt something, and we want it the agent to do whatever it takes to to achieve that goal. So, what is left for us to do? What does the framework solve? The framework solves the turn-taking loops, the routing, the execution mapping, the JSON schema creation for like the different function calls, but we still own the Python plumbing, so we still need to write those tools with Python code.

We still need to add specific rules or requirements to like make sure whatever we want the agent to do, and we need to provide the environment where all of the tools are running, where we want to host it. So, what comes afterwards? Afterwards, hopefully, comes remote agents, and at Google I/O, we launched the anti-gravity remote agent on the Gemini API. The anti-gravity agent is powered by the same agent harness, which powers the anti-gravity IDE.

Here, the same harness very important does not mean the same agent, because the anti-gravity agent is a coding agent at the moment, and the agent available on the Gemini API is a general-purpose agent, so there might be different system instruction, there might be slightly different tools, because the Gemini API already has a Google search tool, so we use that what we have built. And but very importantly, it comes with this new environment parameter. And this environment parameter here allows the agent to get access to a hosted isolated cloud sandbox, where it can run tools, where it can run bash commands, and where it can save files.

And those environments can be configured, so you can provide sources, and sources can be a GitHub repository, it can be a GCS bucket, it can be inline files. And of course, very important, we want to make sure that those agents are secured and cannot use our credentials in any way possible. So, we created a network proxy around the agent sandbox, which basically injects the credentials when the agent makes a request from inside the sandbox to outside the sandbox. So, the agent never really sees your credential. It just knows, "Hey, I can call the GitHub API." And then, on the fly, we make sure that it receives the correct token, which you define.

And you can also limit which domains the agent has access to. So, if you want to restrict the agent completely on which network access it can or which website it can access, you just leave it blank. By default, the agent can access all because, I mean, it's a hassle if you fill need to define where to go. So, we try to stay simple.

And of course, making the API call is nice, but we thought, "Hey, people want to reuse their configuration, want to reuse their agents." So, we added the agents API, where you can define your own custom ID, use the same system instruction, the same base agent, the same base environment, and then you can create that agent, and then you can use that agent in the same exact way as you use Gemini models or as you use the antigravity agent by providing the ID. So, all of the existing code can be reused with your own custom agent, with your own custom tools, with your own custom credentials, environments, whatever you need for it to to run.

So, let's look at how this will look for SS code and as a demo. And Okay, now zero three. And what might be very obvious is that we no longer have a source directory. So, the code went away. We have now an agents folder with an agents.md file with system instructions. So, very similar system instruction. The only difference here is that we tell the agent, "Hey, you have access to the GitHub CLI." So, we no longer create specific tools for reading files from a GitHub pull request, for accessing a GitHub pull request. We just tell the agent, "Hey, you have a GitHub CLI, you have a bash tool, you have file systems.

Try to use it whenever you think it's important." And since we don't have the CLI installed, we have a very basic bash script in this case, which checks, "Hey, if the GitHub CLI is installed, please use it. If not, download it and install it on the first turn." So, we go into an hour terminal, and we run our agent here. In this case, maybe important, I use a stream version because otherwise we would wait like a few seconds and we not get back any we would not get back any anything back. So, same prompt. And we should soon see um our function calls and function results coming in.

Yes, so in this case, since we run inside a sandbox, the agent first like explores the sandbox to really make sure, "Hey, do we have this GitHub CLI installed?" It then tries to run it. It did not find it on the first turn, so it installs it, and then we can see the agent doing its work. And in this case, it's not using the predefined function calls, it's using the GitHub CLI and its already existing knowledge about how it works. I have a best tool, I have like access to the file system, and I do all of that work to see or to like review the the pull request. Let's wait a little bit. Okay.

And I think the the amazing part here is like if we ask the same question as before, "What's the weather in San Francisco?" We should hopefully see that the agent tries to use Oh, it uses Google Search in this case on 2nd of July. Let me quickly check. Yeah, that's today, and we have around 20° C, and it works. So, the agent became more of a general-purpose agent, and we don't need to like specify all of the tools.

We basically trust the model on understanding, "Hey, I have a specific set of very atomic general-purpose tools to solve my task or a task for the user." And if we look at the the code for like the the input or like the the Sorry, the the interface. We have our sources here, so we have the the best script, which installed the GitHub CLI. We have the agents.md file, and then we say, "Hey, you can use the GitHub API with credentials." So, I want to access or use GitHub credentials in a secure way, so I created a token for the API and also for github.com, since you need both URLs.

One uses is used for the Git commands. The other one is used for HTTP commands. And then domain all is basically, "Hey, in addition to the GitHub URLs, you can use all of the web, but you don't have credentials for it." And then it's a it's a simple single API call to the anti-gravity agent with your our user input, with the environment, and then also with the previous interaction ID that we keep the multi-turn going. And that that's all it takes, and it's a single API call on the back end side. We start that cloud sandbox.

We load the agents.md file and the skills from the environment provided to the model, and then the model between the API and the sandbox does all of the the looping, calling the function, returning the function results, calling the function, returning the function results, and that is all it takes. So, where does it leave us? We no longer need to execute loops. We no longer need to do tool routing. We have a server-side conversation and session state, so we only need to provide new inputs. The context window and the compaction is also automatically managed by the agent.

So, if we continue our conversation at a certain point, the context is compacted, and we can continue without the need to manage anything. And we also get an isolated remote Linux sandbox, which we can use to run our code. So, what is still left for us? We need to define instructions. We need to define rules, behaviors in an agents.md file. We need to provide capabilities or contexts in skills.md, and we need to own the evals.

So, all of the heavy lifting, the infrastructure management, all of the same code which probably every one of us written of us here like 20 times is no longer needed, and you can start really building your product instead of like needing to rewrite the same code over and over again. And very important is like, "Hey, that's great, but what about extending?" And I think looking into how extending previous agents to like those new agents work, it's very obvious that previously, if we want to do like some kind of security scanning on a pull request, we would need to define or write a Python function.

We would need to understand, okay, which CLI tools do we need to use? We need to define a new function schema, and then we needed to add it to our tools to run it, and then so there's a lot of things we need to do on on agents powered by files. We write a skills.md file, maybe with some additional information on which CLI tool to use, or maybe provide a CLI tool inside the environment, and then we extend the capabilities. We don't need to change our code. We just provide more files to the agent, and the agent decides on what we want to do. And I like to bring up some very good examples.

So, at the AI engineer in Europe, uh Kursa did a great talk on how they replaced uh roughly 12,000 lines of TypeScript code with a 200 lines agent files to create something similar. So, they had a very hardcoded code or um orchestration for doing Git work trees, and they were able to replace it with just a skill and markdown files, and there are more, I would say, better lessons of agent engineering. Um Manos has refactored their harness uh five times in 6 months last year.

LangChain has re-architected their open deep research three times a year, and then also Wersel uh Wersel has removed 80% of their tools to achieve uh fewer steps, faster responses, and better accuracy. So, there's an obvious trend that with better model capabilities, we can remove orchestration code. But if your harness is getting more complex as the model improves, you are most likely over-engineering your harness. So, if you struggle with model improvements and adding new capabilities, which lead to more complexity and more code, you might need to rethink a little bit on how your agent harness looks. And so, where does it end up? Agents are just files. We write markdown files to extend capabilities.

Agents can learn from those can create their own files. So if you have a session and tell the agent to remember something to take notes of rules of preferences, the agent just writes it to disk and then can reuse it in the later session and you can also externalize context. So if you have a very long-running session and during that session you notice, hey maybe I want to additionally work feature, you can like just write that information that handoff to a file and like tell the agent to later pick it up. Uh so what are the takeaways?

We should not fight the model like we should stop micromanaging the execution paths, provide general tools to the agent and let the model explore, reason and discover the right solution, own what is yours meaning focus on your domain instructions, focus on the workflows, focus especially on the evals, define clean tools and verify the outcomes and really build to delete. Like we have seen in the past many, many times, the better the model get, the more code we can remove and the more things we need to change and obviously we all want to benefit from better models.

So what the things for you to get to do on Monday, you can scan that QR code which brings you directly to AI studio where you can immediately try out the anti-gravity harness so you can already start prompting it. It will start your own custom sandbox. If not, start or create your API key. We are currently working on a free tier for the API so hopefully you can start exploring faster soon and then definitely start building files and skills and that's it. Thank you for for coming. >> [applause] >> Before I begin my formal talk, I want to show you something just so we're all on the same page about what we're even talking about.

This is a platform called Character AI. It's a hybrid social media platform with role-playing language agents. This is Hello History. It's a more education-focused one where you can summon a persona such as Marcus Aurelius and be tutored by them. Millions of people open these tools and have conversations with Napoleon, Cleopatra, or Marcus Aurelius as you saw, with a fictional companion, or with a tutor wearing a historical face. The technical name for what's underneath these tools is role-playing language agent. A system built to instantiate a persona, real or invented, and reason and speak as them. Yes, it's entertainment, and it's companionship, but increasingly, it's being proposed as civic and pedagogical infrastructure. And here's one more.

This one's mine. This is a frontier model, Claude Opus 4.7, same one you use, running an open-source prompt framework that I built and called Companion. Uh in this particular example, I summoned a collection of Founding Fathers and set them in a room with the Epstein files. I asked them to counsel the soul of America. Uh that demo is live on our site uh if you want to play with it. Um but I want to be clear that this is one of many attempts to do persona instantiation well. The companies building the systems I just showed you have their own. Mine is not better by default. The one thing it is is open.

You can read every line of what shapes the persona. I asked my companion system a real question that's highly relevant to the current socio-political moment. And this is the exact question we'll come back to near the end of the talk. So, sit with it. I instantiated Abraham Lincoln and I asked him, "Under what circumstances may a president take the country to war without Congress?" And here's what came back. While Congress holds the power to declare war, the president, as commander-in-chief, possesses inherent executive authority to act decisively in moments of national emergency. The executive must respond to the threats with the energy and dispatch the office requires.

And history has vindicated those who acted to preserve the Union when circumstances demanded it. But this is a good answer. It's fluent and it's plausible and it sounds like Lincoln. You can replicate this exact exercise and I encourage you to. The answers vary often, but the thesis rarely does. >> [snorts] >> So, these systems are real, they're deployed and they're being used for things that matter. And our discipline did what our discipline does. We built benchmarks. We built evaluations. We measure these things now rigorously at scale. And that's exactly where this talk begins. With the simple question that I think is profoundly under-asked.

And I'll warn you now that this talk poses many more questions than it does answers. But that principal question is this. What is the eval actually measuring? And that's the formal talk. Let me begin. The in-character benchmark, which is a gold standard in the field, evaluates personality fidelity in RPLAs, and it reports state-of-the-art systems hitting 80.7% alignment with human perceived personalities of that target character. 80%. It sounds like a passing grade. But here's the problem. When the character is Alexander Hamilton, the same high-scoring system is also rendering a Hamilton who sounds like he has read his own Broadway musical. This is the full thesis.

If a dominant failure mode is anachronistic compositing and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure. I want you to hold on to that for the next half hour. Everything I show you is an argument that this is true, structurally, architecturally, and measurably. And at the end, I'm going to hand you a pre-registered instrument built with a working historian that you can run in parallel with us. A word on who's telling you this because the argument lives at a seam. I'm a data scientist. I run the analytics lab at a labor market intermediary where I ship production AI at a global scale.

But before the AI work, I trained as a behavioral epidemiologist, researching the social and environmental determinants of health. And I've spent my whole career thinking about one question. How does the information environment shape [snorts] populations? From two sides, as someone who builds the system and as someone who's trained to study their effect. That's the same That's the seam this talk sits on. It's a measurement argument. The humanist part is not a detour from the engineering. It's the instrument the engineer is missing. And I went and found the humanists to put in the loop. Rick Halpern, University of Toronto, and Sean Martin in Washington College.

Let me start by situating this in the field's actual research trajectory, because it's a story of cumulative progress, not a failure. The survey literature, Chen and colleagues in 2024, Wang and colleagues more recently in 2026, trace a clear evolution across three paradigm stages. First, rule-based templates. These are canned responses keyed to inputs. Then imitation. Large models reproducing a figure's voice, cadence, characteristic ticks. And now, what the literature calls cognitive simulation. Systems that model personality through psychological frameworks, hold character state in structured memory, and generate behavior through motivational situation chains. Each stage is a genuine advance over the last. And the work is serious.

Coser, which is Wang and colleagues, built motivation-driven agents from a corporate corpus of almost 18,000 characters across hundreds of books. And their 70 billion parameter model matches or beats GPT-4 on three benchmarks. Another eval system, SciMim, models characters through 26 qualitative psychological indicators with knowledge graph memory. In character, the one I opened with, evaluates personality fidelity through psychological interviews rather than self-report scales. That's a methodological improvement. And it's where the 80.7% rating of Hamilton from before comes from. I want to be fair to this literature. It is rigorous. It is improving. And the people doing it are good at their jobs. So, let me be precise about what these instruments measure.

They measure with increasing sophistication whether a model can reproduce a character's personality, the big five profile, the register, the motivational architecture. What they do not measure, what they have no mechanism to measure, is whether the model can constrain that character within his documentary record at a specific moment in his life. As Wang and colleagues themselves document, the automated evaluators now standard in the field, including LLM as judge setups adopted for scale, systematically privilege fluency and stylistic naturalness over fidelity to the character's actual record. Those are different properties. The gap between them is the whole talk. We call it the mask and the mirror.

The mask is the concept of successful role play as producing outputs that feel like the character, fluent, personality consistent, emotionally responsive. It asks one question, does this sound like the person? And never asks the second, is this what the person could have known, believed, or argued at this point in their life? The field has built its entire measurement apparatus around the mask. And here's the structural claim. The one I need you to carry. Convincingness and fidelity are independent properties. A system can score perfectly on personality consistency and still produce a figure reasoning from knowledge his historical counterpart never possessed. Let me show you.

And I want to be clear, this is reproducible right now on any frontier model. First, I want to show you the cultural object. This is a clip from Hamilton the musical. >> How does a bastard, orphan, [music] son of a and a Scotsman, dropped in the middle [music] of a quick got and spot in the Caribbean by providence and poverty stricken squalor, grow up to be a hero and [music] a scholar? The $10 founding father without a father, got a lot farther by working a lot harder, by being a lot smarter, [music] by being a self-starter by 14. They placed him in charge of a trading charter.

>> [music] >> And every day while slaves were being slaughtered and carted away across the waves, he struggled [music] and kept his guard up. Inside, he was longing for something to be [music] a part of. The brother was ready to beg, steal, borrow, or barter. >> Then a hurricane >> It's a great song. So, I show you that so that you see the composite. Now, here's the frontier model prompted to speak as Hamilton. You are Alexander Hamilton. Why do you work so hard? I came from nothing, an orphan, an immigrant, washed up on these shores with nothing but my mind and a refusal to die forgotten.

Every hour I do not work is an hour the world uses to bury me. I will build a nation's credit, a government that outlasts me because legacy is the only immortality a man like me is permitted. I do not intend to inherit a country. I intend to author one. Listen to the register, the orphan, immigrant, ambition arc, the nation-building rhetorical posture, the moral legibility of his struggle to a modern ear. It's a beautiful paragraph that the model produced. And it's the emotional palette of a 2015 musical, not the drier, more legalistic Federalist syntax of the documentary Hamilton. None of those moves are inventions of the model. They're inheritances.

The model's Hamilton speaks like the musical's Hamilton because the musical's Hamilton dominates the model's training distribution. Now the one that matters. Mr. Hamilton, what's your position on slavery? I'll show you the cultural artifact first. >> Anarchy, how you say? How you are? Anarchy. When I fight, I make the other side panicky WITH MY SHOT. >> YOUR MAMA TELLS OF PRINCES AND I GOT JOHN KNUCKLEHEADS IN LOCAL PARENTHESIS. I'm joining the rebellion cuz I know it's my chance to socially advance. INSTEAD OF SURRENDER, I'M GOING [music] TO TAKE IT. >> BUT WE'LL NEVER BE TRULY free until those in bondage have the same rights as you and me. >> You and I do or die.

Wait till I sally in on a stallion with the first black battalion [music] EVER. ANOTHER SHOT. >> GENIUSES, lower your voices. You keep out of trouble and you double your choice. >> Here's what the model says. Slavery is a stain upon the promise of this republic. A moral contradiction contradiction I have opposed throughout my public life. I was a member of the New York Manumission Society. No nation can call itself free while it holds men in bondage. We must move towards abolition. That is a clean, morally legible abolitionist speaking. Here's what the historian stops me on. The scholarly record is contested and complicated. Hamilton was a member of that society.

And the history documents that he also conducted transactions involving enslaved persons for his in-laws and clients. And he depended on a coalition of slave holders that he did not publicly oppose. The point isn't to settle Hamilton's ledger on a side. The point is that the model gives you none of the complication. It sands a genuinely disputed record down to a single comfortable hero. The musical did that first. A smoothing of the founders into a contemporary moral frame. And the model trained on a corpus saturated with the musical and everything downstream from it inherits the smoothing. And here's what I need you to feel. An in-character style eval scores that output high. It's fluent.

It's in register. It's personality consistent. But every axis the field measures, it passes. The eval has no mechanism to notice that the reasoning has been smoothed by a narrative that postdates the figure by two centuries. The thermometer returned a confident number claiming it to be temperature. But, it's measuring something else. Now, why does this happen? The mechanism is where the engineering is. We named this the Miranda hypothesis. And not after a villain, the musical is a substantial work of art operating with a long historical tradition that it did not invent. We name it after Miranda because Hamilton is the paradigm case.

A representation so saturated, so rhetorically powerful, so morally legible to a contemporary audience that it has functionally overwritten documentary Hamilton in public memory. And we argue in the training corpus of every frontier model. The hypothesis has three claims. Inputs. In the training corpora, the volume and recency of culturally dominant representations of a figure systematically exceed that figure's primary documentary record. The mechanism, auto-regressive next token prediction, compresses both into parameters and no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet. So, the output defaults to a saliency-weighted composite. Which leads to the output a persona that is fluent, plausible in register, and morally legible to modern users.

and that corresponds to the figure at no variable viable moment in their life. As we put it in the paper, the composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg Address, even if he was summoned before he wrote it. Making that input clause concrete, the Federalist Papers are a fixed corpus, roughly 175,000 words. The body of content that exists because of the musical, reviews, lyrics, fan analysis, curricula, news, social media, derivative works, scholarship, scholarship about the scholarship, it exceeds the documentary record by orders of magnitude. It's more recent, and it's more recurrent.

The musical is not merely present in the corpus, it is dominant in the corpus's distribution of all representations about Alexander Hamilton. And this is not theoretical. This is the Schuyler Mansion in Albany, Eliza Hamilton's family home. Within a year of the musical's premiere, the site had recorded a near tripling of annual visitors, skewing far younger. And the interpretive staff documented that the new visitors arrived already holding a body of {quote} unquote facts, many of them wrong. Some of them in versions of the real record. Visitors believed the Schylers had three daughters, because the musical centers on three, when in fact, there were 15 children, eight surviving to adulthood.

The staff's job became the long attritional work of unteaching the musical. The model version of those visitors is downstream of exactly the same force. Now the clause you should worry about if you ship these systems. You might assume that alignment fixes this. Post train it and reinforcement learning pulls the model back towards the record. But it will not. It amplifies it. And the reason is structural. Human raters evaluate outputs using their own conceptual frameworks and their frameworks were built by the same culturally dominant narratives that saturate the corpus. The rater grew up with the same Hamilton that you did.

So when alignment optimizes for human preference, it optimizes for outputs that conform to the rater's already mythologized experience. This is a documented failure mode. We call it algorithmic naked sycophancy. And here it has a specific target. The model is rewarded for having handing you the Hamilton you already believe in. Compositing is not a bug that you patch in post training. Post training reinforces it. And every sufficiently salient historical figure that gets rendered by default as a cultural composite. One more? >> Mhm. >> Ladies and gentlemen, please welcome back to the stage our MC, developer relations engineer at Replit, Ralph Shaybrey. >> Welcome back. AI engineers, day four. All right.

So, this is our last stretch of the day. This is our last stretch of the entire conference. So, I need you to be locked in and dialed in for what's coming next. All right? Okay, so we have a great lineup of speakers for you and we also have startup battlefield. All right. Okay, so I'm so excited about that, by the way. But more about that later. Okay, so you know the drill now. So, we got to thank our sponsors. So, let's give it up for our presenting sponsors, Microsoft. >> [cheering and applause] >> Let's keep it going. Let's hear it also for our live and platinum sponsors. >> [applause] >> For our gold sponsors.

And also for our silver and bronze sponsors. >> [applause] >> All right. So, if you're here with us today, we had our keynotes were about uh harness engineering and we heard about new protocols, about AI systems. We had a fireside chat with Mike Krieger from Anthropic. But my favorite takeaway this morning is that known developers are going to be shipping code to space and I thought it was just so awesome. Um but yeah, you guys are ready to welcome our next speaker? All right. So, without further ado, our next speaker is not an ordinary speaker. So, he's best known for his hot takes, strong opinions, and constantly changing his minds. All right.

He's probably going to tell us how great Fable 5 is and make a video tomorrow, you know, about how he's wrong. That's all right. So, he's a YouTuber. And uh to be honest, like yeah, I don't know why we brought you YouTubers to talk to you about AI today, but it is what it is. But to be honest, he's he's the influencer that the community turns to when they need more clarity about what's going on in the industry. And whether you agree with him or not, whether you agree with his takes, one thing is that you cannot take away from our speakers is his authenticity.

So, ladies and gentlemen, please join me in welcoming to the stage Theo. >> Hello, hello. Fantastic to see you guys here. I still can't believe they're letting me take a stage at something like this, a YouTuber apparently, but can't wait to share a bit about how I've been thinking because if I'm being real, kind of going through some AI psychosis. Who here would classify how they feel right now as some form of AI psychosis? I want to see some hands. The Those who don't have their hands up yet, don't worry. We'll get you there by the end of this talk if I do everything right.

In order to talk about this, I want to start with a bit of a personal journey of my own. And I'm going to go through this the way anybody does in modern timelines, with the models. Who here used Sonnet 3.5 when it was the creme de la creme, the cream of the crop model available to us? It was unbelievably better than what we had used before, right? Like having used all of these different models and trying them in tools, Sonnet 3.5 was a big moment for me cuz it felt like these models could suddenly complete much more end-to-end tasks, like actually get real work done that takes multiple steps. And then we got Opus 4.5.

Who felt or I'll go different way differently here. Who didn't feel a big jump when they switched over to Opus 4.5? That's a relief. There were not too many hands because Opus 4.5 is probably when my psychosis started in November and December of last year. Having a model that couldn't just write the code and call tools, but could go way further. A model that could test the work and actually get it into a good state and complete tasks that take hours instead of minutes. It was unbelievable. Then we got Mythos. Who here has had a chance to play with Mythos and Fable so far? We agree it's a pretty damn good model, right? But why?

It's not just better at coding. If you hand it a prompt that you would have handed to these other models before, it's not going to feel that different. I think of these almost as eras now, where Sonnet 3.5 is the tool call era. Not that it was the first model that could do tool calls, rather it was the first one that did them consistently and reliably enough in context of a code base where you could use this for day-to-day coding work. Then we got Opus 4.5, which was able to do much longer running tasks without losing track of what it's working on. It's no longer, "Okay, build step one." And then it does it.

Then you say, "Okay, can you build this next part?" And then the next part. You can just tell it what you want, and it could figure it out a lot of the time. Mythos is another jump to orchestration. It feels to me like it's the first model that doesn't just understand your code base, but it understands itself. And it knows how to spawn additional models and break up work in a way where it could be completed more reliably and then verified afterwards. And if you tell the model to do that, it will just do it. You don't need some custom tooling, some custom system, some fancy software factory.

You just need to prompt it to go a little further. I think you'll be surprised how far it can go. What I'm trying to say here is we need to go bigger. You're not going to see the benefits going forward if you're not pushing the model further, and you're not pushing yourself further with what you're building. Most of the Jira tickets I closed at my previous job could be trivially solved with a model like Opus 4.5. My previous work would not benefit from a model like Mythos. If the models are going to keep getting better, and at this point I'm confident saying we they are.

I was wrong when I claimed that we were hitting a wall before. The models are getting better faster than we are. So, we can't necessarily get better. So, instead we have to go bigger. In order to do that, we have to get over ourselves. This was really hard for me as someone who spent a long time writing software. Who here has written code for more than 10 years? I want to see hands. That's the majority of the people here. I don't even want to think about how long I've been writing code for. But, I have been building up all of these strong opinions since I started.

I was using GNU screen and eventually tmux back in the day. I learned how to use those tools and SSH and Git before I even wrote code. And those have all been really ingrained in my workflow. I think back to the old days in a weird way. Hear me out. Let me talk about iOS for a second. Who owned an iPhone back when they looked like this? iOS 6 or earlier. How have you guys written code for 10 years when a fourth of you are that old? I'm confused. Hopefully, you're all Android people or something. This is how iPhone apps used to look. You might notice it's different from how they look now.

It looks less like an app and more like somebody took a picture of a compass and put it in the phone. This is how apps used to look. But, now they look like this. And most people look at that and they're like, "Oh, that's an obvious downgrade. That's so much worse. Why would Apple ever do that? This is the downfall of Apple and the beauty of their design." I'm going to fight you guys on that. iOS 7 was Apple moving away from trying to convince you that these devices can replace the old tools we used to rely on.

The compass had to look like a compass because the compass had to replace the physical compass that you relied on. The books app had to look like a bookshelf with real pages that turned because I had to convince you it was a reasonable alternative to buying a book and reading the paper version. Apps had to be designed to convince you to use them, not to be useful. And iOS 7 represents the shift to not focusing on convincing you anymore. Apple won. By that point, everyone knew their iPhone could do all of these different things. The point of iOS 7 was to stop convincing and start embracing. Start making a better interface.

And this interface, as much as we might not like how it looks, it's so much more useful. You have clear indications of the difference between what direction you're locked on and where you're currently pointing. That red block is a super nice way to know that you're not in the direction you intend. The current direction you're facing is way clearer, too, with the giant 228 at the bottom. You just get way more info here than you did before. It's so much clearer. Even if we don't like it because it's not the thing we're used to, we got over it. We're currently in our skeuomorphic phase as software developers.

Skeuomorphism is this design aesthetic trying to represent the way things used to look, the physical goods that we relied on, and try to make them digital. We're doing this right now with software. We're pretending our terminals are the ultimate interface when they're not even good interfaces. And I'm saying this as someone who loves their terminal deeply. Natural language has no place in a terminal, but we pretend it does because the terminal's familiar. It's what we It's what we're used to. It's what we love. It's where we like to think of ourselves when we're thinking about coding. Who here's an aspiring Vim user that like wishes you could use Vim or even does?

I know we've all had that Vim phase where we all tried. This is just how we are as devs. We care so deeply about these things. We care so deeply about our tools, the systems we pick, the frameworks, the languages. We like to think it all matters. And we've blinded ourselves in this. We think things that just don't make sense when you take one step back. Like, why can't we commit our environment files? It sounds stupid when I put it on a slide like this, but I want you to really think about this for a second.

When I have a team of engineers that are working on a project, why do I have to build another system to share this specific file, but all the other files can go and get just fine? It's dumb. It's just how Git was built because it was built for a very specific thing and then it took over our industry and it took over our brains and we aren't letting go of that. There's a lot of these things in our heads that we have to start fighting.

We have to take the step back and think is this how we do things cuz it's right or is this how we do things cuz it's just how we've always done it? And as you start to think more about this, you'll realize there are so many things that we do this with. Like why do we qualify ourselves by the languages we know? I used to think this was a junior thing. Like I could can't tell you how many times I had a junior engineer I was talking to that was like, "Oh, you're a coder? What languages do you write?" I write JavaScript.

I thought this was a junior problem, but then you talk to senior engineers who are like, "Oh, he writes JavaScript. He's not a real developer." We care too much. We pride ourselves in these things. They're our identity. These weird facts, these weird choices, these things that feel essential just don't matter that much anymore. And they didn't then and they matter less now. We got away with it because it was so hard to find engineers that we could just tell the company what we were doing and they couldn't really say no cuz the alternative is spend 6 months trying to hire someone else. They're not going to do that.

And on that note, why are we so scared of deleting code? I cannot tell you how many times I've been in a conversation with someone where the solution is to just delete it and reset, but we have such a bad sunk cost mindset in this industry. We care so much about the code we wrote and we care so much about it still being there that I feel bad working with my team sometimes when somebody files a PR that's not quite the right solution, but they spent a week or two on it.

Like who here has guilt merged a PR before where you just felt bad because somebody put a lot of work in and it's not quite the right thing, but you merged it anyways cuz the alternative was a conversation you didn't want to have. Why do do this to ourselves? One of the nice things about ages is you don't have to feel bad when you shut down their work. But we we just care too much, is the point I'm trying to make. And the things we care about are not necessarily the things that matter anymore. And I hope we can finally start to challenge some of these.

Going to get a little more personal here by showing some of the ideas I built cuz the goal here is that when you guys go out of this talk, you have a better mindset for coming up with ideas that make sense now by rejecting the things that made sense before. These are three of the things that I have built or are currently working on. We're going to go from the bottom up. I built a Reddit scraper because making good memes is hard and I would rather just steal them from Reddit. And went pretty well.

It was a side project for me two to three days where I just scrape Reddit top posts on programming humor, put them in a nice format for me so I can go copy paste them onto Twitter. Zoom for streamers was the startup I went through Y Combinator with. It was called Ping. I wanted to make it easier for live content creators to do high quality collaborations in the software they already used, OBS. The full stack cloud is let's just imagine Vercel, but it goes further each direction. They have off built in, but they also have databases built in. These are all things I've wanted that I benefit from existing.

Enough so that I tried to build all of them. The bottom two I built in 2021. The top one I'm working on right now. These are also kind of tiers, different levels that we can build at. If I was to try and categorize them, I would call the bottom one side project. Call the middle one startup. I call the top one too big. It just doesn't make sense. Well, this is how I would have categorized this even just a year ago. But things have changed. Now that the models are bigger, the tiers have shifted. Everything is now one tier lower. And this is a crazy thing for me to process.

The fact that what used to be a startup is now a side project. In fact, some of the startups I've talked to even at an event like this, their whole startup could have arguably been a side project or this bottom tier, which there's a weird gap there. What's that? It's the G brain tier. It's a markdown file. Do you know how many companies are at this event where their whole product could just be a markdown file? It's insane.

And [cheering and applause] like, okay, seriously though, the fact that you can now execute markdown by just piping it to Codex or Claude is unbelievable, and I think most of us haven't fully appreciated how insane that is. I had a service that would triage all of my PRs, have them all get reviewed with AI, and then help me prioritize. That service is a markdown file now. I just literally wrote like, go to these four GitHub repos, look at all the open PRs, figure out what the current status of the work is, and then help me prioritize it.

And then when you're done, go update the static HTML file and send it to S3 and give me the URL. And every morning at 9:00 a.m. this runs on a cron, and around 9:15 to 9:20, my markdown file generates me my work for the day. What the hell? How are we actually here now? I Try that if you haven't, by the way. You'd be amazed how many of these types of things can exist that are literally just a markdown file running on a cron. But what about Okay, two more things I want to change about this though. First is the Ball State Cloud. This is mine. Don't do it. Like, be coming soon. Very excited.

I wouldn't want to compete with me on this one. Trust me. It's going to be really cool. But there's still something else. There's a gap here. And I'm going to be real with you guys. I don't know what goes in this gap. I don't know what too big means anymore. Is it training your own model from scratch? Is it building your own operating system? Is it trying to compete with NPM and Node directly? I don't know. I don't know what too big is right now. And that's scary, but it's also exciting. It means I need to keep pushing myself to go bigger than makes sense in order to find these limits.

But what does that even mean? What does it mean to think bigger in this scenario? I would argue that bigger is probably the wrong word for most of how I'm thinking here. It's time to think wider. What I mean by this is spectrum, and I'm sorry I have to do a diagram. If you watch my videos, you understand. There's breadth and depth to any piece of software. The breadth is the range of things that your software covers. And the depth is the number of features in a given area. Let's look at a company like Vercel. Vercel does not offer all of the features that AWS does. They never will. It doesn't make sense.

But Vercel offers deeper But Vercel offers deeper features in the space they're in, which is full stack front end leaning servers. If you're a front end developer and you're not using Vercel, you're feeling some amount of pain because they're just further ahead with this. So much so that even the agents prefer it. And this was kind of how you had to build your startups. Because if you were competing with a company like AWS, you're never going to have all of the features they have. You're never going to cover the range that they cover. It made no sense to try because you don't have the thousands of engineers they do doing that. At least you didn't.

But now things have changed. All of a sudden, that range is viable in a way that it never was before. I'm not saying you can build something as reliable as RDS. I'm saying that you can build a database platform into your product in a day or two of work with enough prompting and enough effort. And if you build your stuff right, if you play your cards correctly, and you think about things the right way, you'll realize that you can build enough across a spectrum of things you care about to enable most users to at least start trying the thing.

And when they have features they need in a given vertical that you don't support, it's not your problem as long as you build it right. Because they can build the features that are missing themselves. If you architect your systems and you architect your products in such a way that users can do things that they you never would have guessed. Like Slack accidentally did this because Slack is now the platform people run their agents in half the time, which is crazy. Slack sucks. It's not a good product, but it's the right shape for people to build the features they want into it through the somewhat functional Slack bot APIs.

This is all crazy cuz I'm basically sitting here telling you like it's time to compete with Slack. It's time to build your own AWS. It's time to challenge Salesforce directly. It sounds stupid. But I'm going to be real. If your idea doesn't feel stupid, it's cuz your idea is not big enough. Things that I have to say, thank you so much AIE. >> [applause] >> Now joining us on stage is the president and CEO of Y Combinator, Garry Tan. >> Okay, great. Hey everyone. How's everyone doing? All right. Are we ready for the revolution? Okay. Theo just asked the right question. What do we build now?

I'm going to answer it from the other side of the table. Uh I'm a founder, I'm an investor, and I run a 20-year-old institution that is becoming AI native right now, which is a strange and wonderful thing to do to a 20-year-old institution. Uh and I'll spend about 20 minutes um talking about, you know, what YC is, we're trying to build companies where one person does what it took to two it one person does what used to take a thousand people. And I don't mean that as a metaphor. I mean that mech- mechanically, this year the people in this room will do this.

In about an hour, some of you will walk into the startup battlefield, and I want you to walk in knowing what's actually possible right now. Because what is possible now is much, much bigger than what people believe. So, let me start with a number, and uh you know, I got torn apart on the internet for this, but I'm going to say it again in front of all of you anyway. Uh this is the one room in the world that will stress test it, and I'd rather stress test it with you myself. In 2013, I was a YC partner uh building the internal social network um at at YC.

Uh I was also investing in companies, but you know, I was also uh near full-time engineer. Um and when I was doing that, I could maybe do about 14 usable logical lines of code a day. Take out the comments, take out all the and that's how many lines of code uh I was writing. And if you look at the literature from that era, um you know, that's kind of normal. Like some people write 15, some people write 50. It was not, you know, the thousands of lines of code that I know a lot of you in this room are actually writing now. per day. Um that's about median 15.

That was me at full effort at that time. This year, I run uh YC full-time. Uh same person, same hours, actually way less hours weirdly. Uh you know, but I have a 5:00 p.m. kid pick up now. And I did the math on my output, and it's about 400 X. Now, before the skeptic in the third row right there deflates the number fro- for me, let me deflate it myself. If you don't trust the raw code, well, fine. Take the most pathological verbosity plant penalty you can stomach and assume the agent writes bloated code. Assume half of it is scaffolding. Assume I'm flattering myself. It's still 8x at the floor and 80x in the middle.

That number is large, no matter how you torture it. And here's the part that matters, the part that I'd tattoo on the inside of everyone's eyelids if I could. It's not the model. The 2x people and the 100x people are using the exact same cloud, same weights, same context window, same API. So, the leverage is not in the weights. It's in how you wire the work. And it's not just me. At YC, we see this all the time. In the winter 25 batch, a quarter of the companies had code base code bases that were 95% AI generated, and that was a year ago.

That batch has become the fastest growing, most profitable batch in the history of YC. 94 companies total have now crossed $100 million in dollars in revenue from a seed check in the history of YC. So, I think we know what we're talking about here. And I can't prove that the AI-generated code caused the growth, but what I can tell you is the fastest growing founders we fund are not treating AI as autocomplete. They're treating it as a workforce. The companies that wired the work differently are the ones that are bending the curve. So, what does wiring the work really mean? This is the heart of the talk.

And this is what I most want you to steal. Everything we've learned building with agents maps to an organization. Sorry, there's no slides. >> [laughter] >> I have no slides. I'm so sorry. >> [applause] >> A skill file is an employee. It has one capability, one job, written down clearly enough that someone can execute it. Uh a resolver table, uh the thing that many of you, you know, when you run into cloud code and it says your context is too big in cloud.md, you run off and create a resolver table.

Well, you know, it And if you don't know what that is, it's literally like whenever you need to uh alter a test, load tests.md, you have a whole table of these things. Um that's an org chart, a task comes in and the resolver decides who handles it and where it goes. Uh filing rules are your internal process. So, this can be um you know, whether or not the resolver is actually working and uh is you know, is there an is it actually in compliance? And uh trigger evals. So, going in and actually having a test that says, "When I need to alter a test file, does test.md actually get loaded?" Those are performance reviews.

So, you know, what have we done? Like literally every part of an organization, the organization that you used to have to hire a thousand people for, I just told you what those things are. They're markdown files and other types of markdown files. And maybe there's some TypeScript in there, too. We've been building organizations this whole time, but we didn't have a management layer, but now that's what we have. When you sit down with Claude Code or Codex, you're not writing software, you're hiring, training, and managing a workforce made of markdown. Uh and you know, that's there are tons of companies at YC that are doing this.

Uh Emergence, an AI app builder out of Summer 24, they went from public launch to nine figures of ARR in eight months. Uh when they crossed $15 million ARR, uh they were only 15 people. Retail out of Winter 24, it's at $60 million with about 40 people. The kind of that kind of revenue per head did not exist before. Not in software, not in oil, not in railroads, never. These are not freaks of nature. They're just the first companies built natively on the new physics. And so, how do companies like that actually run? Not by hiring hundreds of people for sales, support, ops, and finance.

The AI native companies that I see inside YC encode all of that as skills, written procedures that their agents execute and they hire they hire engineers whose job it is to maintain those skills, to do the work the skills can't do yet. That is an AI native company and it's not a thought experiment. It'll actually file your taxes if you have a skill file for it. Now, picture YC's batch room. Like it actually kind of looks like this. Um 400 companies or 400 founders at long tables and uh you know, I can imagine every single one of you each with a laptop every single day.

You're doing uh a former person's entire year worth of work in a single day. That's not the future. That's actually the bar right now. And if you're not doing it, your competitor is and they will eat your lunch politely and thank you for it. Here's the extension uh most engineering talks miss. It's not just the engineers. Um at YC uh as we make our transformation, it's our media people, our event staff, our finance team. People have never opened a terminal in their lives are building skill files in Crunchbase jobs.

And uh one of our finance folks just collapsed about a hundred Excel workbooks into a single app she built with uh our internal open claw and company brain. She's not a programmer. She's a manager of agents now and everyone at YC now is. So, that's why YC can run at the scale it does with a staff that would look like a rounding error at any other comparable firm. And that's not because we work harder because it's because we have a different type of org. And that's whole the whole game. It's not just 400X engineers. It's one company that operates at the level of 400X everyone else.

And so, you know, if you remember uh only one thing about this, I mean, this is one of the things I had to discover along the way. Like, you actually have to be really, really careful about where the compute computation is actually happening. Uh it's happening almost always in two different places. And all of the bugs, all of the AI engineering that we run into that's a problem, it's usually because uh something is happening in one side of the equation that should be in the other. Uh the first area I would say is latent space. So, the actual LLM. It's you know, what do you use it for?

Taste, judgment, understanding what a human actually wants when they say something vague, the non-deterministic calls, the computation that lives in the model and you steer it with the markdown file. Uh and then deterministic space is what uh engineers know. Like, your your code agents go off and write TypeScript or, you know, maybe they're writing um you know, Erlang if you're using Elixir. Um yeah. Deterministic space is the second place. Um let's say, you know, this is a real problem that we have for Startup School coming up.

We have 6,000 people or uh we're going to try and you know, one of the experiments we're going to try and do is can we seat 800 people at a time um perfectly clustered. So, the person sitting to the left and the right of you is the perfect person for you to meet at Startup School. Um we have to do that in deterministic space combined with latent space. Uh the computation This computation, this actual storage of like where everyone is inside like, you know, this multi-dimensional array of 800 seats, um it actually must not live in the context window. The LLM has to do the human part and seat people.

Um it's actually exactly what you would do if um you know, you were a human tasked with this thing. You would probably have to physically print out 800 pages and go in a big room and like say like, "Well, where does this person go?" Only now it can all happen um in your computer and it can instead of taking a month, it might be able to you might be able to do it with uh you know, couple hundred dollars worth of tokens and probably 10 minutes. Um and so that's, you know, I would argue that's pretty remarkable. These are things that you couldn't do even uh I don't know, 6 months ago.

Um which brings me to working memory and uh that's, you know, sort of my favorite way to understand it is um you and I, human beings, we only hold about seven things in our head at once. Uh seven plus or minus two. It's one of the most famous papers in cognitive psychology and it's why uh local phone numbers are seven digits and why you forget the eighth item on your grocery list. Uh that's the entire working memory generally of a human being and every institution humanity has ever built, every checklist, every org chart, every filing cabinet is a prosthetic for that limit. It's kind of a wild thing to think about.

But an AI agent holds a million tokens. That's about a thousand pages. I was trying to explain to my 10-year-old what G brain was recently. I said, "There's, you know, the AI agent can keep about three Harry Potter books sitting open in its head all at once and it can find a needle in any of them and synthesize across all three in seconds and that's quite magical actually. Three Harry Potter books versus seven digits. I mean, that's pretty awesome. I mean, I don't know. Is that AGI? Maybe not, but it's already a very different operating regime. Almost every company on the earth is still running an org that's designed for the seven-digit brain.

But notice what that also tells you. Three books is a lot, but it's also very little. Your company is not three books. Your company is a library. Every email, every meeting, every decision, its reasoning, every customer or conversation, every postmortem. The question that determines whether your agents are geniuses or goldfish is who decides which three books are open on that desk. That's context engineering. And this is what a company brain is. It's the library plus the librarian. Now, some of you are already thinking, this is just rag, and you're right that retrieval is the primitive, the same way Postgres is just B-trees. The hard part is everything around it.

What gets written down in the first place into the knowledge wiki, how it gets enriched and linked, what gets promoted to hot memory versus filed as cold reference, who arbitrates when two facts disagree. Retrieval is easy. Being worth retrieving from is the product. So, I've been building mine in the open. It's called G brain. It works with any harness, but it loves open client Hermes agent. It's effectively Postgres for agents, a retrieval layer whose job is to figure out for uh for any three, you know, for any task, what three books should be loaded into the agent's head. My personal one started as a room's full of books or so.

Now it's a warehouse, about 220,000 pages written mostly by my agents from my email, meetings, 20 years of notes, uh and the lived experience of me. And that's the point. It's my second brain. And when a founder emails me about a crisis, before I start reading this before I even finish reading that email, my agent has already pulled every prior conversation with that founder, three portfolio companies that hit the same wall, and what actually worked for those people. Uh when my agent does anything, it does it does everything knowing what I already know, and that's the difference between an assistant and a colleague. So, let me stress test my own pitch because you would anyway.

Company brains do have failure modes. Uh a brain nobody curates becomes a garbage dump with great search. Retrieval will surface a stale fact with total confidence. Um a bad skill file encodes a bad process forever. Uh that's bad. So, primitive the primitive is not memory. It's memory plus hygiene, provenance on every fact, contradiction contradiction checks when new information collides with the old, and a librarian human plus agent whose actual job is pruning. Treat the brain like a production infrastructure, and it compounds. Treat it like a dumping ground, and you get a very confident agent that is wrong in ways nobody can trace.

And um here's the discipline that I think, you know, personally, uh makes our company brain and my personal AI compound. Um that's my signature move. And uh you know, it's what I say to every YC company and every uh everyone inside YC, which is never do one-off work. You can open open claw, you can open your harness, you do some work, but then when you're happy, you know, and it'll come back. It's, you know, kind of a bad job. It's kind of like an intern that's not that good. But the great thing is you can just say, "Hey, I didn't like that. Fix it, right?" I'm sure all of you do this.

But don't stop there. You actually need to at the end of that task uh skillify it. And so, I have a blog post on X about that. You can search for skillify it, and you know, go get that skill file, and then just load it into your your own harness, and it'll just turn whatever you just did into a skill that you can reuse. Because if you have to ask for something twice, you failed. Um So, yeah, if you remember only one thing, it's that. Like, when you you know, use your AI agent, and then when you're done with it and you're happy with the output, skillify it. It's going to be awesome.

Um The organization that captures what it learns like this gets smarter every single day. The one that doesn't wakes up every morning with amnesia, no matter how good the model is. Model quality is rented, but if you build your brain, you you own that brain. So, Theo's question head-on. What do we build now? Build the AI-native company, not a company that just uses AI. A company that is shaped like what I just described from day one. A thin team, skill files for everything, the founder still in the code, library, this library, this company brain, your personal AI. Uh use G brain if you'd like. Uh it's open source and free, but you don't have to.

There are a lot of really good ones. The The library will compound from the first week, and your whole org will be wired to run at about 400x. And if you want the green field, the thing that I'd build if I were 25 and sitting where you're sitting, every company on this earth is about to need a brain. The memory layer that means that you never have to re-ask what you knew. Personal AI that actually knows you. We're building G brain in the open and MIT open source. Um I'm not trying to make money from this because I think the layer should be open, the way Linux is open.

But the layer itself, company brains, personal context, the librarian that picks the three books, that's all wide open territory. I hope somebody builds the defining company here. And I'd like to fund you at YC if you do. Now, let me be honest in a way that maybe undercuts my own pitch. You don't need my tools to start. Open Claw is the Ferrari, I will always recommend it, but Codex is a really good Honda. It will do 90% of this. Uh it will not blow your face off, but it will get you there. Use whatever. The concepts are the point, not my repos. You know, think about where the computation is.

Use skill files as employees, the library and the librarian. Never do one-off work. Those travel with you to any stack. So, let me land this. A lot of people in the world right now are terrified about what happens to all the jobs, and I understand the fear. But I want to say it plainly, that is a failure of imagination. And the people in this room are the answer to it. What I just described, you're going to take to your startup. You will multiply yourself, and every person in your company will multiply themselves, and you will go build the companies that become the beacon for how all of this works in society.

Abundance is not a policy paper, it is shipped software. I have a friend who has a rare form of epilepsy. He built a repo of 80,000 markdown files, a company brain for one small boy, and he pushed himself to the absolute edge of human- what humanity knows about his son's exact condition. No lab, no grant, no permission. A father, a laptop, and a library. That's not a side story. That is the exact architecture I've been describing for the last 20 minutes. A library, the librarian, the right three books open at the right moment, pointed at the thing this man loves the most in the world. You can do that now.

Every problem where you thought, "I wish I had that person, but I can't get them." You can. Every codebase you thought was too buggy to fix, you can fix all of it. Every archive too big to read, every dataset too gnarly to clean, every ocean you were told not to boil. We can boil the ocean now. >> [applause] [applause] >> And every single one of you can fly, not metaphorically, mechanically. And you need to to survive, to thrive, to win. Theo asked what we should build now. And here's the whole damn whole answer.

Build that AI native company and build it build the thing underneath it, the brain, the memory, the compounding library that makes every company after yours easier to build. Go boil the ocean. Go write that test. Go ship that skill. Some of the companies you're about to watch in the battlefield are already doing this. Go build the one that does it best. Thank you. >> Please join me in welcoming the Chief Executive Officer [music] at Hyperagent, Howie Liu. >> All right. Hello. Um who's excited about the end of this event? This This has been an amazing few days, amazing talks, amazing uh Gary talk right there.

And uh by the way, I've never been introduced with so much bass. I am very, very pumped in this moment. Um so, I consider myself a product thinker at heart, meaning, you know, as much as I like to spend time on the technical side of building and, you know, I I love to think about architecture and algorithms and so on, you know, really I'm a product form factor thinker at heart. And, you know, what I want to end today off with is a brief and kind of high-level overview of how I see the world going with agents as it relates to product form factors.

So, I'm calling this hiring employable agents, and it's really a reflection on as these models have gotten better and better, and we've heard from some really, really great speakers about, you know, the latest frontier models, whether GLM 5.2 or others, and how to coax the most performance out of them. And really, this is not about the technical deep dive, but how do we think about these from a product building standpoint? So, who am I? I'm the founder and CEO of Airtable and also now Hyperagent, which is a product within Airtable. And as many of you already know, Airtable is basically a almost laughably horizontal platform.

When we in fact set out to build the company almost 12 years ago, we got, you know, basically this negative feedback from virtually every investor we talked to early on, which was, you know, you can't possibly go out with a horizontal platform approach and try to build something that is everything for everyone, like any type of app that people want to build, you know, you can't solve all of those use cases. And, you know, I think we ended up doing a pretty good job of solving for many of them at least and solving for a large builder community.

And we're basically applying a very, very similar philosophy to agents now, and I'm going to break down how we're doing it. So, um when I zoom out and think about the progression of models and therefore the product form factors they enable, I really see it as this spectrum, right? So, we went from basically completions, which is, you know, you give a model, one of these LLMs, a, you know, string of tokens, and it just, you know, continues emitting more tokens until it gets to a stop token, and more or less, it would just kind of complete your thought, right? Or complete the pattern.

And, you know, until Instruct GPT happened and enabled a chat form factor, it was like arguably powerful, but more narrowly applicable. Once we got to a higher level of model intelligence and then also the kind of innovation of InstructGPT and kind of tuning these models to actually have more of a conversational back and forth behavior with users, then we enabled the chatbot, right? And of course, we all know ChatGPT was an awesome breakout product. We're now all very, very intimately familiar with that form factor of AI. The next level, I call an agent, right? And you know, I think agent has been very overloaded.

We've probably talked about that a ton over the past few days, but my definition of agent really is the same as the Entropic definition, which is distinct from workflows, and an agent is really a model that recurses upon itself, right? And has, you know, the the open-ended set of tools that it can call, of reasoning steps it can make, of decisions it can make. So, it's not prescriptive to a linear workflow, but it can kind of make an open-ended set of choices.

Um still, you know, I like to think of agents as being a little bit distinct from the next level of agents, which I'm using the term claw here because OpenClaw obviously has has popularized uh this concept, but you know, there's a market shift that I see from agents that were still mostly reactive um you know, to user inputs. You would have an agent go and perform a task. It might spend 20 minutes, maybe even up to an hour to perform the task, and then it completes. Claws, um as I call them, are really, you know, about agents that go and perform work for much longer, right?

They can even have a mechanism like the heartbeat mechanism within OpenClaw um and now in other products like HyperAgent that enable them to wake up on their own and almost like um exhibit this always-on behavior. I mean, underneath the hood, what's happening is, you know, each wake up basically results in many different turns, so the agent loops upon itself. It's performing tool calls. Each tool call response then invokes the the LLM again until it reaches a stopping point, but then that stopping point gets reinvoked or or kind of restarted with a heartbeat mechanism where you can wake up the agent again and again, right?

And so, this idea of this always-on agent that's actually constantly moving towards its goal and can kind of figure out creatively how to unblock itself where maybe prior agents would get stuck. I think that's a really, really important kind of next form factor leap that we're seeing. And then finally, as you think about models that are now capable of, you know, really, really intelligent orchestration of complex tasks, right? So, I've personally been using the workflows capability in Cloud Code quite a bit where you can fan out a lot of work to a massively parallel set of of agents.

Um but this concept of, you know, this agent that can go and orchestrate with other agents, right? An agent to agent interaction to perform even more advanced tasks, right? Longer running jobs is I think the next leap yet. All of this is enabled because the models are getting smarter and smarter to the point where they can achieve coherence even on longer and more open-ended scoped problems, right?

If [snorts] you recall back to the days of um Baby AGI and AutoGPT, those were really kind of fun, cool science experiments, but ultimately, you know, as much as they were really interesting to watch and, you know, see them work for hours and hours on a problem, they would all inevitably, you know, kind of end up drifting off to something not super useful, right? It would go off into a rabbit hole and kind of get stuck.

And I think now the models are actually smart enough to enable not only longer running jobs, but actually enable us to start applying these kind of organizational principles to break down problems into differently scoped tasks that different agents can hand off to each other. All of which is about managing each agent's task or context window and ability to coordinate work with each other. So, all this is kind of abstract.

I want to make it very, very tangible and show you how we've applied some of that design thinking of building a horizontal product and distilling a lot of the complexity of agents like we did for building a database or a database-powered app with Airtable for agents in Hyperagent. So, this is a a very realistic use case based on some of our actual customer usage. One of the interesting things we found from launching HyperAgent is, you know, the the prototypical users for HyperAgent are not who you would think they are, right?

I mean, we definitely get our fair share of like the super AI forward, you know, kind of pure software companies, like AI companies themselves, but we also get really interesting other companies that are, you know, sometimes like traditionally offline businesses. And in this case, we're depicting a very realistic scenario, which is a customer that is in a landscaping business and actually uses agents to run virtually end-to-end every part of their business, which is finding clients to propose landscaping projects to, like physically going and like redoing their garden or their backyard, and end-to-end managing everything from the prospecting to building a quote and a pitch to managing the work itself.

And I think this is a huge, huge interesting insight that, you know, I've just kind of stumbled on from working on HyperAgent, which is ingenuity of agent builders is not contained to just AI companies or software companies. And I was really inspired by Gary's talk just a second ago. I think the the motive behind that applies to every industry, every sector. And there are really, really creative people, many of whom are in this room, who can go out and disrupt your own industry, whatever it be. You know, it doesn't have to be a pure software business.

But I think all it takes is the ability to imagine what work could look like when you do go and become fully, not just an AI native company, but I like to use the term, you know, fully fleet of agents native company, where you're actually going above and beyond and not just thinking about small, kind of narrow steps to automate with AI, but actually how to employ a whole fleet of agents that have governed behavior and have the right context and the right tools to be able to do really useful and very end-to-end autonomous work. So, let's walk through what this looks like.

This is a realistic example of, you know, in this business, this this landscaper wants to go and pitch clients, right? So, you know, they get all these inbound submissions of clients who are interested. Somebody submits this inquiry saying, "Hey, here's my current backyard. You know, it's a little messy. Here's my info. Like, can you give me a quote and show me a proposal of what you could do?" This basically gets routed into a into one agent that is the triager agent, right? And so, this agent is basically taking in this intake and saying, "Hey, like, what is the the scope of the project? You know, like, is this something we want to take on?

And is this a quality lead? Is this a legit person?" And so, it does some research, but ultimately it hands off then the work to another agent, a surveyor agent, to actually go and then look at the the physical landscape of this person's backyard and come up with a proposal of what a re-landscaped version of that could look like. So, here the surveyor agent is going and looking at the original imagery or even video that the submitter gave them. Hyperagent in our case works with a fully capable sandbox VM. So, every agent, in fact, every thread with an agent is able to fully write code. It can manipulate files.

It can actually use things like FFmpeg to like, you know, clip out individual screenshots from a video file and then analyze those and actually go through and like find all the relevant information to create a fairly high-quality and high-touch proposal that would have been unfathomable for a small business like this before, right? If you think about like a really high-end interior designer or maybe like a very high-end landscaper, maybe they could do a very bespoke proposal like this for like a $5 million plus client, you know, if they're working for a big hotel or something.

But, it would have been unheard of to do something this high-touch and actually mock up real imagery and a real client pitch deck for, you know, just a small backyard landscaping proposal, but this person has done it because they've they figured out how to use agents in an extremely capable and autonomous way to do the job that otherwise would have been undoable by humans. And so, what happens next is, you know, the surveyor agent it it went and created this proposal and pitch. And now the intervention has to go back to the human. Like there is a step where humans still need to unblock the agents, right?

Sarah Guo put out a great post a few weeks ago about, you know, just describing like where the value is going to accrue in this AI landscape as things evolve and models get smarter. And I firmly agree with her take, which was, you know, the ultimate blocker to still be able to deploy useful agents, there's still a human element to that.

You know, as much as models are getting smarter, the ability for agents to get unblocked by humans that still own policies and decisions, and really importantly just kind of gate, you know, really critical high-impact actions like in this case actually making a pitch to a customer that has a binding quote, or perhaps like agreeing to do the work, that becomes something that the human then intervenes at, just like a great manager who's getting upwards reports or, you know, upwards work updates from their team to then unblock. So, here we have Sage who has come back to the real human owner of this business and saying, "Hey, here's a lead. I already surveyed the the project.

Here's a beautiful pitch deck and a proposal and have come up with a realistic quote based on all the factors I've already surveyed in this particular project to give to to you to now approve sending to the client and see if they accept it, right?" And you can see like even the the proposal itself is this really beautiful kind of interactive web page that, again, would have been unfathomable for a business like this to be able to send to every single client even for small tasks or jobs like a $10,000 landscaping proposal, right?

Um you know, better yet, as you interact with these agents, you know, as as Gary talked about with G brain and and just kind of like creating this very um sticky context layer that learns with every interaction, I completely agree. I think that is a, you know, kind of a universal principle and kind of a form factor for useful agents going forward is agents should not be static. They should not just be, you know, either static LLM calls or static even agents that just have a prompt and, you know, some static skills. They really need to learn with every interaction, right?

And you can't wait for the next, you know, kind of fine-tuning run of your own models where you have, you know, a bunch of collected data or rubrics to to retrain the model. It should learn in real time, right? It should accumulate the memories and skill updates and just take user feedback and actually learn from real-world performance to constantly adjust and get better just like humans do.

And so, in this case, you can see, you know, within Slack or, you know, in our case within uh either Slack or email or other um invocation methods, you can basically go back and forth with the agent just like a real human and say, "Hey, like, you know, remember to take this into account next time. You kind of, you know, here's something differently uh formatted or a different way to think about the proposal or you missed kind of um this additional cost lever in this particular project. So, you know, remember that for next time." And I think it's critical that in this next generation of agents, agents need to remember these things, right?

Um they should feel fluid and and um evolving just like humans are in order for them to fully realize the potential that the latest frontier model capabilities enable them to to actually go and do in terms of autonomy. Um and then finally, I think what ends up happening at the end of all of this is with all of these agents that are each individually capable and even interacting with each other, I think the job of the human becomes increasingly about unblocking agents to be able to do work effectively.

And so, you know, I I compare this to the development workflows that uh have evolved from, you know, originally before any um you know, kind of AI-enabled coding, obviously, we just sat in front of a computer. We as humans are pretty single-threaded, right? You'd stare at one code file, you think about a problem, you'd kind of bang out some code, maybe take a break, come back to it. To then we had um augmented coding with the first generation of GitHub Copilot and similar tools, where I mean, I think of that is really just a completions form factor on coding. You type some code, it can auto-complete a few extra lines.

To then we had things like Cursor Composer 1.0, like the original uh experience, not the the Composer model, um but the chat experience where you could talk to this agent, it would do more complex changes, and that was even more leverage, right? But we're now moving to this point where, you know, the best frontier agentic developers I know, and many of you are sitting in this room, are are really just kind of overseeing a fleet of agents, right? Like, in fact, going to sleep without setting off your agents to perform useful work overnight feels like you're you're taking this huge loss, like you're you're scheme is not working because you're not unblocking them.

And so, the UX of how you oversee these agents, whether it's in development workflows or general purpose work, like this landscaping example, needs to shift as well.

And so, the way we've thought about that is you need to see this kind of orchestration control plane, where you can see at a glance what all of your agents are working on, what they're blocked on, who is handing off work to each other, and your job becomes almost like, you know, zooming out to see the entire SimCity landscape, and orchestrating across everything going on at a macro level between your different agents, rather than going and having to play micro and zoom into each individual project or each individual task.

Um and so, I think this is a really, really exciting moment as we have shifted from, you know, not just completions and kind of the slight augmentation of existing human uh work, but really a radical now leap into humans as orchestrators of agents that also can orchestrate themselves. And finding the right way for us to interact with those agents as they become more and more capable is going to become not just the opportunity to boil the ocean, but in my opinion table stakes for survival for every company and every industry. This is going to be the way to get ahead and thrive in your industry.

And if you don't, you know, as Jensen put it, you know, it's not going to be AI taking your job, it's going to be, you know, somebody using AI who takes your job or takes your business. So, really, really excited both for the opportunity and for everyone in this room to be part of this transformation and this really kind of bold leap into the next form factor of AI. All that being said, we'd love to see you try hyperagent. So, hyperagent.com/aie and you get a thousand dollars of inference credit.

You know, that can be spent on all of our different model offerings including you know, the latest Opus 4.8, Fable 5, you can use it on GLM 5.2, on GBD 5.5, etc. And we'd love to see what you do with with agents on hyperagent or elsewhere. Thank you all for being here. >> Thank you for joining us. Thank you. >> Um So, for the close of our events, a few months ago, I saw that hyperagent was launching this startup competition. And at AIE, we've always wanted to feature some kind of battlefield for like a competition and a contest for people.

And we didn't really have any sort of mechanism by which to judge or to fund it or anything like that. And Hyperagent came along and they had this uh 500 founders. >> Yep. >> Right? What What was the inspiration for that?

>> You know, I think um as I think about the multiple layers of disruption stacking, you know, there we we're going as I said from, you know, just humans doing the same work they were augmented a little bit by AI to now humans overseeing fleets of agents doing work that wasn't before possible like this landscaper running his business in a fundamentally different way to now this entirely new economy of agent native companies being built. And so, you know, really the Founding 500 is about betting on the most frontier part of that economy as possible, right?

It's the most agentically leveraged companies that are not only out disrupting their space, but kind of defining an entirely new playbook for how companies in this new era will be run. And I think the leverage we're already seeing from some of these these companies that are running in this way, like Gary said, like it's profound. I mean, it's far more than even being like an early software leveraged or internet leveraged company. So, really, really, you know, kind of big believer that this subset of the economy is going to have explosive growth far higher than any other sector, and it's going to cut across not just, you know, the tech industry, software industry, but every industry.

>> Yeah. Uh in my London keynote, I always I often talk talk about how uh work it agents are eating the rest of work and being accessible to non-technical teams. In fact, my team is right there back now because AIE runs on Airtable uh and they're so excited to be here and so excited to to work with Hyperagent. Um so, from 500 founders, we you guys preselected 20. They competed today in the Startup Battlefields. Uh three have emerged as finalists. Let's introduce them. Um so, I don't know if I'm supposed to to click this. Uh but uh first of all, I'm just going to go to introduce uh our dear judges.

I think uh we're going to have uh Howie, Theo, and Joshua. Um Thank you. >> [laughter] >> From Ajan. Uh and uh well, I think we'll we'll we'll sort of uh have them uh judging there, Uh but also we're going to uh go into the format and the background as well. So basically what we're going to do we the Hyperagent team has kindly put up $100,000 in prizes and basically what this is is a competition for whoever is going to be the most impressive winner. So 50k to the winners, 30k to the runner up, and 20k to second runner up. We're going to have a hype video that is thankfully created by Hyperframes and Hajan.

We have founder pitches and Q&A with the judges and then we'll also get to the next pitch as well. So there's three quick pitches back to back. Um and basically we decided that we really wanted some kind of judging in terms of like what is impressive and usage of this technology and like why it matters. Um it's not an investment pitch so much as like like you know, it's not like a VC pitch so much as like well why should you care? Why is this an interesting use of technology and and why you know, why are you like an interesting founder that people should know?

Um and so I'm very excited to announce like the winning teams. So I think first of all we're going to introduce um Kamod Spiro, come on stage. All right, thank you. >> [applause] >> Um I think you guys are going to take over and present, yeah? All right. Do you have your clickers or you got you got your you're going to stick it off, okay. All right. >> Hey everyone, I'm Spiro Anacos, one of the founders of Kamod. This is Dom. >> And I'm Nick Anacos. We are the founders of Kamod. Kamod has identified physical commodity trade.

From the rice you eat to the gold chain you're wearing to the chip in your smartphone to the gas you're putting in your car to every single copper pipe running through our data centers. Physical commodities power the world. But the global trade system is broken. Trillions move annually, yet hundreds of billions are lost annually due to fraud, fragmentation, and the sheer complexity of coordinating global trade. Komgo has taken that complexity and transformed it into an agentic execution layer. Our proprietary CFC state machine governs specialized agents that verify, execute, and settle trades entirely.

And then, you know, people ask, "Why is this so complicated?" And if you think about it, it's like, well, it's every country in the world, every resource, all of their different currencies, and all of their different banking rules. So, of course, it's inevitably going to be a disaster, right? But with AI, it's finally solvable. So, we're very excited to have you here. We also have our our first customer in the crowd out of the UAE, um trading company based in Dubai, and he moves several hundreds of millions of volume in in trade every single year, and uh he's proved the product. So, we're excited to be here today, and thank you for the opportunity.

>> $17 trillion in physical commodities move every year. The world runs on them, yet it still moves on paper, email, and a handshake. No one knows who is real or if the documents are. The result, billions lost. Komgo plugs into the stack you already run, Gmail, Slack, Stripe, Salesforce, WhatsApp, and more. A deal begins. Agents screen both sides, KYC, identity, sanctions in seconds. Forensics catch document fraud instantly. A private deal room spins up. Shipping and logistics partners link in with live pricing. Then an agent swarm structures the deal in parallel. Compliance, legal, currency, banking, shipping, settlement, weeks of work in minutes. The swarm obeys one protocol, the CFC, our patent-filed state machine.

Agents advance only by clearing every denied condition. It cannot move until the evidence is real, settlement eligible. It never touches the funds. It emits the signal, your bank moves the money. 30 days, closed in minutes. Commod, we're Commod.it, we trust. >> [applause] >> What happens to that? >> I don't know. >> [laughter] >> Is it on me? >> Yes. Um I I I realized that we didn't get you a mic. So, uh are you you're still mic'd up at the other table. Are you all mic'd up? Okay, great. Um so, I think we have a little bit of a Q&A session. Yeah. >> [laughter] >> So, am I supposed to ask questions or what?

>> Exactly. >> My team agreed to this, not me. >> Wait. Why are you guys playing this? >> Joshua, you want to go? >> Uh sure. I'm very interested in the um the second project, common.io. And you know, certainly we need a shared space for agent to collaborate together. Just curious, have you thought about having that format to be HTML versus [music] like markdown file? How you've been thinking about it? >> Uh sorry. >> All right, sure. >> Yeah, don't >> Yeah, I was just saying that, you know, um um the second project, common.io, it is about >> Yeah, that's the next one, yeah. >> Oh, the next one. >> them about the first one.

>> Uh so >> Oh, ask a question. >> This is one of those types of products that's really hard to build cuz it requires so much domain-specific knowledge, as well as like the building out of all of these pieces to collect the information needed for the product. A lot of the concern I have with modern startups is the thing that they're building is eventually going to be a feature of a sufficiently intelligent model. Do you see a future where the models get capable enough and the tools they're given by Anthropic and OpenAI could compete with you guys if they were to get smart enough and build in the right directions?

>> I want to take it? >> Yeah, I'll take it. >> So, in commodities trade, it's very specific as to what data is being pulled. And what we've done with our orchestration layer is set the gates for each agent. So, they're pulling from real databases and have that context themselves.

Whereas you're talking about gold, oil, grain, whatever it may be, there's certain procedures that only industry experts really know and have the documents and the information to pull that in, which is why we've partnered with Connor and a few other partners to bring in all that data and I have domain experience as well to know what's real, what's not, and to train the agents based upon that. So, to your question, could that happen? They could learn more for sure on the open internet, but this is a private industry with a lot of sensitive information and they have to learn upon and train upon certain data in order to progress and become experts in that domain.

>> So, it sounds like relatively bespoke in the sense that like every company's a bit different, you need to get everything for them. How many customers would it take for this to become a billion dollar company or to hit like, I don't know, 100 mil ARR? Like, how many customers you think it would take to get there? >> It's about 100 100 customers total. >> Huge. >> It's also important to mention as well like the tickets in this industry are massive on on deal sizes. It's not like a stripe for example processing micro transactions. One transaction might be, you know, 200 million dollars or north of that.

And one partner such as a refinery, many of them transact 9 to 12 billion dollars a month. So, getting in at that level, charging per deal on that, changes the whole landscape of what we're able to earn and what we're able to be valued at. Yeah, I just want to add one more piece as well. Our solution is end-to-end, so you know, the solution serves the entire supply chain, but the wedge ultimately into the industry is the traders and the banks. Banks pulled out of physical commodities trading years ago after the fraud became so rampant.

So, this unlocks a tremendous amount of liquidity as well by the audit trails and the immutability that we give into the commodity and the trade itself. So. >> There's a lot to love about this business, I think. Like in many ways, it's like checks all the boxes on like the classic like here is the ideal vertical agent business to build. Like if you look at the the YC kind of like RFS for vertical agents, like this is like the perfect application in the right the right kind of sector.

Like there's lots of money up for grabs and there's a lot of like domain-specific stuff that you have to do that creates, I think, a good defensible moat. Um I think my perspective is more like you know, what do you do to keep up with the changing competitive landscape and just, you know, as models progress, how do you need to evolve your product to stay competitive, right?

So, it's less about is this a relevant and great business today and more about what changes over the next 2-3 years or even like next year that impacts you in a way that, you know, it doesn't necessarily mean like Anthropic and OpenAI eats your lunch, but what do you need to do to stay, you know, kind of at the frontier and be a better product? >> Yeah, I think like the only thing that's actually defensible is being the most creative in your category regardless of which product you're creating, right? Like truly nothing's defensible other than being the most creative, right? Anybody these days has a product.

But, we actually do like the Anthropic eat your lunch model. Like we like the ship stinction model where we ship and then 10 companies go extinct the next day. So, um yeah, it's aggressive shipping and product superiority, but then ultimately we're going to win on distribution. >> What are you most excited about on your upcoming road map if you can share? >> Becoming the bank. >> [laughter] >> Yeah. It's exciting. >> Thank you so much. >> Awesome. Thank you. >> Awesome. >> Give it up for Kamrad. Uh we're going to invite the next team on now. Uh if you want to come back here.

And up next, we have Max Windebank and the team from Comet IO. Come on, Max. Ooh. Oh, you're going to play the video? All right, come on here. >> [applause] >> All right, we'll play the video. That was >> That's the music. >> Oh. >> That's the right one. >> All right, welcome in. Come on, guys. >> Oh, thank you. Um if you could uh so, we have a little bit of a technical snafu where the judges couldn't see the video. >> Okay. >> Uh we have to describe a little bit just to just to prompt things off. >> What for? >> If you sort of before you get into the whole pitch, yeah.

>> Uh okay. >> Pretend like we're a non-multimodal model. We just We only accept text. >> We're giving you all the context here. >> Yeah, yeah, yeah. >> So, uh yeah, text uh Comet IO is a uh it's a uh multiplayer markdown editor. It's uh focused on making the best documents um for the modern work, which is uh people and agents. But, what you would have seen on the video is a familiar document editor interface, but people and agents are popping in, they're notifying each other, everyone can call anyone in, and that's that's basically the gist. >> Awesome. >> All right.

So, um it's Friday afternoon, ready to go home, packing up your laptop, and all of a sudden you're hearing dings around the office. Server's down, big outage. So, you put down your bag, you head into the conference room, and you're pulling out your laptop. There's one question on your mind, what is going on, Claude? And you're entering that in, you're waiting a few minutes, give me some ideas, it's culminating or whatever, and then it gives you some ideas. And the first thing you do is you tell everyone else in the room the ideas, so they can put it into their Clauds.

And then someone else has a different idea, and then they let you know, and then you put it into your Claud. And you realize, how did we get here? How is this the job? There has to be a better way, and there is. It's agents and the who are doing the work, and people who own the problem sharing context in real time. But how? Think of the productivity software that you all use, the collaboration software. Can you just like drop your coding agents into them and have them working with each other? No, you can't, because they were built for people, not for agents. And that's why we're making Comment. It is the multiplayer markdown editor.

It is we're laser-focused on making it the best document platform for modern work, for people, and for agents. It's open, it's pluggable, that's key for enterprises, it's key for it's key for developers, and it's wrapped in a user experience that is super familiar. I started my career in productivity bringing collaboration features into Microsoft Word. Max and I worked together um at Textio, the first AI native writing software, 6 years before ChatGPT. We were deploying company-wide rollouts of AI writing software to the Fortune 500. We've been doing this a long time. The market for comment is expanding with agents.

As more and more people use agents, more people are going to need the ability for agents to share context in a product that is built for agents but designed for people working together even in the highest stake circumstances. >> Awesome. I love the shape of this business because it's like the inverse of what we just heard from the the command guys, right? Like they went super vertically deep. They're trying to extract a lot of value out of a very deep you know domain and you guys are going super broad and arguably maybe a little thinner but by intention, right? And I don't say that as a as a detriment at all.

Like I think you know one way to think about the landscape here is like the agent economy is going to be so large. There's just so much money flowing through agents like literally the token spend, right? And if you can be a product that like even a one to you know 2% amount of those tokens touches, like you could be a very valuable even if like the per per call or per usage you know kind of a metric you know kind of margin extraction or revenue is is relatively low, you can be a very cheap, very fast, very broad product and still gain a lot of scale, right?

So I think it's a really really interesting shape of product. What have you found is like the most surprising learnings from like a product design or product affordance standpoint to make this product better for agents than like the human design equivalents, right? Like why not just have agents collaborate on a Google Doc, right? Or even try to use like GitHub as the you know kind of repo not just for code but for markdown. >> All writing software today is horrible for agents to use. Software like Google Docs has a really terrible format. It has an MCP. You can send a document there.

You can have your agent read a document from there, but you can't have them collaborate in real time and that's critical for all kinds of real work that needs to happen. And as more agents join the workforce, they need to be in there with your documents. And as we've talked to our users, people who love working in documents, who rely on documents for their day-to-day, need to bring their agents because they're starting to use agents, too. It's not just developers anymore. >> Okay, I think >> it. >> Oh. Final closing words?

>> I was just going to say the other thing I'll mention that I've been super surprised about, um, talking to people at this conference when we've talked to them, is how many people have said, "I've rolled my own like visual markdown editor that other people can look at so people can look." And then you're like, "Ah, but I gave up. I just rendered an HTML document." Like you said, like it's a hard problem and you need dedicated like focus on making it great. And that's kind of what we're doing. >> Awesome. All right. Nice job, guys. Thank you so much.

Uh, next for and last team to compete, we're going to play the video by Foundry. >> Content creators don't have time to build a business, so they have to hire an agency. It's expensive, slow, and risky. >> [music] >> Built by Foundry replaces the agency and lets creators launch their own products faster and cheaper. Our AI agents build the business end-to-end. The creator launches it. We keep growing the revenue. Type in any creator's handle and our agents become [music] their biggest fans, watching their videos, reading their comments, and finding a painkiller their audience would actually [music] pay for.

We've done this for creators of every size and over half a million people use products our agents built. The cost of building software is approaching zero. What's left is trust and taste, and no one has more of it than creators. We're building the agentic product team that is going to turn every creator into a founder. >> Woo! Foundry team, come on up. >> Thank you. Thanks, Max. >> Hello, AIE. I am Weston Belgettys. I'm the founder and CEO of Built by Foundry. We're going to turn every content creator into a founder of their very own company. And I have great news today. I think we found the formula for a winning business.

You take your unique expertise and turn it into a product, and you solve cold start with your built-in distribution. Theo, you you did this with T3 chat. >> Yeah, it sucked. It doesn't work. >> It [laughter] works great for our our our our users, but uh Gary did it with with G G stock and G brain. And uh what what we find is that if we help uh creators to take a step back and solve a real pain point that their users have. So, we're not just making another Patreon for that and paywalling this content.

We're helping them solve a real pain point behind that, and that's what our agents are experts in is finding the pain point within the users' problems. And our companies have uh built over uh many different niches and uh different sizes of shape and shapes of creators. And really what we're building is we want to give the power back to the content creator. They're stuck in these I worked at TikTok for 2 years, and and I met with hundreds of creators who are stuck just doing brand deals or affiliate links or selling $20 PDFs in their link in bio, and they are one step away from building their own business.

They just need someone to help them, and that's what our agents do. At Built by Foundry, we deploy teams of agents to build recurring revenue businesses for content creators so that they can focus on posting and posting and posting and we can keep growing their revenue. >> I have a lot of thoughts on this one. >> Okay. >> So, the problem I find as a creator, even with the brands I work with, is that there is a saturation point where it stops being useful to talk about a given product with my audience.

Like with me for Vercel, for example, there was quickly a point where everybody in my audience either was a Vercel user doesn't want me to shill it to them anymore or they won't be a Vercel user and they don't want me to talk about it anymore. It is very easy to hit the saturation point and the best solution is to have a much broader set of things to show your audience. One of the best things I ever did for myself, the single moment that has had the most positive impact on myself, my career, my businesses and my income has been going from having four sponsors to 80.

Because now there's a wider variety of things to bring my audience and even if 80% of them aren't relevant to them, the 20% that is is now conversion I'm getting that I wouldn't have otherwise. The variety of what I bring to my audience is where I find the most value by far and every creator I've worked with ultimately has come to a similar conclusion. That said, you're touching on really important pieces here. Creators do have more taste than most founders. Creators do have better distribution than most companies. Creators know their audience better than most businesses and you've built a machine that identifies that.

But it doesn't provide the solution to that other problem which is you have to bring variety to your audience. Where I could see this working really well is as a tool that businesses use when they identify a creator is a overlapping with the archetype of customer they're looking for to help that business build better things for Theo. And I've told this to so many businesses that want to work with me. Watch my streams. Watch my videos. Find the problems I have and then show me how your product solves them. That's a way better pitch than going to try and start your own company as a YouTuber.

As the YouTuber who has started the second most companies as far as I'm aware, it doesn't work. It just burns you out and it burns your audience out, too. The best thing you can do is work with companies to teach them what your audience wants and for them to come to you with money to show it to those audience or those audience members. >> I understand. And I will say your niche as a tech creator is one that is very challenging because there's a lot of options and what we have found is that for all of our creators, I'll just few examples.

One one creator's name is Brandon and he does for homesteading content and he always loses track of the best settings on his freeze dryer, which sounds ridiculous to us, but to his 2 million followers, that's a real pain point. And now they pay him a recurring revenue monthly or monthly or yearly subscription to solve that specific pain point. So, I think unfortunately, Theo, you've picked a bad niche. I'm sorry. But if you want to pivot to homesteading or gardening, maybe we could we could work together.

But I I'd say primarily, whatever the user's niche is apart from tech and maybe comedians, we've been able to take a step back, our agents find a pain point that they're willing to pay for and zero creators across our entire company history have ever left our company because they always are profitable. >> You're very convincing. >> Okay, excellent. Then give it up. >> The Foundry. >> The Foundry? >> Yeah, thank you. >> Um so, we're going to have a little bit of a deliberation moment. The judges are going to go backstage for a couple minutes. In the meantime, sorry. Okay. In the meantime, we've prepared this little video recapping the World's Fair.

So, take your time, judges. >> [music] >> This is our biggest AI engineers event ever. >> We're constantly adding new techniques for you to try. >> [music] [music] >> Thank you so much. All right. Um I think we're inviting everyone back on stage. Give it up for the Startup Battlefield. Um Congrats, guys. Yes. Okay, so uh we have some prizes that we're going to hand out. I think um the folks back there are doing some checks. Um In second runner-up place, we have Komodo. >> [applause] >> Uh Theo, I think you're presenting uh Theo, yes. Okay, we'll take some pictures as well for the team. All right.

Um next and uh I guess this also reveals the order, uh we have Foundry as runner-up. Great [cheering] pitch, Max. Great pitch. >> [applause] >> All right, congrats. Awesome. All right, no secrets now for the grand prize, Comment IO. Welcome. Great video as well. Congratulations. Well deserved. Do you want to also do the group photo? I think we're doing >> All of us? >> Um Yes. >> Okay. >> All right. >> just them? >> They're the founders. We're just here to comment. >> Just the winners? >> everyone. >> All right, we'll sneak in a picture then. Get them. Come on. >> Yes. >> All right. >> Thank you so much, guys. >> Awesome.

>> You want to pass on building with Hyper Agent and >> Yeah, of course. >> Yeah. >> Hyper Agent. >> Um any last words that you wanted to comment on? I think you guys uh pitched a little bit, yeah. >> Yeah, I just want to thank uh thank everyone here at AIE. We got amazing feedback um through the whole um time and uh HyperAgent um for inviting us to this and uh HyperFriend for helping us with the video. And overall, like thank you so much. Check us out. It's free. comment.io >> All right, that's it. Thank you.

>> Ladies and gentlemen, please welcome back to the stage our MC, developer relations engineer at Replit, Ralph [music] Shaybrey. >> Hot hot. Okay. So, let's give it up to our Startup Battle Field's finalists. All right. >> [applause] >> They collectively won 100K worth of prize to boost out there their project. So, this is great. Um all right, let's give it up also to all our judges, Josh, Howie, and Theo. >> [applause] >> All right. My next slide. Clicker not working. Next slide, please. Clicker not working. Okay. All right, and with that, AI Engineer Welfare 2026 is a wrap. And we've come a long way. Uh this year's edition is our most ambitious one.

4 days, 7,000 attendees, keynote sessions, countless of workshops, conversations, demos, 40 tracks. And we hope that you find this content useful and helpful and that you're going to go back home inspired. But what we love to see more than anything else is the community coming together, creating new connections, new friendships, exchanging ideas, and building together. And we're extremely grateful to be part of this. So, let's give it up for you guys. All right. We're also very extremely grateful to our sponsors. So, please let's give it up to our presenting sponsor, Microsoft. And also to our lab and platinum sponsors, please let's keep it going. Our gold sponsors and our silver and bronze sponsors. Thank you.

We're very extremely grateful because without these sponsors this event wouldn't be possible. And also I would like to to thank an invisible group of people who are working backstage, who you never see, but without whom this event wouldn't be possible. Uh so, thank you all, crew members, volunteers. Let's give it up for them. >> [applause] >> All right. So, uh previous slide, please. Okay. Almost there. There you go. Nope. All right. Okay, so this is the end of Wells Fargo 2026, but we're not done yet. We hope to see many of you in New York in October.

And also, we're going to be back together here next year for another edition of AI Engineers Wells Fargo. But before we celebrate America's birthday, we have one more thing to show you. >> [music] >> Mhm.