Model Whisperers: How Evals and Prompts Shape Agent Behavior

Official Schedule Context

Getting an AI agent to behave the way you want isn’t just about writing better prompts. In real

systems, behavior emerges from a loop: prompts->evals->iteration->feedback. Small changes in any

part of that loop can completely change outcomes. We saw this while building a seed asset agent - a

system that turns messy, real-world advertising creatives (low quality images, cluttered visuals,

heavy text overlays) into clean, reusable assets for downstream Gen AI tools. The agent acts like an

editor, simplifying visuals, removing unnecessary elements, and isolating core content so that

additional context (like text or CTAs) can be added back in a more controlled, brand-safe way. But

the real challenge wasn’t just building the agent - it was making it reliable. And prompting alone

wasn’t enough. What actually moved the system forward was how we defined success—and how we used

evals to reinforce it. Over time, evals stopped being just a way to measure quality. They became

part of how the agent learned what “good” looks like. In this talk, we’ll cover: Why prompting alone

doesn’t give you stable agent behavior How evals act like feedback signals, not just scorecards How

we built evals sets that reflect the real-world Using agent trace logs to understand why things fail

(not just that they fail) How to iterate without breaking things you already fixed By the end,

you’ll have a set of patterns you can apply to any system dealing with messy/continuously changing

data and how to tweak your prompt and evals to accommodate such changes.

No official session recording transcript was found by exact title match on the AI Engineer YouTube channel during this run.

Pending transcript synthesis when an official recording or confirmed matching video is available.