Markdown source

Transcript: HTML is All You Need (for Agents to Make Graphics) - Amol Kapoor, Nori

Source Video

Local Cache

Transcript

[music] >> Hi, I'm Amol, CEO of Nori Atentic. >> [music] >> We deploy an AI employee that understands your company, your code, docs, Slack, [music] and other kinds of data. We spend a lot of time thinking about how coding agents really work. Most people think coding agents only write code. But if you ask me, that's just bad marketing. >> [music] >> Forget the name for a second. Coding agents can do almost anything. >> [music] >> There's just one trick. You have to be able to think like an agent to get it to do what you want it to do.

[music] >> Today, we're going to talk about how we use coding agents to do something most people think agents are terrible at, make visual artifacts, like slides, docs, [music] and yeah, even video. >> [music] >> Every day, the world pours something like 34,000 human years into making slide decks. Most of that time isn't the thinking, it's the fiddling. [music] A deck that takes 10 hours should really take about 25 minutes once you remove all the formatting and the branding and the moving things around. Say you need to make a slide. What do you do? You open a tool, PowerPoint, Slides, Figma, Canva, and then you start manipulating a canvas.

Every one of these tools is built for human hands and human eyes. Click, drag, drop, resize, snap to grid, all motions and patterns that make sense for our geospatial view of the world. There is a data structure underneath, but it's in a format that only the application can read. What happens when you hand these tools to an agent? Well, the output comes out all wrong. Things overlap in weird ways, you can't see the text, there's no alignment, It's just garbage. AI skeptics say that it's not just the tools. Agents fundamentally can't reason about space. And there are whole benchmarks like Arc AGI that are built exactly around that premise.

There's a famous little test for this from developer Simon Willison. He asks every new model the same thing. Can you draw a pelican riding a bicycle? But there's a trick. The agent is only allowed to use SVG. It's a quick gut check for whether a model can reason about space at all. Here's some examples of what the models actually give you on this test. And [music] yeah, these are pretty bad. Like genuinely deeply really bad. >> [music] >> So, does that mean it's hopeless? Agents are just doomed to be bad at graphics? No, I don't think so. If you ask me, it's not the model, it's the medium.

If I asked you, someone who is presumably human, to handwrite an SVG of a pelican, you wouldn't be able to do that either. >> [music] >> SVGs are just a wall of numbers. You can't go from a wall of numbers to a pelican, you just can't see that way. That's just not how people think. We think graphically. So, we build tools that let us draw on a canvas. Figma MCPs, PowerPoint CLIs, screenshot and replace loops, what do all of these agent tools have in common? They all approach the problem like a human. But an AI is not a human.

Asking an AI to use a canvas is like asking a human to write SVG by hand. It doesn't really make sense. You need to give the AI tools based on how it thinks, not in pixels, in language. Words, tokens, structure, that is its native medium. Imagine a language that's incredible at describing layout. That models have seen and trained on billions of examples of. That they understand intuitively. that renders to pixels and can run everywhere. Oh, right. HTML lets the model think in structure. HTML tags have meanings built into the language. A heading, a chart, a grid, and the browser turns it all into pixels.

So, the model never actually places a coordinate, and you can get all sorts of visual effects, charts and layouts, fonts and motion, all of it for free. Remember that pelican from earlier? Now, ask it to do the same exact task, but in HTML. Same bird, but now it's in a structure that the model can reason about. And you can read and theme and edit every single line of it. I spent my whole life building slide decks with PowerPoint. So, I always thought that those two things, slide decks and PowerPoint, were synonyms. [music] But that's just not really true, is it? PowerPoint is a tool that you use to make slide decks.

The deck itself, that's just the presentation mode. And as it turns out, no one in your audience is going to care how you got to the presentation mode. The editing format is totally arbitrary. >> [music] >> So, you can just pick the editing format that the agents are already good at, HTML, and if you need to render to a different format like PDF later on. We use this HTML trick to build all of our slide decks, our board decks, and our sales decks. >> [music] >> These are real things that we actually present and send out constantly. We use it for our docs, too.

It gives our docs color and vibrancy, all while following our brand. And of course, we also use it to make videos like this one. What you're watching is just HTML and CSS. It's literally just divs all the way down. Almost everything is better with a little structure and a little bit of color. Plain text is a choice, generally a choice of convenience, but it's usually the wrong one if you're actually trying to create something of use. Now, I do want to take a quick beat here and point out that a beautiful deck on its own is generally not worth anything.

You still have to go and get all of that content, all of the things that actually populate that deck, right? Well, again, we can think like the model. If you just give the model access to your data, say your call transcripts or your emails, you can have the model build the deck end to end. Let your agents do all the grunt work while you focus on vision and story. That's what Nori Sessions lets you do. I've built entire board decks for my phone on the subway during my commute. Why? Because our Nori bot lives in the fabric of our company.

Of course, Nori ships with everything you need to make this all work, so don't bother reinventing the wheel. That's my little spiel. Thanks for listening. If you have just one takeaway, it's this. Stop thinking like a user. Think like the model. Give it the right language, and for graphics, all you need is HTML. >> [music]