Markdown source

Transcript: How Lovable self-improves every hour — Benjamin Verbeek, Lovable

Source Video

Local Cache

Transcript

All right. Let's get this party started. Do we have any Lovable users in here? Good, good, a few. Very nice. A warm welcome to to this talk on how Lovable is improving every hour, including this hour as I'm speaking. I'm Benjamin van Beek. I'm a member of technical staff of Lovable. It's so amazing that we are filling up this bonus room. I think we're around three times as many people as would have fit in the original. So, thanks to the organizers. Um This is my background. I used to work with satellites, particle physics, and fusion reactors. So, I have a physics background.

And today, I'm working at Lovable and working towards what is maybe the holy grail of AI engineering right now, which is continuous learning at scale. How do we learn from mistakes? So, you've probably all experienced working with an agent and having the feeling, "Why do I have to explain the same thing over and over again?" This is what we want to avoid. We want to have a mistake happen once and then never again. So, you need to learn from that. And in this talk, I want to share two ways how we are working towards this goal at Lovable. This is the Lovable interface.

Uh we were actually one of the first uh I think we coined the term vibe coding. So, coding without actually looking at code. So, you have chat interface where you describe what you want to happen, and you have a model sandbox, for example, where you can see directly what you're creating. And I think in many ways, this is the way we should be building software. The code was always just an annoying technical layer in between to create what we wanted. But ideally, you just say what you want, you see it, you test it, and you ship it.

We have some nice benefits with the Lovable platform, which is that people stick to one project for quite a long time. This chat can go on forever, essentially. And people have like this one artifact that they are really passionate about and they want to ship. So, we can learn a lot about this one thing versus, for example, a chat agent where you have very, very short conversations. We can actually get to know the user in quite detail. Another exciting thing is that we're building for the 99% who can't code. So, probably not the people in this room, actually.

There are still quite a lot of people, evidently, who use our product, but we are trying to unlock software creation for everyone. In the future, you will not have to look at code to be able to create software. I think that is the world we all want to move towards, and we are trying to do that right now. And in many ways, this is like building for the future, I think. So, a reason that startups succeed with new technology is that they are a bit naive, and they don't really think about all the issues that can happen. They don't think too much about all the details in implementation. They just go until it works.

This is how I like to see our users. They just go until it works, and they're not sort of held back by old paradigms. This is a wonderful group of people to to build for. Um and it scaled quite rapidly. Uh I joined Lovable around a year ago. At that time, we had a few thousand users. Now, we're creating over 200,000 projects per day. That's a significant percentage of all internet websites being created on Lovable. And that scaling journey has been incredibly fun. We can do so many cool things now, like the things I will talk about in this talk. Uh it's also very hard.

My first day, GitHub banned us because we were creating too many ripples. And we've taken down a number of cloud providers along the way. How do we succeed with AI? Um, I like to think about it in this way. Technical personas generally have this amazing moment with AI where they are just building and building. You're accelerating 10 times, 100 times faster than you used to. But sometimes we hit some friction points, this yellow part, and you have to intervene or like from the bit harder. And sometimes you even get stuck. Maybe you have to manually go in and change a config somewhere.

You need to really do a lot of manual work, change the setting, add an end bar, API key, etc. And then you move on and you really get the good parts of AI and you can work past the bad parts. You're still annoyed by it, but you can work past it. And non-technical persona is not really like this. They might prompt their way past the first friction, but as soon as this technical block happens, they generally walk away and they give up and they actually still never experience successful AI. This is what we need to minimize. We can never get stuck to a point where a non-technical user cannot get back get get past it.

And this is a very exciting problem to work on. Luckily, models are getting better and the requirements of these people is generally to a point where we can achieve them. But for the past year, we have been working on making sure none of these red bars happen. People can never get stuck. How do we do that? First of all, we want to define like what does it mean to be stuck? We probably want to find those cases and then learn from them. So, one clear signal of someone being stuck is that they ask for the same thing more than once. Probably stuck or it didn't work as smoothly as they could.

Maybe they're complaining about how something was implemented or that it failed very explicitly or they gave up on a session that they would otherwise have continued. And we can have a LLM judge that is just looking at sessions and trying to look for these situations and they can flag it and say, "Hey, I think this user is stuck." I want to split being stuck into two different ways. You can be stuck in a way where it's possible to solve. This is maybe the yellow part that I showed. And you can get past it with the right prompting. Some users are more willing to go put in that effort and get past that point.

Others will give up, but it is possible to solve with the current structure of our product. And then there are tasks that are not solvable even if you say all the right things. Maybe we're just not supporting it for some reason. Um And there there are two classes. There's the things that are just dumb that we don't support. Maybe it's a bug. Maybe it's a very very simple thing we could add to the product. Uh And then there are the actual hard things that would take like weeks of engineering AI-assisted engineering effort. These first two things we should be able to improve on very rapidly, I think.

There's no excuse to not improve on these things. If it is solvable, then it should work for everyone. And if it is just easy to do, we should just ship it. So, how do we do that? Um some of you might have heard of Stack Overflow. It's very very ancient thing. We had an idea to build a Stack Overflow for Lovable. So, we basically learn from uh whenever someone is stuck and has an issue, we try to figure out the solution and give that to the agent. This could look something like this. Um I'm complaining to the agent about my website being laggy when I scroll. It's bad performance. Maybe I'm not stuck yet.

This is just me complaining about something and giving it a new task. The agent probably replies something like, "I fixed it. Maybe I quantized the animations to make it more performant." And it was lying. The agent has failed and it actually made things worse. Now the website is jumpy and laggy. Terrible. Okay, this is a clear case of someone being stuck. They asked the same thing again. They probably wanted wanted agent to try again. And they're complaining about an implementation that didn't do what they wanted. This might repeat for a while where they keep iterating several times. Some people give up at this stage, but at least some people will continue.

And eventually the agent might find the true solution. Which in this case happened to be that overlay text had individual gradients which made the animation super low performing. And now we see that is stuck changes back to false. It succeeded. So this we can flag. We can notice whenever it went from being stuck to not being stuck. And ideally not because the the user gave up. This we can flag and now we have a high signal sample of a problem that was high friction or not solved that was solved. So we have gotten the solution right here.

And the question we just ask is what should we have injected at the start of this query to jump straight into the solution so the next user does not experience this friction. What we do is we create a new lovable Stack Overflow knowledge entry. And in general we actually do some clustering at this step where we look for similar issues. We look for what is the actual information we should give so that's not overfitting to the specific issue. We don't want like a million overflow pages Stack Overflow pages that are all talking about if you get this exact prompt then you should do this exact thing. It's not very helpful. We do some clustering.

And then we have an external reviewer, generally an agent and then maybe in some cases we have a human if we are uncertain, but in most cases it's actually just an agent that generates and runs a quick eval on this and sees if this did this resolve the specific examples that we had in the set. From that we get a full bank of level Stack Overflow problems and solutions that's being continually updated. Whenever the model is working, we have a lightweight model that tries to inject that context when needed. If it detects there is an issue and there is an answer, it injects that context into the main agent.

And sometimes it detects that I should inject this, but we inject a blank. So, we don't actually send anything for a small sample of use cases. And this allows us to with very high signal review whether this solution was actually useful in production. And we rate it. We compare the the group of projects where it was injected and where we it could have been injected, but it wasn't. And we say which of these projects were actually more successful overall. It was more successful, then we show it more. And if it was less successful, we should show it less. And this loop is incredibly important. I cannot emphasize enough how important this step is.

Because things are moving around this set of knowledge all the time. It gets stale whenever a new model is released. It gets stale whenever we change features. It gets stale incredibly quickly. So, very often, we have to rebalance this, but we also have to throw away a lot of this context. And this allows us to really be at the frontier of what is solvable right now, but not have a lot of old deprecated knowledge that is giving context rot, um and also in many cases just hampering the actual uh agent. And this is working at scale. Uh this is a very very early data actually, where we're doing quite a lot better now.

Um but the number of messages with fixing tent or people being stuck is dropping significantly. And we have also significant number of people that deploy more. This is one of our key metrics. That they actually finish a project. This is a very very strong signal that it has worked all the way through. They've never gotten stuck so bad that they give it given up and abandoned the project completely. Um we also do internal ranking with this. So, it's really interesting to see how the models perform on this set of problems that we have collected. And with the Stack Overflow information, uh all models in the top of our ranking use this information.

There's a few hidden entries here, which I unfortunately cannot talk about. Um but it really really makes a significant boost in our internal ratings. And then there's a second set of uh being stuck, which is uh when it's not actually solvable with the current setup. Maybe there's a bug in our product. Um but in these cases we think it should be easy in principle. And what happens if you think something should be possible, but it just doesn't work? You feel significant frustration. And what would humans do if this happened?

Well, if you were given a task and you just don't have the tools to do it, you would probably complain to your boss or go venting in Slack. So, we thought, why not do exactly that, but for the agent? So, we basically are asking Lovable, "How are you doing? Can we help you in any way?" And we gave it an outlet to let out its frustrations. So, we asked ourselves, "What if we could uh let the Lovable agent give direct feedback to its creators?" This sounds very scary, I have to say, and it sounds like an absolute insane idea. Um what's even more crazy is that it's kind of working.

This kind of detecting issues is something I've heard during this conference, but also a few other times, where you might have an external reviewer looking at the conversation and asking, "What could have been done better to reduce friction?" One problem with that is that you get a pretty low signal-to-noise ratio. Which you're basically forcing an answer on every iteration. And the reality is that most iterations actually just work pretty well. And you will overfit to noise. In this case, it's prompted to only send feedback if it is really frustrated. And you can tune that balance until you get a lot of signal. So, we gave our agent a vent tool.

A way to complain to its creators. Um it looks something like this. It's a vent send feedback tool. You should use this when tooling docs or platform behavior material is low so it degrades your work. For example, missing or unsuitable tools, unclear tool names, parameters or schemas that are not matching what you were expecting, confusing or conflicting docs or instructions, broken or unexpected platform behavior, repeated failed attempts at causing caused by environment limitations. So, basically a list of all the things we think it should we should be able to solve and that we hope is not an issue, but if it is, please tell us.

And our users generally don't know what the cause of a problem is. When I'm using Lava when I get stuck, I generally don't know. But the agent actually has a lot more context. It has literally been working on the issue, sometimes several turns. And it generally has a lot of context on this problem. So, it can look something like this. It actually gets sent directly to our Slack. And the agent says something like, "I'm so annoyed at Frame Motion's TypeScript types for generating I think this is a cubic bezier. I just want to be able to send a list of four numbers. That should be fine. That's all I'm sending.

I don't need all this casting gymnastics." Now, we can ask ourselves is this a relevant vent or not? Um but in this case, it it probably just was not relevant for its use case and we could have simplified things a lot. Another benefit of this setup is that it's very easy to understand as a human. It's very relatable that you're complaining about your workflow and that means that engineers sort of already have all this implicit context on how to alleviate the issue. Another problem that got reported by the agent was our copy tool was struggling with certain file names. It couldn't copy it to another place where we store store documents.

And we were so confused by this. Like we checked the tool, it's working. Um we didn't even know that this tool was failing so regularly until the agent told us. And it said whenever there is a space or raw space in the file name, it's failing to copy. And we got like 20 complaints about this in the first hour of launching this tool, which is crazy. It turns out that there was an error where it could not copy files with a space in their name. We fixed it and we told it whenever there's a space, just replace it with an underscore. And we kept getting the reports.

And then we realized that when you screenshot something in WhatsApp or on Mac, it it enters a non-breaking space. Which we did not replace in our regex. And this kept complaining for various other special characters until we did solve it properly. And now this issue never happens again. I think this is an a prime example of something that's pretty hard to detect in other ways. You can see the tool failure, but this is just a such a clear case of how to fix it. And we just do it. I will skip this one. Um It's essentially this is the flow. The agent experiences an issue, it uses the event tool, it's sent straight to Slack.

At first I was very uncertain if this would work. And I didn't want to spam spam us. It was like it closed down channel. I didn't invite so many people. Our head of product was very excited to see this. Uh he was reading every single message. Now it's a bit more balanced and we actually have an agent that's monitoring removing duplicates and investigating and creating a PR for all these issues all the time. And we're still at the point where devs are reviewing and then in many cases actually merging this to prod. This is number of events over time. Uh do you have any guesses what these spikes are?

Why do we see spikes in the number of events cost per time? So, that's server went down. Server went down. It's an incident. Yes. Something broke in the platform. At some point, our sandboxes broke. Something something else broke. And the agent is very upset about this. It's complaining a lot. So, it turned out that this was actually a prime place to notice when our product was having an incident. And it actually gave a pretty good sense for what the problem was. It was complaining about right things in general. Um very brief I'll keep this brief. Um you get the strong model intelligence versus an external reviewer.

You generally don't want top frontier level intelligence to look through a lot of context. But if it's in line, it's very very cheap comparatively. Um this was an example where the agent was actually giving meta feedback on the venting tool. It said it's too easy to send feedback and I can't pull it back. It was being ashamed of what it had sent to Slack. Um and it's actually the case now that I just get uh review requests on my phone from this automation to hey, review this PR that was completely automated. I look through it quickly and we can merge it.

And we're continuing to work to close this loop of detecting a shortcoming, merging a fix, and then continuously review and eval that. And if you think this is exciting, uh you should join us and help us close this loop and fully automate uh continual improvements. Thanks for listening.