Markdown source

Transcript: Your agent is blindfolded — Johan Lajili, Poolside AI

Source Video

Local Cache

Transcript

Hi everyone. So I'm Joan from Poolside. >> Hi everyone. So I'm Joan from Poolside. If you haven't heard of us, we're one of If you haven't heard of us, we're one of If you haven't heard of us, we're one of the handful of companies that are making the handful of companies that are making the handful of companies that are making their own foundational model from their own foundational model from their own foundational model from scratch their own LM and coding agents. scratch their own LM and coding agents. scratch their own LM and coding agents. Check us out if you if you're not aware Check us out if you if you're not aware Check us out if you if you're not aware some cool stuff and you should hear more some cool stuff and you should hear more some cool stuff and you should hear more soon. soon. soon. But what I want to talk to about today But what I want to talk to about today But what I want to talk to about today is this. You have is this. You have is this. You have people seeing AI and using AI and people seeing AI and using AI and people seeing AI and using AI and getting vastly different experiences. If getting vastly different experiences. If getting vastly different experiences. If you're on Reddit, if you're on Twitter, you're on Reddit, if you're on Twitter, you're on Reddit, if you're on Twitter, you're going to see people that say, oh you're going to see people that say, oh you're going to see people that say, oh yeah, I'm never like touching code yeah, I'm never like touching code yeah, I'm never like touching code anymore. The AI is is doing everything anymore. The AI is is doing everything anymore. The AI is is doing everything for me. It's fantastic. for me. It's fantastic. for me. It's fantastic. And others that say, what are you And others that say, what are you And others that say, what are you talking about? I'm trying it in my talking about? I'm trying it in my talking about? I'm trying it in my production app. It produces absolute production app. It produces absolute production app. It produces absolute garbage. garbage. garbage. What are you guys working on to do apps? What are you guys working on to do apps? What are you guys working on to do apps? What are you doing? What are you doing? What are you doing? And there is multiple ways to try to And there is multiple ways to try to And there is multiple ways to try to to understand what's going on there. One to understand what's going on there. One to understand what's going on there. One way is to say, way is to say, way is to say, this guy is a shill from open AI trying this guy is a shill from open AI trying this guy is a shill from open AI trying to sell you AI. to sell you AI. to sell you AI. Or this one is an entity that doesn't Or this one is an entity that doesn't Or this one is an entity that doesn't care about anything and is just like care about anything and is just like care about anything and is just like lying. He didn't even try it. Another lying. He didn't even try it. Another lying. He didn't even try it. Another is to say, well, maybe someone is is to say, well, maybe someone is is to say, well, maybe someone is working on a greenfield app and that's working on a greenfield app and that's working on a greenfield app and that's nice and easy. nice and easy. nice and easy. Whereas someone else is working on Whereas someone else is working on Whereas someone else is working on brownfield on a legacy application. brownfield on a legacy application. brownfield on a legacy application. That's complicated and agents are not That's complicated and agents are not That's complicated and agents are not there yet. there yet. there yet. But personally, I think that doesn't But personally, I think that doesn't But personally, I think that doesn't quite hold up. quite hold up. quite hold up. We've seen people using AI in legacy We've seen people using AI in legacy We've seen people using AI in legacy applications with good success. I have applications with good success. I have applications with good success. I have myself. So at least on my own myself. So at least on my own myself. So at least on my own experience, I know that can work. So experience, I know that can work. So experience, I know that can work. So what's the difference? What's the what's the difference? What's the what's the difference? What's the difference really between difference really between difference really between brownfield and greenfield? The brownfield and greenfield? The brownfield and greenfield? The difference is that with greenfield, the difference is that with greenfield, the difference is that with greenfield, the agent's intuition is correct. The agent's intuition is correct. The agent's intuition is correct. The agent's writing the code and expect, you agent's writing the code and expect, you agent's writing the code and expect, you know, if I write the components here, if know, if I write the components here, if know, if I write the components here, if I write the service, it's going to work. I write the service, it's going to work. I write the service, it's going to work. I I think that's going to be fine. And I I think that's going to be fine. And I I think that's going to be fine. And he's right cuz it has very good he's right cuz it has very good he's right cuz it has very good intuition. intuition. intuition. On brownfield, however, On brownfield, however, On brownfield, however, there be dragons. You're going to have there be dragons. You're going to have there be dragons. You're going to have things that the agent is not expecting. things that the agent is not expecting. things that the agent is not expecting. Maybe you know, like dead ends, code Maybe you know, like dead ends, code Maybe you know, like dead ends, code that's not used anymore, that's not used anymore, that's not used anymore, things that it is not aware of in in things that it is not aware of in in things that it is not aware of in in different part of the code that it different part of the code that it different part of the code that it hasn't even looked at. And that's where hasn't even looked at. And that's where hasn't even looked at. And that's where the big difference between those two is the big difference between those two is the big difference between those two is the feedback loop. And everybody is the feedback loop. And everybody is the feedback loop. And everybody is somewhat talked about it in in in the somewhat talked about it in in in the somewhat talked about it in in in the background of the talks we've seen over background of the talks we've seen over background of the talks we've seen over the past three days. the past three days. the past three days. But I think that's the difference with But I think that's the difference with But I think that's the difference with getting these results. So you you've got getting these results. So you you've got getting these results. So you you've got your agent that says, yeah, I've your agent that says, yeah, I've your agent that says, yeah, I've implemented the new auth flow and it's implemented the new auth flow and it's implemented the new auth flow and it's all working perfectly. all working perfectly. all working perfectly. What the agent means really is, well, to What the agent means really is, well, to What the agent means really is, well, to the best of the the best of the the best of the of my capabilities, to the best of what of my capabilities, to the best of what of my capabilities, to the best of what you have given me, that sounds like it you have given me, that sounds like it you have given me, that sounds like it should work. Maybe the agent was able to should work. Maybe the agent was able to should work. Maybe the agent was able to verify its work. Maybe it wasn't. But as verify its work. Maybe it wasn't. But as verify its work. Maybe it wasn't. But as far as it knows, it's working. far as it knows, it's working. far as it knows, it's working. If you're a skeptic of AI, you're going If you're a skeptic of AI, you're going If you're a skeptic of AI, you're going to see that first quote, see that it's to see that first quote, see that it's to see that first quote, see that it's indeed not working and just sees, you indeed not working and just sees, you indeed not working and just sees, you know, I'm a liar, I'm incompetent or know, I'm a liar, I'm incompetent or know, I'm a liar, I'm incompetent or like you cannot trust the AI. And that's like you cannot trust the AI. And that's like you cannot trust the AI. And that's I think where this cleavage in between I think where this cleavage in between I think where this cleavage in between those two type of users. Like the first those two type of users. Like the first those two type of users. Like the first category is going to see some things. Oh category is going to see some things. Oh category is going to see some things. Oh yeah, actually, you know what? It's not yeah, actually, you know what? It's not yeah, actually, you know what? It's not working agent. Can you try again? Check working agent. Can you try again? Check working agent. Can you try again? Check those logs or whatever. those logs or whatever. those logs or whatever. Whereas those ones are just going to Whereas those ones are just going to Whereas those ones are just going to give up. But I think we can give up. But I think we can give up. But I think we can make this still better. make this still better. make this still better. At Poolside, I've created a little CLI At Poolside, I've created a little CLI At Poolside, I've created a little CLI tool called Spoolside. Yeah, I might be tool called Spoolside. Yeah, I might be tool called Spoolside. Yeah, I might be good at programming. I'm not good at good at programming. I'm not good at good at programming. I'm not good at naming things. naming things. naming things. That basically allows it to test our That basically allows it to test our That basically allows it to test our applications effectively. Some of the applications effectively. Some of the applications effectively. Some of the stuff you've already seen in things like stuff you've already seen in things like stuff you've already seen in things like G stack for instance being able to take G stack for instance being able to take G stack for instance being able to take screenshots of the applications, screenshots of the applications, screenshots of the applications, being able to take a snapshot, that is being able to take a snapshot, that is being able to take a snapshot, that is to say like a very token compressed to say like a very token compressed to say like a very token compressed version of what's going on on a web page version of what's going on on a web page version of what's going on on a web page and use it. But our application is not a and use it. But our application is not a and use it. But our application is not a web page. It's an extension within VS web page. It's an extension within VS web page. It's an extension within VS code. So already lucky takes an extra code. So already lucky takes an extra code. So already lucky takes an extra step to get there. step to get there. step to get there. But with that, our AI can interface with But with that, our AI can interface with But with that, our AI can interface with it just like it would with a normal web it just like it would with a normal web it just like it would with a normal web page. But we take it further. page. But we take it further. page. But we take it further. We have thing to extract logs from We have thing to extract logs from We have thing to extract logs from different services, from the back end, different services, from the back end, different services, from the back end, from the front end, ways to restart from the front end, ways to restart from the front end, ways to restart different services, different services, different services, high level commands. Can you access a high level commands. Can you access a high level commands. Can you access a specific menu? Can you go to this page? specific menu? Can you go to this page? specific menu? Can you go to this page? Can you send a message to the agent? Can you send a message to the agent? Can you send a message to the agent? Wait for it to reply, send another Wait for it to reply, send another Wait for it to reply, send another message, upload an image and do that and message, upload an image and do that and message, upload an image and do that and it can stack things like somewhat it can stack things like somewhat it can stack things like somewhat efficiently and a bit like we've talked efficiently and a bit like we've talked efficiently and a bit like we've talked with coding tools this morning. with coding tools this morning. with coding tools this morning. And And And that's pretty useful cuz then the agent that's pretty useful cuz then the agent that's pretty useful cuz then the agent is able to test what it's doing. If it's is able to test what it's doing. If it's is able to test what it's doing. If it's working on a bug, it can actually working on a bug, it can actually working on a bug, it can actually reproduce the bug before it starts reproduce the bug before it starts reproduce the bug before it starts working on that on that. The agents are working on that on that. The agents are working on that on that. The agents are very eager. Yeah, I know what's going very eager. Yeah, I know what's going very eager. Yeah, I know what's going on. You just need a margin there. You on. You just need a margin there. You on. You just need a margin there. You just need to go and add this code. But just need to go and add this code. But just need to go and add this code. But until it reproduces the bug, I don't until it reproduces the bug, I don't until it reproduces the bug, I don't trust you. And that's the big thing. trust you. And that's the big thing. trust you. And that's the big thing. It's maybe without that the agent is It's maybe without that the agent is It's maybe without that the agent is able to still have good intuition and able to still have good intuition and able to still have good intuition and fix the issues. But I don't trust it and fix the issues. But I don't trust it and fix the issues. But I don't trust it and then I'm going to have to go and verify then I'm going to have to go and verify then I'm going to have to go and verify it myself and I'm wasting time and I it myself and I'm wasting time and I it myself and I'm wasting time and I cannot then take that agent and start cannot then take that agent and start cannot then take that agent and start running it overnight for instance. running it overnight for instance. running it overnight for instance. This is a first step to trust. And the This is a first step to trust. And the This is a first step to trust. And the point of this is not point of this is not point of this is not Spoolside. It's not something you're Spoolside. It's not something you're Spoolside. It's not something you're going to find on GitHub to use for going to find on GitHub to use for going to find on GitHub to use for yourself. It's to build your own. I yourself. It's to build your own. I yourself. It's to build your own. I think as engineers, think as engineers, think as engineers, that's our new role. that's our new role. that's our new role. We are going to have to focus less on We are going to have to focus less on We are going to have to focus less on the product and more on trying to make the product and more on trying to make the product and more on trying to make the AI work on the product. How can we the AI work on the product. How can we the AI work on the product. How can we make it easy for it? make it easy for it? make it easy for it? That can be those tools. That can be That can be those tools. That can be That can be those tools. That can be improving the code base so that it's improving the code base so that it's improving the code base so that it's easier to work on. That can be improving easier to work on. That can be improving easier to work on. That can be improving knowledge bases. You can implement this knowledge bases. You can implement this knowledge bases. You can implement this as a CLI, as a skill, as an MCP. In my as a CLI, as a skill, as an MCP. In my as a CLI, as a skill, as an MCP. In my case, it's a CLI cuz I like things case, it's a CLI cuz I like things case, it's a CLI cuz I like things simple. But like there are many simple. But like there are many simple. But like there are many different variations of it and I think different variations of it and I think different variations of it and I think it's going to be different from people it's going to be different from people it's going to be different from people to people and problem to problem. to people and problem to problem. to people and problem to problem. But yeah, I think in But yeah, I think in But yeah, I think in 2025, we had product engineers 2025, we had product engineers 2025, we had product engineers that were, you know, like very focused that were, you know, like very focused that were, you know, like very focused on doing everything with the product. on doing everything with the product. on doing everything with the product. But now that AI is getting quite good, But now that AI is getting quite good, But now that AI is getting quite good, you want to focus more on making sure you want to focus more on making sure you want to focus more on making sure that that velocity is not a trap, that that that velocity is not a trap, that that that velocity is not a trap, that you're not going to multiply errors or you're not going to multiply errors or you're not going to multiply errors or compound errors, that you're going to compound errors, that you're going to compound errors, that you're going to actually verify what you're doing, actually verify what you're doing, actually verify what you're doing, making it easy for you to to verify as making it easy for you to to verify as making it easy for you to to verify as well with like well with like well with like presenting the work that the AI done and presenting the work that the AI done and presenting the work that the AI done and everything around that. And so I think everything around that. And so I think everything around that. And so I think we're all going to become AIX engineers we're all going to become AIX engineers we're all going to become AIX engineers essentially. And yeah, it's a bit like essentially. And yeah, it's a bit like essentially. And yeah, it's a bit like when you're in an airline when you're in an airline when you're in an airline and they say, oh, you put your mask on and they say, oh, you put your mask on and they say, oh, you put your mask on yourself before you fit you put your yourself before you fit you put your yourself before you fit you put your your mask on your children. It's the your mask on your children. It's the your mask on your children. It's the same with AI. You need to put the mask same with AI. You need to put the mask same with AI. You need to put the mask on the AI. You need to make sure that on the AI. You need to make sure that on the AI. You need to make sure that it's self-served before you try to work it's self-served before you try to work it's self-served before you try to work on features. Even if it slows you down on features. Even if it slows you down on features. Even if it slows you down right now, it's an investment that pays right now, it's an investment that pays right now, it's an investment that pays off as soon as you start multiplying off as soon as you start multiplying off as soon as you start multiplying agents and running things over time. agents and running things over time. agents and running things over time. And that's me done with two minutes to And that's me done with two minutes to And that's me done with two minutes to spares. spares. spares. So thank you very much. So thank you very much. So thank you very much. Any question from anybody? Any question from anybody? Any question from anybody? Yes. I think they're quite ephemeral in the I think they're quite ephemeral in the sense and that I guess is a personal sense and that I guess is a personal sense and that I guess is a personal preference. But I do feel like automated preference. But I do feel like automated preference. But I do feel like automated test can be sometimes a bit too rigid test can be sometimes a bit too rigid test can be sometimes a bit too rigid and hard to predict and hard to like and hard to predict and hard to like and hard to predict and hard to like work over time. So I like something that work over time. So I like something that work over time. So I like something that mimics the way I would test it like a mimics the way I would test it like a mimics the way I would test it like a human would go and test the application. human would go and test the application. human would go and test the application. The way I find those is The way I find those is The way I find those is in the first stage where I work with the in the first stage where I work with the in the first stage where I work with the AI, AI, AI, I try I try I try even though, you know, like I can see even though, you know, like I can see even though, you know, like I can see that the button is a bit that the button is a bit that the button is a bit to the left and I want to tell the AI to the left and I want to tell the AI to the left and I want to tell the AI that. What I want is the AI to realize that. What I want is the AI to realize that. What I want is the AI to realize that by itself. So whenever I encounter that by itself. So whenever I encounter that by itself. So whenever I encounter that sort of problem, I take a step back that sort of problem, I take a step back that sort of problem, I take a step back and try to think on how to make the AI and try to think on how to make the AI and try to think on how to make the AI realize the problem by itself. realize the problem by itself. realize the problem by itself. Another thing is re-attractive loop. Another thing is re-attractive loop. Another thing is re-attractive loop. After you've done that many times, look After you've done that many times, look After you've done that many times, look past over your logs, ask an AI, past over your logs, ask an AI, past over your logs, ask an AI, did you notice any issues, any did you notice any issues, any did you notice any issues, any any stink? Are you doing sleep for any stink? Are you doing sleep for any stink? Are you doing sleep for instance? Calling sleep parentheses 15 instance? Calling sleep parentheses 15 instance? Calling sleep parentheses 15 everywhere is a stink that like there is everywhere is a stink that like there is everywhere is a stink that like there is something that should be wait for something that should be wait for something that should be wait for whatever like a comment that could be whatever like a comment that could be whatever like a comment that could be there. there. there. Is there anything that you keep doing Is there anything that you keep doing Is there anything that you keep doing over and over like running storybooks? over and over like running storybooks? over and over like running storybooks? And another thing is really think about And another thing is really think about And another thing is really think about like your own product. If you're making, like your own product. If you're making, like your own product. If you're making, say, say, say, a game in Unity, like a game in Unity, like a game in Unity, like do you want an ASCII representation of do you want an ASCII representation of do you want an ASCII representation of your 3D world in for for your AI? If your 3D world in for for your AI? If your 3D world in for for your AI? If you're making something with lots of you're making something with lots of you're making something with lots of permissions, different logins that your permissions, different logins that your permissions, different logins that your AI can take very easily. It's it's AI can take very easily. It's it's AI can take very easily. It's it's really your new role to think about all really your new role to think about all really your new role to think about all that. That's what I think anyway. that. That's what I think anyway. that. That's what I think anyway. And I think we're it for time. Thank you And I think we're it for time. Thank you And I think we're it for time. Thank you very much.