Markdown source

Transcript: Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

Source Video

Local Cache

Transcript

Hello, my name is Angel Ermanlee, and Hello, my name is Angel Ermanlee, and I'm a software engineer at Duolingo. I I'm a software engineer at Duolingo. I I'm a software engineer at Duolingo. I work on security for the Duolingo work on security for the Duolingo work on security for the Duolingo English Test. Today's talk is the human English Test. Today's talk is the human English Test. Today's talk is the human in your loop isn't thinking. Build AI in your loop isn't thinking. Build AI in your loop isn't thinking. Build AI systems for discernment, not approval. systems for discernment, not approval. systems for discernment, not approval. So, let's get started with some So, let's get started with some So, let's get started with some background. What is human-in-the-loop background. What is human-in-the-loop background. What is human-in-the-loop AI? Human-in-the-loop AI is a framework AI? Human-in-the-loop AI is a framework AI? Human-in-the-loop AI is a framework where a system or process is actively where a system or process is actively where a system or process is actively involving a human involving a human involving a human who's participating in the operation, who's participating in the operation, who's participating in the operation, supervision, or decision-making for an supervision, or decision-making for an supervision, or decision-making for an automated system. automated system. automated system. Humans are involved because they're able Humans are involved because they're able Humans are involved because they're able to ensure the accuracy, safety, or to ensure the accuracy, safety, or to ensure the accuracy, safety, or ethical decision-making for that system. ethical decision-making for that system. ethical decision-making for that system. They are a piece of that decision-making They are a piece of that decision-making They are a piece of that decision-making where where where AI normally falls short. You can kind of AI normally falls short. You can kind of AI normally falls short. You can kind of think of this as a linear process where think of this as a linear process where think of this as a linear process where a model provides some sort of output, a model provides some sort of output, a model provides some sort of output, and a human sees that and makes a and a human sees that and makes a and a human sees that and makes a decision. decision. decision. Now, in the age of AI, trust is a big Now, in the age of AI, trust is a big Now, in the age of AI, trust is a big part of the conversation. Recently, as part of the conversation. Recently, as part of the conversation. Recently, as technology has developed, a lot of technology has developed, a lot of technology has developed, a lot of things that used to be manual cognitive things that used to be manual cognitive things that used to be manual cognitive tasks are now part of technology. So, no tasks are now part of technology. So, no tasks are now part of technology. So, no one really memorizes phone numbers one really memorizes phone numbers one really memorizes phone numbers anymore. They're just sitting in our anymore. They're just sitting in our anymore. They're just sitting in our contact list on our smartphones. When contact list on our smartphones. When contact list on our smartphones. When you're driving over somewhere, you just you're driving over somewhere, you just you're driving over somewhere, you just put the directions into your GPS, and put the directions into your GPS, and put the directions into your GPS, and you don't really think about the details you don't really think about the details you don't really think about the details of that route. Instead, you just trust of that route. Instead, you just trust of that route. Instead, you just trust the GPS to come up with the optimal way the GPS to come up with the optimal way the GPS to come up with the optimal way for you. for you. for you. When you're searching something in the When you're searching something in the When you're searching something in the search engine, you used to look at the search engine, you used to look at the search engine, you used to look at the top results, and now, as AI is becoming top results, and now, as AI is becoming top results, and now, as AI is becoming more increasingly integrated into our more increasingly integrated into our more increasingly integrated into our day-to-day life, you might be seeing day-to-day life, you might be seeing day-to-day life, you might be seeing some AI summaries at the top instead. some AI summaries at the top instead. some AI summaries at the top instead. Here's an example of Googling Here's an example of Googling Here's an example of Googling COVID-19 symptoms, and you see that COVID-19 symptoms, and you see that COVID-19 symptoms, and you see that Gemini summary at the top. And instead Gemini summary at the top. And instead Gemini summary at the top. And instead of looking at the CDC website or the WHO of looking at the CDC website or the WHO of looking at the CDC website or the WHO website, you might actually look to that website, you might actually look to that website, you might actually look to that as your final answer to the question as your final answer to the question as your final answer to the question that you were asking to the search that you were asking to the search that you were asking to the search engine. engine. engine. And as these pieces of AI are becoming And as these pieces of AI are becoming And as these pieces of AI are becoming more integrated into your day-to-day more integrated into your day-to-day more integrated into your day-to-day life in these atomic little things like life in these atomic little things like life in these atomic little things like searching for an answer or getting a searching for an answer or getting a searching for an answer or getting a result from an app, result from an app, result from an app, your trust of AI systems will increase your trust of AI systems will increase your trust of AI systems will increase and caution will decrease. And this is and caution will decrease. And this is and caution will decrease. And this is something that's happening across all of something that's happening across all of something that's happening across all of society. society. society. A study at Wharton was looking exactly A study at Wharton was looking exactly A study at Wharton was looking exactly that. How is AI reshaping human that. How is AI reshaping human that. How is AI reshaping human reasoning and what does it mean for us reasoning and what does it mean for us reasoning and what does it mean for us as we continue to use it day-to-day? as we continue to use it day-to-day? as we continue to use it day-to-day? They've observed an interesting They've observed an interesting They've observed an interesting phenomenon called cognitive surrender. phenomenon called cognitive surrender. phenomenon called cognitive surrender. When a human foregoes deliberation and When a human foregoes deliberation and When a human foregoes deliberation and adopts AI output as their own with adopts AI output as their own with adopts AI output as their own with minimal scrutiny. minimal scrutiny. minimal scrutiny. They saw this in a study where humans They saw this in a study where humans They saw this in a study where humans were asked to look at reasoning exams were asked to look at reasoning exams were asked to look at reasoning exams and they were given AI resources to and they were given AI resources to and they were given AI resources to answer those questions. answer those questions. answer those questions. They realized that AI can either They realized that AI can either They realized that AI can either supplement or supplant a human's supplement or supplant a human's supplement or supplant a human's thinking and the human might not even thinking and the human might not even thinking and the human might not even know it's happening. know it's happening. know it's happening. For example, for questions where the AI For example, for questions where the AI For example, for questions where the AI was right, the human performance was right, the human performance was right, the human performance increased by 25 percentage points. increased by 25 percentage points. increased by 25 percentage points. Whereas when the AI was wrong, it Whereas when the AI was wrong, it Whereas when the AI was wrong, it decreased by 15. decreased by 15. decreased by 15. This suggests that the humans were This suggests that the humans were This suggests that the humans were taking in that AI cognition and not taking in that AI cognition and not taking in that AI cognition and not really thinking critically about what really thinking critically about what really thinking critically about what that answer is and instead using those that answer is and instead using those that answer is and instead using those answers as part of their own answers as part of their own answers as part of their own reasoning reasoning reasoning together with their human instinct and together with their human instinct and together with their human instinct and deliberate reasoning amplifying those deliberate reasoning amplifying those deliberate reasoning amplifying those results. results. results. Most interestingly, they saw that 80% of Most interestingly, they saw that 80% of Most interestingly, they saw that 80% of participants were accepting those AI participants were accepting those AI participants were accepting those AI answers even when they were wrong. So, answers even when they were wrong. So, answers even when they were wrong. So, they were lowering that barrier to entry they were lowering that barrier to entry they were lowering that barrier to entry and just trusting the AI and just trusting the AI and just trusting the AI even if they weren't really critically even if they weren't really critically even if they weren't really critically examining the correctness of that examining the correctness of that examining the correctness of that result. result. result. So, at the Duolingo English Test, we So, at the Duolingo English Test, we So, at the Duolingo English Test, we wanted to look at the same thing and we wanted to look at the same thing and we wanted to look at the same thing and we published some research titled When published some research titled When published some research titled When Machines Mislead. It was a case study on Machines Mislead. It was a case study on Machines Mislead. It was a case study on the English exam, specifically in AI the English exam, specifically in AI the English exam, specifically in AI human interaction. human interaction. human interaction. So, for context, what is the DET? The So, for context, what is the DET? The So, for context, what is the DET? The DET is a high-stakes exam for that DET is a high-stakes exam for that DET is a high-stakes exam for that measures your English proficiency and measures your English proficiency and measures your English proficiency and it's fully online and remotely it's fully online and remotely it's fully online and remotely proctored. You can take it on your own proctored. You can take it on your own proctored. You can take it on your own machine in the comfort of your home and machine in the comfort of your home and machine in the comfort of your home and it still provides high-quality results it still provides high-quality results it still provides high-quality results that 6,000 programs worldwide are that 6,000 programs worldwide are that 6,000 programs worldwide are trusting day-to-day. trusting day-to-day. trusting day-to-day. As you can imagine with a fully online As you can imagine with a fully online As you can imagine with a fully online assessment, there are some interesting assessment, there are some interesting assessment, there are some interesting things that from a technological things that from a technological things that from a technological standpoint that have to happen to ensure standpoint that have to happen to ensure standpoint that have to happen to ensure good quality security. This includes good quality security. This includes good quality security. This includes identity verification, a lockdown identity verification, a lockdown identity verification, a lockdown testing environment, as well as a testing environment, as well as a testing environment, as well as a variety of ways in which AI assisted variety of ways in which AI assisted variety of ways in which AI assisted monitoring is happening to predict monitoring is happening to predict monitoring is happening to predict different types of cheating. different types of cheating. different types of cheating. Lastly, we have a final Lastly, we have a final Lastly, we have a final round of round of round of scrutiny where we have human proctors scrutiny where we have human proctors scrutiny where we have human proctors review all of the video footage of the review all of the video footage of the review all of the video footage of the exam as well as all of those AI results. exam as well as all of those AI results. exam as well as all of those AI results. Specifically for the study, we wanted to Specifically for the study, we wanted to Specifically for the study, we wanted to target one of our AI target one of our AI target one of our AI cheating detection detection systems, cheating detection detection systems, cheating detection detection systems, which is copy typing. Copy typing is a which is copy typing. Copy typing is a which is copy typing. Copy typing is a form of cheating where you're writing form of cheating where you're writing form of cheating where you're writing down down down uh information that you're reading uh information that you're reading uh information that you're reading currently and supposed to something that currently and supposed to something that currently and supposed to something that you're thinking right now at this you're thinking right now at this you're thinking right now at this moment. moment. moment. So, as you can imagine, your typing So, as you can imagine, your typing So, as you can imagine, your typing patterns differ when you're transcribing patterns differ when you're transcribing patterns differ when you're transcribing versus composing. And this is something versus composing. And this is something versus composing. And this is something that our custom model is measuring. It's that our custom model is measuring. It's that our custom model is measuring. It's looking at anomalies in keystroke looking at anomalies in keystroke looking at anomalies in keystroke patterns and it's flagging them for uh patterns and it's flagging them for uh patterns and it's flagging them for uh sessions that are unusual. So, our model sessions that are unusual. So, our model sessions that are unusual. So, our model is highly conservative and so we're is highly conservative and so we're is highly conservative and so we're prioritizing fairness for our test prioritizing fairness for our test prioritizing fairness for our test takers. Uh so, this is not a very common takers. Uh so, this is not a very common takers. Uh so, this is not a very common flag. However, when humans are human flag. However, when humans are human flag. However, when humans are human proctors are taking a look at these, proctors are taking a look at these, proctors are taking a look at these, they're well trained to they're well trained to they're well trained to um properly examine those segments and um properly examine those segments and um properly examine those segments and see if this is a flag. see if this is a flag. see if this is a flag. So, for experiments, we wanted to answer So, for experiments, we wanted to answer So, for experiments, we wanted to answer the question, would a skilled reviewer the question, would a skilled reviewer the question, would a skilled reviewer catch a false alarm or would they just catch a false alarm or would they just catch a false alarm or would they just rubber stamp it? rubber stamp it? rubber stamp it? So, we know that our proctors are highly So, we know that our proctors are highly So, we know that our proctors are highly accurate at detecting various forms of accurate at detecting various forms of accurate at detecting various forms of cheating. cheating. cheating. We wanted to see specifically if we give We wanted to see specifically if we give We wanted to see specifically if we give them a fake signal, how well would they them a fake signal, how well would they them a fake signal, how well would they be able to discern that this is totally be able to discern that this is totally be able to discern that this is totally legit session. legit session. legit session. The way we did that is by selecting The way we did that is by selecting The way we did that is by selecting sessions that had no cheating sessions that had no cheating sessions that had no cheating whatsoever. They were totally okay, and whatsoever. They were totally okay, and whatsoever. They were totally okay, and we made it look like they have this AI we made it look like they have this AI we made it look like they have this AI signal that says, "Hey, check if there's signal that says, "Hey, check if there's signal that says, "Hey, check if there's copy typing behaviors present at a copy typing behaviors present at a copy typing behaviors present at a specific moment." We presented them to specific moment." We presented them to specific moment." We presented them to the proctors as part of their normal the proctors as part of their normal the proctors as part of their normal workflow. workflow. workflow. They thought it was just another session They thought it was just another session They thought it was just another session that they were taking a look at. Um and that they were taking a look at. Um and that they were taking a look at. Um and this did not impact our test takers in this did not impact our test takers in this did not impact our test takers in any way. any way. any way. Their results were quite interesting. Their results were quite interesting. Their results were quite interesting. So, we found that despite the fact that So, we found that despite the fact that So, we found that despite the fact that our human reviewers are consistently our human reviewers are consistently our human reviewers are consistently scoring above 90% on their accuracy scoring above 90% on their accuracy scoring above 90% on their accuracy calibration metrics, they actually calibration metrics, they actually calibration metrics, they actually accepted 50% of these fake signals, accepted 50% of these fake signals, accepted 50% of these fake signals, meaning they were falsely accusing meaning they were falsely accusing meaning they were falsely accusing people of cheating half half the time. people of cheating half half the time. people of cheating half half the time. And this coin flip rate is something And this coin flip rate is something And this coin flip rate is something that is a strong suggestion of that is a strong suggestion of that is a strong suggestion of automation bias. This means that they're automation bias. This means that they're automation bias. This means that they're deferring their judgment to AI deferring their judgment to AI deferring their judgment to AI uh without giving a second thought to uh without giving a second thought to uh without giving a second thought to whether or not that there's evidence to whether or not that there's evidence to whether or not that there's evidence to corroborate this and if this is a true corroborate this and if this is a true corroborate this and if this is a true uh cheating session. uh cheating session. uh cheating session. Uh again to reiterate, these were not Uh again to reiterate, these were not Uh again to reiterate, these were not sessions that were actually going to our sessions that were actually going to our sessions that were actually going to our test takers. These were historical test takers. These were historical test takers. These were historical sessions, so there was no impact to our sessions, so there was no impact to our sessions, so there was no impact to our customers. However, this is something customers. However, this is something customers. However, this is something that was definitely very alarming to the that was definitely very alarming to the that was definitely very alarming to the team because in a high-stakes exam where team because in a high-stakes exam where team because in a high-stakes exam where these results are going to influence these results are going to influence these results are going to influence college admissions and visa decisions, college admissions and visa decisions, college admissions and visa decisions, this is not something that we can take this is not something that we can take this is not something that we can take uh uh uh we can just accept. we can just accept. we can just accept. We knew that the problem was not the We knew that the problem was not the We knew that the problem was not the model. model. model. Our model has a 1% false positive rate, Our model has a 1% false positive rate, Our model has a 1% false positive rate, and also these were sessions that were and also these were sessions that were and also these were sessions that were negatively predicted. We knew that our negatively predicted. We knew that our negatively predicted. We knew that our problem was not the people. Our people problem was not the people. Our people problem was not the people. Our people are highly skilled and very experienced are highly skilled and very experienced are highly skilled and very experienced with detecting cheating. So, we wanted with detecting cheating. So, we wanted with detecting cheating. So, we wanted to take a look at the interface, and to take a look at the interface, and to take a look at the interface, and that's where we targeted our solution. that's where we targeted our solution. that's where we targeted our solution. So, we decided to take a look a look at So, we decided to take a look a look at So, we decided to take a look a look at the human-AI interaction loop. the human-AI interaction loop. the human-AI interaction loop. Specifically, our proctors are given a Specifically, our proctors are given a Specifically, our proctors are given a series of guidelines about how to series of guidelines about how to series of guidelines about how to interact with the system that they have. interact with the system that they have. interact with the system that they have. And we wanted to just update that And we wanted to just update that And we wanted to just update that proctoring guideline to emphasize two proctoring guideline to emphasize two proctoring guideline to emphasize two things. First, the AI signal is just a things. First, the AI signal is just a things. First, the AI signal is just a preliminary alert. They're the final preliminary alert. They're the final preliminary alert. They're the final decision-maker. And second, when they're decision-maker. And second, when they're decision-maker. And second, when they're making that decision, they must find making that decision, they must find making that decision, they must find independent evidence in the video independent evidence in the video independent evidence in the video footage before upholding a flag. footage before upholding a flag. footage before upholding a flag. This simple copy change led a 21% This simple copy change led a 21% This simple copy change led a 21% increase in rejection rates, meaning increase in rejection rates, meaning increase in rejection rates, meaning that from 50% only half of sessions that that from 50% only half of sessions that that from 50% only half of sessions that were rejected were rejected were rejected uh the majority 71 were now determined uh the majority 71 were now determined uh the majority 71 were now determined to be totally okay. to be totally okay. to be totally okay. And this is really good because it is And this is really good because it is And this is really good because it is more closely matching more closely matching more closely matching sessions in production. sessions in production. sessions in production. And for an engineer, this means that we And for an engineer, this means that we And for an engineer, this means that we didn't have to tweak the model and we didn't have to tweak the model and we didn't have to tweak the model and we didn't have to tweak uh the UI in some didn't have to tweak uh the UI in some didn't have to tweak uh the UI in some way. We simply had to take a look at the way. We simply had to take a look at the way. We simply had to take a look at the copy change and this really changed the copy change and this really changed the copy change and this really changed the way that people were interfacing with way that people were interfacing with way that people were interfacing with that AI result. that AI result. that AI result. So, what are the implications of this? So, what are the implications of this? So, what are the implications of this? As an AI engineer, your interaction loop As an AI engineer, your interaction loop As an AI engineer, your interaction loop determines how effective your AI system determines how effective your AI system determines how effective your AI system is and also what you can learn from it. is and also what you can learn from it. is and also what you can learn from it. The way that you measure efficacy and The way that you measure efficacy and The way that you measure efficacy and the way that you can uh and the data the way that you can uh and the data the way that you can uh and the data that you collect from that to then drive that you collect from that to then drive that you collect from that to then drive your next iteration are really your next iteration are really your next iteration are really important. important. important. Earlier we talked about the Earlier we talked about the Earlier we talked about the uh human in the human in the loop AI as uh human in the human in the loop AI as uh human in the human in the loop AI as a linear interaction where a model goes a linear interaction where a model goes a linear interaction where a model goes to a human goes to a decision. However, to a human goes to a decision. However, to a human goes to a decision. However, in reality in reality in reality that process isn't linear, it's that process isn't linear, it's that process isn't linear, it's cyclical. cyclical. cyclical. Models Models Models provide some sort of output that goes provide some sort of output that goes provide some sort of output that goes into an interaction that goes into a into an interaction that goes into a into an interaction that goes into a human behavior which yields data for human behavior which yields data for human behavior which yields data for your evals and then later on your model. your evals and then later on your model. your evals and then later on your model. You can't really change the way a human You can't really change the way a human You can't really change the way a human behaves unless you are that human. But, behaves unless you are that human. But, behaves unless you are that human. But, what you can do is tweak that what you can do is tweak that what you can do is tweak that interaction such that it can elicit interaction such that it can elicit interaction such that it can elicit different results from that human different results from that human different results from that human behavior. And that data is golden. It behavior. And that data is golden. It behavior. And that data is golden. It leads to better analytics, model leads to better analytics, model leads to better analytics, model improvements, and future iterations, improvements, and future iterations, improvements, and future iterations, including the next generation of your including the next generation of your including the next generation of your model, or maybe some different product. model, or maybe some different product. model, or maybe some different product. Now, zeroing in on that interaction Now, zeroing in on that interaction Now, zeroing in on that interaction portion, we can look at how to evaluate portion, we can look at how to evaluate portion, we can look at how to evaluate your system. Ask yourself a question, your system. Ask yourself a question, your system. Ask yourself a question, "How can I measure and improve the "How can I measure and improve the "How can I measure and improve the efficacy of my AI system?" efficacy of my AI system?" efficacy of my AI system?" Looking at structured interactions that Looking at structured interactions that Looking at structured interactions that were designed intentionally, you can see were designed intentionally, you can see were designed intentionally, you can see that those interactions yield good data that those interactions yield good data that those interactions yield good data that are labeled signals that can then that are labeled signals that can then that are labeled signals that can then become training data and evals for a become training data and evals for a become training data and evals for a better model. And that is a flywheel better model. And that is a flywheel better model. And that is a flywheel that compounds. As you have the better that compounds. As you have the better that compounds. As you have the better model, you can have better data that model, you can have better data that model, you can have better data that then has better interactions with your then has better interactions with your then has better interactions with your human, which then continues to cycle. human, which then continues to cycle. human, which then continues to cycle. Think of your structured interactions as Think of your structured interactions as Think of your structured interactions as a system property that specifically a system property that specifically a system property that specifically yields high-quality data. That yields high-quality data. That yields high-quality data. That high-quality data is unlocking high-quality data is unlocking high-quality data is unlocking meaningful insights and quicker meaningful insights and quicker meaningful insights and quicker development iterations. You're not going development iterations. You're not going development iterations. You're not going to be spending to be spending to be spending days cleaning your data or trying to days cleaning your data or trying to days cleaning your data or trying to find useful insights. Instead, you're find useful insights. Instead, you're find useful insights. Instead, you're going to have structured data that is going to have structured data that is going to have structured data that is directly impacting your next iteration. And this is a compounding effect. Uh and And this is a compounding effect. Uh and it's cyclical. If you are not being it's cyclical. If you are not being it's cyclical. If you are not being intentional with that design, you can intentional with that design, you can intentional with that design, you can get stuck in a vicious cycle where your get stuck in a vicious cycle where your get stuck in a vicious cycle where your model is making confident calls, and model is making confident calls, and model is making confident calls, and because your interface is not actually because your interface is not actually because your interface is not actually uh eliciting that deliberation from a uh eliciting that deliberation from a uh eliciting that deliberation from a human, they just end up rubber stamping human, they just end up rubber stamping human, they just end up rubber stamping it. And that positive signal is logged it. And that positive signal is logged it. And that positive signal is logged as truth. So, over time, your model as truth. So, over time, your model as truth. So, over time, your model becomes more confident, and the human is becomes more confident, and the human is becomes more confident, and the human is not encouraged to think further. So, not encouraged to think further. So, not encouraged to think further. So, they continue to defer to the AI, and they continue to defer to the AI, and they continue to defer to the AI, and the AI becomes the person in the driving the AI becomes the person in the driving the AI becomes the person in the driving seat. seat. seat. Uh instead, what you want is a virtuous Uh instead, what you want is a virtuous Uh instead, what you want is a virtuous cycle. cycle. cycle. Uh, if you have an interface that forces Uh, if you have an interface that forces Uh, if you have an interface that forces independent judgment, uh, this allows independent judgment, uh, this allows independent judgment, uh, this allows the human to really think critically the human to really think critically the human to really think critically about the decisions they're making and about the decisions they're making and about the decisions they're making and have real disagreements surface. That have real disagreements surface. That have real disagreements surface. That means that you have true positive and means that you have true positive and means that you have true positive and negative labels that are honest and get negative labels that are honest and get negative labels that are honest and get logged to then continue to have model logged to then continue to have model logged to then continue to have model improvements that are targeting exactly improvements that are targeting exactly improvements that are targeting exactly where the model goes wrong. where the model goes wrong. where the model goes wrong. So, let's take a look at some examples. So, let's take a look at some examples. So, let's take a look at some examples. For example, we have other cheating For example, we have other cheating For example, we have other cheating flags that are driven by AI. As an flags that are driven by AI. As an flags that are driven by AI. As an example, we have headphone detection. On example, we have headphone detection. On example, we have headphone detection. On the left, you can see, the left, you can see, the left, you can see, uh, a bad pattern that we had where we uh, a bad pattern that we had where we uh, a bad pattern that we had where we were saying, "Hey, the model detected were saying, "Hey, the model detected were saying, "Hey, the model detected headphones at this moment. Can you flag headphones at this moment. Can you flag headphones at this moment. Can you flag it here?" And it was just asking a it here?" And it was just asking a it here?" And it was just asking a simple yes or no question. We realized simple yes or no question. We realized simple yes or no question. We realized that there's actually two questions that there's actually two questions that there's actually two questions hidden in this single, uh, hidden in this single, uh, hidden in this single, uh, single CTA. single CTA. single CTA. Uh, one, we're asking headphones were Uh, one, we're asking headphones were Uh, one, we're asking headphones were detected. Is that true or false? detected. Is that true or false? detected. Is that true or false? Uh, did the model correctly predict that Uh, did the model correctly predict that Uh, did the model correctly predict that at this video segment there are some set at this video segment there are some set at this video segment there are some set of pixels that look like headphones? of pixels that look like headphones? of pixels that look like headphones? And secondly, are we applying a flag And secondly, are we applying a flag And secondly, are we applying a flag that this person had a violation for it that this person had a violation for it that this person had a violation for it not correctly using headphones at this not correctly using headphones at this not correctly using headphones at this moment? moment? moment? Why is this important? For example, if Why is this important? For example, if Why is this important? For example, if there's somebody who has a hearing aid, there's somebody who has a hearing aid, there's somebody who has a hearing aid, that means that the model correctly that means that the model correctly that means that the model correctly predicted that there are headphones or predicted that there are headphones or predicted that there are headphones or earbuds detected. Uh, that is a true earbuds detected. Uh, that is a true earbuds detected. Uh, that is a true signal. signal. signal. However, we don't actually want to However, we don't actually want to However, we don't actually want to penalize this person for wearing, uh, penalize this person for wearing, uh, penalize this person for wearing, uh, hearing aids. So, we don't want to flag hearing aids. So, we don't want to flag hearing aids. So, we don't want to flag them for a violation. Previously, as to them for a violation. Previously, as to them for a violation. Previously, as to not falsely accuse somebody of cheating, not falsely accuse somebody of cheating, not falsely accuse somebody of cheating, we would select no, but that would be a we would select no, but that would be a we would select no, but that would be a bad signal to give to our model. So, bad signal to give to our model. So, bad signal to give to our model. So, breaking this up into two pieces is breaking this up into two pieces is breaking this up into two pieces is really important because we actually get really important because we actually get really important because we actually get more data and it's better quality and more data and it's better quality and more data and it's better quality and we're not harming the model in the long we're not harming the model in the long we're not harming the model in the long run. run. run. Another example is a quality tutor. On Another example is a quality tutor. On Another example is a quality tutor. On the left here, you see a pattern where the left here, you see a pattern where the left here, you see a pattern where the interaction is overwhelming and on the interaction is overwhelming and on the interaction is overwhelming and on the right delightful. the right delightful. the right delightful. So, this is an example of me talking to So, this is an example of me talking to So, this is an example of me talking to an LLM. I asked it to act as a writing an LLM. I asked it to act as a writing an LLM. I asked it to act as a writing tutor and provide feedback for a passage tutor and provide feedback for a passage tutor and provide feedback for a passage that I wrote. The passage was that I wrote. The passage was that I wrote. The passage was specifically written as if it was a specifically written as if it was a specifically written as if it was a English language learner, so it had a English language learner, so it had a English language learner, so it had a few mistakes or awkward sentences. few mistakes or awkward sentences. few mistakes or awkward sentences. It was a relatively brief paragraph and It was a relatively brief paragraph and It was a relatively brief paragraph and instead instead instead this LLM decided to give me 400 lines of this LLM decided to give me 400 lines of this LLM decided to give me 400 lines of text and text and text and it was very overwhelming and hard to it was very overwhelming and hard to it was very overwhelming and hard to parse. First, it decided to praise me, parse. First, it decided to praise me, parse. First, it decided to praise me, then give some sort of feedback and then then give some sort of feedback and then then give some sort of feedback and then completely rewrite my passage even completely rewrite my passage even completely rewrite my passage even though I never asked it to. though I never asked it to. though I never asked it to. The feedback was not direct. It was hard The feedback was not direct. It was hard The feedback was not direct. It was hard to tie feedback to specific parts of the to tie feedback to specific parts of the to tie feedback to specific parts of the input and it was so large that it would input and it was so large that it would input and it was so large that it would be difficult to actually improve your be difficult to actually improve your be difficult to actually improve your writing through this sort of writing through this sort of writing through this sort of interaction. interaction. interaction. And most importantly, this did not feel And most importantly, this did not feel And most importantly, this did not feel natural. If you were to send an email to natural. If you were to send an email to natural. If you were to send an email to your friend saying, "Hey, can you take a your friend saying, "Hey, can you take a your friend saying, "Hey, can you take a look at this thing that I wrote?" you look at this thing that I wrote?" you look at this thing that I wrote?" you would not expect them to send you an would not expect them to send you an would not expect them to send you an essay double the length telling you all essay double the length telling you all essay double the length telling you all the things that you did right and wrong the things that you did right and wrong the things that you did right and wrong and I'll also give you a whole new and I'll also give you a whole new and I'll also give you a whole new version. version. version. Instead, you would probably want it to Instead, you would probably want it to Instead, you would probably want it to look something like this. look something like this. look something like this. Here you have a screenshot of a Duolingo Here you have a screenshot of a Duolingo Here you have a screenshot of a Duolingo style writing tutor where the text is style writing tutor where the text is style writing tutor where the text is directly marked up. The green is directly marked up. The green is directly marked up. The green is signifying that this is something that signifying that this is something that signifying that this is something that was good and also providing some sort of was good and also providing some sort of was good and also providing some sort of feedback when you hover over it. Yellow feedback when you hover over it. Yellow feedback when you hover over it. Yellow signifying something is awkward or a signifying something is awkward or a signifying something is awkward or a little bit off and red being direct little bit off and red being direct little bit off and red being direct mistakes. When you hover over all of mistakes. When you hover over all of mistakes. When you hover over all of these markups, you can get direct these markups, you can get direct these markups, you can get direct actionable feedback that's concise and actionable feedback that's concise and actionable feedback that's concise and you can accept the suggestions in line. you can accept the suggestions in line. you can accept the suggestions in line. Most importantly, Most importantly, Most importantly, all the feedback is directly tied to a all the feedback is directly tied to a all the feedback is directly tied to a portion of text. portion of text. portion of text. This is more closely mimicking human This is more closely mimicking human This is more closely mimicking human behavior. behavior. behavior. Specifically, like in an academic Specifically, like in an academic Specifically, like in an academic setting, if you're handing your essay to setting, if you're handing your essay to setting, if you're handing your essay to a classmate, you would expect feedback a classmate, you would expect feedback a classmate, you would expect feedback in line as they mark up your text rather in line as they mark up your text rather in line as they mark up your text rather than an essay back. than an essay back. than an essay back. This is useful because over time, if you This is useful because over time, if you This is useful because over time, if you get these sorts of feedback get these sorts of feedback get these sorts of feedback [clears throat] [clears throat] [clears throat] points, you can improve points, you can improve points, you can improve in incremental steps in your writing. in incremental steps in your writing. in incremental steps in your writing. And this is really important for And this is really important for And this is really important for surfacing the exact same AI output. surfacing the exact same AI output. surfacing the exact same AI output. Lastly, and probably most applicably to Lastly, and probably most applicably to Lastly, and probably most applicably to an engineering audience, is a coding an engineering audience, is a coding an engineering audience, is a coding agent. Now, there are a lot of coding agent. Now, there are a lot of coding agent. Now, there are a lot of coding coding agents out there, and there's a coding agents out there, and there's a coding agents out there, and there's a lot of different UIs, people have their lot of different UIs, people have their lot of different UIs, people have their personal preferences, but I've seen that personal preferences, but I've seen that personal preferences, but I've seen that there's two patterns that a lot of out there's two patterns that a lot of out there's two patterns that a lot of out of the box coding agents are falling of the box coding agents are falling of the box coding agents are falling into. So, one pattern is that if you ask into. So, one pattern is that if you ask into. So, one pattern is that if you ask it to do something, it's going to touch it to do something, it's going to touch it to do something, it's going to touch a lot of files and give you a giant a lot of files and give you a giant a lot of files and give you a giant diff, basically doing the thing that you diff, basically doing the thing that you diff, basically doing the thing that you requested all in one go. requested all in one go. requested all in one go. This leads you probably to approve all This leads you probably to approve all This leads you probably to approve all the changes and then take a look at them the changes and then take a look at them the changes and then take a look at them maybe on GitHub, maybe on GitHub, maybe on GitHub, so that you can see all the things that so that you can see all the things that so that you can see all the things that it did. it did. it did. Another approach is that it will ping Another approach is that it will ping Another approach is that it will ping you a notification every single time you a notification every single time you a notification every single time it's changing some sort of file or it's changing some sort of file or it's changing some sort of file or function, and you have to keep clicking function, and you have to keep clicking function, and you have to keep clicking yes, yes, yes, just so that you can get yes, yes, yes, just so that you can get yes, yes, yes, just so that you can get to the result. to the result. to the result. Either way, in both of these cases, Either way, in both of these cases, Either way, in both of these cases, you're just becoming a rubber stamp. you're just becoming a rubber stamp. you're just becoming a rubber stamp. You're accepting all of the changes, and You're accepting all of the changes, and You're accepting all of the changes, and then you're taking a look at them all in then you're taking a look at them all in then you're taking a look at them all in one go and debugging later. Anything one go and debugging later. Anything one go and debugging later. Anything that went wrong is your responsibility that went wrong is your responsibility that went wrong is your responsibility to then identify, fix, re-prompt, or do to then identify, fix, re-prompt, or do to then identify, fix, re-prompt, or do manually. manually. manually. And And And it's not a very delightful experience. it's not a very delightful experience. it's not a very delightful experience. Instead, you probably want a coding Instead, you probably want a coding Instead, you probably want a coding agent that acts like a junior developer. agent that acts like a junior developer. agent that acts like a junior developer. You wouldn't want a junior developer to You wouldn't want a junior developer to You wouldn't want a junior developer to come onto your team and just be like, come onto your team and just be like, come onto your team and just be like, "Oh, sure, I can do that." and disappear "Oh, sure, I can do that." and disappear "Oh, sure, I can do that." and disappear for a day and then give you a thousand for a day and then give you a thousand for a day and then give you a thousand line PR. You also probably don't want a line PR. You also probably don't want a line PR. You also probably don't want a junior dev who's asking you a question junior dev who's asking you a question junior dev who's asking you a question every five minutes at your desk. every five minutes at your desk. every five minutes at your desk. Instead, you want someone who's able to Instead, you want someone who's able to Instead, you want someone who's able to plan, ask good questions, and document plan, ask good questions, and document plan, ask good questions, and document those design decisions, giving you a those design decisions, giving you a those design decisions, giving you a good spec and breaking up their PRs in good spec and breaking up their PRs in good spec and breaking up their PRs in meaningful and reviewable ways. meaningful and reviewable ways. meaningful and reviewable ways. This means that it's delightful for you This means that it's delightful for you This means that it's delightful for you as a mentor or in this case as a mentor or in this case as a mentor or in this case developer who's partnering with a coding developer who's partnering with a coding developer who's partnering with a coding agent to experience things in a way that agent to experience things in a way that agent to experience things in a way that is easily reviewable. You can highlight is easily reviewable. You can highlight is easily reviewable. You can highlight assumptions and maybe answer questions assumptions and maybe answer questions assumptions and maybe answer questions about them and you can easily see about them and you can easily see about them and you can easily see a plan and the things that the decisions a plan and the things that the decisions a plan and the things that the decisions that it's making before things go wrong. that it's making before things go wrong. that it's making before things go wrong. Also, the steps are manageable and Also, the steps are manageable and Also, the steps are manageable and you're not being pinged too many times. you're not being pinged too many times. you're not being pinged too many times. And then going a little bit into the And then going a little bit into the And then going a little bit into the data of side of this, if you have data of side of this, if you have data of side of this, if you have agentic agentic agentic coder and it's leading an interaction coder and it's leading an interaction coder and it's leading an interaction where a developer is encouraged to just where a developer is encouraged to just where a developer is encouraged to just quickly skim everything and not really quickly skim everything and not really quickly skim everything and not really discern. discern. discern. This means that you're just collecting This means that you're just collecting This means that you're just collecting these binary signals of accepted and these binary signals of accepted and these binary signals of accepted and rejected, usually skewing towards rejected, usually skewing towards rejected, usually skewing towards accepted, and that doesn't really give accepted, and that doesn't really give accepted, and that doesn't really give you that much information because it's you that much information because it's you that much information because it's just on some coding block. just on some coding block. just on some coding block. Instead, if an agent is acting as a Instead, if an agent is acting as a Instead, if an agent is acting as a partner with a developer steering, you partner with a developer steering, you partner with a developer steering, you can get rich data because it's can get rich data because it's can get rich data because it's structured specifically around different structured specifically around different structured specifically around different parts of the development cycle and it's parts of the development cycle and it's parts of the development cycle and it's capturing nuanced structured decisions capturing nuanced structured decisions capturing nuanced structured decisions about the different parts of about the different parts of about the different parts of the tasks that it's doing. For example, the tasks that it's doing. For example, the tasks that it's doing. For example, it can look at assumptions that were it can look at assumptions that were it can look at assumptions that were bad, the tradeoffs that it made, bad, the tradeoffs that it made, bad, the tradeoffs that it made, stylistic preferences and approaches stylistic preferences and approaches stylistic preferences and approaches that the developer's taking. And all of that the developer's taking. And all of that the developer's taking. And all of those pieces of data can then continue those pieces of data can then continue those pieces of data can then continue to make your experience as a like as to make your experience as a like as to make your experience as a like as your AI system coding system better. your AI system coding system better. your AI system coding system better. So, let's look at some design So, let's look at some design So, let's look at some design principles. First, you can engineer the principles. First, you can engineer the principles. First, you can engineer the reasoning. Think about what reasoning reasoning. Think about what reasoning reasoning. Think about what reasoning pattern do you want to elicit from a pattern do you want to elicit from a pattern do you want to elicit from a human and how does your interface human and how does your interface human and how does your interface challenge that? So, if you have a system challenge that? So, if you have a system challenge that? So, if you have a system that is specifically optimizing for that is specifically optimizing for that is specifically optimizing for independent judgment, reframe your independent judgment, reframe your independent judgment, reframe your system to think of the human as system to think of the human as system to think of the human as investigator, not just a validator. investigator, not just a validator. investigator, not just a validator. Don't just give a hey, is this looking Don't just give a hey, is this looking Don't just give a hey, is this looking good, but instead have it be a more good, but instead have it be a more good, but instead have it be a more thoughtful engaged effort. thoughtful engaged effort. thoughtful engaged effort. Surfacing assumptions is something Surfacing assumptions is something Surfacing assumptions is something that's very valuable when you care about that's very valuable when you care about that's very valuable when you care about a lot about the quality a lot about the quality a lot about the quality of a long-running task. If the model is of a long-running task. If the model is of a long-running task. If the model is stating them earlier on and asking for a stating them earlier on and asking for a stating them earlier on and asking for a sign-off, you can prevent any future sign-off, you can prevent any future sign-off, you can prevent any future miscommunication or having to go back miscommunication or having to go back miscommunication or having to go back and fix your assumptions. and fix your assumptions. and fix your assumptions. Weighing trade-offs is also very Weighing trade-offs is also very Weighing trade-offs is also very important because very often important because very often important because very often an LLM might give you some sort of an LLM might give you some sort of an LLM might give you some sort of option and it thinks it's best without option and it thinks it's best without option and it thinks it's best without having to actually weigh the different having to actually weigh the different having to actually weigh the different different different different trade-offs and design decisions that trade-offs and design decisions that trade-offs and design decisions that it's making in the background. it's making in the background. it's making in the background. Presenting those options and the Presenting those options and the Presenting those options and the reasoning for why allows the user to reasoning for why allows the user to reasoning for why allows the user to continue to be in control over the continue to be in control over the continue to be in control over the output and they can opine earlier on so output and they can opine earlier on so output and they can opine earlier on so that they can get the thing that they that they can get the thing that they that they can get the thing that they actually want. Lastly, for sustained actually want. Lastly, for sustained actually want. Lastly, for sustained attention, you want to build in friction attention, you want to build in friction attention, you want to build in friction exactly where the stakes are high. If exactly where the stakes are high. If exactly where the stakes are high. If you want the person to slow down and you want the person to slow down and you want the person to slow down and think deliberately, friction is your think deliberately, friction is your think deliberately, friction is your friend, which leads us to the next friend, which leads us to the next friend, which leads us to the next principle to match the friction to the principle to match the friction to the principle to match the friction to the stakes. stakes. stakes. In a high-stakes example like ours where In a high-stakes example like ours where In a high-stakes example like ours where the Duolingo English Test really matters the Duolingo English Test really matters the Duolingo English Test really matters for people and their livelihood, we want for people and their livelihood, we want for people and their livelihood, we want our human reviewers to be deliberately our human reviewers to be deliberately our human reviewers to be deliberately slow and very thoughtful with the slow and very thoughtful with the slow and very thoughtful with the decisions that they make. This means decisions that they make. This means decisions that they make. This means that we want to be adding review gates that we want to be adding review gates that we want to be adding review gates that are well-structured and friction so that are well-structured and friction so that are well-structured and friction so they can slow down and think about these they can slow down and think about these they can slow down and think about these things so that they can never become a things so that they can never become a things so that they can never become a rubber stamp. rubber stamp. rubber stamp. You [snorts] want these people the You [snorts] want these people the You [snorts] want these people the people to slow down people to slow down people to slow down and you want the output to remain high and you want the output to remain high and you want the output to remain high quality. You don't want noise and you quality. You don't want noise and you quality. You don't want noise and you want clarity. So, this means that you want clarity. So, this means that you want clarity. So, this means that you have to be actively thinking about what have to be actively thinking about what have to be actively thinking about what friction friction friction like what speed bumps you want to put like what speed bumps you want to put like what speed bumps you want to put along the road and how and what along the road and how and what along the road and how and what checkpoints. checkpoints. checkpoints. For systems where you have low For systems where you have low For systems where you have low oversight, for example, something oversight, for example, something oversight, for example, something delightful where you're just chatting delightful where you're just chatting delightful where you're just chatting with an AI, you want to make sure that with an AI, you want to make sure that with an AI, you want to make sure that it's relatively frictionless. Your it's relatively frictionless. Your it's relatively frictionless. Your design has to optimize for a happy design has to optimize for a happy design has to optimize for a happy someone who's happy with that output and someone who's happy with that output and someone who's happy with that output and doesn't really feel like there's a lot doesn't really feel like there's a lot doesn't really feel like there's a lot of of of stopping points or scenarios where it's stopping points or scenarios where it's stopping points or scenarios where it's kind of difficult to interact with the kind of difficult to interact with the kind of difficult to interact with the system. Instead, you want it to be system. Instead, you want it to be system. Instead, you want it to be absolutely seamless. absolutely seamless. absolutely seamless. Next principle is every interaction is Next principle is every interaction is Next principle is every interaction is already a label. already a label. already a label. Instead of having to select a portion of Instead of having to select a portion of Instead of having to select a portion of your data and have human annotators your data and have human annotators your data and have human annotators get you a better cleaner data set, you get you a better cleaner data set, you get you a better cleaner data set, you can kind of already start thinking about can kind of already start thinking about can kind of already start thinking about that specific interaction piece as that specific interaction piece as that specific interaction piece as already providing you labels and signals already providing you labels and signals already providing you labels and signals for your next iteration. So, if the for your next iteration. So, if the for your next iteration. So, if the agent plan was approved or a suggestion agent plan was approved or a suggestion agent plan was approved or a suggestion was accepted, this means that you did was accepted, this means that you did was accepted, this means that you did something right and your output matched something right and your output matched something right and your output matched the intent of the user. However, if the the intent of the user. However, if the the intent of the user. However, if the output was modified or in a output was modified or in a output was modified or in a recommendation was overridden, that recommendation was overridden, that recommendation was overridden, that probably signifies something bad. probably signifies something bad. probably signifies something bad. However, a lot of systems don't really However, a lot of systems don't really However, a lot of systems don't really capture that diff and the model falls capture that diff and the model falls capture that diff and the model falls short because what you get on that short because what you get on that short because what you get on that decision is a yes or no. And the human decision is a yes or no. And the human decision is a yes or no. And the human clicked yes, then manually went in and clicked yes, then manually went in and clicked yes, then manually went in and changed something or erased it changed something or erased it changed something or erased it completely. completely. completely. If you don't capture that diff, which If you don't capture that diff, which If you don't capture that diff, which could be something for the could be something for the could be something for the the AI falling short or maybe completely the AI falling short or maybe completely the AI falling short or maybe completely being wrong, you actually capture a being wrong, you actually capture a being wrong, you actually capture a false signal that then can pollute your false signal that then can pollute your false signal that then can pollute your data sets. Instead, think about how can data sets. Instead, think about how can data sets. Instead, think about how can you measure that diff? you measure that diff? you measure that diff? Lastly, there are also questions that a Lastly, there are also questions that a Lastly, there are also questions that a human might be asking. For example, human might be asking. For example, human might be asking. For example, explanations or follow-ups. And this explanations or follow-ups. And this explanations or follow-ups. And this could mean that the human has a low could mean that the human has a low could mean that the human has a low trust in the system or there was trust in the system or there was trust in the system or there was something wrong with the AI. something wrong with the AI. something wrong with the AI. Uh so, take a look at the kind of Uh so, take a look at the kind of Uh so, take a look at the kind of questions that are being asked and what questions that are being asked and what questions that are being asked and what sentiment they have. sentiment they have. sentiment they have. Another principle is to stop asking how Another principle is to stop asking how Another principle is to stop asking how to evaluate the model. to evaluate the model. to evaluate the model. Instead of building your system and then Instead of building your system and then Instead of building your system and then being like, "Hey, what kind of data can being like, "Hey, what kind of data can being like, "Hey, what kind of data can I capture to see if this is good?" I capture to see if this is good?" I capture to see if this is good?" Proactively think about what defines Proactively think about what defines Proactively think about what defines success for your AI system. Think about success for your AI system. Think about success for your AI system. Think about how you can be measuring that with how you can be measuring that with how you can be measuring that with concrete metrics and what data will you concrete metrics and what data will you concrete metrics and what data will you need to improve the system from the need to improve the system from the need to improve the system from the start. start. start. Those decisions will be informing the Those decisions will be informing the Those decisions will be informing the way you design that interaction such way you design that interaction such way you design that interaction such that you can ensure that you have the that you can ensure that you have the that you can ensure that you have the hard evidence for that next step. Every hard evidence for that next step. Every hard evidence for that next step. Every interaction should be helping the user interaction should be helping the user interaction should be helping the user to teach them how to improve the system to teach them how to improve the system to teach them how to improve the system and be a partner with the AI. This leads me to my final point, which This leads me to my final point, which is all of those things are leading you is all of those things are leading you is all of those things are leading you to engineer that interaction. There are to engineer that interaction. There are to engineer that interaction. There are different ways to engineer that different ways to engineer that different ways to engineer that interaction, which I've covered briefly, interaction, which I've covered briefly, interaction, which I've covered briefly, but some of the principles are this. but some of the principles are this. but some of the principles are this. First, you can have structured inputs First, you can have structured inputs First, you can have structured inputs and outputs. That way you don't just and outputs. That way you don't just and outputs. That way you don't just have vibes or giant walls of text. have vibes or giant walls of text. have vibes or giant walls of text. Instead, you have specifics. You can Instead, you have specifics. You can Instead, you have specifics. You can think of this in terms of forms that the think of this in terms of forms that the think of this in terms of forms that the user is filling out, structured output user is filling out, structured output user is filling out, structured output in forms of tables, markup UIs, for in forms of tables, markup UIs, for in forms of tables, markup UIs, for example, that example, that example, that writing tutor or maybe it's a design writing tutor or maybe it's a design writing tutor or maybe it's a design thing where you can highlight specific thing where you can highlight specific thing where you can highlight specific elements. All of those ways are having elements. All of those ways are having elements. All of those ways are having targeted interactions between the human targeted interactions between the human targeted interactions between the human and the AI ensuring a better result. and the AI ensuring a better result. and the AI ensuring a better result. Next, highlighting that those Next, highlighting that those Next, highlighting that those assumptions. If you are having assumptions. If you are having assumptions. If you are having assumptions being made by the model, if assumptions being made by the model, if assumptions being made by the model, if you surface them proactively and ask if you surface them proactively and ask if you surface them proactively and ask if it's legit, that's going to save you it's legit, that's going to save you it's legit, that's going to save you some tokens down the line so that the some tokens down the line so that the some tokens down the line so that the human is not having to fix or argue or human is not having to fix or argue or human is not having to fix or argue or correct with the correct the model correct with the correct the model correct with the correct the model output. Instead, it's being proactive output. Instead, it's being proactive output. Instead, it's being proactive and the having those things that are and the having those things that are and the having those things that are core core core pieces of information that are going to pieces of information that are going to pieces of information that are going to inform later decisions. inform later decisions. inform later decisions. Next is building in friction and review Next is building in friction and review Next is building in friction and review gates. That deliberate slowing down so gates. That deliberate slowing down so gates. That deliberate slowing down so that you can have good quality thought that you can have good quality thought that you can have good quality thought and just required for good quality and just required for good quality and just required for good quality decision-making and good quality data is decision-making and good quality data is decision-making and good quality data is all important because if you're bringing all important because if you're bringing all important because if you're bringing in that friction and review gates, in that friction and review gates, in that friction and review gates, people are going to start thinking people are going to start thinking people are going to start thinking clearly as if it's really their own clearly as if it's really their own clearly as if it's really their own thing rather than just something that AI thing rather than just something that AI thing rather than just something that AI is is is supplanting. supplanting. supplanting. Lastly, collecting explicit feedback is Lastly, collecting explicit feedback is Lastly, collecting explicit feedback is really important because as we've really important because as we've really important because as we've covered, every one of those interactions covered, every one of those interactions covered, every one of those interactions is a piece of data. So, all of those is a piece of data. So, all of those is a piece of data. So, all of those signals are going into your next signals are going into your next signals are going into your next generation of the model, and collecting generation of the model, and collecting generation of the model, and collecting explicit feedback, not just a thumbs up, explicit feedback, not just a thumbs up, explicit feedback, not just a thumbs up, thumbs down, but instead feedback at the thumbs down, but instead feedback at the thumbs down, but instead feedback at the correct touch points, correct touch points, correct touch points, uh with the correct amount of nuance is uh with the correct amount of nuance is uh with the correct amount of nuance is something that can really drive something that can really drive something that can really drive improvements directly. improvements directly. improvements directly. So, yes, So, yes, So, yes, uh you should be designing for uh you should be designing for uh you should be designing for discernment. Sometimes the fix is not a discernment. Sometimes the fix is not a discernment. Sometimes the fix is not a better model or more oversight, it's better model or more oversight, it's better model or more oversight, it's just engineering the interaction itself. just engineering the interaction itself. just engineering the interaction itself. Thank you.