---
title: "Transcript: AI System Design: From Idea to Production - Apoorva Joshi, MongoDB"
category: "transcripts"
videoId: "T0HhO4YtTfE"
sourceLabels: ["YouTube transcript", "Cached transcript markdown"]
wordCount: "4200"
---

# Transcript: AI System Design: From Idea to Production - Apoorva Joshi, MongoDB

## Source Video
- [YouTube](https://www.youtube.com/watch?v=T0HhO4YtTfE)

## Local Cache
- `raw/sources/youtube-transcripts/T0HhO4YtTfE.txt`
- 4,200 words

## Transcript

Hi everyone. I'm Apurva and I'm a data scientist turned developer advocate currently at MongoDB. I spent the first of my career building machine learning applications for various cybersecurity use cases and I now use that applied machine learning knowledge to help AI builders successfully build AI applications with MongoDB and Voyage AI. In this talk you'll learn how to think through building AI systems end-to-end from idea to production. We'll take a real-world use case and walk through all the steps of designing it and hopefully in the end you'll walk away with a repeatable framework that you can apply to any AI system you design.

Now, you might be thinking in the age of AI do we even need to think about what we build? Just wipe code it and ship it, right? But that's actually where I want to start. Now, here's the problem with that. Wipe coding works great when you're building for fun, the stakes are low, and you can easily eyeball whether the output of what you're building is right. But the moment you're building something real, something other people depend on, something with real consequences, just ship it is actually kind of dangerous. Uh and it's not just me saying it, folks from Anthropic and OpenAI who are so bullish on AI coding are saying the same thing.

These are actually quotes from their talks over the past few months. Specs are the new code. The art is in defining the product requirements, the system design, and evaluate criteria so you can be confident that your AI coding buddies are building the right thing. And the rest of the talk is just about that. How do we do this well? It's useful to think about it as a framework. Four phases. You start with product requirements. What are you actually building, for whom, and what are the constraints? Then system design, the data, the architecture, the patterns that actually help you meet these requirements. Then comes evaluation and monitoring.

How do you know what's being built actually works before and also after you ship? And finally, how do you optimize for cost, not just for accuracy, but also for cost, latency, and reliability before you ship and or as you find gaps in production. Instead of talking abstractly through a framework, I thought why not apply it to a real-world use case so you can see how exactly each decision flows from one stage of the framework to the next. So let's take the example of a health insurance claims review system.

So adjudicating health insurance claims has historically been an extremely manual process where human medical reviewers have to cross-reference clinical guidelines, insurance coverage policies, patient history, and stuff to decide whether or not a medical treatment, procedure, or medication is covered by a patient's health insurance policy. Now, if you live somewhere that has free health care, even there there's things that are covered and things that are not by your health care system. So you're still subject to some version of this process. Now, our goal with building this system is to see if we can improve or simplify the experience for medical reviewers with the help of AI.

So let's see what the product requirements stage of the framework for this application might look like. So first thing to do is quantify the business problem. The business problem should focus on something specific, uh clearly state who the users of the application are, uh the current state of things, and quantify the users' pain point. It should not prescribe what the system is going to be, whether it's going to be an agent, a multi-agent system, something else. Uh that will come later. So, here's what the business problem for our claims review system could look like.

Medical reviewers at MDB Health spend an average of 2 days processing claim review requests, which is four times the industry standard for non-urgent cases, and 12 times the industry standard for urgent ones. Delays at this scale postpone patient care, uh particularly for time-sensitive treatment. So, as you can see, this business problem is user-specific. It says this is meant for medical reviewers. Uh it states the current state, which is like this manual process that leads to them spending 2 days processing uh claim reviews. It's measurable because it states some baselines. Uh it's solution-agnostic. It doesn't really tell you uh what the system should look like, uh and it's very focused on a specific problem.

Next, you want to gather any business constraints for the application. So, any regulatory uh compliance requirements, any constraints on data leaving your organization, procurement constraints, meaning are certain vendors uh not approved for use within the organization. Uh all of these would be good to know before you even start designing the system. So, for our application, say these are some of the business constraints. Patient data has to stay within the approved cloud environment. Um only models available on the approved cloud can be used. Um there's some use cases where human review is required. So, any complex use cases require a senior physician's review. Um what else?

Any denial decisions must be uh reviewed by a human reviewer. So, there's cases which you can't fully automate with AI. So, yeah, all of these are useful design inputs. So, you want to have these sooner rather than later. Also good to know of any performance requirements. Uh do we need sub-millisecond responses or do we have some leeway there? Uh how much can we spend per month on LLM inference? Any uptime SLAs to be aware of? Next, if you're building an AI product, identifying the role of AI uh in your product is helpful. And I like to think about the role of AI along three dimensions. First, is AI critical to your product or complementary?

In this case, complementary since um claims are already happening without claim reviews already happening without AI, but they are just slower. Next, will this be a reactive or proactive system? Meaning, is it triggered by a user or an event or does it proactively do things? In this case, it's going to be a reactive system since uh it's triggered by a claims application actually being submitted. And finally, what's the level of autonomy uh for AI in the application? So, in our case, the business constraints prescribe human review in certain cases. So, our system can be semi-autonomous at max.

And finally, as part of defining what you're solving for, you also want to come up with one to two uh success metrics aligned with the business problem so you can capture what success looks like. A good success metric is specific, measurable, achievable, relevant, and time-bound. And here's an example of a success metric for our claims review application. So, it says it says reduce the average processing time for urgent claim review requests from 2 days to 1 hour within 90 days of launch. So, it's super specific. It's measurable because it's saying uh the processing time needs to be 1 hour.

Uh it seems achievable because um there'll be AI in the loop, so hopefully any information gathering uh getting an initial recommended recommendation will be faster. Uh it's very relevant and rooted in the business problem from before, and it's time-bound because it gives you uh a window for when we need to start seeing success. Okay, so at this point we figured out uh the what. Let's look at the how. The first thing you want to think about here is your data strategy. What does uh what data does your application need? Do you even have access to it? Where does it reside? Uh is it ready for consumption in its raw form, and so on?

So, for our claims review application, these are the data sources that we might need, where they reside, and what their raw access formats might look like. So, we need uh clinical guidelines probably, uh which describe what care is appropriate for a given condition. Uh MDB Health's coverage policies that state what MDB Health will and will not pay for. And finally, the patient's claims history to know any uh past conditions, uh procedures that were performed, when they were performed, and so on. Let's assume that the clinical guidelines and coverage policies are stored as PDFs in something like Confluence and claims are stored in MongoDB as and when they are processed.

The next thing to think about is data update frequency. How often do the data sources update? Now, if you're doing any sort of data processing on these raw data sources and you typically are, these pipelines are going to have to run at the right cadence to keep up. Otherwise, our system is going to work off of stale information and that's not good, especially in such a high-stakes use case. So, let's work this out for our claims review system. Clinical guide These probably update annually. Uh internal coverage policies may be quarterly and the patient's claim history needs to be updated whenever a new claim is processed.

And in our success metrics, if you recall, uh we've said we want to aim for a 1-hour time frame for adjudicating urgent claims, so these will probably get updated hourly. Next, can the data be consumed directly by our application or does it need any processing? For clinical guidelines, which are typically long documents, you probably want to chunk them, embed the chunks, and also extract metadata such as procedure names, publication dates to make it suitable for retrieval using something like vector search or hybrid search.

Same goes for coverage policies, but for patient claims history, which is already well-formatted and stored in MongoDB, we might just need to remove personally identifiable information before passing it to an LLM. Next, we want to think about what retrieval techniques would be best suited for each of the data sources. And you kind of already start thinking about this in the data processing step, but let's quickly formalize this. Now, clinical guidelines and coverage policies are well suited for vector search, but they can contain medical terminologies, such as diagnosis codes, procedure codes, that might not be picked up by vector search. So, um but these will be picked up by metadata filters or something like keyword search.

So, for these, you'd either want to use vector search with metadata pre-filtering or hybrid search. For getting the patient's claims history, you can do an exact match on the patient name or ID, whatever the patient identifier is in the system. Next, you want to think about what the system architecture looks like. Now, it can be tempting to jump straight to building an agent. There's so much hype around them, or let a coding agent decide what the system architecture should look like, but you risk ending up with an over-engineered system by doing this.

So, what you instead want to do is start with the simplest design, evaluate it, find gaps during the evaluation, and iterate from there. To decide what the system should look like, let's first map out how an insurance claim would flow through the system, because this will help us inform the architecture. So, first, the claim request and any clinical notes from the physician are received by our system. Then the system needs to retrieve any clinical guidelines and coverage policies that are relevant to the current claim, and also retrieve the patient's claims history.

Then all of this information, along with the claim itself, and any system prompts on how the LLM should generate go about generating the recommendations, all of these are passed to the LLM, Um, and it uses those to generate uh, a recommendation of whether the claim is claim should be approved or rejected. Now, if it's a complex case, our uh, system or the LLM at this point should uh, send its initial recommendation um, to a senior physician. And if the LLM's recommendation is to deny the claim, then it should also be escalated to a human medical reviewer.

And finally, the final decision should be logged to MongoDB, which is where we are storing our claim documents uh, along with a reasoning for why the claim was approved or rejected. Now, let's talk about some of the most common AI design patterns that we see uh, in AI systems today and see which of those might apply to our system. So, the first one is rag, uh, short for retrieval augmented generation. You all probably know it at this point. Um, basically, in these systems, you simply augment and LLM's pre-trained knowledge with external knowledge sources. So, information from external knowledge sources.

Then, there's AI agents where you give uh, a single LLM or multiple LLMs in the case of multi-agent systems full autonomy to complete tasks end-to-end with the help of a set of tools. And then, there's agentic systems, which cover uh, a broad spectrum of semi-autonomous AI systems. A common architectural pattern for uh, these types of systems are control flows, where LLMs perform certain tasks within the workflow, but the sequence of steps in the workflow itself is predetermined by design or by code. Then, there's LLM as a router, where LLMs have limited autonomy in that their job is simply to categorize incoming requests and route them to different downstream workflows.

Then you have human in the loop where an LLM or agent can take certain actions, but it needs human intervention somewhere in the process. And finally, there's fine-tuning. So, if you're observing that the LLM's failures are behavioral rather than data or orchestration issues, or if you need superior performance on a domain-specific task, then fine-tuning is a good technique to explore. I don't have the time to go too deep into these here, but if you'd like to learn more about these design patterns, I'm going to leave this resource here for you to check out later. So, we've thought through what the workflow of this system should look like.

We've spoken about a few different AI design patterns. Now, let's apply those and decide what design patterns um we'd need to implement for our system. So, since we need to retrieve clinical guidelines, coverage policies, and stuff to inform the decision, this is This definitely looks like a rag component. So, we need some form of rag. The workflow is pre-structured in that the LLM first provides its recommendation on whether or not a claim should be approved or rejected, and in very specific and well-defined cases, it needs to escalate claims to human reviewers. So, this sounds like a controlled workflow.

You could also argue that this looks a little bit like LLM as a router, especially if you use an additional LLM call to decide what a complex case looks like, unless this is already well-defined. So, yeah, this could be an LLM as a router, but uh for now let's just go with control flow. And finally, there's also an element of human in the loop, right? Since human reviewers are very much a part of the workflow. And finally, in the design phase, you also want to think about the end user experience and how you'll collect feedback from your users to improve the system.

Here are some questions to ask yourself in this stage to help think through uh what the UX and feedback mechanism should look like. So, first, what does the system take as input? In our case, it's going to be a claim request form of some kind. Then, what does the system produce as output? So, in our case, it's going to be uh an approval or denial verdict along with an explanation for why uh a claim was approved or rejected. Where does the system live? Does it have a standalone app? Is it embedded in an existing website? Is it a Slack bot? Uh and so on.

Uh so, for our application, it's very likely going to be embedded within MDB Health's main website, let's say. Uh what triggers the system? So, in our case, very clearly, a claim submission. What's the human's role? So, in our case, they are going to review the system's assessment and make the final decision in some cases. Um how does the system explain itself? In our case, um it explains itself by providing citations, so quoting uh the clinical guidelines and coverage policies that uh informed the system's assessment. And finally, how does it give feedback? So, remember, for our system, um the consumers are medical reviewers.

Um so, they can provide feedback by overriding the verdict of the LLM and also log a reason for why Uh they can also we can maybe also have them flag any irrelevant citations. So, if their LM is hallucinating um clinical guidelines and policies or yeah, quoting the wrong ones for a particular claim, then we can have them flag those as well. All right. So, at this point we've decided what the architecture of our system should look like, but we also need to decide what our stack should look like.

Once again, it might be tempting to have a coding agent recommend what tools to use, but they might not necessarily apply to your system or you might not be able to use them based on the constraints that we identified during the product requirement stage. So, we had constraints around models need to be supported by our cloud provider and things like that. I'm not going to get into this too much since our constraints are well, hypothetical, but I just want to mention that this is the next step and the different tooling decisions to be made such as what model to use.

There's a bunch of data processing tools out there that have some nice out of the box data processing capabilities. If retrieval is part of your system, then and you're using things like vector search, then you might need to decide what embedding models to use, what vector database to use. Um then there's orchestration frameworks and so on. So, I'm just going to leave this here for reference later on. Next, let's talk about evaluation and monitoring. How do you know that what you've built works before and also after you ship? So, evaluation is before, monitoring is after you ship, and you need both of them.

But before we get into evaluation metrics, let's talk about guardrails because these weren't really really a thing pre-LLMs. The reason they've become a topic of conversation is because unlike traditional software, LLM-based systems are going to are probabilistic systems and can produce outputs that are unexpected, incorrect, or even harmful. And guardrails are an attempt to mitigate these risks and ensure that your system behaves within acceptable boundaries. And what those boundaries are are something you need to define. So for inputs to your system, the goal is usually to detect invalid, irrelevant, or harmful inputs. Uh so for our claims review system, an input like write me a poem is irrelevant and our system should reject this claim.

Similarly, for outputs, our goal is to detect invalid, incorrect, or hallucinated, uh or harmful outputs. So for our claims application, we'll consider uh a response to be invalid if the system doesn't provide citations for uh what informed an approval or denial. Now guardrail compliance is not the only thing you want to measure. You want to measure the quality of the responses, uh create and measure domain-specific accuracy metrics, and also general system health. So for input guardrail compliance, uh for our system, we can calculate a claim rejection rate. How often is our uh system rejecting claims because they don't obey uh our input guardrails.

Now if it's rejecting too many times, then that's a call for investigation, but if you didn't measure this in the first place, then you won't have anything to investigate, so uh there we go. That's why you need metrics. For output guardrail compliance, we can have something like a missing citation rate, how many times is the LLM producing responses without citations. For response quality, we can measure faithfulness, which is whether or not the claims approval or rejection is actually rooted in the retrieved uh information like clinical guidelines and policies. Then you want to define at least one domain or application-specific metric.

So, in our case, something like claim processing time might be a good one because that's really uh our North Star. And system-level metrics are usually uh around overall system health. So, uh what's your system's average token cost, token usage, uh average number of turns in a conversation. Uh so, here just to not have another latency metric, uh I put down something like cost for recommendation. So, once you evaluated your system and ship your product in front of users, then comes monitoring the system for regressions in real time.

Now, in production, you still want to track the metrics that you used for uh offline evaluation, so the metrics that we talked about, but you might also be able to track additional metrics that can act as uh implicit indicators of your product's success or like how uh uh how happy users are with your product and overall system health. For example, for the claims review app, you can track how often a human reviewer overrides the AI verdict. Uh now, you want this rate to be low uh because that means the system is not doing its job.

So, if you ever start seeing is seeing it increasing about above a certain threshold, then you need to go and investigate. Or you can also track something like how long does it take for a human to review the AI recommendation. If it takes too long, which means um this could mean that the responses are very verbose, confusing. So, yeah, these would be uh some good indicators to track uh once you ship your product to production. Now, once you have a working prototype that you've evaluated, accuracy looks good, it's time to move to production, right? Well, what about cost, latency, and reliability? Uh these constraints become absolutely non-negotiable as you're moving your product to production.

So, you might need to do a few more iterations, evaluations, and testing uh between accuracy looks good and moving to production. So, let's talk about some optimization techniques for each of these. So, optimizing for accuracy. In LLM-based applications, it really comes down to optimizing the information that uh ends up in the LLM's context window. And here are some techniques for you to refer to later, but let's quickly talk about which of these uh might apply to our claims review application. So, prompt engineering, of course. Uh reranking is another good one because uh retrieval is a big component of it.

So, uh a reranker will make sure that the information being retrieved um is surfacing information in the right order of relevance, which is important when working with LLMs. Uh we are also persisting patient history uh to MongoDB, so our application already has some form of memory, but you could also think about other types of information that might be uh useful to persist across sessions. Here are some techniques for cost and latency optimization. For our use case, semantic caching could be useful to expedite decisions based on uh similar claims from the past. Uh batch processing, perhaps, to process claims in batches instead of one at a time. And finally, optimizing for reliability.

Most of these would be useful for our use case since they're mainly solving for API failures, but I want to specifically call out structured outputs, which would be a good one to make sure that our system is always producing a structured output which contains not just the decision, but also the citations. This brings me to the end of my talk, but I wanted to leave you with a few takeaways. Think deeply about your product's requirements before having AI generate any code. The product spec is the hard part now, it's not the code anymore. Your latency budget, your cost ceiling, any regulatory requirements, all of these shape every architectural decision downstream.

So, good to identify business and performance constraints before starting to design your application. Design the simplest system that meets your needs. Evaluate it and then iterate from there. The most common mistake I see is over-engineering the solution before knowing what's actually failing or not even evaluating what's actually failing. And that brings me to evaluation. Build evaluation in from the start. You can't improve what you can't measure. Link to our GenAI cookbook that has a ton of examples of different retrieval technique, agentic design patterns, and some of the evaluation and optimization techniques that I was talking about. And yeah, thank you so much for taking the time to listen to my talk.

I'll see you at the conference. >> conference.