RAG couples retrieval, which selects evidence, and generation, which composes answers. Treating that combination as one AI box hides the decisions an interviewer is trying to evaluate.

A strong RAG system design interview answer draws two connected systems. Each has its own latency budget, state, capacity limits, failure handling, and quality measurements. The boundary matters because a fluent but incorrect answer may originate in retrieval, context construction, or generation. An end-to-end result alone cannot tell you which stage failed.

This builds on the prerequisite post’s frame that the model is a dependency, going one level deeper into retrieval.

Use a changing-document assistant as the running example. Employees ask questions about internal policies, and the source documents are updated over time. The design must retrieve the applicable text, construct a bounded prompt, and generate an answer grounded in that text. Before choosing an index or model, establish how quickly document changes must become searchable.

Start with freshness and draw two systems

The first question to ask is: How fresh does the retrieved data need to be?

That answer shapes the ingestion path. If updates can appear the next day, a scheduled batch pipeline may be sufficient. If policy changes must become searchable within minutes, the design needs a more continuous path, along with a defined way to handle failed ingestion, duplicate events, deletions, and documents that change while an older version remains indexed.

Freshness is only one part of the workload contract. Clarify:

• What document types and approximate corpus size are in scope?
• How frequently do documents change?
• What query volume and traffic pattern should the system support?
• Do users have different permissions over the same corpus?
• Must answers include citations or excerpts?
• What should happen when the system cannot find adequate evidence?
• Is the goal conversational first-token latency, complete-answer latency, or an offline result?

Then draw the architecture as two hops:

1. Retrieval: transform the query, search the index, filter and rank candidates, and return selected chunks.
2. Generation: assemble the prompt, invoke a model replica, and produce an answer from the selected context.

An orchestrator connects the two. It carries request metadata, enforces deadlines, constructs the model input, and applies fallback behavior. Even if a managed platform exposes RAG through one API, these logical boundaries still matter for diagnosis. A single product surface may simplify operations, but it does not make retrieval and generation share the same failure causes or quality metrics.

Treat retrieval as a search system with a write path

A complete design includes an ingestion pipeline as well as a query path.

On the write path, an ingestion worker reads source documents, extracts text and metadata, divides text into chunks, computes embeddings where applicable, and writes each chunk into an index shard or index version. Metadata might include a document identifier, source version, section, update time, and access-control attributes. Those fields support filtering, deletion, traceability, and replacement when the source changes.

Chunking affects both retrieval precision and prompt size. Small chunks may isolate precise passages but lose surrounding context. Large chunks preserve more context but increase prompt size and may introduce unrelated material. The appropriate boundary depends on the structure of the documents and the kinds of questions users ask. Headings, paragraphs, code blocks, and tables may warrant different treatment.

For changing policy documents, the system also needs an update model. Replacing an index in place can expose partially updated state. Writing to a new index version and switching reads after validation provides a clearer publication boundary, although it requires additional storage and lifecycle management. Another design may update individual records incrementally. In either case, explain how the system removes stale chunks and prevents a query from mixing incompatible versions of the same document.

On the query path, the retriever may combine several controls:

• semantic retrieval based on a query embedding;
• lexical retrieval for exact names, identifiers, or uncommon terms;
• metadata filters for permissions, tenant, language, or document type;
• reranking to reorder an initial candidate set;
• deduplication or diversity rules to avoid returning several near-identical chunks.

These are not mutually exclusive architectures. They modify different parts of retrieval and are often composed. For example, the assistant might apply an access filter, run lexical and semantic searches, merge their candidates, and rerank the result before context construction.

Caching also needs a named boundary. A cached query embedding avoids repeating one computation. A cached retrieval result avoids search work but can violate the freshness requirement. A cached final answer adds another source of staleness and may be unsafe across permission boundaries. State what is cached, how it is keyed, and what invalidates it.

Split the latency budget by stage

Suppose the interview gives an illustrative target of two seconds to first token. That is an end-to-end design assumption, not a universal RAG target, and retrieval does not receive the full two seconds.

Break the path into measured stages:

• request admission and authentication;
• query normalization or rewriting;
• query embedding, if used;
• index lookup and filtering;
• reranking and context selection;
• prompt assembly;
• model queueing and prefill;
• generation of the first token.

Assign a budget and percentile to each stage rather than naming only an average end-to-end latency. Retrieval p99 matters because a slow search delays every downstream step. Model prefill also matters because retrieved context increases the number of input tokens. Returning more chunks can therefore increase both retrieval work and generation latency.

The design should record request shape alongside latency: query rate, candidate counts, selected chunks, input tokens per request, output tokens per request, and model generation rate under the relevant conditions. These quantities describe different resources. A lower retrieval latency does not by itself prove that fewer machines are required, and a concurrency limit does not measure the amount of work performed.

Capacity estimates should come from the expected request mix and tested per-unit throughput at the required latency and resource thresholds. For retrieval, the binding resource might be index CPU, memory, storage access, or a downstream service. For generation, it may be accelerator memory, prefill throughput, decoding throughput, or available model slots. Size and cost cannot be inferred responsibly until that limiting resource and the required headroom are known.

Handle retrieval failures as answer-quality failures

Retrieval can fail without returning a conventional server error. Two cases deserve separate handling.

First, retrieval may return nothing relevant. The index call still succeeds, but the selected chunks do not contain evidence for the question. If the model is allowed to answer from its general behavior, the result can sound confident while being unsupported by the corpus.

The response depends on the product requirement. The system might decline to answer, ask the user to clarify, broaden retrieval, fall back to a different search method, or label the answer as unsupported. A grounding check belongs at this boundary because transport success does not establish evidence quality. Citations can help users inspect the selected sources, but displaying a citation does not by itself prove that the cited text supports the answer.

Second, retrieval may return too much context. More chunks can introduce conflicting versions, tangential passages, or repeated evidence. It also increases prompt size and prefill work. The model may focus on the wrong passage even though the correct one was retrieved somewhere in the window.

Controls for this case include reranking, score or confidence thresholds, per-document limits, deduplication, version filters, and a context-size policy. These controls can be combined. A threshold decides whether a candidate is eligible; reranking changes its order; context selection decides how much of the ordered set enters the prompt.

Operational failures still exist, including an unavailable index shard, an embedding dependency timeout, or an overloaded model replica. Handle those with deadlines, bounded retries where safe, isolation, admission control, and explicit fallback behavior. Keep them distinct from semantically bad retrieval because the user-visible response and the debugging path are different.

Measure retrieval and generation independently

End-to-end evaluation tells you whether the final answer was acceptable. It does not identify why it was good or bad.

Evaluate retrieval against a set of queries with known relevant passages. The central question is whether the right chunk appears in the top k results. Depending on the evaluation design, teams may track recall at k, ranking behavior, performance by query class, and whether permission or version filters excluded or admitted the correct records. No single value is sufficient without the corpus, labels, and workload definition that produced it.

Evaluate generation using the context that the model actually received. Relevant checks may include whether the answer is supported by that context, whether it follows the requested format, whether citations point to supporting passages, and whether the system abstains when the supplied evidence is inadequate. Groundedness and answer usefulness are related but separate: an answer can be supported yet incomplete, or useful-sounding yet unsupported.

The separation creates an actionable failure matrix:

Retrieval	Generation	Likely next investigation
Relevant context missing	Poor answer	Chunking, indexing, query transformation, filtering, or ranking
Relevant context present	Poor answer	Context assembly, conflicting passages, prompt behavior, or model behavior
Relevant context missing	Acceptable-looking answer	Possible ungrounded generation or evaluation gap
Relevant context present	Supported answer	Validate latency, cost boundary, and behavior across query classes

Reliability tests should preserve the same boundaries. Test what happens when an index shard is unavailable, when an update leaves a stale version, when the embedding service times out, and when a model replica is saturated. Connect detection, interruption, fallback, and recovery time to the service’s stated objectives.

A concise RAG system design interview answer

A strong answer can establish the structure:

First, how fresh does the retrieved data need to be? That determines whether ingestion can run in batches or needs a continuous update path, and how I publish or replace index versions.

I would model this as two systems connected by an orchestrator. The retrieval system ingests, chunks, indexes, filters, searches, and reranks documents. The generation system assembles selected context and invokes the model. I would give each stage its own latency budget, capacity model, failure handling, and evaluation.

For the interview’s two-second time-to-first-token target, I would split the budget across query processing, retrieval, reranking, prompt construction, model queueing, and prefill rather than giving retrieval the full target. I would validate capacity from the request mix and tested throughput at the required percentile.

I would handle irrelevant retrieval by abstaining, clarifying, or using a defined fallback, with grounding checks before presenting an answer. I would handle excessive context through ranking, thresholds, deduplication, version filtering, and a context limit.

Finally, I would measure retrieval independently by checking whether relevant chunks appear in the top results, then measure whether generation is supported by the supplied context. That separation lets us attribute a bad answer instead of treating the entire pipeline as one opaque failure.

From there, the interviewer can choose the deep dive: freshness, indexing, retrieval quality, latency, model capacity, permissions, or reliability. The architecture remains understandable because every decision maps to an explicit system boundary. Separating retrieval and generation lets you explain which stage a proposed change affects and how you would evaluate it.

This post is Formation’s pillar for this month’s AI-Era System Design sub-challenge. Continue with the sub-challenge for interview practice, or start with the August 25 prerequisite post on treating the model as a dependency.