"Design an AI-powered support assistant" can look like a machine learning question. In a common product or platform engineering interview, a useful framing is to treat the model as an unusual dependency: an API that may be slow relative to the rest of the architecture, is often metered by token, is non-deterministic, and can return a fluent, wrong answer behind a 200 status code. The design problem is how the system behaves around that dependency.

This framing does not remove model-specific concerns. Model selection, prompting, context behavior, evaluation, and safety may all belong in a product-engineering discussion. If the role involves ML research or applied science, the interview may also probe training and modeling decisions directly. The scope here is a product or platform role in which the model is a vendor API or internal service rather than something the candidate trains.

Set the product boundary first

Consider a customer-support assistant that drafts responses to incoming tickets using retrieval over company documentation. Before choosing a model or vector store, define whether the assistant drafts for a human agent or answers customers directly, what data it may access, and whether its output can trigger refunds or account changes. Those choices determine the latency target, verification requirements, threat boundary, and acceptable fallback.

The dependency has at least three distinct failure classes: slow responses, explicit failures, and semantically wrong responses that look successful. Giving each class its own budget and handling path demonstrates the relevant system-design judgment without assuming that every interview uses the same scoring rubric.

Latency: two numbers, not one

A generative model call has two latency figures worth tracking separately: time to first token and time to last token. Total generation time depends partly on output length. Streaming does not reduce model generation time by itself; it lets a user read output before generation finishes.

For an assistant that drafts replies for an agent, waiting for the complete response may be acceptable. For direct customer chat, streaming may help when the text is low consequence and can be displayed before full validation. It is not an automatic choice. Output that requires whole-response checks should be buffered, while output whose claims can be checked incrementally may be released only in validated segments. Consequential content, such as a refund decision, should wait for a complete validated response and must not authorize the underlying action.

A timeout also needs defined behavior in the deployed system. Specify what the user sees, whether partial output is retained, which failures are retryable, and how retries fit within the request's latency and cost budgets. A 30-second timeout is only a configured ceiling.

Cost: budget input and output separately

Provider plans and prices change, so use current prices for the proposed deployment rather than relying on a memorized ratio. The request cost may depend on input tokens, output tokens, model choice, and supporting services such as retrieval or reranking. In the support assistant, adding more retrieved documentation increases the prompt size on every uncached model call.

Caching. Exact matching on natural-language input may miss differently worded versions of the same question. Alternatives include normalizing a bounded set of frequent queries, caching retrieval results, or using embedding similarity. A semantic cache adds a correctness and isolation risk: similar questions can require different answers, and a key that omits tenant, corpus, permissions, model, or prompt version can return content from the wrong context.

Model routing. A classification task may be suitable for a cheaper model while customer-facing drafting uses a more capable one. The router's cost and error rate belong in the design. A misroute can affect answer quality as well as savings.

A completion limit is a risk envelope: it bounds one response's maximum length and model spend but does not predict typical usage. Provider request or token limits are admission ceilings, not tested safe capacity. Safe operating capacity must come from load tests against the workload mix, latency targets, and limiting resources.

A compact hypothetical budget shows the arithmetic. Assume an end-to-end p95 target of five seconds under the tested workload, with provisional stage budgets of 500 milliseconds for retrieval and 4.5 seconds for generation, including a 1.5-second time-to-first-token target. Stage percentiles do not add mechanically, so the end-to-end distribution still needs measurement. Assume each model call uses 3,000 input tokens and 500 output tokens, with planning prices of $2 per million input tokens and $8 per million output tokens. That is $0.006 plus $0.004, or $0.01 per call. At 100,000 user requests per day, the pre-cache model estimate is $1,000 per day. If a tenant-safe cache avoids 20% of calls, and half of the remaining 80,000 calls route to a model assumed to cost half as much, estimated model spend becomes $400 + $200, or $600 per day. Allowing at most one retry per uncached request puts the model-spend ceiling at $1,200 per day if every eligible call retries. This excludes retrieval, storage, transfer, and other services. Also, 100,000 requests per day averages about 1.16 requests per second; it says nothing about peak capacity without an arrival model and load test. Substitute measured latency, token distributions, retry rates, traffic shape, and current prices before using the result.

Failure: three behaviors, handled three ways

Slow means elevated latency on a call that may still succeed. Timeouts, streaming where appropriate, and queueing or shedding non-interactive work address this case.

Failed means a visible error or timeout. Selective retries, circuit breaking, and a fallback path apply.

Semantically wrong means a fluent, well-formed, incorrect answer inside a successful response. Search, recommendation, and earlier ML systems can also return semantic errors, but generative models can produce errors across a broad output and present them in language that is difficult to validate. Transport status and latency do not reveal this class of failure. Detection requires separate controls such as source checks, deterministic validation, evaluation, user signals, or human review.

Handling semantic errors starts with a correctness boundary. Suppose a draft includes a refund amount. A schema can verify that an amount has the expected type and format; it cannot establish that the amount is correct. A citation can show which document informed a claim, but it does not prove that the document is current or authoritative. An automated claim checker may itself use a non-deterministic model. Consequential facts should therefore be recomputed or checked against an authoritative deterministic service, and consequential decisions should require explicit application authorization or human approval. Lower-consequence output, such as an entertainment recommendation, may justify lighter controls, but it can still create product or user harm.

Fallback, degradation, and escalation can be combined. Under load, the system might route an eligible request to a smaller model. During a provider failure, it might return approved macros or search results. High-risk categories can go to a human regardless of provider health. Each path should name the failure it handles and preserve the same authorization rules.

RAG is two systems with two budgets

Retrieval-augmented generation is a pipeline: retrieval followed by generation. The retrieval stage may embed the query, search a vector store, and rerank results. Its latency share is deployment-specific, and it can be slow, stale, or unavailable independently of the model. If the documentation index lags behind a refund-policy change, the model may produce an answer grounded in obsolete material. Tracing that chain distinguishes retrieval freshness from generation behavior.

Retrieving more chunks can improve recall, but it also increases input tokens and may add irrelevant context. Reranking may improve precision at the cost of latency and compute. When retrieval returns no relevant authorized source, the safer behavior for this assistant is to say that evidence is unavailable or use the defined fallback rather than ask the model to fill the gap.

The pipeline also creates a threat boundary. Tickets and retrieved documents are untrusted content and may contain prompt-injection instructions. The application should treat them as data rather than instructions that can override system policy. Retrieval must enforce tenant, corpus, and document authorization before content reaches the model, and semantic-cache keys must preserve those boundaries. Sensitive data should be sent, stored, and logged only under the organization's approved tool, retention, and access policies. Any downstream action needs its own authorization based on the authenticated user and authoritative application state; model output is never an authorization grant.

Failure handling around the provider

Retries. A timed-out model call may continue on the provider side, so a retry can add cost and return a different answer. Retry only defined transient failures, use backoff with jitter, and cap the attempts and spend associated with one user request. Idempotency is still required for surrounding writes or actions even if the model call itself has no side effect.

Circuit breaking. A breaker can stop calls when measured error rate or latency crosses a chosen threshold and route requests to the fallback. Recovery probes should test the same path needed for normal service.

Designed degradation. The non-AI path must be maintained and exercised under representative conditions. Otherwise its configuration, permissions, or content can drift and fail during the outage that requires it.

Output-quality monitoring. Availability metrics cannot reveal every quality regression. Human review samples, task-specific evaluation sets, user feedback, and operational signals such as ticket reopen rates can provide competing evidence. Any metric is a proxy, so material prompt, retrieval, or model changes should be assessed against several signals rather than one score.

Define measurable latency and cost budgets, explicit failure behavior, the data and authorization threat boundary, and the verification required before output reaches a user or triggers an action. Model, prompt, cache, and RAG choices should follow from those constraints.