Two things called caching

Prompt prefix caching reuses the model's already-computed attention state for a run of tokens identical to the start of a previous request. It lives inside the inference server or behind the provider's API. Semantic response caching stores a finished answer in an application-layer store and returns it when a new question embeds close enough to an old one. It lives in your code, and it can change the answer, by design.

The first uses deterministic matching to avoid repeated computation. The second is a correctness decision dressed up as a performance optimization. They can be combined, but their savings require different checks.

Why the obvious cache key can fail

The standard caching reflex is to hash the request and look it up. On free-form natural-language input, full-request hits depend on traffic. Add per-user context, a timestamp, or a retrieved document that differs by one line, and exact-match on the full request may miss even when much of the input is unchanged. Paraphrases can further reduce reuse, depending on how requests arrive.

A low full-request hit rate does not rule out exact matching. Before matching on meaning, examine the part of the request that can remain identical: the front.

Prefix caching: exact-match on the part that stays stable

Take a support agent as the running example. Every request it sends to the model is assembled from, in order, a system prompt, tool schemas, few-shot examples, retrieved documents, and the user's turn. Assume the first three remain identical within a deployment.

An autoregressive transformer handles a request in two phases: prefill, where it computes attention key/value state for input tokens, and decode, where it generates output one token at a time. Prefix caching keeps the KV state for an already-prefilled prefix and, when a new request arrives whose leading tokens are identical, skips recomputing them. The key is the exact token sequence. Reuse requires the same model, compatible inference configuration, and a permitted cache scope. There is no similarity threshold to tune. Check the serving system's eligibility rules, retention behavior, and any cached-input or cache-write prices.

The design constraint is worth saying out loud in an interview: put stable content first where prompt semantics and access controls permit it. For the support agent:

• The system prompt, tool schemas, and few-shot examples go first, in a fixed order. Reordering prevents reuse beyond the first differing token; an eligible unchanged leading prefix may still be reused.
• Move timestamps, user names, and other variable content after the stable block where doing so preserves meaning. Do not omit authorization context merely to improve hits.
• Retrieved documents go after the stable block. They may be cacheable across turns or authorized users when the entire leading token sequence matches within the same permitted cache scope. Token equality does not grant access.
• The user turn goes last.

Two operational details follow. First, TTL, eviction, and refresh behavior depend on the provider or deployment; traffic does not necessarily extend an entry's lifetime. An infrequently reused prefix may be evicted before the next request, so prompt structure alone cannot establish the hit rate. Second, if KV caches are replica-local, prefix-aware routing can improve reuse; evenly balanced traffic can still hit warm matching caches.

With compatible state and correct isolation, prefix caching avoids a semantic answer-substitution decision. It still requires access controls, and a hit does not promise identical sampled output.

Semantic caching: what you are actually agreeing to

Semantic response caching operates on the other end of the request. You embed the incoming question, search a vector store for previously answered questions, and if one is close enough, return its stored answer without calling the model.

"Close enough" is doing all the work. Consider two questions the support agent might see:

• "What is our refund policy for EU customers?"
• "What is our refund policy?"

These questions could be close in embedding space while requiring different answers. The EU version may need jurisdiction-specific terms the general version does not mention. Where those terms matter, serving the general answer to an EU customer substitutes the wrong answer, even if the stored answer is current.

The similarity threshold is a classifier decision boundary, not a known wrong-answer rate. The cache is making a binary decision: "this new question can use that stored answer." Tightening the threshold can reduce hits and false matches, but the resulting tradeoff needs evaluation on representative labeled traffic, including near-matches that require different answers. Review it as traffic and source truth change. Evaluate that error rate against the service's correctness requirements, rather than against a cost target alone.

There is a deterministic middle option: exact-match caching of the full response. Constrained UI prompts or button actions can produce repeatable traffic. Use the exact request string or a lossless representation; lossy normalization can collapse meaningfully different requests. Keys and reuse eligibility must also account for tenant and authorization scope, conversation and retrieved context, and relevant model, prompt, tool, and data versions. Expire or invalidate entries when truth or permissions change. Exact text matching avoids similarity matching; it does not by itself make a stored answer safe to reuse. If the traffic shape allows it, propose it before proposing similarity.

When semantic caching is defensible

For this support agent, propose semantic caching only for a slice of traffic that meets all four conditions.

1. High-volume, low-stakes, low-variance queries. Many requests whose answers do not depend on who is asking or on details the embedding might flatten, and where a wrong answer is cheap. Store-hours FAQs may qualify when location is explicit and the scope and freshness safeguards below hold.
2. A tight threshold, measured. Evaluate on labeled traffic representative of the eligible slice, including misleading near-matches. Review both served-answer errors and hits rather than treating the threshold as an accuracy guarantee.
3. A namespace scoped per tenant. Entries from one customer's tenant are never candidates for another's. Authorization still applies within a tenant; a shared namespace must not expose restricted answers.
4. A TTL tied to how fast the underlying truth changes. Set retention to tolerated staleness, not just the publication schedule. Changes to truth or permissions may require immediate invalidation or bypass; a TTL alone cannot guarantee freshness.

Keep this agent's policy and account questions outside that slice. Prefix caching can still apply to eligible shared prompts for those requests.

Model routing is a separate lever

Prefix caching reduces repeated input processing; response caching can avoid a generation call. Routing changes which model you call. They are independent controls that can be combined when each is justified.

The design is easy to describe: evaluate which requests a smaller, cheaper model can handle adequately, then classify and route incoming requests accordingly. Reserve the large model for the rest. The interesting questions are the ones the diagram hides.

First, what happens when the classifier is wrong? Sending a hard question to the small model can degrade the answer. Accepting it carries a quality risk; escalation requires a way to detect inadequate answers and pays for both generation calls. Biasing routing toward the large model may reduce that risk while giving back savings. Sending an easy question to the large model adds cost and may also change latency.

Second, who pays for the classification? If the classifier is itself a model call, you have added latency and cost to each classified request, including ones that would have gone to the small model anyway. Heuristics and dedicated classifiers also have evaluation and operating costs. A classification prompt with a stable system prompt is itself a candidate for prefix caching.

The cost model to bring into the room

For a base cost sketch, assume every request attempts semantic lookup and a miss triggers one generation call. This is lookup plus generation cost, not a production total:

input_cost  = P_shared * (hit_rate * cached_rate + (1 - hit_rate) * input_rate) + P_variable * input_rate
output_cost = O * output_rate
per_request = lookup_cost + (1 - semantic_hit_rate) * (input_cost + output_cost)

P_shared is shared-prefix tokens, P_variable is suffix tokens (retrieved documents plus user turn), and O is output tokens. hit_rate is the measured prefix-hit fraction of modeled generation calls; semantic_hit_rate is the served-hit fraction among lookup attempts. lookup_cost includes embedding and search on every attempt, including misses. input_rate, cached_rate, and output_rate are per-token prices for uncached input, cached input, and output. Use lengths and hit rates from generation misses, not requests served by the response cache.

The formula assumes a fixed shared block fully reused on a hit. For partial-prefix reuse, use measured cached-token quantities instead. If only a subset attempts lookup, weight lookup cost by attempt probability; generation occurs on misses and on requests that bypass lookup. For attempt probability a, the generation fraction becomes 1 - a * semantic_hit_rate in this one-call model.

With routing, replace the single-model generation term with route-weighted costs using each route's token lengths, prices, and prefix reuse. Add classification charges at the stage where classification occurs, weighted by its frequency per incoming request. Add expected extra generation cost for escalations without counting the initial call twice. Include incremental provider cache-write/fill charges, where applicable, at their event frequency, without duplicating input charges already counted. Cache storage and other serving, network, and orchestration infrastructure remain outside this sketch.

Three things this makes visible that "we cache, so it is cheaper" hides:

• Prefix caching discounts only P_shared, and only on hits. It does nothing to output cost. For short prompts with long generations, its savings may be limited; routing savings depend on model prices and acceptable quality.
• The cached fraction of input tokens and the request-level hit rate are different quantities. For equal input lengths and price discounts, a hypothetical 90% hit rate on a prefix that is 30% of the input saves less than a 60% hit rate on a prefix that is 90% of it. Report cached input tokens as a fraction of total input tokens, not of requests.
• Use hit rates measured under representative TTL/eviction, prompt-assembly, traffic, and routing conditions. Steady-load tests can overstate production reuse if entries expire between bursts.

What this looks like in an interview

For interview preparation, be ready to explain what is reused, what makes reuse valid, and which measured quantities support the projected savings. A useful cost estimate identifies both its quality constraints and the charges it leaves out.

From Formation, the publisher of this blog: consider the Fellowship and AI-Era SD mock interview.