LLM inference has two phases with different resource profiles. Prefill processes prompt tokens together; autoregressive decode advances each output sequence one token at a time. Prefill is often compute-bound, while decode at small batch sizes is often memory-bandwidth-bound. That distinction helps explain why batching improves throughput and why a long prompt can disrupt an unrelated response already being generated.

Use one deployment as a working example: a dense, decoder-only transformer with full-context attention, running on one GPU under one scheduler. Model weights and per-sequence key-value (KV) state fit in GPU memory. The workload mixes short chats with long document prompts. Assume ordinary autoregressive generation, without prefix-cache hits, offloading, or speculative decoding.

Prefill vs decode in LLM inference

The prompt is already known, so prefill can process its positions in parallel within each model layer while preserving causal attention. It computes keys and values for the prompt and the logits used to select the first output token. Parallel prompt processing does not mean generating the answer in parallel.

Here is the handoff from prompt processing to the generation loop:

flowchart LR
    P["Prompt tokens"]
    subgraph PREFILL["1. Prefill"]
        A["Process prompt tokens together"] --> B["Create prompt KV cache"]
        B --> C["Select first output token"]
    end
    subgraph DECODE["2. Decode"]
        D["Process latest token with cached KV"] --> E["Append its KV; select next token"]
        E -->|If continuing| D
    end
    P --> A
    C --> D
    B -->|KV state| D

During decode, each iteration consumes the latest token, attends over previous context through the KV cache, and adds that token's keys and values. Reusing cached state avoids recomputing those representations, but attention still reads the retained context. The next iteration depends on the token just selected.

Why the phases have different bottlenecks

Compute-bound means execution is primarily limited by available arithmetic throughput. In prefill, applying the model to many prompt positions creates large matrix operations with substantial reuse of loaded weights. There can be enough arithmetic per byte moved, called arithmetic intensity, to keep the GPU's compute units busy.

Decode offers only one new position per sequence per iteration. There is still parallel work within that position, but little work to amortize loading the model's weights when the batch is small. Reading weights and attention's KV state can dominate execution time. The bottleneck is memory bandwidth, not necessarily insufficient memory capacity. A growing KV cache also increases attention's work and memory traffic.

These labels describe workload-dependent bottlenecks, not fixed properties of every kernel. Short prefills may not saturate compute; sufficiently large decode batches can become compute-intensive. Hardware, model shape, context length, and batch composition all matter. Batching can help short prefills too. The useful question is which resource limits this workload at the required latency, rather than whether a phase always belongs in one category.

Continuous batching improves decode throughput

Batching independent sequences lets a worker reuse weights across their decode steps. Each sequence remains sequential, but the GPU can advance multiple sequences during the same model iteration. That raises aggregate throughput without requiring the next token of any individual sequence to be known early.

A fixed batch keeps the same group together until it finishes. In the mixed workload, a short chat may finish while a document response still has tokens to generate. Continuous batching updates membership between iterations: completed sequences leave, and eligible work can join without waiting for the longest response to finish. Orca's iteration-level scheduling provides a concrete research example of this approach.

Continuous does not mean unlimited. Each admitted sequence needs KV storage, and larger batches can lengthen iterations. For this worker, test the chat and document classes together, retaining their different prompt and output lengths. Select concurrency against both memory use and latency targets; a configured request ceiling alone does not establish safe operating capacity.

How long prompts interrupt concurrent decode

Suppose the worker is decoding short-chat responses when a long document prompt becomes eligible. If the scheduler runs that prefill as one large iteration, the chats cannot advance until the iteration completes. Even a mixed iteration containing both prefill and decode work can delay the next decode step when the prefill portion is large. This is the prefill-decode interference studied by Sarathi-Serve.

The affected boundary is the shared scheduler and GPU, not every machine or tenant behind an endpoint. This resembles a noisy-neighbour problem, but the mechanism is specific: a long iteration occupies resources needed for repeated, latency-sensitive decode steps. It does not require requests to share their KV contents.

For this example, define the tail metric as p99 inter-token delay: the 99th percentile of intervals, in seconds, between consecutive token emissions across the short-chat streams during a stated observation window. Keep that distinct from p99 time to first token and p99 request completion time. A prefill stall can worsen this distribution; whether it moves p99 depends on its frequency and the sampled population.

Chunking and disaggregation control different dimensions

A practical baseline to test for this mixed worker is continuous batching combined with chunked prefill. Batching supplies parallel work across requests; chunking controls how much prompt processing enters an iteration. They address different dimensions and can be used together.

Chunked prefill divides the document prompt into smaller pieces, giving ongoing decodes opportunities to advance between pieces or alongside them in mixed batches. Smaller chunks can reduce decode stalls, but may delay the document request's first token and reduce prefill efficiency. Larger chunks favor prompt processing while allowing longer decode interruptions. Choose the chunk size against both latency objectives, then test the actual mix rather than only isolated requests. Chunking also does not eliminate the eventual KV state required for the full prompt.

Disaggregated serving changes placement: prefill and decode run on disjoint accelerator pools, potentially on different hosts. DistServe demonstrates this design. It removes direct prefill-decode contention on the same GPU and allows phase-specific allocation, but adds a state handoff. The decode worker needs the request's KV cache. Merely putting two workers in separate processes on one GPU does not provide that accelerator isolation.

For the document workload, a large prompt now produces a larger KV transfer rather than a long prefill iteration on the chat decode GPU. Its later decode can still compete with chats for bandwidth and KV capacity. Evaluate the transfer and each pool's queueing against first-token and streaming targets. Separate pools still need batching, and prefill workers can still use chunking. If the colocated baseline meets both targets, splitting phases is not automatically warranted; it introduces more coordination without an established benefit for that workload.

What changes when using a hosted API

A hosted API hides these scheduling choices, but it does not make the phases irrelevant. A request can wait a long time for its first token and then stream quickly, or start quickly and generate slowly. Client-observed delay also includes networking and queueing, so the two-phase model is a diagnostic hypothesis, not proof that a provider is experiencing prefill interference.

For the same chat-and-document application, record prompt length, generated length, time to first token, and streaming timing separately. Compare workload classes at controlled application concurrency. Removing irrelevant document text is a reasonable experiment for reducing prompt work; shortening requested answers targets the number of sequential generation steps instead. Preserve answer quality as a test condition. A histogram with two latency peaks does not, by itself, establish that prefill and decode caused them.

Keep billing separate from hardware utilization. When an API prices input and output tokens separately, estimate token charges using reported usage in each billing category and its applicable rate. Account for cached-input categories if offered, rather than treating every prompt token identically. Those charges cover the token-billing boundary, not necessarily tools, retrieval, storage, or the rest of the application. Longer prompts raise the input-token component at an unchanged rate and billing category; they do not necessarily dominate the total bill.

The resource distinction explains why input and output are different work. It does not establish a provider's pricing rationale. Do not turn an estimate of GPU efficiency into a projected API discount; use the billing contract.

Time to first token vs tokens per second

For an interview, make the measurement boundary explicit before proposing an optimization. Time to first token measures the wait before output starts; tokens per second measures the rate afterward and needs a scope: one stream or the entire service.

• Client time to first token (TTFT), in seconds: elapsed time from sending the request to receiving its first content token. This includes transport, server queueing, prompt processing, and any other work before that token reaches the client. It is not a pure prefill timer.
• Per-stream output rate, in tokens/second: for an output of N > 1 tokens, use (N - 1) / (last_token_time - first_token_time), with timestamps in seconds. This excludes first-token waiting and averages the rest of that stream.
• Aggregate output throughput, in tokens/second: total output tokens across requests divided by a specified measurement interval. More aggregate throughput does not establish faster individual streams.

An average stream rate can also hide pauses, which is why the inter-token tail matters. At an API boundary, streaming messages may contain several tokens. Record chunk sizes and timestamps, and label chunk-normalized estimates honestly; packet arrival times do not reveal exact internal token-emission times.

To practice the distinction, use one endpoint and compare the two workload classes without changing everything at once. If first-token delay is the problem, investigate where time is spent before streaming starts. If an established stream pauses, examine its token intervals. Choose a serving change only after identifying which measurement must improve and what tradeoff the application can accept.