Interactive streamed LLM responses need separate targets for first-token responsiveness and sustained delivery. At the client boundary, time to first token (TTFT) runs from request start to the first generated output token, not headers or heartbeats. For N > 1 output tokens, request-average time per output token (TPOT) is the elapsed time between the first and last tokens divided by N-1. End-to-end time through the last token is TTFT plus (N-1) times TPOT. Collapsing that into one number hides which phase breached and where to investigate.

Why one number cannot describe a streamed response

Take two responses that both complete in three seconds. In the first, the first token lands at 300ms and the rest stream steadily behind it. In the second, the user stares at an empty box for 2.8 seconds and then receives the whole answer at once. Same total, opposite experience. The first feels responsive; the second feels broken, even though a request-response latency dashboard cannot tell them apart.

The reverse failure exists too. A first token in 200ms followed by a stream that stalls every few tokens reads as a struggling system, and an end-to-end percentile again shows nothing unusual if the total lands within budget.

An end-to-end number alone cannot distinguish these failures. That is the argument for separate SLOs, and it is also the argument for keeping the end-to-end target rather than replacing it. A TTFT target and a TPOT target compose with an end-to-end budget; they do not substitute for one.

Prefill and decode stress resources differently

Prefill, the forward pass over the prompt before the first output token is sampled, can dominate TTFT. Long uncached prompts can be compute-bound, depending on model and hardware. A retrieval-augmented request with 6,000 tokens of context can require much more prefill work than a 200-token question, though reusable cached prefixes reduce that work. Client-observed TTFT also includes queueing and other service and delivery overhead.

Decode is a major contributor to TPOT: typically one forward pass per generated token, each producing a single token per sequence. At small batch sizes, decode can be memory-bandwidth-bound because each step reads model weights and KV-cache data for relatively little arithmetic. Context length, batch composition, cache reuse, and saturation can change the bottleneck. Inter-token latency (ITL) is each successive token gap at the same client boundary. Request-average TPOT can hide ITL stalls; percentiles of request averages and individual gaps describe different populations.

Different resources mean different fixes. Long TTFT can point to prompt length, prefix caching, prefill scheduling, or admission control. High TPOT can point to batch size, KV-cache memory pressure, or a machine serving more concurrent sequences than it can decode at target pace.

Batching trades one number against the other

Continuous batching admits new requests into the running batch between decode steps instead of waiting for a batch to drain. It can improve utilization and aggregate token throughput, but per-request pace and queueing depend on workload, scheduler, batch size, and the saturated resource. When prefill shares constrained resources with ongoing decodes, it can delay them, producing ITL spikes.

Two serving mechanisms can manage this interference.

The first is chunked prefill, described in the Sarathi-Serve work and documented for vLLM v0.8.2 with a token budget per scheduler step. A long prompt is split into chunks processed across several iterations alongside decodes, limiting prefill interference rather than guaranteeing latency. Smaller chunks can protect token pace at the cost of longer TTFT; larger chunks favor prefill progress.

The second is prefill/decode disaggregation, described in DistServe, which runs the two phases on separate GPU pools so each can be sized against its own targets. It reduces direct phase interference but adds KV-cache transfer and deployment complexity; contention within pools remains.

Neither is free, and neither is universally right. A batching policy must balance latency and throughput against the product's objectives.

Product shape sets the targets

Three workload classes make the point; their objectives need not match.

Interactive chat. A user is watching. TTFT governs the initial wait, while TPOT and stalls affect readability. Smaller effective batches may help under contention; prefix caching helps when prompts share reusable prefixes. Admission control can protect existing streams, but queueing delays new requests and rejection denies service.

Unattended summarization. Nobody is watching. A nightly job summarizing ten thousand documents can prioritize completion time and cost without product SLOs for TTFT or TPOT. Both metrics remain useful for diagnosis. Select batch sizes by tested useful throughput for the input/output mix, available resources, and completion deadline. Memory is a constraint, not proof of the cost-optimal batch size; long inputs may make prefill the bottleneck.

An eight-hop agent. An agent making eight sequential model calls, each consuming the previous output, is a different shape again. No user reads the intermediate hops, so per-token pace inside a hop matters only insofar as it adds to the hop's total time. The agent pays TTFT eight times; prefill work depends on retained context and cache reuse. Allocate the trajectory budget using observed hop-duration distributions from representative trajectories, not an equal division by eight. Reserve time for non-model work and any final user-facing stream, which warrants its own TTFT and TPOT targets.

Shared pools can work when objectives are met. If chat and summarization contend, separate pools or scheduler priority classes can protect chat, at the cost of less flexible capacity or longer batch-job waits.

Here on Formation's blog, the sub-challenge is a place to practice that workload-specific choice.

Percentile targets and what happens when you miss them

For interactive streams, state TTFT and request-average TPOT targets as percentiles, tied to the product, alongside request deadlines. One possible reporting policy evaluates each interactive workload class over rolling 30-day windows: TTFT across requests and request-average TPOT for outputs with N > 1. Include failed and timed-out requests in breach accounting rather than reporting only successful completions. Any ITL percentile uses individual gaps rather than request averages.

I will not propose thresholds here because the right values depend on model size, hardware, prompt distribution, and what the product team has tested with users. An objective can precede testing. Load tests establish throughput and concurrency at target only for the tested traffic mix, model, hardware, scheduler, and test duration. That demonstrated capacity should inform admission limits.

Configured limits are not capacity. A server's max_num_seqs or token budget per step describes the ceiling the scheduler will attempt, not the load at which TPOT stays on target. Observed concurrency describes what traffic did last week. The admission limit is what you refuse above. These concepts can have coincident values, but that requires justification.

Timeouts are where streaming stops resembling request-response. A generated-first-token deadline tracks content, unlike a transport first-byte check that headers or heartbeats can satisfy. An inter-token stall timeout limits mid-stream gaps. An overall completion deadline is also needed: a stream can avoid stalls yet continue too long. These request-level deadlines are distinct from population-level SLOs. Mid-stream failure is the hard case because partial output already reached the user.

That forces a product decision, not just an engineering one. Options include showing the partial text with an explicit truncation marker, discarding it and retrying the whole request, or resuming generation from the delivered prefix on a different replica if the serving layer supports it. Truncated output may be misleading. Retries can repeat prefill and generation work; their cost depends on cache reuse, retry count, context, and how much generation already completed. Prefix resumption requires agreement on the delivered prefix. None of these choices undo what the user has already seen.

Cost is the third axis, attached to every latency decision

Each lever above has a price. For decode-bound workloads, generated tokens per GPU-second while the TPOT target holds measures decode efficiency. For prefill-bound long-prompt and summarization workloads, also measure tokenized input processed per GPU-second, accounting for cache reuse. Neither phase-specific ratio is total serving cost. Reducing batch size can raise per-token compute cost if it reduces useful throughput on the binding resource; that loss needs measurement.

Disaggregating prefill and decode buys isolation with separately provisioned pools and transfer overhead. Prefix caching spends GPU memory to save prefill compute, and the saving depends on how much prompt is actually shared. Reserving headroom for burst-time p99 targets can leave paid capacity idle at lower load.

None of these are reasons to skip the improvement. They are the reason the SLO has to be stated first, because the SLO is what tells you which cost is worth paying.

The Aug 4 companion post, More Servers Is Not a Scaling Plan, makes the broader capacity argument. Chat may justify extra capacity to hold both streaming targets at the required throughput. An unattended pipeline can accept slower individual streams if that meets its completion deadline at lower measured cost.