In a full-attention, GPU-resident baseline without sharing, every prompt or generated token adds state to the KV cache. For Llama 3.1 8B, the per-token cost follows from the model configuration: 32 layers, 8 key-value heads, a head dimension of 128 (4,096 hidden dimensions divided by 32 query heads), and two tensors (K and V) per layer. In FP16 that is 2 × 32 × 8 × 128 × 2 bytes, or 131,072 bytes per token, exactly 128 KiB. Two-byte weights for roughly 8 billion parameters take about 16 GB.

Use decimal GB and k = 1,000 tokens throughout. For this capacity estimate, set vLLM v0.8.5's gpu_memory_utilization to 0.9 on one 80 GB H100: 72 GB for the engine. Subtract weights and a 4 GB allowance for activations, workspaces, and CUDA graphs, leaving 52 GB for cache. That is roughly 396,729 token-equivalents, rounded to 400,000 before block-allocation constraints. Execution overhead depends on workload and settings; the allowance needs profiling.

Divide that pool by request length and the memory-only concurrency ceiling falls out: about 49 resident requests at 8k tokens, 6 at 64k, 3 at 128k, before output-growth headroom. Nothing else about the deployment changed. Context length is a capacity parameter, and KV cache arithmetic is central to long-context capacity planning.

Why the cache exists

A full-attention decoder generates one token per step, attending over every previous position. Attention needs the key and value projections for those positions at every layer. Without a cache, step t would repeat K/V projections for prior tokens. The KV cache stores them when first computed, so each decode step projects only the newest token and reads the rest. This avoids repeated projections, not the growing attention reads: with a fixed initial prompt, total full-attention decode work over an n-token generation still grows quadratically in n. For this GPU-resident baseline, that stored state occupies the accelerator's high-bandwidth memory (HBM).

It grows, and it is held per request

It grows linearly with sequence length: prompt tokens during prefill, then one more token's worth on every decode step. Active decoding needs that state, but exact-prefix sharing can reduce physical duplication. Preemption, eviction, or offload can change GPU residency, requiring restoration or recomputation before decoding resumes.

This is where token throughput and concurrent residency come apart. Ten requests at 100k tokens and a thousand at 1k initially hold the same million tokens of cache without sharing. Their per-request allocations differ by 100x. Neither workload necessarily holds memory longer. Request-size distribution, output growth, and scheduling determine subsequent occupancy. Instantaneous resident bytes impose the hard capacity limit. Token-time integrates occupancy for utilization analysis; it does not replace that limit. Equal tokens per second does not establish equal memory use.

The concurrency ceiling

After weights and workload-dependent execution overhead, the remaining HBM budget sets the unshared-cache envelope:

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element
tokens_in_flight ≈ (hbm × utilization − weights − execution_overhead) / kv_bytes_per_token
resident_requests ≈ tokens_in_flight / mean_resident_length

The boundary is the replica and its per-GPU allocations, not the host: an 8-GPU box running eight TP=1 replicas has eight independent pools, each with its own weights copy. One TP=8 replica shards weights and cache across the GPUs, subject to per-GPU limits. mean_resident_length is the mean length of requests resident at the same moment; requests with longer service durations are overrepresented.

The connection-pool parallel is useful here, up to a point. A database pool is a finite set of slots. In a one-connection-per-request design, a pool slot is held while the request uses the connection. Exhaustion can produce waiting, timeouts, or rejection. The difference is that a pool slot is fixed-size and a KV allocation is not. A request admitted at 2k tokens can be 40k tokens deep by completion, so capacity in requests shrinks while requests run. A single connection does not grow to consume six slots mid-query. That growth is why admission based only on the prompt's current size can overcommit the pool.

What happens when you run out

Depending on the engine and serving policy, the scheduler may queue new requests or preempt active work for recomputation or supported offload. The serving layer may instead reject or cancel requests, and clients can time out. vLLM v0.8.5's V1 scheduler uses recomputation after preemption rather than swap, potentially repeating prefill work before decoding resumes.

Queued requests can see higher time-to-first-token; preempted requests can stall and later resume decoding. GPU utilization can stay high during recomputation, and aggregate throughput can hide long waits if shorter requests keep completing. When short requests dominate, median latency can also stay low. Watch p99 time-to-first-token and inter-token latency, split by prompt length, alongside cache occupancy and the engine's preemption counter.

The ceiling also depends on latency, not only memory. Decode can be memory-bandwidth-bound: each step reads weights and the relevant resident K and V. Growing context can push per-token latency past its target even though blocks remain. In vLLM v0.8.5, max_num_seqs limits sequences processed per iteration; it is a scheduling cap, not a capacity estimate. That makes the 400,000-token figure above a memory envelope rather than a promise. The safe operating concurrency is whatever a load test at your real request mix and length distribution shows meets your time-to-first-token and per-token latency targets.

Naive allocation can waste most of it

Reserving a contiguous cache region per request sized to the maximum sequence length accounts for the unknown final length. When requests finish far short of that maximum, much of the reservation goes untouched. The PagedAttention research reported 60–100% KV-memory waste from over-reservation and internal/external fragmentation in the systems and workloads studied, reduced to under 4% with PagedAttention.

PagedAttention allocates fixed-size blocks on demand, maps logical positions to physical blocks through a table, and reclaims unneeded blocks. Within-block tail waste is at most one partially filled block per sequence, not a bound on every source of runtime overhead. Prefix-cache blocks may be retained after completion for reuse.

For capacity planning, assuming every request holds max_model_len tokens can understate achievable concurrency when requests are shorter. Ignoring the length distribution can overstate it.

One replica under a mixed workload

Take the Llama 3.1 8B replica from the opening, roughly 400,000 tokens of cache, serving two request classes: chat turns that resolve around 8k tokens total, and document-analysis requests that run to about 64k. The classes are separate because their resident footprints differ by 8x. At those lengths without sharing, four resident 64k requests hold 256k tokens, leaving room for about 17 chat requests. Six resident 64k requests leave room for only one chat request within the unrounded 52 GB budget. Twelve do not fit concurrently; excess requests must wait or face the serving policy's overload response.

The available levers are mostly composable, and each moves a different term:

• FP8 KV cache halves bytes_per_element, doubling the envelope to roughly 800,000 tokens. It halves KV payload bytes read per step, not all decode traffic. Quality impact is model-dependent; evaluate it with the supported kernels.
• Grouped-query attention is already in this model (32 query heads, 8 KV heads), which is why the per-token figure is 128 KiB rather than 512 KiB. This is a model-architecture decision rather than a serving knob.
• Prefix caching shares blocks for identical token prefixes under compatible model/configuration conditions. If the 64k requests share a reusable 40k-token prefix, each additional one needs about 24k new tokens of cache, subject to block granularity.
• Chunked prefill breaks a 64k prefill into pieces interleaved with decode steps to reduce interference with active streams. It does not reduce the final per-request KV footprint.
• Adding replicas multiplies independent pools but pays about 16 GB of weights per copy. TP=2 instead shards one logical weights copy across two GPUs. With even weight/cache sharding and 4 GB overhead per GPU, cache totals 2 × 72 − 16 − 2 × 4 = 120 GB, about 2.31 times the TP=1 pool. That memory gain does not guarantee proportional throughput.
• max_num_seqs caps scheduled sequences. Set it from the load test, not from memory arithmetic.

If 64k requests turn out to be 5 percent of traffic but 40 percent of resident tokens, isolating them onto a replica with a lower scheduling cap can protect chat latency. Separate pools, however, can strand spare capacity.

The design consequence

Every token in a prompt is a claim on this pool while resident. That is one reason long prompts are expensive, and it makes several product decisions capacity decisions. Stuffing the full conversation history into every turn, or returning thirty retrieved chunks because the window can hold them, spends cache even when those tokens do not improve the answer. A context budget belongs in the design document next to the latency target: how many prompt and output tokens a request class is allowed, and what gets cut when it exceeds that.

For retrieval, this is the operational argument for precision. Returning the right three chunks rather than a plausible thirty is more than a quality question. At 128 KiB per token, a 20k-token difference in retrieved context is 2,621,440,000 bytes, about 2.62 GB of unshared KV state per request. Multiply by the concurrency you want and compare with the 52 GB you have.

The context length on the model card is the maximum one request can attend over. It says nothing about how many requests can be doing so at once. Treat it as a per-request limit you budget against, size concurrency from a load test at your actual length distribution, and watch cache occupancy, preemptions, and per-length tail latency alongside throughput to detect pressure on the pool.