Theoretical Bottlenecks for Scaling LLM Inference to Get Higher Token per Second

2026-07-05 · source · source ↗ · wiki page

Freddie Spirit (@freddie_spirit) · Jul 2, 2026

Let's establish the fundamental accounting identity that governs everything else. For any workload on any accelerator, execution time is bounded by max(compute_time, memory_time, communication_time). Compute time is FLOPs / peak_FLOPS; memory time is bytes_moved / bandwidth; communication time is bytes_exchanged / interconnect_bandwidth (plus latency terms). Tokens/sec is just the inverse of per-token time, so every bottleneck we discuss is a claim about which of these three terms dominates and why.

The roofline model formalizes this: define arithmetic intensity I = FLOPs / bytes moved from memory. Your achievable performance is min(peak_FLOPS, I × bandwidth). Every accelerator has a "ridge point" — the intensity at which you transition from bandwidth-bound to compute-bound. For an H100 SXM: ~989 TFLOPS (fp16 dense) / 3.35 TB/s HBM3 ≈ 295 FLOPs/byte. Memorize the shape of this argument, because we will return to it for every single bottleneck: where does the workload sit relative to the ridge point, and what would it take to move it?

Prefill processes the full prompt in one pass — for a prompt of length S, you perform roughly 2·N·S FLOPs (N = parameter count) while moving the weights once: 2N bytes in fp16. Arithmetic intensity is therefore ~S FLOPs/byte. With S = 2048, you're at intensity ~2048, far above the ~295 ridge point — prefill is compute-bound and can saturate tensor cores.

Decode is the opposite pathology. Each generated token requires a full forward pass over one token: ~2N FLOPs, but you must still stream all 2N bytes of weights through the memory hierarchy (weights don't fit in on-chip SRAM — an H100 has ~50 MB of L2 against tens of GB of weights). Intensity ≈ 1 FLOP/byte at batch size 1. That's two-and-a-half orders of magnitude below the ridge point. Decode is catastrophically bandwidth-bound, and this single fact explains most of the inference optimization literature.

Let's derive the batch-1 decode ceiling concretely, since this is the number practitioners most often want. Take a 70B parameter model in fp16: weights = 140 GB. Per decoded token, you must read every weight once (each parameter participates in exactly one multiply-accumulate per token, and nothing is reused across the single token). On an H100 with 3.35 TB/s: t_token ≥ 140 GB / 3.35 TB/s ≈ 41.8 ms, so the ceiling is ~24 tokens/sec per GPU — and that's ignoring KV cache reads, activations, and everything else. Note what this ceiling does not depend on: compute throughput. You could double the TFLOPS and gain exactly nothing at batch 1. Check this against reality: real systems achieve maybe 60–80% of this bound because of KV cache traffic, kernel launch overheads, non-fused elementwise ops, and imperfect bandwidth utilization on small matrix-vector products. Here's your first back-of-envelope exercise: an 8B model in fp16 on the same hardware — what's the ceiling? (16 GB / 3.35 TB/s → ~210 tok/s. Real systems hit ~150–180.)

The KV cache introduces a second bandwidth term that grows with context length, and a capacity constraint that caps batch size. Per token, per layer, you store K and V vectors: 2 × n_kv_heads × d_head × bytes_per_element. For a 70B-class model (80 layers, 8 KV heads with GQA, d_head = 128, fp16): 2 × 8 × 128 × 2 × 80 = 327 KB per token. At 32K context, that's ~10.5 GB per sequence. Two consequences. First, latency: during decode, attention must read the entire KV cache each step, adding S × 327 KB to bytes-moved-per-token; at long contexts this term overtakes the weight-read term and your tokens/sec degrades linearly with context length. Second, capacity: an 80 GB GPU holding 140 GB... wait, it can't — so assume tensor parallelism across 2 GPUs, leaving ~20 GB for cache, i.e., only ~2 sequences at 32K context. Capacity directly limits achievable batch size, which — as we'll see next — limits throughput. This is why MQA/GQA exist: reducing n_kv_heads from 64 to 8 shrinks both the bandwidth term and the capacity term by 8×. It attacks the S-dependent term in our bound, not the weight-read term.

Batching is the canonical lever for moving decode up the roofline, and it's worth deriving why it works and where it stops. With batch size B, one weight read (2N bytes) is amortized across B tokens' worth of compute (2·N·B FLOPs), so arithmetic intensity on the weight GEMMs scales linearly: I ≈ B. To reach the H100 ridge point of ~295, you'd need B ≈ 300. Below that, adding batch is nearly free in latency terms — you're bandwidth-bound, and the extra FLOPs hide under the memory transfer — so throughput scales almost linearly with B while per-user tokens/sec barely degrades. Above the ridge point, you're compute-bound and further batching trades latency for throughput one-for-one. But there's a critical subtlety: the KV cache does not amortize. Each sequence has its own cache, so attention's bytes-moved scales with B while its intensity stays flat. At large B and long S, attention — not the weight GEMMs — becomes the bandwidth bottleneck, and the effective ridge-point batch is never reached. The practical batch limit is usually KV cache capacity hit long before compute saturation.

Now distinguish rigorously between the two limits people conflate constantly: latency limits and throughput limits. Throughput (aggregate tokens/sec across all users) is bounded by compute and memory capacity, and batching plus better scheduling (continuous batching, chunked prefill) can push it toward hardware peaks. Latency — single-stream tokens/sec for one user — is bounded by something more fundamental: the irreducible sequential dependency of autoregressive decoding. Token t+1's computation depends on token t's output; the forward passes cannot be parallelized across the sequence dimension at generation time. This is a dependency-chain limit, not a resource limit. No amount of additional hardware attacks it directly: adding GPUs via tensor parallelism reduces per-device bytes but adds communication latency; adding batch improves throughput but not single-stream speed. Formally: single-stream tokens/sec ≤ 1 / (t_layer_chain), where t_layer_chain is the critical-path time of one forward pass, itself lower-bounded by (bytes_per_device / bandwidth) + (n_layers × interconnect_latency_per_layer) under tensor parallelism. Every technique in this space should be classified by which limit it attacks. Batching: throughput only. Quantization: both. Speculative decoding: latency specifically. Keep this taxonomy in your head.

Tensor parallelism is the standard tool for reducing single-stream latency, and its scaling limit is instructive to derive. Shard the weights across P GPUs: each device now reads 2N/P bytes per token, so the bandwidth term shrinks as 1/P. But each transformer layer requires (typically) two all-reduces per layer to reassemble activations — for an 80-layer model, ~160 collective operations per token, each incurring a latency floor. NVLink all-reduce latency for small messages is on the order of 5–10 μs; 160 × 7 μs ≈ 1.1 ms of pure communication latency per token, irreducible by bandwidth. Meanwhile the memory term for our 70B model at P = 8 is 140 GB / 8 / 3.35 TB/s ≈ 5.2 ms. So at P = 8 you're at ~6.3 ms/token (~160 tok/s ceiling), but notice the communication term doesn't shrink with P — it's a fixed floor set by n_layers × latency. Push P higher and you hit Amdahl's law: as the memory term → 0, tokens/sec asymptotes to 1 / (comm_floor) ≈ 900 tok/s regardless of GPU count. Crossing node boundaries (InfiniBand, ~2–10× worse latency than NVLink) makes this floor dramatically worse, which is why tensor parallelism rarely spans nodes.

Quantization attacks the bytes term directly, and its benefit is exactly predictable from our roofline analysis — which is what makes it satisfying. In the bandwidth-bound decode regime, tokens/sec ∝ bandwidth / bytes_per_token, so fp16 → int8 gives ~2× and fp16 → int4 gives ~4× on the weight-read term, essentially for free in latency (the dequantization FLOPs hide under the memory transfer, since we're far from compute-bound). Our 70B batch-1 ceiling goes from ~24 tok/s to ~96 tok/s at 4-bit. Note the corollary: quantization's speedup evaporates in compute-bound regimes. At large batch or during prefill, weight-only quantization gives little to nothing (you still compute in fp16 after dequantizing), and you need quantized arithmetic (int8/fp8 tensor cores) to gain there. KV cache quantization is a separate lever attacking the S-dependent term — increasingly important at long context, where the cache reads dominate. The Amdahl framing: quantization compresses the weight-read term; its ceiling is set by whatever term remains — KV reads, communication floors, and kernel overheads — which is exactly why measured speedups fall short of the naive 4×.

Speculative decoding is the only mainstream technique that attacks the sequential dependency itself, so understand its mechanism as a bet on bandwidth-compute asymmetry. A small draft model proposes k tokens autoregressively (cheap: small model, small bytes), then the target model verifies all k in a single forward pass. The key insight: verifying k tokens costs nearly the same wall-clock time as generating one, because the target's forward pass is bandwidth-bound — you read the weights once whether processing 1 token or k, and k·2N FLOPs still sits far below the ridge point. If the draft's acceptance rate is α per token, expected tokens per target pass is roughly (1 − α^(k+1)) / (1 − α), and speedup ≈ that quantity divided by (1 + k·c) where c is the draft's relative cost. Derive the ceiling: with α = 0.8, k = 5, negligible draft cost, you get ~3.3 tokens per verification — a hard Amdahl ceiling set by acceptance rate, which is set by how well a cheap model predicts an expensive one. Note also that speculative decoding spends spare compute to buy latency; in throughput-saturated serving (large batch, near compute-bound), that spare compute doesn't exist, and speculation can reduce aggregate throughput. Latency tool, not throughput tool.

There's a layer of "soft" bottlenecks below the roofline terms that explains the gap between theoretical ceilings and measured numbers — and at high tokens/sec, these become the binding constraint. Kernel launch overhead: each CUDA kernel launch costs ~3–10 μs of CPU-side latency; a naive decode step with hundreds of unfused kernels can spend more time launching than computing, which is why CUDA graphs and megakernels exist. Memory bandwidth utilization: matrix-vector products (the batch-1 decode workload) achieve maybe 60–80% of peak HBM bandwidth due to access patterns and DRAM row activation behavior — your effective bandwidth is lower than the spec sheet. Attention kernel efficiency: naive attention materializes the S×S score matrix to HBM; FlashAttention's contribution is precisely a roofline argument — restructure the computation to keep intermediates in SRAM, raising attention's arithmetic intensity. KV cache fragmentation: PagedAttention exists because contiguous cache allocation wastes 60–80% of capacity on internal fragmentation, and capacity is the batch-size limiter. Each of these is best understood as recovering a constant factor between the theoretical bound and reality — collectively worth 2–5×, which is why systems work matters as much as algorithms.

Let's assemble the complete picture as a hierarchy of ceilings, tightest first, for single-stream decode: (1) sequential dependency — no bound on tokens/sec in principle, but it forbids parallelizing across generated positions, forcing everything through the per-token critical path; (2) communication latency floor under parallelism — n_layers × all-reduce latency, ~1 ms/token territory, immune to more hardware; (3) memory bandwidth — (weight_bytes/P + KV_bytes) / effective_bandwidth, attackable by quantization, GQA, and sharding; (4) soft overheads — launch latency, kernel efficiency, worth constant factors. For aggregate throughput, the hierarchy is different: (1) KV cache capacity caps batch size; (2) compute saturation caps FLOPs; (3) scheduling efficiency (continuous batching) determines how close you run to those caps.

Your closing exercise: a 405B model in fp8 (405 GB) on a node of 8 H100s with tensor parallelism — derive the batch-1 ceiling. Memory term: 405/8/3.35 ≈ 15.1 ms; comm floor: ~126 layers × 2 all-reduces × 7 μs ≈ 1.8 ms; total ≈ 16.9 ms → ~59 tok/s. Then ask: which technique attacks the biggest term next, and what's its Amdahl ceiling? If you can run that analysis unprompted for any model-hardware pair, you understand the theoretical bottlenecks — everything else in this field is engineering against these bounds.