Setting up KV cache quantization in llama.cpp and vLLM

When we deploy local language models to process long documents, deep coding tasks or intensive multi-step conversations, we run into a physical memory barrier almost immediately. Where the static weight of the model rests permanently in graphics memory (VRAM), the key-value (KV) cache grows dynamically with every token supplied and generated. With context windows of 32k, 64k or even 128k tokens, the memory footprint of this cache often exceeds the total size of the base model itself. Anyone wanting to maximize context space without immediately investing in extra graphics cards ends up compressing these temporary attention vectors through quantization.

In this article we walk through the exact runtime flags and configuration parameters for KV cache quantization in both llama.cpp and vLLM. We cover how FP8, INT8 and INT4 precision work, analyze the trade-offs between memory gains and computational degradation, and explain the effect on language ability. To understand the theoretical foundations of weight compression and bit reduction, the overview article on quantization covers the underlying principles of rounding, scaling factors and matrix transformations.

Why the KV cache grows linearly with context length

In autoregressive transformers, the model has to look back at every generated step to all preceding tokens in the active sequence. To avoid recomputing earlier matrix multiplications for key and value pairs at every new token, the inference engine stores these intermediate results per attention layer in VRAM. For an in-depth understanding of how these mechanisms work mathematically, the explanation of KV caching in transformer architectures sets out exactly how keys and values are structured and stored per attention layer.

The default precision for this cache has traditionally been 16-bit floating point (FP16 or BF16), which amounts to 2 bytes per element. The theoretical formula for memory usage per token is as follows:

Geheugen per token (bytes) = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element

As an illustrative worked example, take a model with 32 layers, 8 KV heads (through Grouped-Query Attention) and a head dimension of 128 (as is common in modern 8B architectures). In FP16, every token in the cache costs: 2 * 32 * 8 * 128 * 2 = 131.072 bytes, that is, exactly 128 KB. At a context of 4,096 tokens this requires roughly 512 MB of VRAM. As soon as we push the context to 128,000 tokens, however, the KV cache alone swallows a full 16 GB of VRAM. Add the roughly 5 to 6 GB of VRAM for a 4-bit quantized 8B base model and a 24 GB consumer card is nearly full for just one active session.

To work out how these parameters land for specific model configurations and hardware, the VRAM calculator for local models offers a systematic method for estimating memory limits in advance. In addition, the interactive KV cache calculator lets you vary context lengths, batch sizes and precision levels dynamically to compute exact allocations.

Quantization types for the KV cache: FP8, INT8 and INT4

Compressing the KV cache differs fundamentally from quantizing model weights. Model weights are static and can be carefully optimized in advance using calibration datasets (as with AWQ, GPTQ or GGUF k-quants). The KV cache, by contrast, arises dynamically while text is being generated. Quantization therefore has to happen on the fly during inference, with minimal computational delay.

In practice we distinguish three common formats for cache compression:

• FP8 (E4M3 and E5M2): Uses 8-bit floating-point numbers. The E4M3 format (1 sign bit, 4 exponent bits, 3 mantissa bits) offers excellent dynamic range for activation vectors and comes very close to FP16 quality. It halves memory usage straight to 1 byte per element and is hardware-accelerated on modern GPU architectures.
• INT8 (Q8_0 in llama.cpp): Quantizes numbers uniformly to 8-bit integers with a scaling factor per block (often per 32 values). This format works universally on virtually all compute hardware and shows extremely little loss of numerical precision.
• INT4 (Q4_0 in llama.cpp): Reduces vectors to 4-bit integers (roughly 0.55 byte per element including scaling factors). This yields a memory saving of about 70% relative to FP16, but can cause slight quality degradation on tasks that demand extreme context precision across long distances.

Setting up KV cache quantization in llama.cpp

Within the ecosystem of llama.cpp (and derived wrappers), two specific parameters control the format of the cache: --cache-type-k (for the keys) and --cache-type-v (for the values). By default, llama.cpp allocates f16 for both buffers.

Key vectors determine the attention distribution through the inner product with the query vector, while value vectors carry the actual content representations. Because key vectors are more sensitive to quantization noise than value vectors, you can choose an asymmetric configuration (for example Q8_0 for K and Q4_0 for V) when maximum memory savings are necessary.

The example below shows the startup command for llama-server with a symmetric Q8_0 cache and a context of 65,536 tokens:

llama-server \
  -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 65536 \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --host 0.0.0.0 \
  --port 8080

When you want to use maximum context capacity on a graphics card with limited VRAM (such as 16 GB), a 4-bit configuration can be set for both keys and values:

llama-server \
  -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 65536 \
  -ngl 99 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --host 0.0.0.0 \
  --port 8080

For further guidance on distributing buffers and preventing memory fragmentation, the guide on optimizing the context window locally describes how to set context lengths and buffer space in balance.

Configuration in vLLM for high throughput and an FP8 cache

Where llama.cpp is optimized mainly for local single use, vLLM targets servers with high processing capacity and multiple concurrent requests. vLLM uses PagedAttention, which divides KV memory into virtual memory pages to counter internal and external fragmentation.

vLLM supports FP8 KV cache quantization through runtime arguments. On architectures with native FP8 Tensor Cores (such as modern datacenter and consumer chips) this delivers direct hardware acceleration. On older generations the engine can fall back to software conversion, which preserves the memory saving but requires extra instruction cycles.

To start a vLLM server with an FP8-quantized cache, activate the flag --kv-cache-dtype fp8:

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --port 8000

For in-depth instructions on multi-GPU setups, scheduling and server optimization, consult the extensive guide on configuring a vLLM server on Linux. If you are unsure which engine suits a specific server setup, the comparison of vLLM and Ollama for production environments offers a systematic evaluation of memory management and throughput under concurrent load.

Within vLLM, FP8 not only halves the memory footprint per token but also increases the effective capacity of the PagedAttention pool. This lets the system hold substantially more concurrent sessions in video memory before requests have to be swapped to slower system RAM.

Effect on language consistency and quality trade-offs

An important aspect of cache compression is its potential effect on semantic consistency across long text sequences. Unlike model weights, where quantization noise is distributed globally, noise in the KV cache directly affects the attention distribution between distant tokens.

In academic literature and general benchmarks, 8-bit variants (FP8 and INT8) typically show a negligible deviation in perplexity relative to FP16. With 4-bit quantization (such as Q4_0), very long sequences (above 32k tokens) can show subtle deviations on tasks that require strict syntax, such as formal logic or retrieving exact facts from large text documents.

When formulating complex Dutch instructions — where bracket constructions and longer dependencies between verbs and subjects occur — 8-bit quantization keeps working robustly. To check whether the model's responses stay natural and syntactically correct, the article on optimizing AI performance in Dutch offers useful techniques for consistent output quality.

Impact on throughput: latency and prefill

Compressing the KV cache affects two phases of the inference process: the prefill phase (processing the initial prompt) and the decode phase (generating new tokens).

During the prefill phase, the computed activation vectors have to be converted to the lower precision format before they are stored. This requires a small amount of extra compute. During the subsequent decode phase, memory bandwidth between the GPU cores and VRAM is almost always the primary bottleneck. Because 8-bit and 4-bit formats require considerably less data per token to travel across the memory bus, generation speed can actually increase with large contexts.

The real gain in tokens per second depends heavily on the ratio between context length and GPU compute power: with short prompts (below 2,048 tokens) the effect on speed is minimal, while with long documents the reduction in memory traffic can deliver a clear speed-up.

Troubleshooting and preventing out-of-memory situations

Even with a compressed cache, runtime interruptions can occur when context space fills up unexpectedly. A common cause in llama.cpp is allocating a context size that fits within theoretical VRAM space but does not account for temporary CUDA compute buffers needed during peak load.

When a process stops unexpectedly through lack of memory, this can often be traced to an overestimate of available free VRAM or too high a number of concurrent slots. Should the inference engine hang or refuse to start, the troubleshooting guide on resolving out-of-memory messages and slow inference offers concrete steps for locating memory conflicts step by step.

Privacy and local data processing

Quantizing the KV cache is a purely mathematical memory optimization that takes place entirely within the local working memory and VRAM of your own system. No intermediate activation layers, embeddings or prompt data are sent to external servers during this process. All context information stays strictly bound to the local machine.

For organizations deploying local language models to comply with strict privacy guidelines, the overview of working with local AI in a privacy-friendly way describes how data flows can be isolated and managed in line with regulations.

Practical configuration guidelines

When setting up a local inference environment, the following guidelines apply:

• Modern GPUs with hardware FP8 support: Preferably use FP8 in vLLM (--kv-cache-dtype fp8) or Q8_0 in llama.cpp (--cache-type-k q8_0 --cache-type-v q8_0). This halves the memory footprint of the cache while preserving numerical stability.
• Systems with older GPU architectures or CPU inference: Choose q8_0 in llama.cpp for a balanced trade-off between memory gains and compatibility. Avoid FP8 in vLLM if the GPU offers no native support and compute speed is the priority.
• Maximum context capacity on tight hardware: Apply a 4-bit configuration (such as --cache-type-k q4_0 --cache-type-v q4_0 in llama.cpp) when long documents absolutely have to fit within a single graphics card, and spot-check whether output stays consistent on large-scale prompts.