Configuring speculative decoding for faster LLM tokens

Anyone deploying local language models typically works through a fixed path from selection to setup, server optimization, and performance tuning. Once the base installation is running, larger models of 32 billion or 70 billion parameters run almost immediately into a fundamental limitation of autoregressive generation: for every single generated token, the system has to move the full set of weights across the memory bus. Before starting with advanced inference acceleration, it is wise to check the physical system requirements in the overview of what hardware local LLMs require.

Speculative decoding offers a proven method for substantially reducing the wait time per token without compromising final model quality. In this article we cover how speculative decoding works, how to configure it in llama.cpp, vLLM, and Ollama, how to select compatible draft models, and the interplay between memory budgets and acceptance rates across different applications.

How speculative decoding works technically

During standard inference, a neural network generates text token by token. For every token, the compute cores of a graphics card (such as CUDA cores or Tensor Cores) have to fetch all of the tens of gigabytes of model parameters from working memory or VRAM. Because the actual mathematical operation per individual token is relatively light compared with the enormous volume of data that has to be moved, local inference at a batch size of 1 almost always runs at the limit of memory bandwidth (memory-bandwidth bound). The processor spends most of its compute time waiting on data transfer.

Speculative decoding resolves this bottleneck by running two models in tandem: an extremely compact and fast draft model (0.5B to 1.5B parameters, for example) and the full target model (such as a 32B or 70B variant). The small draft model predicts a sequence of consecutive candidate tokens at minimal computational cost (often denoted $K$, typically between 3 and 8 tokens). The large target model then performs a single parallel verification (forward pass) across that entire sequence.

Because verifying several tokens in parallel costs the memory bus almost as much time as processing a single token, this check requires hardly any extra bandwidth. Tokens that match the probability distribution of the large model are accepted immediately. As soon as a predicted token deviates, the target model rejects it, generates the correct alternative on the spot, and restarts the draft cycle. The mathematical foundation proves that the output remains statistically identical to a generation produced purely by the large model. If you want to study the exact formulas and probability distributions, all the background is in the explanation of how speculative decoding increases generation speed.

VRAM budgeting and model choices

The most important practical trade-off in speculative decoding is available video memory. Both models have to fit in memory simultaneously, including their respective KV caches. When the VRAM budget is tight, adding a draft model may force you to compress the main model more aggressively. More detail on the impact of lower bit precision on quality and memory use can be found in the overview of quantization and compression of large models.

To keep the graphics card from running out of memory, an accurate memory calculation is essential. Use the VRAM calculator for model memory and context size to determine in advance whether the desired context window for both models fits within your hardware capacity.

Configuring speculative decoding in llama.cpp

The open-source ecosystem around llama.cpp offers excellent and direct support for speculative decoding through both llama-cli and the standalone llama-server binary. The configuration rests on specifying a primary model with -m and a secondary draft model with -md (model draft).

When fine-tuning the parameters, the following settings determine performance:

• --draft-max (or -draft): The maximum number of tokens the draft model may speculate ahead per cycle (usually set between 4 and 8).
• --draft-min: The minimum acceptance threshold for keeping the speculative cycle active.
• --draft-p-min: A threshold based on token probability that aborts speculation early when the draft model shows low confidence.

Example 1: Interactive CLI session

The command below illustrates how to start a CLI session with a 32B main model and a 0.5B draft model, moving all layers to the GPU:

# Starten van llama-cli met hoofdmodel en draft-model
llama-cli \
  -m models/target-model-32b-q4_k_m.gguf \
  -md models/draft-model-0.5b-q8_0.gguf \
  -ngl 99 \
  -ngld 99 \
  -c 8192 \
  -cd 8192 \
  --draft-max 6 \
  --temp 0.0 \
  -p "Leg in heldere bewoordingen uit hoe caching werkt in moderne computerarchitectuur."

The flag -ngl 99 transfers all network layers of the main model to the graphics processor, while -ngld 99 gives the same instruction for the draft model. With -cd 8192 the context memory of the helper model is matched exactly to the context size of the main model.

Example 2: Server mode with an OpenAI-compatible API

For continuous integration with applications and web interfaces, llama-server can be started as a service with draft functionality enabled:

llama-server \
  --host 127.0.0.1 \
  --port 8080 \
  -m models/target-model-32b-q4_k_m.gguf \
  -md models/draft-model-1.5b-q4_k_m.gguf \
  -ngl 99 -ngld 99 \
  -c 16384 -cd 16384 \
  --draft-max 8 \
  --draft-p-min 0.4

Configuring speculative decoding in vLLM

In professional and multi-user environments, vLLM is one of the most widely used frameworks for high throughput. vLLM supports both a full parallel draft model and prompt lookup decoding through n-gram matching. For administrators setting up a central Linux server, the guide on Configuring vLLM for high throughput on Linux offers valuable guidance on network and process configuration.

When using a neural draft model, the arguments --speculative-model and --num-speculative-tokens are passed when initializing the vLLM OpenAI server:

# Starten van vLLM met een complementair draft-model
python3 -m vllm.entrypoints.openai.api_server \
  --model /pad/naar/target-model-70b \
  --speculative-model /pad/naar/draft-model-1b \
  --num-speculative-tokens 5 \
  --use-v2-block-manager \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --port 8000

If the VRAM budget leaves no room for a second model, n-gram matching can be enabled instead. This method looks for repeating token patterns in the prompt and history already supplied, which works excellently for document analysis, code edits, and structured data extraction:

# vLLM prompt lookup decoding zonder extra neuraal draft-model
python3 -m vllm.entrypoints.openai.api_server \
  --model /pad/naar/target-model-32b \
  --speculative-model [ngram] \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 3 \
  --ngram-prompt-lookup-min 1

Speculative decoding in Ollama (Modelfile and CLI)

Inside Ollama, the underlying runtime is powered by llama.cpp. Although speculative handling is driven automatically in some versions when compatible architectures are detected, it can also be set explicitly in a custom Modelfile.

This defines the path to both the base model and its accompanying draft model:

# Modelfile met speculatieve configuratie
FROM ./target-model-32b.gguf

# Parameterinstellingen voor het draft-mechanisme
PARAMETER draft_model "./draft-model-0.5b.gguf"
PARAMETER num_draft 5
PARAMETER temperature 0.2
PARAMETER top_p 0.9

SYSTEM """Je bent een deskundige assistent die bondige, feitelijke antwoorden formuleert in het Nederlands."""

Once the file is saved, the model can be created and started with the standard CLI commands:

ollama create aangepast-speculatief-model -f Modelfile
ollama run aangepast-speculatief-model

Evaluating and optimizing the acceptance rate

The real return on speculative decoding stands or falls with the acceptance rate ($\alpha$). This is the ratio between the number of approved speculative tokens and the total number of predicted draft tokens. When the acceptance rate is high (75% or more, for example), the system generates several tokens in parallel in nearly every iteration. If that rate drops below roughly 35% to 40%, however, the computation can become slower than standard autoregressive generation because of the overhead of the draft passes.

The main variables that influence the acceptance rate:

• Shared vocabulary and tokenizer: The target model and the draft model must use exactly the same tokenizer architecture and token ID distribution. Combining models from different families inevitably leads to failed verifications.
• Content type and predictability: Source code, JSON structures, and formal reports contain many repeating syntactic patterns, which leads to a considerably higher acceptance rate than poetry or creative prose.
• Sampling parameters: A low temperature (such as temperature = 0.0) maximizes the agreement between the models. As temperature rises, entropy increases and the number of accepted tokens falls.

To verify that a specific configuration actually delivers a speedup, it is advisable to use an objective measurement method. For concrete measurement protocols, see the guide on measuring latency, throughput, and tokens per second.

A simple benchmark script

A standardized Python script lets you determine the actual throughput of a local OpenAI-compatible API precisely:

import time
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="lokaal")

test_prompt = """Schrijf een modulaire Python-klasse voor het beheren van een 
in-memory cache met een Time-To-Live (TTL) mechanisme. Voeg type hints toe."""

tijd_start = time.perf_counter()
respons = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": test_prompt}],
    temperature=0.0,
    max_tokens=400
)
totale_duur = time.perf_counter() - tijd_start

aantal_tokens = respons.usage.completion_tokens
doorvoer_tps = aantal_tokens / totale_duur

print(f"Gegenereerde tokens: {aantal_tokens}")
print(f"Totale tijdsduur: {totale_duur:.2f} seconden")
print(f"Berekende snelheid: {doorvoer_tps:.2f} tokens/seconde")

The influence of the Dutch language on the acceptance rate

When processing Dutch-language text, the acceptance rate can deviate somewhat from English-language reference values. Many smaller draft models (on the order of 0.5B to 1B parameters) have a shallower representation of Dutch grammar, sentence structure, and compound nouns than of English. As a result, the helper model makes prediction errors more often on specific inflections or less frequent words.

Although the large target model corrects such deviations seamlessly, this can lower the effective acceptance rate by a few percentage points compared with identical tasks in English. In practice, the overall speed gain on tasks such as summarizing, analyzing, and programming nevertheless remains substantial. Additional techniques for optimizing language quality and instruction following are covered in the article on getting AI to perform better in Dutch.

Privacy and data protection

An essential foundation of local LLM infrastructure is that all processing stays inside your own machine or local network. That applies undiminished to speculative decoding: both the draft model and the target model run locally on the graphics card or in shared system memory. No intermediate steps, logits, or prompts are sent to external servers.

The processing therefore remains fully in line with GDPR requirements and internal compliance guidelines for sensitive data. Extensive information on setting up a secure local processing environment can be found in the guide on using AI in a privacy-friendly way.

When speculative decoding is not the right choice

Despite its clear advantages, speculative decoding has specific situations in which enabling it is actually counterproductive:

• High concurrent server load: When an inference server continuously handles dozens of simultaneous requests with large batches, the bottleneck shifts from memory bandwidth to raw compute (compute bound). Running additional draft computations then lowers the system's overall throughput.
• Highly creative tasks at high temperature: At a high randomness setting (such as temperature > 0.85) the statistical overlap between the draft and main model drops sharply, which can push the acceptance rate below the profitable level.
• Insufficient physical memory: If adding a draft model forces you to quantize the main model extremely aggressively (to 2-bit or 3-bit, for example), the potential speed gain rarely outweighs the loss of reasoning ability and accuracy.

Recommendations for practice

For individual work on a local workstation — programming, summarizing documents, and interactive chat — speculative decoding is a particularly effective optimization for noticeably improving the response time of heavy models. Adding a 0.5B or 1.5B helper model to a 32B or 70B main model can reduce wait time considerably, provided memory is carefully budgeted in advance and both models share the same tokenizer.