Comparing models fairly on your own hardware

By Ivo Donker - 3 August 2026

Comparing large language models (LLMs) online is often based on general benchmarks, tables, and other people's experiences. However, anyone who wants to run a model locally themselves quickly notices that published figures rarely match reality on their own computer. A model that runs smoothly according to an online table may respond slowly or consume an unnecessary amount of memory on your specific configuration. To determine which model performs best for your applications, a structured and objective measurement method is necessary.

In this guide, we cover how to compare different models against each other on your own hardware in a scientific and repeatable way. We look at the variables that influence the outcome, the units of measurement that actually matter, and the steps you need to take to prevent distortion from external factors.

Why external figures rarely hold up

The performance data that appears on forums and in benchmark overviews is collected under specific test conditions. When you try to reproduce these figures, you run into inherent differences in both the underlying hardware and the software settings. Anyone following general advice for a choosing a local model, notices that local factors strongly dictate the final performance.

There are four primary reasons why published figures deviate from your own experience:

• Hardware architecture: The ratio between memory bandwidth, video memory (VRAM), working memory (RAM), and core clock speeds differs per system. A graphics card with a wide memory bus processes tokens differently than a processor that depends on general system memory.
• Quantization formats: Not every quantization is implemented identically. A 4-bit quantization via a particular scheme can produce a different memory footprint and processing speed than a 4-bit version on a different scheme or a different type of engine.
• Context length: Increasing the context window requires an exponentially or linearly growing amount of memory for managing the key-value cache (KV cache). A benchmark performed on a context of 512 tokens gives no representative picture of performance at 8192 tokens.
• Runtime settings: The chosen inference engine, the number of allocated CPU threads, and the distribution of the model layers across different systems directly affect speed.

Fixing variables for a fair comparison

A reliable measurement requires all boundary conditions to remain constant, except for the model you're directly testing. When you compare two models with different settings, you're measuring the influence of the settings, not the quality or efficiency of the model itself.

Make sure the following variables are explicitly fixed for each test series:

• Quantization level: Compare models of the same precision level, for example both at Q4_K_M or Q8_0. Comparing an unquantized model with a heavily compressed model gives a distorted picture.
• Context length: Set the maximum context size to a fixed value (for example 4096 tokens) in the runtime configuration.
• Batch size: Keep the parallel processing parameters (such as logical_batch_size or physical_batch_size) identical.
• Hardware offloading: Fix the number of layers sent to the GPU. Make sure the entire model fits into VRAM, or keep the ratio between VRAM and RAM exactly the same for comparable model sizes.
• Sampling parameters: Disable randomness during performance tests, or fix it. Use a fixed random seed, a fixed temperature (for example temperature = 0.0 for deterministic results), and consistent values for top-p and top-k.
• Exact model version: Document the exact revision number or the exact file name (SHA256 hash) of the model file.

Prefill versus decode: two separate phases

When a language model processes a request, two technically distinct phases take place: prompt processing (prefill) and token generation (decode). Combining these two phases into a single average speed hides crucial performance information.

During the prefill phase the model reads in the input text (the prompt) and builds up its internal representation. This process is primarily bound by the processing power of the CPU or GPU (compute-bound). Especially when submitting long documents or large chunks of code, the speed of the prefill phase is decisive for the responsiveness of the system. More details on how to specifically isolate these speeds can be found via measuring speed on the benchmark hub.

The decode phase concerns generating new tokens, one at a time. Because the model's entire weight matrix has to be pulled through memory again for each new token, this phase is primarily bound by memory bandwidth (memory-bound). A model can have a slow prefill but a fast decode, or vice versa. For this reason, both values must be recorded separately in the measurement protocol.

Relevant performance indicators

A single value is not enough to assess a model on your own hardware. A complete analysis rests on four pillars:

Never measure these indicators over a single run. Perform each test series at least five times under identical conditions. Then calculate the average and the standard deviation. A model that averages 25 t/s with large fluctuations is, in practice, less predictable than a model that consistently operates at 22 t/s.

Preventing distortion: temperature and background processes

Hardware doesn't behave statically. Two common sources of noise during local measurements are thermal throttling and unforeseen background activity from the operating system.

When a graphics card or processor is heavily loaded for an extended period, the temperature of the chip rises. Once the maximum operating temperature is reached, the hardware automatically lowers the clock speed to prevent damage. If you test model A on a cold graphics card and immediately afterward model B on a warmed-up card, model B will consistently score worse purely because of the higher temperature.

Therefore, take the following precautions on your machine:

1. Perform a warm-up run: Before the official measurement, let the hardware run one short generation to fill the memory and start up the cooling. Discard the results of this first run.
2. Keep rest breaks: Build in a fixed rest period (for example 10 to 15 seconds) between consecutive tests to give the cooling system a chance to let the temperature drop.
3. Close background processes: Disable browsers, communication apps, automatic backups, and other heavy software. Make sure the system's idle load is minimal.
4. Check the cooling: Monitor the temperature and clock speed of the hardware during the tests to determine whether throttling is occurring.

Measuring quality with a fixed Dutch test set

Speed is only half of the comparison. A model that generates text extremely fast but delivers factually incorrect or incoherent answers is unsuitable. Assessing quality requires just as much structure as measuring speed.

For the qualitative evaluation, build a small, custom test set of 10 to 20 fixed prompts. Make sure these tasks are representative of the work you actually want the model to perform. If the intended application is Dutch-language, the test set should consist entirely of Dutch-language tasks. Quality tests in English say little about a model's language comprehension and syntax in Dutch. For specific guidelines, see the article on testing language models in Dutch.

For the assessment, set up a fixed evaluation form with specific criteria, such as:

• Factual accuracy: Does the answer contain factual inaccuracies or hallucinations?
• Instruction following: Is the requested format (for example a JSON structure, a bullet list, or a fixed length) followed exactly?
• Language use and grammar: Does the Dutch sentence structure flow naturally, or does the answer show clear traces of literal English translations?
• Reasoning ability: Is a logical step-by-step reasoning correctly built up for a complex problem?

When running qualitative benchmarks, it's important to standardize the test tasks in a way that ensures the reproducibility of results .

Practical measurement protocol in steps

To get started right away, you can follow the step-by-step plan below for each comparison session on your own machine:

Step 1: Preparing the environment

Close all non-essential applications. Check via your operating system's task manager or monitor whether the CPU and GPU load is close to zero percent. Make sure your latest drivers and the desired runtime (such as llama.cpp or Ollama) are up to date. If needed, review the instructions on running LLMs locally for a stable base setup.

Step 2: Determining the test tasks

Choose three fixed prompts for the speed measurements:

1. Short prompt / Short response time: A simple question (for example 20 tokens in, 50 tokens out) to test the initial latency.
2. Long prompt / Short response time: A large source text of 2,000 tokens with the request for a short summary, to measure the prefill speed.
3. Short prompt / Long response time: A short instruction requesting an extensive article of 1,000 tokens, to isolate the decode speed.

Step 3: Running the test series

Follow the steps below for each model:

• Load the model with the established parameters (for example Q4_K_M, 4096 context).
• Perform one warm-up run and ignore the timing measurement.
• Run test 1, 2, and 3 consecutively. Repeat this series 5 times. Wait 10 seconds between each run.
• Note the raw measurement values from the log file or the runtime's terminal output.

Step 4: Recording in the results log

Keep track of the data in a simple log or spreadsheet. Use a consistent structure so you can later compare old measurements when you buy new components or adjust settings.

Common measurement mistakes

When testing local models, methodological mistakes that make the outcome unreliable are regularly made. The most common pitfalls are:

Comparing apples to oranges: Comparing a 7-billion-parameter model at Q8_0 with a 14-billion-parameter model at Q2_K. Although they may take up the same amount of memory, the qualitative characteristics and processing characteristics differ fundamentally. More background on the impact of compression on accuracy can be found in the overview of quantization explained.

Measuring with a cold cache: Some runtimes store previous contexts in RAM. If you send the exact same prompt twice in a row, the second run may show an extremely high prefill speed because the context has already been processed. Clear the cache or restart the runtime between tests to avoid misleading results.

Drawing conclusions from a single prompt: One specific prompt may happen to turn out very favorably for a particular model because of the specific training material. Only once a model consistently scores well across multiple varied tasks can you draw a well-founded conclusion about its suitability for your workflow.

Further reading

• Choosing a local model
• Quantization explained
• running an LLM locally
• Measuring speed on the benchmark hub
• Reproducibility of results
• Testing language models in Dutch