Speed tests

By the end of this page you have measured three speed effects in your own app, and you know which one matters for which job.

All three tests use the speed line under each answer. Set the temperature to 0 in Settings first. Then a model always picks the most likely token, so the same prompt gives the same answer, and the speed is the only difference.

The results below are from one run on an M3 Max with 36 GB and Ollama 0.40.14. Your numbers will differ, because memory bandwidth decides most of the speed. Look at the ratio between the two models, not the absolute numbers.

To start an empty chat between runs, press Cmd+N on macOS or Ctrl+N on Windows. Picking another model unloads the old one, so two large models never share memory.

Test 1: dense against mixture of experts

The two models are almost the same file size.

1. Pick qwen3.8:27b-q4_K_M, set thinking to Off, and send the prompt below.
2. Start a new chat, pick qwen3:30b, and send the same prompt. It always thinks, so there is no thinking menu.

Explain how a hash map works to a junior developer. Use about 150 words and one short Python example.

The mixture of experts wrote about six times faster, with a file that is almost 1 GB larger. It reads about 3B weights per token instead of 27B. It also wrote more tokens, because its thinking counts too. Tokens per second is a rate, so the comparison still holds.

Test 2: thinking on against off

Thinking does not change the speed. It changes how much the model writes before it answers.

1. Pick qwen3:8b and click the Thinking pill so it is off. Send the prompt below.
2. Start a new chat, turn thinking on, and send it again.

A train leaves at 14:35 and the trip takes 2 hours and 50 minutes. When does it arrive? Answer in one sentence.

Both runs gave the same answer, "The train arrives at 17:25." With thinking on, the model first wrote about 430 tokens of reasoning, so the answer took about 45 times longer, at almost the same rate. That is the trade. Thinking helps on hard problems, and on easy ones it only costs time. Models with levels, such as gpt-oss:20b, let you pick how much.

Test 3: speculative decoding

A model writes one token at a time, and every token means reading all the active weights. Checking several proposed tokens costs about the same as writing one. Speculative decoding uses that. Something cheap guesses a few tokens ahead, and the full model checks the guesses in one pass. It keeps every guess up to the first wrong one.

qwen3.8:27b-mtp-q4_K_M has the guesser built in. Its extra multi-token prediction (MTP) heads predict the next few tokens. The file is the same 17.7 GB as the plain version.

The guesses are only good when the next tokens are predictable. An edit is the best case, because most of the answer repeats code that is already in the prompt.

1. Pick qwen3.8:27b-q4_K_M, set thinking to Off, and send the prompt below.
2. Start a new chat, pick qwen3.8:27b-mtp-q4_K_M, set thinking to Off, and send the same prompt.
3. Do the same with a normal question: "What causes the seasons on Earth? Answer in about 150 words."

Change the late fee from 2% to 3% per week. Return the whole function, nothing else.

```python
from datetime import date


def invoice_total(lines, issued, paid=None, country="NL", discount_code=None):
    """Return the amount due for an invoice, including VAT and late fees."""
    vat_rates = {"NL": 0.21, "DE": 0.19, "BE": 0.21, "FR": 0.20}
    if country not in vat_rates:
        raise ValueError(f"Unknown country: {country}")

    subtotal = 0.0
    for line in lines:
        quantity = line.get("quantity", 1)
        price = line["price"]
        if quantity <= 0 or price < 0:
            raise ValueError(f"Invalid line: {line}")
        subtotal += quantity * price

    if discount_code == "WELCOME10":
        subtotal *= 0.90
    elif discount_code == "PARTNER20":
        subtotal *= 0.80
    elif discount_code is not None:
        raise ValueError(f"Unknown discount code: {discount_code}")

    vat = subtotal * vat_rates[country]
    total = subtotal + vat

    paid_on = paid or date.today()
    days_late = (paid_on - issued).days - 30
    if days_late > 0:
        weeks_late = days_late // 7 + 1
        late_fee = total * 0.02 * weeks_late
        total += min(late_fee, total * 0.10)

    return round(total, 2)
```

On the edit the MTP model was about 2.5 times faster. On the question it was about 1.7 times faster. Both pairs wrote the same number of tokens and the same text. Speculative decoding changes how fast the answer arrives, not which answer you get.

Editing is the job where speculative decoding pays off most, and it is a large part of what a coding agent does all day. That is the next page.

Check

You have a speed line under each answer. The mixture of experts is several times faster than the dense model. Thinking on writes far more tokens at about the same rate. The MTP model is faster on both prompts, and the gap is larger on the edit.