Run local AI with Ollama

By the end of this page Ollama runs on your computer, you have chatted with a model in the terminal, and you can look at a model name and tell whether it fits on your machine and roughly how fast it runs.

How Ollama works

Ollama has two parts.

1. A server. A small program that runs in the background on your own computer. It keeps your models on disk, loads one into memory when something asks for it, and answers at the address http://127.0.0.1:11434. 127.0.0.1 means "this computer". Nothing goes over the internet.
2. Clients. Anything that talks to the server. The ollama command in your terminal is a client. The app you build later is a client too.

The server is light. It only uses real memory while a model is loaded. A model loads on the first question, and after five minutes without questions the server unloads it again.

On macOS and Windows the Ollama app starts the server when you log in. On Linux the install script sets it up as a service that starts with the computer.

# macOS with Homebrew (the app, not the command-line-only formula)
brew install --cask ollama-app

# Linux
curl -fsSL https://ollama.com/install.sh | sh

ollama -v

ollama run qwen3:1.7b

ollama run qwen3:1.7b --verbose --think=false "Say hi in three words."

ollama run qwen3:8b --verbose --think=false "Say hi in three words."

What fits on your computer

A model must fit in the memory your graphics chip can use. First check how much you have.

• Mac: Apple menu, then About This Mac. Read Memory. The graphics chip shares this memory, and macOS lets it use about two thirds.
• Windows: press Ctrl+Shift+Esc to open Task Manager. Go to Performance, then GPU, and read Dedicated GPU memory. That is the memory on your graphics card, not your normal RAM.
• Linux with an NVIDIA card: run nvidia-smi and read the memory total.

That number is not all yours. On a Mac, the graphics chip can use about two thirds of it. On Windows, the graphics card can use all of its own memory.

A model needs its file size plus about 2 GB for the conversation.

If "needs about" is smaller than your room, the model fits. For example, a Mac with 16 GB has about 10 GB of room. qwen3:8b fits, gpt-oss:20b does not.

No graphics card, or too little memory on it? Ollama then runs the model on the processor, from normal RAM. That works, but it is much slower. Stay with qwen3:1.7b.

To check a model you are running, use ollama ps. 100% GPU in the PROCESSOR column means it fits. A split such as 30%/70% CPU/GPU means it does not fit, and it writes much slower.

Why a model is this big

A model is mostly a long list of numbers, the weights. To run it, all of them must sit in memory. The size follows from two things.

size in bytes = parameters x bits per weight / 8

qwen3:8b has 8.2 billion parameters. At 4 bits per weight that is 8.2 x 4 / 8 = 4.1 GB. The real file is 5.2 GB, because the common Q4_K_M format keeps some layers at higher precision.

You can work out the real average from what Ollama reports. These are the models used in this course, as ollama list and /api/tags showed them on one machine with Ollama 0.40.14.

The last column is the file size in bits divided by the parameter count, so it also includes the small extras in the file.

What quantization does

Models are usually released with 16 bits per weight. Quantization stores each weight with fewer bits, for example 4 instead of 16. The file gets about four times smaller and the model gets a little less precise. For chat and coding on a laptop, 4-bit models are the usual choice, because the size difference decides whether a model fits at all.

The suffix in a model name tells you the format. q4_K_M is a 4-bit format from llama.cpp, the engine Ollama builds on. MXFP4 is the 4-bit format gpt-oss ships in. When a tag has no suffix, ollama show <model> prints the quantization.

Dense or mixture of experts

How fast a model writes depends on how many weights it reads per token. On a laptop, reading the weights from memory is the slow part, not the math.

A dense model uses every weight for every token. qwen3.8:27b reads all 27 billion weights to produce one token.

A mixture of experts splits most of its layers into many small blocks, the experts. For each token a small router picks a few of them, and only those run. qwen3:30b has 128 experts per layer and uses 8 for each token, which comes to about 3 billion of its 30 billion weights. It reads roughly a ninth of what the dense 27B model reads, so it writes much faster. You measure exactly how much faster on the speed tests page.

The catch is memory. The router can pick any expert for the next token, so all of them must be loaded. A mixture of experts is fast, but it is not small. gpt-oss:20b is a mixture of experts too, with 32 experts and 4 in use per token.

The expert counts come from the model_info field of Ollama's /api/show endpoint, which the next page calls.

ollama pull gpt-oss:20b

ollama pull qwen3.8:27b-q4_K_M
ollama pull qwen3:30b
ollama pull qwen3.8:27b-mtp-q4_K_M

Check

ollama list
ollama show qwen3:30b
ollama ps

ollama list shows every model you pulled with its size. ollama show qwen3:30b prints the architecture (qwen3moe), 30.5B parameters, the context length, the quantization, and the thinking levels the model accepts. ollama ps shows the model that is loaded now. A model stays loaded for five minutes after the last request, then Ollama frees the memory.