Glossary

A “4-bit” file actually averages about 4.85 bits per number, because it also stores scaling information and keeps a few sensitive parts at higher precision. Multiply this by the number of parameters and divide by 8 to get the model’s size in bytes.

Writing text is mostly limited by memory speed. Making pictures and video is limited by raw arithmetic, and that is where NVIDIA cards are furthest ahead of Macs.

Every model has a maximum, often 128,000 tokens or more. Apps usually set a much smaller limit by default, because a longer window needs more memory. In Ollama you can raise it with the num_ctx setting.

When a conversation outgrows the window, the oldest parts fall out and the model forgets them.

Technically called the KV cache, for the “keys” and “values” it stores at every layer for every token. It saves the model from recomputing the whole conversation for each new word, at the cost of memory: for an 8-billion-parameter model, a 32,000-token conversation takes several gigabytes on top of the model itself.

It starts from static and, over many passes, removes the noise that doesn’t look like what you asked for. Each pass is heavy arithmetic, so diffusion models lean on a graphics chip’s raw computing power more than on memory speed.

Training from scratch costs millions; fine-tuning an open model costs far less. Hermes, from Nous Research, is a fine-tune of Llama and Qwen. A fine-tune keeps the size, and the memory needs, of the model it started from.

A single file that holds a model’s weights plus everything an app needs to run it. Ollama and LM Studio both use it. You rarely need to handle GGUF files yourself: Ollama downloads and manages them for you.

Training a model takes data centres and months. Inference, using it, is what your Mac or PC does when you chat. It needs far less computing power, which is why it can happen at home.

A project that makes models run efficiently on ordinary hardware, including Macs, graphics cards and plain processors. It introduced the GGUF format and many of the quantization schemes in common use.

Ollama listens on localhost:11434. Coding tools, note apps and scripts can send it a message and get the model’s reply back, without anything leaving your machine. You can also let other computers in your home use it.

A desktop PC often has 32 or 64 GB of main memory, more than any consumer graphics card. Models can run from it, but slowly, because it reads data at a fraction of a graphics card’s speed.

To write each token, the computer reads the whole model out of memory once. So a machine that can read 1,000 GB a second will write about ten times faster than one that reads 100, whatever its processor.

This is why graphics cards and Apple’s Pro and Max chips are so much faster for AI than their raw processing power suggests.

Qwen 3.6 35B-A3B has about 35 billion parameters, but only about 3 billion are “active” for any one token: that’s the A3B. The whole model must still fit in memory, but it writes at the speed of a much smaller one.

That makes these models a good fit for Macs and PCs with plenty of memory but modest memory speed.

An alternative engine to llama.cpp, built by Apple for its own chips. LM Studio can run models with it on a Mac. Models for it come in their own format, not GGUF.

Macs call theirs the Neural Engine; Copilot+ PCs have one too. Built-in features like Apple Intelligence and Phi Silica use them. Ollama and most local apps use the graphics chip instead, which is much faster for big models.

The tool our guides use. It runs on Mac, Windows and Linux, picks sensible settings for your hardware, and serves models on localhost:11434 so other apps can use them.

Llama, Gemma, Qwen, Mistral and gpt-oss are all open-weight. Each comes with its own licence: some, like Apache 2.0 and MIT, let you do almost anything; others add conditions, or rule out paid work. The big cloud models, such as Claude, ChatGPT and Grok, are not available this way.

Strictly, “open source” would also mean publishing the training data and code. A few labs, such as Ai2, do.

A model is, in the end, a very long list of numbers. The “B” in a model’s name counts them in billions: Qwen3 8B has about 8 billion, Llama 3.3 70B about 70 billion.

More parameters usually means a more capable model, and always means more memory and slower writing.

Reading happens many tokens at a time, so it is limited by processing power rather than memory speed. For a short question it is instant; for a long document it can take a few seconds before the first word appears. Ollama reports it as prompt eval rate.

Models are trained with 16 bits per number. Quantization rounds those numbers so each takes about 4, 5 or 8 bits instead. A 4-bit model is under a third of the original’s size and writes about three times as fast, and for everyday tasks most people can’t tell the difference.

The names you will see, like Q4_K_M, describe the exact rounding scheme. Headroom calls them 4-bit, 5-bit, 8-bit and 16-bit.

Ollama does this automatically. The model still works, but every token now waits for the slow part, so even a small spill can halve the speed. ollama ps shows how much is on the card: 100% GPU means none of it spilled.

Models don’t see letters or whole words. They split text into tokens: common words are one token, rarer ones are two or three, and punctuation gets its own. A thousand tokens is roughly 750 English words.

Tokens are the unit for everything: how fast a model writes, how long a conversation it can hold, and what cloud services charge for.

The number Ollama prints as eval rate when you run a model with --verbose. Most people read at roughly 5 to 8 tokens a second, so around 10 feels comfortable, 30 feels quick, and anything above 60 appears faster than you can follow.

It is set mostly by memory speed and model size, not by how powerful the processor is.

A Mac doesn’t have separate graphics memory, so a 64 GB Mac can give a model far more room than most graphics cards. The trade-off is speed: unified memory is slower than a high-end card’s, and the model shares it with every other app you have open.

macOS also reserves part of it for the system, which is why a model can’t use all of it.

On a Mac, macOS lets the graphics chip use about two-thirds of memory on smaller machines and about three-quarters on larger ones. On a PC, Windows and the graphics driver keep a little of the card’s memory back. Headroom’s checker always compares against this figure, not the number on the box.

On a PC, a model runs fast only if it fits in the graphics card’s own memory: 8 GB on an RTX 4060, 24 GB on an RTX 4090. Windows and the display take a little of it for themselves.

When people say “download the weights”, they mean the file holding all of a model’s parameters. For local use it is usually a single file of a few gigabytes to a few dozen, which the app loads into memory before the first word.