Start here · Step 3 of 7

1. What is a local LLM
2. Hardware
3. Words
4. Pick a model
5. Install
6. First chat
7. Next

Step 3 of 7 · about 6 minutes

Words

Parameters are the size of the model, in billions. More parameters usually means a larger file and more memory. For a mixture-of-experts model, the page says both the total (what you store) and the active count (what it uses per token). Qwen3-30B-A3B on this site is 30.5B total and 3.3B active. Memory follows the total.

Quantization (Q4, Q5, Q8) is a smaller, slightly lossy copy of the weights. Q4_K_M is the default on the memory table because it is a common home-hosting choice. Q8_0 is closer to the original, and larger. The bits-per-weight averages come from llama.cpp’s quantize README. I have not measured a quality side-by-side, so I do not rank the quants by a score.

GGUF is the file format llama.cpp and many local runners use. It is not a model family. Official GGUF files, when the publisher ships them, are linked on the model page. If only official full-precision weights exist, the page says that.

Context length is how much of the conversation, and any attached text, the model can keep. A card may advertise 32k or 128k. The memory table here is computed at 8,192 tokens unless a measured row uses another length. Longer context uses more memory (the KV cache) even if the weights stay the same.

Tokens per second is how fast the model writes. I report generation tok/s as the median of three runs, and prompt tok/s as uncached prefill. If there is no Frankenstein (HP ProLiant DL380p Gen8) row, the cell says not yet measured. I do not invent an estimate to fill the gap.