Start here · Step 4 of 7

1. What is a local LLM
2. Hardware
3. Words
4. Pick a model
5. Install
6. First chat
7. Next

Step 4 of 7 · about 5 minutes

Pick a model

Start from what you have, not from a leaderboard. Open What can my rig run? and type the RAM and the GPU. It stays in the browser.

A sensible first pick for most people is a small instruct model in the 4B–8B range at Q4_K_M. These are suggestions by size and license, not a ranking of quality, and not a promise of speed on your machine.

Computed Q4_K_M weights plus KV at 8,192 tokens. Not a speed.
ModelLicenseQ4_K_M + 8k
Qwen3-4BApache-2.03.4 GiB Computed
Qwen3-8BApache-2.05.8 GiB Computed
Llama 3.1 8B InstructLlama 3.1 Community License5.6 GiB Computed
Gemma 3 4BGemma terms2.5 GiB Computed

Qwen3-4B and Qwen3-8B are Apache-2.0, and I have already run them on Frankenstein (HP ProLiant DL380p Gen8). Llama 3.1 8B Instruct is the name many people already know; read its license before you download. Gemma 3 4B takes text and an image; its terms are not Apache-2.0.

Match the job: chat, coding, reasoning traces, or a home assistant that can see an image. The “best for” field on each model is a judgment from the card’s stated jobs and the model’s size. It is not a speed ranking.

Read the license before you download. Apache-2.0 and MIT are straightforward. Llama Community, Gemma terms, and “Apache plus a usage policy” are not the same thing.

If the computed Q4_K_M + 8k total does not fit, pick a smaller model or a shorter context. Do not start with a 70B.