Start here · Step 2 of 7

1. What is a local LLM
2. Hardware
3. Words
4. Pick a model
5. Install
6. First chat
7. Next

Step 2 of 7 · about 6 minutes

Hardware

VRAM is the memory on a discrete GPU. If the weights and the working context fit in VRAM, generation is usually much faster than on the CPU. The model pages compute a Q4_K_M and a Q8_0 total at 8,192 tokens. Those figures are labeled computed. I do not invent a speed for a card I have not timed.

System RAM is what a CPU-only box uses. More RAM can hold a larger quant or a longer context. It does not, by itself, make tokens per second high. Frankenstein (HP ProLiant DL380p Gen8) is the CPU-heavy machine I have measured: two old Xeons, a lot of DDR3, and a Tesla P40 when the GPU is in use.

Apple unified memory is one pool shared by the CPU and GPU. A 16 GB or 32 GB Mac does not have “16 GB of VRAM plus 16 GB of RAM.” It has one number, and the operating system and the model both sit in it. I have not measured a Mac, so this site has no Mac tokens-per-second figure.

CPU-only is how a used Xeon or a small office PC can still run a model. It will be slower. The cheap 8B chat build and the CPU-only used Xeon build are the place to go when you want parts. Their prices stay dated and typed as sold or asking.

A fit check, without a made-up speed: compare the computed memory on the model page with the memory you actually have, and leave room for the operating system. Then use What can my rig run? rather than guessing.