Start here
Run a model on a computer you have.
I’m Adam. This is measuredllm, my notes for running language models on machines I actually own.
I got tired of guessing. A forum post says a card does some number of tokens a second. A listing says a server is a bargain. Neither one is in my house. So I write down what my machines do, on the day I measure them, and I leave the cell blank when I have not measured it yet.
The machine I have measured the most is Frankenstein (HP ProLiant DL380p Gen8): two old Xeons, a lot of DDR3, and a Tesla P40. Those rows are labeled measured. If a number is an estimate, the page says estimated and why. If a number comes from someone else, or from a vendor’s model card, it is labeled that way. It is not mixed in with my runs. A price has a date, and it says whether it was a sold price or an asking price. I do not add those two together.
Seven lessons. You can skip ahead. Amber on this path is only the lesson marker, not a data badge.
- 1What is a local LLMWhat it is, and why run one at home.
- 2HardwareVRAM, system RAM, unified memory, and CPU-only.
- 3WordsParameters, quants, GGUF, context, and tokens per second.
- 4Pick a modelA first model that fits the machine you already have.
- 5InstallOne official runner. Not all three.
- 6First chatOne real prompt, and what to do if it fails.
- 7NextWhere the rest of the site is, once the first chat works.