Start here

Run a model on a computer you have.

I’m Adam. This is measuredllm, my notes for running language models on machines I actually own.

I got tired of guessing. A forum post says a card does some number of tokens a second. A listing says a server is a bargain. Neither one is in my house. So I write down what my machines do, on the day I measure them, and I leave the cell blank when I have not measured it yet.

The machine I have measured the most is Frankenstein (HP ProLiant DL380p Gen8): two old Xeons, a lot of DDR3, and a Tesla P40. Those rows are labeled measured. If a number is an estimate, the page says estimated and why. If a number comes from someone else, or from a vendor’s model card, it is labeled that way. It is not mixed in with my runs. A price has a date, and it says whether it was a sold price or an asking price. I do not add those two together.

Seven lessons. You can skip ahead. Amber on this path is only the lesson marker, not a data badge.

  1. 1
    What is a local LLM
    What it is, and why run one at home.
  2. 2
    Hardware
    VRAM, system RAM, unified memory, and CPU-only.
  3. 3
    Words
    Parameters, quants, GGUF, context, and tokens per second.
  4. 4
    Pick a model
    A first model that fits the machine you already have.
  5. 5
    Install
    One official runner. Not all three.
  6. 6
    First chat
    One real prompt, and what to do if it fails.
  7. 7
    Next
    Where the rest of the site is, once the first chat works.

Step 1: What is a local LLM