Hardware for Local Models

Running models on your own machine is now practical, and the hardware advice circulating is mostly wrong because it optimises for the wrong quantity.

Memory is the binding constraint

A model must fit in memory to run at useful speed. If it does not fit, the system swaps to disk and performance collapses — not by a factor of two, but by orders of magnitude.

Processing speed determines how fast tokens are produced once the model fits. It does not determine whether the model runs at all.

The practical consequence: a machine with plenty of memory and modest compute runs larger models slowly. A machine with fast compute and insufficient memory runs them not at all. Memory first.

What size fits where

Model sizes are quoted in parameters, and memory needed depends on the precision at which weights are stored.

Full precision requires roughly two bytes per parameter. Common quantisation to around four bits requires roughly half a byte, plus overhead for context.

A rough guide with moderate quantisation:

8 GB of usable memory: small models, in the low single-digit billions of parameters. Useful for summarising, extraction and classification. Noticeably weaker than a hosted assistant.

16 GB: models in the 7 to 14 billion range comfortably. This is the point where output becomes genuinely useful for drafting, rewriting and routine coding.

32 GB: models in the 30 billion range, or smaller models with long context. A substantial capability step.

64 GB and above: the larger open models. Approaching hosted quality for many tasks.

Leave headroom. The operating system and the context window both need memory beyond the weights.

Architecture matters more than raw specification

Unified memory — where processor and graphics share one pool — is unusually well suited, because the model can use system memory rather than being confined to a graphics card's separate memory. A laptop with a large unified pool outperforms a desktop with a small discrete card for this specific task, which is the opposite of the usual advice.

A discrete graphics card with substantial dedicated memory is the fastest option where the model fits within that memory. Where it does not, performance falls off sharply.

Memory bandwidth is the second constraint. Generation speed is largely determined by how fast weights can be read, which is why systems with high bandwidth feel much faster than their processing specifications suggest.

CPU-only works and is slow. Acceptable for batch processing, frustrating for conversation.

Quantisation, practically

Reducing the precision at which weights are stored. This is what makes local models feasible.

Moderate quantisation costs little. Around four to five bits per weight is the common sweet spot, and quality loss is small for most tasks.

Aggressive quantisation degrades noticeably, particularly on reasoning and code.

A larger model heavily quantised usually beats a smaller model at full precision, within limits. If you must choose, the bigger model at moderate quantisation is generally the better bet.

What to buy

If you already have 16 GB, try it before buying anything. The software is free and an afternoon tells you whether the quality suits your work.

If buying a laptop and this matters, prioritise memory. The difference between configurations is usually modest against the capability difference.

Do not buy a graphics card with limited memory for this purpose. A card with substantial memory is expensive, and a card with modest memory constrains you to small models regardless of its speed.

Consider whether you need it at all. For occasional confidential work, a slower machine that produces an answer in a minute is fine. Speed matters for interactive use, not for batch tasks.

The software layer

Several applications now make this straightforward, handling download, quantisation selection and serving. They vary in how much they phone home.

Check the network behaviour. A local model served by an application that logs prompts is not the privacy position you were seeking. Read what the application does before assuming the model being local is sufficient.

Testing before you upgrade

Before spending money, establish what the constraint actually is on the machine you have.

Run a model that fits comfortably and note the tokens per second. This is your baseline for what the machine can do.

Run one that barely fits and watch memory pressure. If the system starts swapping, you have found the ceiling, and it is a memory ceiling rather than a speed one.

Time the same task on a hosted assistant for comparison. Frequently the local model is adequate and the wait is the only difference, which changes the purchasing question entirely.

Try a heavily quantised larger model against a lightly quantised smaller one on your own tasks. The result is not obvious and it determines which direction to spend.

Only then decide. Most people who conclude they need new hardware have not established whether the bottleneck is memory, bandwidth or the model choice, and the wrong upgrade is expensive and unhelpful.