Running Models Locally

Running a model on your own machine used to be a hobbyist exercise producing output too weak to use. That is no longer true for a substantial range of tasks, and the reason is that small models improved faster than large ones.

What you actually get

Privacy that is structural rather than contractual. The text does not leave the machine. For confidential documents, client material, medical notes, unpublished work or anything under a duty of confidentiality, this is a different category of assurance from a policy promise.

No per-use cost. Electricity and hardware. At high volume this matters; at low volume it does not.

Offline operation.

Stability. The model does not change under you. Behaviour today is behaviour next year.

No rate limits, no quota, no service outage.

What you give up

Capability, at the top end. The strongest hosted models remain ahead. The gap is smaller than it was and it is real for the hardest tasks.

Speed on modest hardware. A model that fits a laptop runs at laptop speed. This is the difference people notice first.

Convenience. Setup, updates, model management.

The surrounding features. Search, code execution, file parsing, image handling. Some local applications provide these; most provide less.

Where local is genuinely good enough now

Summarising and rewriting your own documents.

Drafting where you will edit anyway.

Extraction — pulling structured data out of text.

Classification and tagging.

Translation for comprehension rather than publication.

Explanation of concepts within the model's knowledge.

Coding assistance for common languages and routine tasks.

For this list, a mid-sized model on decent consumer hardware produces output that is not obviously worse than a hosted assistant, and the privacy difference is absolute.

Where it is not

The hardest reasoning and coding problems.

Anything needing current information, unless you add retrieval yourself.

Long context. Local models handle large inputs less well and slower.

Niche knowledge, where the smaller model simply learned less.

Hardware, practically

Memory is the constraint. The model must fit, and the surrounding software needs room too.

Around 16 GB runs small models comfortably and mid-sized quantised ones with care.

Around 32 GB is the point where the useful range opens up.

Unified memory architectures are unusually well suited, because the model can use system memory rather than being confined to graphics card memory. A laptop with a large unified memory pool outperforms a desktop with a modest discrete card for this specific purpose.

A discrete graphics card with substantial memory is the fastest option where you have one.

Quantisation — reduced-precision weights — is what makes this feasible. Moderate quantisation costs little quality and cuts memory requirements substantially. Aggressive quantisation degrades noticeably.

The wrapper caveat

A local model served by an application that logs prompts, syncs history or contacts a server is not the privacy position you thought you had.

Check what the application does, not just what the model is. Read the network behaviour if you can, or use a tool with a clear statement and a good reputation. The model being local is necessary and not sufficient.

The mixed arrangement most people end up with

Local for anything confidential. Client documents, personal material, work under NDA.

Hosted for everything else, where capability and convenience matter and the content is not sensitive.

This is not a compromise so much as the correct answer. The two have different strengths and the sorting criterion — is this confidential — is one you can apply in a second.

Worth trying even if you do not switch

Downloading a model and running it once tells you more about how these systems work than any amount of reading. You see the memory use, the speed, the effect of quantisation, and the quality gap in your own tasks rather than in a benchmark.

That is worth an afternoon regardless of what you end up using daily.

Trying it before deciding

The whole question resolves quickly if you actually run one, and most people arguing about it have not.

Download a small model first, in the low single-digit billions of parameters. It will run on almost anything and it tells you whether the workflow suits you.

Run your own five tasks through it, the same set you would use to compare hosted assistants.

Then try one size larger, and notice where the quality step is for your work. It is frequently smaller than expected — several tasks are handled adequately by models much smaller than people assume.

Measure the speed on your machine, not from a specification. Tokens per second determines whether the experience is usable for conversation or only for batch work.

Check the memory pressure while it runs. If the system is swapping, the model is too large for the machine regardless of what the specification suggested.

An afternoon settles the question with evidence about your hardware and your tasks, which no comparison article can provide.