Running Language Models Locally: A Beginner's Guide
Running a language model on your own machine buys you privacy, offline access, and zero per-token cost. The price is that you become responsible for the arithmetic of fitting a model into your hardware. This guide covers the three numbers that decide what you can run: parameters, quantization, and context.
Parameters and memory
A model's size is quoted in parameters — 7B, 13B, 70B. At full precision each parameter costs 2 bytes (FP16), so a 7B model needs about 14 GB just for weights. Nobody runs full precision locally. Quantization stores weights at lower precision: at 4 bits per parameter (Q4), that same 7B model needs roughly 4 GB, with a modest quality cost that most users cannot detect in chat. Rule of thumb: budget about 0.6 GB per billion parameters at Q4, plus overhead.
The KV cache: context is not free
Beyond weights, the model keeps a working memory of the conversation — the key-value cache — which grows with context length. On smaller models it is a few hundred megabytes; on large models at long contexts it can exceed the weights themselves. When a tool says a model "supports 128K context," that is a capability, not a promise that your RAM survives it. Start at 8K–32K and grow only if your tasks need it.
Choosing a model size for your machine
- 8 GB RAM/VRAM: 3B–8B models at Q4. Capable for chat, summaries, and light coding help.
- 16 GB: 8B–14B comfortably; the current sweet spot for quality per dollar.
- 24–32 GB: 20B–32B models, including mixture-of-experts designs whose active parameters are far fewer than their total.
- 64 GB+: 70B-class models at Q4 become practical, and noticeably stronger.
Tools of the trade
Most people start with a one-click runner (Ollama or LM Studio) and the GGUF model format, which is built for CPU/GPU hybrid inference on ordinary hardware. Apple Silicon machines are unusually good at this because CPU and GPU share one pool of memory — the unified memory is the VRAM. Whatever you choose, download models from their official repositories; model files are executables in every way that matters.
Setting expectations
A local 14B model is not a frontier model, and no setting makes it one. What it is: instant, private, free to run, and genuinely good at drafting, summarizing, transforming text, and answering questions about documents you give it. For many daily tasks, that is the right trade.