Running Language Models Locally: A Beginner's Guide

Tutorial · Local AI · 9 minute read

Running a language model on your own machine buys you privacy, offline access, and zero per-token cost. The price is that you become responsible for the arithmetic of fitting a model into your hardware. This guide covers the three numbers that decide what you can run: parameters, quantization, and context.

Parameters and memory

A model's size is quoted in parameters — 7B, 13B, 70B. At full precision each parameter costs 2 bytes (FP16), so a 7B model needs about 14 GB just for weights. Nobody runs full precision locally. Quantization stores weights at lower precision: at 4 bits per parameter (Q4), that same 7B model needs roughly 4 GB, with a modest quality cost that most users cannot detect in chat. Rule of thumb: budget about 0.6 GB per billion parameters at Q4, plus overhead.

The KV cache: context is not free

Beyond weights, the model keeps a working memory of the conversation — the key-value cache — which grows with context length. On smaller models it is a few hundred megabytes; on large models at long contexts it can exceed the weights themselves. When a tool says a model "supports 128K context," that is a capability, not a promise that your RAM survives it. Start at 8K–32K and grow only if your tasks need it.

Choosing a model size for your machine

Tools of the trade

Most people start with a one-click runner (Ollama or LM Studio) and the GGUF model format, which is built for CPU/GPU hybrid inference on ordinary hardware. Apple Silicon machines are unusually good at this because CPU and GPU share one pool of memory — the unified memory is the VRAM. Whatever you choose, download models from their official repositories; model files are executables in every way that matters.

Setting expectations

A local 14B model is not a frontier model, and no setting makes it one. What it is: instant, private, free to run, and genuinely good at drafting, summarizing, transforming text, and answering questions about documents you give it. For many daily tasks, that is the right trade.