How Large Language Models Actually Work
A large language model is, at its core, a machine for predicting the next token. That single sentence, taken seriously, explains most of what these systems do well, most of how they fail, and most of how you should work with them.
Tokens: the atoms of text
Models do not read characters or words; they read tokens — chunks of text, often a syllable or a short word, drawn from a fixed vocabulary of tens of thousands. English averages roughly 0.75 words per token. Tokenization explains odd behaviors: models are famously shaky at counting letters in a word, because letters are not the units they see.
Prediction as the training task
During training, the model reads enormous amounts of text and, at every position, tries to predict what token comes next. Errors adjust billions of internal weights, slightly, trillions of times. To predict text well, the model is forced to internalize grammar, facts, styles, reasoning patterns, and a great deal of the structure of the world that produced the text. Prediction is the task; understanding — in a functional sense — is the side effect.
Attention: reading with context
The transformer's key mechanism is attention: when processing each token, the model looks back over the tokens in its context window and weighs which ones matter for predicting the next. Attention is why a model can resolve "it" to the right noun three sentences back, and why everything the model can use must fit inside the context window — the window is its entire working memory.
Generation: one token at a time
To answer you, the model predicts a token, appends it to the context, and predicts again. Nothing is planned in advance in the way a human outlines an essay; coherence emerges because each new token is conditioned on everything before it. This is also why errors compound: an early wrong turn becomes context, and the model tends to continue consistently with its own mistakes rather than correct them.
What this predicts about behavior
- Fluency outruns truth. The training objective rewards plausible text. Facts are recalled well when they were common in training, and invented confidently when they were not.
- Context is king. Examples and relevant documents placed in the window influence output more than any instruction phrasing.
- Longer is not smarter. A bigger context window holds more, but attention over very long inputs is uneven — material in the middle is used least reliably.
Hold onto the prediction frame and most "prompt engineering" stops being folklore: you are arranging context so that the most probable continuation is the answer you want.