How LLMs Actually Work: The Mental Model Every AI Engineer Needs

Every article on this site — agents, RAG, prompt engineering, evals — assumes you have a working model of what an LLM actually does under the hood. Most people don’t, and it’s not their fault: the marketing describes these systems as “understanding” and “reasoning,” which is true enough to be useful and wrong enough to cause real bugs. Here’s the mental model that actually holds up when you’re building something.

A language model predicts the next token, over and over

Strip away the framing and an LLM does one thing: given a sequence of tokens, predict a probability distribution over what token comes next. Generation is that prediction repeated — pick a token, append it, predict again — until a stop condition is hit. There’s no separate “thinking” step and no persistent internal state between calls. Everything the model “knows” about the conversation so far is re-derived from the token sequence every single time.

That single fact explains a surprising amount of downstream behavior: why models can contradict themselves paragraphs apart, why the order you present information in changes the output, and why “just tell it to remember” doesn’t work across separate API calls — there’s nothing to remember into.

Tokens are not words

Text is broken into tokens — subword chunks — before any prediction happens. Common words are usually one token; rarer words, numbers, and code get split into pieces:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
print(enc.encode("Debugging tokenization"))
# [86058, 2408, 2065]  — three tokens for two words

print(enc.encode("supercalifragilisticexpialidocious"))
# 10 tokens — the model has never seen this word as a single unit

This matters practically in three places. First, cost and rate limits are metered in tokens, not characters or words — roughly 4 characters per token for English prose, worse for code and non-English text. Second, models are measurably worse at exact character-level tasks (count the letters in a word, reverse a string) because they don’t see individual characters — they see token IDs. Third, oddly-tokenized input — a typo, unusual formatting, a rare proper noun — can degrade output quality in ways that look like “the model is being dumb” but are really “the tokenizer chopped this into unfamiliar pieces.”

The context window is working memory, not a filing cabinet

The context window is the maximum number of tokens the model can attend to in a single call — prompt plus conversation history plus the response being generated. Two things follow from this that trip people up constantly:

• Nothing persists outside it. A model has no memory of your last API call unless you re-send the relevant history as part of the next prompt. “Memory” in a chat product is an engineering layer outside the model, re-injecting prior turns.
• Bigger isn’t free. Attention cost scales with context length, and quality doesn’t stay flat as you fill the window — models reliably attend better to information near the start and end of a long context than to information buried in the middle. Stuffing everything you might need into the prompt is not the same as the model actually using all of it well.

If you’re managing a long agent session or a big RAG context, this is the underlying reason curation beats volume — see Managing Context in Long AI Coding Sessions for the practical version of this problem.

Sampling is why the same prompt gives different answers

Given the probability distribution over the next token, the model doesn’t always pick the single most likely one. A temperature parameter reshapes how sharply that distribution favors the top candidates:

| Temperature | Behavior | Use case | |---|---|---| | 0 | Pick the argmax every time — as close to deterministic as it gets | Extraction, classification, code generation | | 0.7 (typical default) | Sample proportionally, favoring likely tokens but allowing variety | Conversational, creative writing | | 1.5+ | Flatten the distribution — increasingly unlikely tokens get picked | Rarely useful; mostly produces noise |

A subtlety worth knowing: temperature 0 is close to deterministic, not guaranteed to be. Floating-point operations aren’t perfectly associative under concurrent batched execution, and some providers route requests across different hardware or mixture-of-experts paths run — so you can still see occasional variation even at temperature 0. If you need true reproducibility for testing, pin the model version and treat “mostly the same” as the realistic guarantee, not “always identical.”

This one surprised me — Josh: I once pinned temperature to 0 and wrote a snapshot test asserting the model’s exact output, assuming 0 meant deterministic. It passed locally for days, then started flickering red in CI with no code change I could point to. It took me embarrassingly long to accept what this section says — batched execution and hardware routing make temp 0 close to deterministic, not identical. I rewrote the test to assert on the output’s structure, not the exact string, and it’s been stable since.

Hallucination is the default, not a bug

Because generation is next-token prediction over a learned distribution, there is no built-in mechanism for the model to know what it doesn’t know. A confident-sounding sentence about a real API and a confident-sounding sentence about a nonexistent one are produced by the exact same process — the model isn’t distinguishing “verified fact” from “plausible continuation.” It’s why hallucination shows up as convincingly formatted, syntactically perfect code calling methods that don’t exist: the model is doing exactly what it’s built to do, extending the pattern, without a ground-truth check.

This is also why “just ask it to be more careful” is a weak fix and grounding techniques are a real one. If the model needs facts it wasn’t trained on or can’t reliably recall, retrieval is what closes that gap — see RAG vs Fine-Tuning for how to actually make that call.

The mental model in one paragraph

An LLM turns your prompt into tokens, predicts a probability distribution over the next token conditioned on everything in the context window, samples from that distribution according to your temperature setting, appends the result, and repeats — with no memory outside the context you provide and no built-in way to flag its own uncertainty. Every technique covered elsewhere on this site — prompting patterns, RAG, agent loops, context management — is a way of working with that process, not around it.