What a Language Model Is Doing

Almost every complaint about AI tools — it made something up, it forgot what I said, it gave a different answer the second time — follows directly from how the thing works. Ten minutes on the mechanism prevents a year of confusion.

The core operation

A language model predicts what text plausibly comes next, given the text so far. That is the entire operation. Everything else is built on top of it.

It does this by having been trained on an enormous quantity of text, adjusting internal parameters until its predictions matched what actually followed. The result is not a store of facts. It is a very large set of numbers encoding statistical relationships between fragments of language.

When you ask a question, the model is not looking anything up. It is generating a plausible continuation of a conversation in which someone asked that question. That the continuation is frequently correct is a consequence of correct text being common in the training data, not of any retrieval step.

What follows from this

It has no sense of true and false. It has a sense of likely. These correlate strongly, because accurate statements are more common in text than inaccurate ones on most subjects. They diverge exactly where the training data was thin, contradictory, or where the plausible-sounding answer differs from the correct one.

It cannot tell you what it does not know. There is no internal flag distinguishing a confident recollection from a fluent invention. The same process produces both, which is why a fabricated citation is delivered in the same tone as a real one.

The same question can produce different answers. Generation involves sampling from a probability distribution. Most tools introduce deliberate randomness because it produces more natural text. Turning it down makes output more repetitive rather than more accurate.

It is strongest where language patterns carry the answer — summarising, rephrasing, translating, drafting, explaining a well-documented concept, transforming text from one format to another.

It is weakest where the answer requires a fact that is specific, recent, or rare. Exact figures, precise dates, obscure people, current events, anything where being nearly right is being wrong.

Why it seems to reason

Models produce chains of steps that look like reasoning, and this genuinely improves accuracy on multi-step problems. Working through steps produces text that constrains the next prediction more tightly than jumping to a conclusion.

Whether that constitutes reasoning is a philosophical argument. What matters practically is that the visible steps are not necessarily how the answer was produced. The model generates a plausible explanation alongside a plausible answer. When the answer is wrong, the explanation frequently justifies it convincingly.

Do not treat a model's account of its own reasoning as an audit trail.

What is bolted on

Most of what people use is not the raw model.

Retrieval — the tool searches, then hands the model the results to work from. This grounds answers in real sources and is why a tool with search is more reliable on current facts.

Tools and function calling — the model can invoke a calculator, run code, query a database. This offloads the tasks it is bad at to systems that are good at them.

A system prompt — instructions the operator sets before your message, shaping tone, refusals and behaviour.

Fine-tuning and preference training — adjusting the base model to be helpful and to follow instructions. This is what turns a text predictor into an assistant.

When comparing products, much of the difference is in this layer rather than in the underlying model.

The practical consequences

Verify anything specific. Numbers, names, dates, citations, quotations, legal and medical claims. The model's confidence carries no information about its accuracy.

Prefer tools that show sources, and check that the source says what the summary claims. Retrieval reduces fabrication; it does not eliminate misreading.

Give it material rather than asking it to recall. Pasting a document and asking questions about it is far more reliable than asking what a document says.

Ask for the answer in a form you can check. A list of steps you can verify beats a conclusion you must trust.

Treat it as a first draft generator and a thinking partner, not an authority. That is where the mechanism actually delivers.

What it is not

Not a search engine. Search retrieves documents that exist. A model generates text that might not correspond to anything.

Not a database. Nothing is stored as a retrievable record.

Not deterministic. Same input, different output.

Not aware of its own limits. It does not know what it does not know, and it will not tell you.

Understanding those five sentences puts you ahead of most people using these tools daily.

Where to put the boundary in practice

A working rule that sorts most tasks correctly.

Give it material, ask for transformation. Summarise this, restructure this, explain this passage, translate this, find the contradictions in these three documents. The answer is in what you supplied, and the model is doing reading comprehension. This is where it is most reliable.

Ask it to recall, and verify everything. What is the population of X, who wrote Y, what does regulation Z require. The answer comes from statistical patterns in training data, and it may be right, and you cannot tell from the output.

Ask it to reason about something you supplied, and check the steps. Here it is doing real work and it can go wrong in the middle of a plausible chain.

Ask it to invent, and judge the result yourself. Names, options, framings, first drafts. There is no correct answer, so there is nothing to fabricate.

The reliability ordering is consistent: transformation of supplied material first, invention second, reasoning third, recall last. Most disappointment comes from using it at the recall end without realising that is what you are doing.