The difference between "thinking" and predicting the next word
A language model can write complex code, solve differential equations and discuss Borges's influence on contemporary literature…
A language model can write complex code, solve differential equations and discuss Borges's influence on contemporary literature. But show it a simple visual pattern it has not seen before and ask it to complete it, and the system fails.
Why can something that has mastered the syntax of programming not solve puzzles that an eight-year-old figures out in minutes? The answer is not a lack of knowledge but the architecture of how it processes. As I explained in an earlier post, LLMs predict the next word based on the geometry of all the previous ones. That is statistical anticipation, not "logical deliberation": masters of prediction rather than navigators with a compass.
The mirage of reasoning
The 2022 publication of a paper titled "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" was a turning point. Its premise was simple: if we ask a model to "think step by step" before answering, its performance on logic problems improves sharply. That made it possible to solve math problems it used to fail. But an AI researcher, François Chollet, also drew an important distinction: showing reasoning steps is not the same as reasoning.
At a fundamental level, today's LLMs work as stores of memorized templates of immense capacity. When you ask one how to solve a linear equation, it does not deduce the answer from first principles. It retrieves a probabilistic template it saw thousands of times during training and runs it. It works brilliantly as long as the problem is familiar. But when the task is genuinely new, something for which it has no stored template, the system hits its limit.
Interpolation vs. extrapolation
Chollet created a benchmark called ARC (Abstraction and Reasoning Corpus) to measure exactly this difference: the ability to solve new problems from very few examples. Humans solve about 80% of those problems, and the best current models solve 31%. That figure will most likely keep rising in the coming months.
The reason for the gap is technical, and deep. Models based on Transformers are "interpolation engines." They can navigate inside the space of the known (their training data) and find midpoints with great precision. But real reasoning requires extrapolation: the ability to take known concepts and apply them in entirely unfamiliar situations.
This is where the lack of a body and physical experience shows. With no "grounding" in the real world, models struggle to grasp basic principles of causality that are intuitive to a human.
Beyond prediction
Recent research suggests that even when models seem to reason, they are often applying superficial heuristics. A 2024 study of performance on ARC showed that models tend to fail when irrelevant variables or minor changes are introduced that should not affect the underlying logic. That shows they are not following robust thinking but replicating the "form" of a logical answer.
What looks like a criticism is really a necessary distinction: LLMs operate in a different cognitive regime from humans, and understanding that lets us design better systems. The near future probably lies not only in building bigger models but in hybrid architectures.
We are seeing the emergence of systems that combine the linguistic fluency of LLMs with modules for symbolic reasoning, program search or formal verification. The goal is ambitious: to finally join statistical prediction with the structure of logical thought. For now, thinking in the strict sense, synthesizing new solutions in the face of the unknown, remains human territory.
Related reading on my blog:
- From words to numbers: how AI builds meaning
- The illusion of continuity: how AI weaves memory without having a past
- Brains in jars, grounding, or why AI needs a body
References
- Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903.
- Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547.
- Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.
- Hu, S., et al. (2024). Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus. arXiv:2403.11793.