AI models flub these intelligence tests. Can you fare any better?
2026-08-26 · MIT Technology Review
Why puzzles remain a core benchmark for AI
Puzzles and games have been used to evaluate AI since the field’s earliest days. The article traces this lineage back to 1959, when IBM scientist Arthur Samuel popularized the term “machine learning” in work on a checkers-playing algorithm. Chess and Go later became iconic test beds as well.
By puzzle performance alone, AI has improved rapidly. A Columbia University team reported that in late 2024, even top models solved only 18% of New York Times *Connections* puzzles; by early 2025, some models were solving them almost perfectly. Still, the article argues that puzzles are valuable not just for showing progress, but for exposing where models succeed, where they fail, and where humans still hold advantages.
Fast gains, uneven capabilities
Despite broad progress, current models continue to fumble in specific domains:
- Spatial reasoning, especially mental rotation;
- Adaptation to subtle variations in familiar logic formats;
- Abstract visual rule induction, including 2D grid tasks.
These are not random misses. They reveal systematic differences between machine and human cognition.
Spatial reasoning: a persistent human edge
The piece starts with a domain where humans still excel: spatial reasoning. Mental rotation tasks ask whether differently oriented images represent the same object. Even multimodal language models, which can process images, often perform poorly here.
The broader implication is that, despite discussion around “world models,” LLMs still seem limited in mentally manipulating 3D structures the way strong human spatial thinkers—such as architects or mechanical engineers—can.
Memory strength can become a weakness
Frontier LLMs have enormous factual memory from training, which helps in trivia-like settings. But this can backfire when a new puzzle closely resembles one seen before. Models may retrieve a memorized pattern and miss small but crucial differences.
The article cites a 2024 study by researchers from Google and the University of Illinois Urbana-Champaign using variants of Knights and Knaves puzzles (where knights always tell the truth and knaves always lie). Slight modifications were enough to trip models.
A similar pattern appears in SimpleBench: questions may look like complex problems, but often hinge on noticing a simple trick. Humans frequently catch that trick; even top models can stumble.
Abstract and visual reasoning limits in ARC-AGI
AI struggles are not limited to 3D perception. Two-dimensional visual abstraction can also be difficult, which is central to performance on ARC-AGI, a well-known puzzle benchmark.
ARC-AGI tasks require inferring general transformation rules from example grids and applying them to a new case. The article notes:
- Models often do better when grids are provided as numeric color encodings rather than raw images.
- Even correct model answers may rely on convoluted, non-generalizable heuristics.
- Humans tend to use simpler, more transferable visual concepts.
Although model performance on ARC-AGI has improved significantly over the past year, some puzzles still resist current systems.
The article’s core message
The puzzle set is presented as a practical comparison between human and machine reasoning. The takeaway is nuanced:
1. AI is advancing quickly on many puzzle benchmarks.
2. Progress is uneven, with clear blind spots.
3. Humans still show strong advantages in spatial intuition, sensitivity to subtle changes, and compact rule abstraction.
In short, puzzles function both as a scoreboard for AI progress and as a diagnostic tool for its limitations. Even when models approach near-perfect scores on some tasks, fundamental differences in reasoning style remain visible.