# A strong general-purpose VLM still fails on a chessboard

> We tested GPT-5.4 and Claude Opus 4.7 on reading 72 chess positions into FEN. Claude Opus 4.7 understood the boards and still lost to square-level errors, and the reason only showed up in the reasoning traces.

- Category: Benchmarks
- Published: 2026-05-19 (updated 2026-09-22)
- Authors: Sankalp Nagaonkar
- Canonical: https://videodb.io/blog/claude-chessboard-spatial-reasoning
- HTML: https://videodb.io/blog/claude-chessboard-spatial-reasoning · Markdown: https://videodb.io/blog/claude-chessboard-spatial-reasoning.md
- Tags: claude, vision, vlm, chess, spatial reasoning

---
We were building [Chess Lens](https://github.com/video-db/chess-lens), an educational chess assistant that looks at a board and helps a player think through the position. The plan was to use VideoDB to ingest the board visually, follow the game as it changes, and turn the current position into structured data for teaching, analysis and next-move guidance. None of that works unless the first step is exact. The system has to read the board correctly, every square, every time.

This post is about how we tested that step. We ran GPT-5.4 and Claude Opus 4.7 through the same board-reading task under several thinking configurations. Claude Opus 4.7 trailed the strongest GPT-5.4 runs throughout, and the aggregate scores did not explain why. The explanation was in the reasoning traces and in the individual failures, and it turned out to be a narrow, specific limitation rather than a general one.

## What we measured, and why FEN

Chess Lens needs the current position in FEN, the standard notation for one exact chess position. FEN writes the board as eight rows, from rank 8 down to rank 1. Each row is a string of piece letters and digits: an uppercase letter is a white piece, a lowercase letter is a black piece, and a digit counts consecutive empty squares. A row like `PP3P2` means two white pawns, three empty squares, one white pawn and two empty squares.

We scored only the piece-placement field. The model received a board image and had to return which piece was on which square, in that format. We compared its output with the expected FEN for the same point in the game, and a position counted as correct only if the whole field matched.

That is strict, and it is strict in the way Chess Lens needs. If a pawn is shifted by one file, the position may still read as plausible in prose, but the FEN is wrong, and everything downstream of it is wrong too.

_Figure 1. The task. A board image goes in and the piece-placement field of FEN comes out, one row per rank from rank 8 to rank 1. Uppercase letters are white pieces, lowercase letters are black pieces, and a digit counts empty squares, so the row <code>PP3P2</code> describes two pawns, three empty squares, a pawn and two empty squares._

## How the test was set up

We took one game and evaluated 72 board positions from it. For each position the model had to work out the board orientation, identify every visible piece, assign each piece to its square, and emit the piece-placement field.

The score is exact-match accuracy: the number of positions where the returned field matched the expected FEN, out of 72. We ran both models under several thinking configurations, from the default with no extended thinking up to the highest settings each model offers. The configuration names below follow the API settings we used: the thinking level, and whether the run returned a reasoning summary.

## The token budget changed the result

Our first run capped the output at 1024 tokens. That turned out to be too low for the high-reasoning configurations. Some runs spent the whole budget thinking and never produced the final structured answer, which the scorer counts as a parse error.

| Run | Accuracy | Exact / Eval | Notes |
|---|---:|---:|---|
| GPT-5.4 low summary | 100.00% | 72 / 72 | best overall |
| GPT-5.4 default | 93.06% | 67 / 72 | no reasoning summary |
| Claude Opus 4.7 xhigh summary | 76.39% | 55 / 72 | 6 parse errors |
| Claude Opus 4.7 high summary | 69.44% | 50 / 72 | 4 parse errors |
| Claude Opus 4.7 default | 66.67% | 48 / 72 | 1 parse error |
| GPT-5.4 xhigh summary | 4.17% | 3 / 72 | 69 parse errors |
| Claude Opus 4.7 max summary | 2.78% | 2 / 72 | 70 parse errors |

The very low scores for GPT-5.4 xhigh and Claude Opus 4.7 max are truncation failures, not vision failures. Those runs used too many tokens and usually did not reach the answer.

We raised the limit to 4096 tokens and ran everything again.

| Run | Accuracy | Notes |
|---|---:|---|
| GPT-5.4 low summary | 100.00% | best overall |
| GPT-5.4 default | 97.22% | strong, no summaries |
| GPT-5.4 xhigh summary | 94.44% | strong but expensive, still some parse errors |
| Claude Opus 4.7 max summary | 84.72% | best Claude Opus 4.7 run |
| Claude Opus 4.7 high summary | 80.56% | no parse errors, still mapping mistakes |
| Claude Opus 4.7 xhigh summary | 79.17% | similar to high |
| Claude Opus 4.7 default | 63.89% | weaker without thinking |
| Claude Opus 4.7 medium summary | 58.33% | worst Claude Opus 4.7 config |

_Figure 2. Exact-match accuracy on the 72 positions at the two output limits, rows in the order of the 4096-token results. At 1024 tokens the two highest thinking settings collapse because they run out of budget before the answer. At 4096 tokens every configuration completes, and the ranking that remains is the real one. The tables above carry the exact figures._

With the larger budget, Claude Opus 4.7 improved substantially, most of all at max thinking, where it went from 2.78% to 84.72%. It still did not catch up with GPT-5.4, whose best runs were at or near 100%. The gap survived the fix, so the token budget was not the explanation.

## Why the aggregate scores hid the problem

At that point we stopped looking at the totals and read the intermediate reasoning summaries and the failed outputs instead.

GPT-5.4's summaries were procedural and row-specific. They walked the board rank by rank:

> For row 8, I see: a8 has a black rook, f8 has a black rook, h8 has a black king...

> Moving to rank 2, I see the white pawns at a2, b2, c2, with gaps where pieces have moved...

Claude Opus 4.7's summaries were more often global descriptions of the position:

> I'm looking at a chess board layout with pieces positioned across the rows, showing what appears to be a mid-game or puzzle position...

> Black has pawns scattered across the board with the king on f6, white has a rook on d1, a king on e3, and a few pawns positioned strategically.

That difference explains a lot of the remaining errors. Claude Opus 4.7 usually understood the board as a chess position, with the right pieces and the right general shape, but it was less reliable at preserving the exact square-by-square layout when writing it down.

## What the failures looked like

We went through the failed positions by hand. Most errors were local. A piece shifted by one file. One extra or one missing empty square in a row. A gap left by a moved pawn that was not recorded. The right piece family on the wrong square. The right piece with the wrong color, which in FEN is the wrong case. Three examples:

| Expected | Claude Opus 4.7 Output | What Went Wrong |
|---|---|---|
| `p1b1pk2` | `p2b1pk1` | empty-square counts shifted |
| `PP3P2` | `PP4P1` | pawn moved one file over |
| `1BNP1N1P` | `1BNB1N1P` | wrong piece at a specific square |

_Figure 3. One square off. The expected row <code>PP3P2</code> and the returned row <code>PP4P1</code> contain the same three pawns, but the third pawn has moved from the f-file to the g-file. Both rows are valid FEN, both add up to eight squares, and only one is the position on the board._

This pattern matters because of how FEN compresses a row. The piece count can be right, the row can add up to eight squares, and the string can still describe a different position. Claude Opus 4.7 was not failing at chess understanding. It was failing at precise spatial localization, and FEN punishes exactly that.

## What the model card already says

The result lines up with Anthropic's own [vision documentation](https://docs.anthropic.com/en/docs/build-with-claude/vision):

> Spatial reasoning: Claude's spatial reasoning abilities are limited. It may struggle with tasks requiring precise localization or layouts, like reading an analog clock face or describing exact positions of chess pieces.

FEN generation is that task in its purest form. It requires exact localization across 64 squares, and a model that recognizes the board but places one piece one file over is wrong.

## What this means for downstream vision tasks

Chess Lens exposed a narrow but important failure mode. Understanding the board is not enough when the output format requires exact coordinates. Claude Opus 4.7's outputs were often reasonable at the level of the position, but FEN is scored at the level of the square. One shifted piece changes the board state, one wrong empty-square count changes the row, and a model can be directionally right and still fail the task.

The practical lesson for us was that downstream tasks need their own evaluations. The model that performs best in general is not automatically the best model for the specific step you are trying to automate, and the only way to know is to test that step directly, with a scorer as strict as the task.

## Cite this work

S Nagaonkar, "A strong general-purpose VLM still fails on a chessboard", VideoDB Labs, May 2026.

```bibtex
@article{nagaonkar2026strong,
  author = {S Nagaonkar},
  title = {A strong general-purpose VLM still fails on a chessboard},
  journal = {VideoDB Labs},
  year = {2026},
  month = {may},
  note = {https://videodb.io/blog/claude-chessboard-spatial-reasoning},
}
```