How Puzzle Difficulty Is Measured: Solve Rates to Dead Ends

Puzzle difficulty is often measured by watching people: a puzzle that most solvers miss is hard, and a rating built from thousands of attempts turns that solve rate into a single number ^[1]. Researchers also use signals hidden in the puzzle itself, such as how many tempting first moves lead nowhere ^[2]. Another is the length of the shortest solution, whose worst case is known exactly for small sliding tile trays ^[3].

Each method sees a different side of difficulty. This article walks through them one by one, shows what each reveals, and explains where each one falls short.

What does a solve rate tell you about a puzzle?

The most direct measure of difficulty is simple. Show a puzzle to many people and count how many get it right. A puzzle that nearly everyone solves is easy, and a puzzle that most people miss is hard.

Large online puzzle collections push this idea much further. On one large chess puzzle platform, the difficulty of every puzzle is expressed as a rating derived from human solve rates ^[1].

The catch is volume. A paper published on 15 October 2025 pointed out that a reliable rating needs thousands of human attempts, which makes it impractical to rate a new puzzle quickly ^[1]. A brand new puzzle has no history at all, so its true difficulty stays unknown until enough people have tried it.

How can a puzzle have a rating, like a player?

The rating idea comes from two-player games. Each attempt is treated like a short match between the solver and the puzzle. When a strong solver fails, the puzzle gains a lot of rating. When a weak solver succeeds, the puzzle loses some.

Over many attempts, the rating of the puzzle drifts toward the level of solvers who succeed about as often as they fail. The beauty of the scheme is that puzzles and people end up on the same scale. A solver can then be offered puzzles close to their own level, neither trivial nor hopeless.

The 2025 paper also reviews an earlier, 2024 edition of the same research competition, in which one entry simulated matches between puzzles to infer their ratings ^[1].

Every rating also comes with a measure of doubt. In the data used by the 2025 paper, each puzzle carried both a rating and a rating deviation, which describes how much the true rating could differ from the estimate ^[1]. A puzzle with few attempts has a wide margin of doubt. A puzzle with many attempts has a narrow one.

Why are ratings from a few attempts misleading?

A rating needs time to settle. With too few attempts, a hard puzzle has not yet climbed to its true level, and an easy one has not yet fallen to its true level. Both end up sitting too close to the average.

The same paper ran straight into this effect. Its test puzzles had only 25 to 50 solving attempts each, far fewer than the training puzzles, and that pulled their ratings toward the mean ^[1]. Without a correction for this shift, the error of the model rose by 38.6 percent ^[1].

The largest corrections were needed at the extremes. The paper notes that the very easy and very hard puzzles are the least stable ones ^[1].

The practical lesson is worth remembering. A rating based on a handful of attempts probably understates how easy or how hard a puzzle really is. Extreme puzzles need the most attempts before their numbers can be trusted.

Can a computer predict difficulty from the puzzle alone?

Yes, at least approximately. Because human ratings are slow to collect, researchers train models to estimate them from the puzzle itself, with the aim of reflecting how people solve rather than how a perfect program would play ^[1].

The same paper describes one such model. It learned from a training set of 4.7 million chess puzzles, each with a known rating and a complete solution ^[1]. The model looks at the board as 64 squares, at the moves of the solution, and at estimates of how often players of different strengths would solve the puzzle ^[1]. Those estimates come from a separate program built to imitate human play ^[1].

Two design choices stand out. First, before learning ratings, the model practised a warm-up task: some squares of the board were hidden, and it had to guess what stood on them. That warm-up alone cut the prediction error by 22.3 percent compared with training from scratch ^[1].

Second, the human-like solve-rate estimates added a great deal. The paper credits much of its improvement over an earlier version to these signals, and concludes that knowledge of human behaviour adds information that the board position alone does not contain ^[1]. Difficulty, in other words, belongs partly to the puzzle and partly to the people solving it.

The paper also made a telling simplification. When the model was taught to find solutions, it looked only at the first move of each solution, on the reasoning that the first move usually holds the key insight while later moves are often forced replies ^[1].

How accurate are these predictions?

Reasonably close, but far from perfect. The model placed third in a 2025 research competition, with a mean squared error of 55.9 thousand on the test puzzles, against 52.3 thousand for the best team ^[1].

Mean squared error is easier to picture after taking its square root. By our own arithmetic, that square root comes to roughly two hundred and thirty-six, so a typical prediction missed the human rating by a couple of hundred rating points.

That is useful for sorting new puzzles into broad bands such as easy, medium and hard. It is not yet a replacement for real attempts by real people.

Do dead ends make a puzzle harder?

They do, and designers use them on purpose. A dead end is a move that follows the rules but can never lead to the goal.

Peg solitaire makes the idea concrete. In this puzzle, pegs hop over neighbours, each hopped peg leaves the board, and you must end with a lone peg ^[2]. A paper published on 9 June 2017 argued that a hard puzzle needs a large number of dead ends, and therefore lots of possible opening jumps ^[2]. Its ideal hard position offers as many opening jumps as possible, with only one of them leading to a solution ^[2].

The search turned up striking examples. On the 33-hole board, one position with 17 pegs offers 17 different first jumps, and only one of them leaves a position that can still be solved ^[2].

The paper contrasts these with its symmetric puzzles, many of which it finds easy to solve by hand ^[2]. The reason is that a few mirrored jumps often shrink a symmetric pattern to a smaller one that has been solved before ^[2].

So one simple measure of difficulty is a ratio: how many moves are possible, and how many of them actually work. The more tempting moves that fail, the more often a solver will go wrong.

Does the number of solutions matter?

It does, and it cuts both ways. Few working moves give the solver many ways to go wrong. Yet a solver who knows that only one opening works can use that knowledge as a clue ^[2].

The peg solitaire paper spells out how. Any jump that was open before the correct opening stays useless afterwards, so the next useful jump must be one that the opening created ^[2]. Following that chain step by step narrows the search sharply.

So a count of working moves is not a pure difficulty score. Telling the solver the count can make the same puzzle easier.

Is a longer solution always a harder puzzle?

Not always, but length is one of the easiest signals to compute. For the sliding tile puzzle, every position of the 3 by 3 tray with 8 tiles can be solved in at most 31 moves, and every position of the 4 by 4 tray in at most 80 ^[3].

A computer can check such numbers with breadth-first search. Starting from the solved tray, it collects all positions reachable in one move, then those needing two, working outward ring by ring until no new position turns up. Our own run of this search on the smaller tray reaches the same ceiling of thirty-one moves.

Here is a worked example. Put the eight tiles in reverse order, from the highest number down to the lowest, and leave the empty square at the lower right. We checked this position by computer: its shortest solution takes thirty moves. That is just one short of the 31-move maximum for this tray ^[3].

Length still misses a lot. Finding the shortest solution is itself hard: for sliding tile puzzles of any size, finding the minimum number of moves has been proved NP-hard, a label for problems with no known fast method ^[3].

The method a solver uses matters too. A paper published on 10 July 2015 showed that a greedy method, which puts tiles in place one at a time, needs about 8/3 times n cubed moves on average for an n by n tray ^[3]. The same puzzle can take very different numbers of moves depending on how it is approached.

Chess puzzles show the flip side. In the data of the 2025 paper, solutions typically ran just 1 to 5 moves ^[1]. A short solution can still be very hard to find when the right first move is hidden among many natural-looking alternatives.

How do the main difficulty measures compare?

Each measure answers a slightly different question, and each has its own blind spot.

Which measure should you trust?

No single number tells the whole story. Solve-rate ratings measure people directly, but they arrive slowly and are least stable for the very easiest and hardest puzzles ^[1]. Structural signals, such as dead ends and solution length, can be worked out before anyone tries the puzzle ^[2], ^[3].

A sensible approach combines both kinds. Structure gives a first estimate, and real attempts correct it over time. Prediction models aim to fill exactly that gap: a usable rating for a new puzzle before thousands of attempts have been collected ^[1].

How can you measure difficulty yourself?

You do not need a computer to use these ideas. Try them on the next riddle you meet.

• Note your first idea. If it was wrong, count how many other first ideas felt just as natural. That count is your own dead-end measure.
• Ask whether the riddle has exactly one answer. If a second answer fits every clue, the riddle is easier to finish but less satisfying.
• Count the steps in your final solution. A long chain of steps usually means a harder riddle, though a short one with a hidden first step can be harder still.
• Compare notes with friends. A riddle that most of them miss is hard, whatever its length.

The riddles below put these signals to the test, each with a single answer to find.