Every puzzle on dxc4.com shows a rating, such as “Predicted ~1300”, and nobody has played any of them yet. The number comes from a model I trained to guess how hard a puzzle is before anyone tries it, and its best clue turned out to be a program that predicts what human players would play.
A puzzle rating works like a player’s rating. On Lichess, the free chess site, each puzzle has its own rating (Glicko-2, the same system as for players), and every attempt at a puzzle counts as a game between you and the puzzle. Solve it and the puzzle loses points, fail it and the puzzle gains them. After enough attempts the rating settles, and a higher number means a harder puzzle.
A new puzzle has had no attempts, so it has no rating. That was the problem with the puzzles I wrote about in 1,010 Checkmate Puzzles, Each One Proven: every one was proven correct, and none of them came with a fair idea of how hard it was.
The First Guess
The forge started with a formula I wrote by hand. It gave a puzzle points for a longer mate, for a quiet first move or a sacrifice, for each defence Black has, and for a first move Stockfish misses when it looks only a short way ahead.
I checked it against 6,000 Lichess puzzles whose ratings are known. On a typical puzzle it was off by about 870 points, and the order it put them in barely matched the order Lichess players had put them in.
Learning From Lichess
Lichess publishes its whole puzzle collection for anyone to use: 6.16 million puzzles, each with its rating. I kept the 4.04 million that had been played at least 100 times, so their ratings had settled, and trained on about 400,000 of them. A tenth stayed out of training, so I could test the model on puzzles it had never seen.
None of those puzzles is on dxc4. The forge makes its own, and it keeps a list of every Lichess puzzle’s starting position so it can refuse any that match.
What Makes a Chess Puzzle Hard
The model is a common kind that learns from examples (LightGBM, for anyone who wants the name). It reads 132 numbers about each puzzle. Some describe the position: how much material each side has, where the kings stand, whether the first move is a check, a capture or a quiet move. Others come from Stockfish, the strongest chess engine: how much better the right move is than the next best, and how long the solution runs.
The most useful numbers come from Maia-3, a program from CSSLab at the University of Toronto that plays like a person rather than like an engine. Given a position, it predicts the move a player of a given rating would actually choose. The forge asks it about seven players, rated 800 to 2600, and notes the lowest rating at which the right move is the one Maia expects. A puzzle whose answer an 800 player would play by instinct is easy. One that only a 2300 player would see is hard.
A small extra network looks at the board as a picture and makes its own guess, which the main model reads as one more number.
On the puzzles it had never seen, the model was off by about 210 points on a typical puzzle, and its order matched the Lichess order closely.
I also trained it again with each group of numbers left out. Without Maia the typical error grew by 32 points, without the picture network by 26, and without Stockfish by 4. Stockfish knows the answer to every puzzle, so it has no sense of how hard that answer is to see. Maia, which plays like a person, does.
Others have found the same. Predicting a Lichess puzzle’s rating was the IEEE Big Data Cup challenge in 2024, and FedCSIS ran a second edition in 2025 on about 4.55 million Lichess puzzles. At least one entry there built its features on Maia too.
Puzzles That Look Different
A model like this is only reliable on puzzles that resemble the ones it learned from, and my first puzzles did not. I trained a second model to tell forge puzzles from Lichess puzzles, and it almost never got it wrong. Lichess puzzles come from real games, where the side to move is never ahead in material and only one move wins. The forge’s early positions often left White ten points of material up, with several moves that won anyway.
So I changed how the forge builds its positions. It now sets up boards with the material, piece placement and king positions Lichess puzzles have, and it throws out any puzzle where a second first move also wins.
The second model can still tell the two apart. For the mates it is nearly as easy as before, and for the new best-move puzzles it is clearly harder. That is why the site says “Predicted ~1300” and not 1300.
The Build
Python, with LightGBM for the model, PyTorch for Maia and the picture network on the Mac’s graphics chip, and Stockfish 19 for the engine numbers. I built it with Claude Code over two days in October, and the model the site uses was trained on 327,022 puzzles on 25 October. It all runs on my Mac and calls no online service. The forge’s code is private.
Still a Prediction
The ratings say how hard Lichess players would find these puzzles. Whether dxc4’s players agree is the next thing to measure, and it needs solve times that dxc4 does not collect, since what you solve stays in your own browser.
