6 minutes
Training a Chess Reasoning Agent
TLDR: I trained an LLM reasoning agent to play chess, with model sizes 0.8B, 2B and 9B. The agent calls tools which help it keep track of the board state, which enables good performance with a relatively small number of parameters.
The demo code is available on Github.
Background
I became excited about the power of RLVR after the Deepseek R1 paper came out. They demonstrated that through the use of “thinking” tokens, an LLM could solve verifiable tasks like math problems. Additionally, they described the necessity of bootstrapping the thinking process with supervised examples, which prevents the model from generating a nonsense chain of thought.
Chess is an ideal candidate for a verifiable rewards setup. You could call it “fractally verifiable”: Not only are games themselves verifiable outcomes, but the strength of individual moves is verifiable, and even the individual claims used to justify those moves can be verified.
The majority of prior work in chess engines has had the sole purpose of maximizing playing strength. I’ve written a longer history of chess engines here and here. This would be a much different type of engine: we should favor explainable, intuitive moves over the absolute best moves.
Training procedure
The general procedure for training a reasoning model is Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL). In my case, I used GRPO with Never Give Up for the RL stage.
In the SFT stage, I had originally been using a LoRA with rank 16, but I found that using rank 64 was actually much better. (Similar to the LoRA Without Regret paper, however, I did not see similar benefits in RL, so I kept LoRA rank at 16 there.) I also found that the SFT stage is very data-hungry, and training for one epoch over 30k samples was much better than three epochs over 10k samples.
The Joys of Reinforcement Learning
I spent a lot of time getting GRPO wrong before I got it right. Originally I used a reward proportional to the ratio in win probability between the chosen move and the best move win_prob(chosen) / win_prob(best_move). This made some sense since it would punish blunders and reward good moves. However, this prevented the model from caring about the quality of its moves in positions which were already won or lost. For example, a large mistake which reduced win_prob from 99% to 80% received a reward of around 0.8. A small inaccuracy which drops win_prob from 0.5 to 0.4 in a drawn position would get the same credit assignment.
After that, I briefly tried to use the difference max(0, 1 + win_prob(chosen) - win_prob(best)) which is closer to how Lichess scores inaccuracies, mistakes and blunders, but this formulation had a similar issue.
Since win probability wasn’t getting anywhere, I moved to a new reward simply based on CPL, which I thought would help to sharpen the difference between good moves and best moves: e^(-CPL/temperature) with temperature=100 determined empirically from the CPL distribution of legal moves. With this new reward, a move with CPL 50 received a reward of 0.61, a move with CPL 100 received a reward of 0.37, and CPL of 200 received only 0.14.
Dynamics of GRPO
GRPO itself has interesting behavior, which we can see from the hit@1 and hit@4 metrics, which indicate how often the model’s move was a hit on the first attempt and any of four attempts, respectively. Typical training behavior is for GRPO to cause dramatic improvements in hit@1 vs SFT, but for hit@4 to be close to the same at the beginning and end of GRPO.
During the entire RL run, we are fighting the model’s natural “cooling and hardening” process, known as entropy collapse. As training progresses, the diversity of moves that the model makes decreases. This is desirable on positions where the model is making good moves, since it will be less likely to make bad moves. However, it also makes it more difficult for the model to find any good move if it hasn’t already.
With Never Give Up, we resample failed positions, which has the effect of dynamically scaling compute, giving the model more opportunities to find the best move. GRPO will reward samples which score better than the average reward of the group, but as training progresses and the model cools, positions which are unsuccessful are more likely to remain unsuccessful. To counteract this, we can raise the sampling temperature and apply asymmetric importance sampling clipping.
Why train an agent?
The short answer is, I initially wanted to train a pure LLM, but a 4090 limits candidate base models to around 3 billion parameters. Unfortunately, the smaller models were plagued by confabulation. Reacting to phantom threats, making illegal moves, and just a general lack of grounding in reality. Allowing the model to call tools that track the board state independently removes this type of error.
Do tools help?
Training the model at different sizes, we can see that tools are a very significant contributor to the model strength. Note that the numbers below are after SFT only, and GRPO typically adds 12 more points after that.

While model size gives modest improvement in skill, the availability of tools is a far larger factor.
What tools does it have?
The goal of these tools is to allow the harness to track the board state while the LLM reasons about how to search. They are intended to be simple and atomic operations that a human chess player can easily follow along with. None of these tools are intended to provide the LLM with judgement calls about the position.
| Tool Call | Description |
|---|---|
piece_on(square) |
What piece is on a square |
pieces() |
Every piece on the board and where it is (both sides) |
material() |
Material count for both sides in pawn units |
attackers(square) |
Which pieces (any side) attack a square. If occupied, distinguishes attackers from defenders |
targets(square) |
What enemy pieces the piece on square attacks, and what friendly pieces it defends |
hanging(color) |
What pieces of this color are hanging |
pins(color) |
What pins against the king for color |
forcing_moves() |
Legal checks, captures, and promotions are available in this position |
legal_moves() |
Every legal move in this position, plus whether the side to move is in check |
try(move) |
Push a move onto the stack to evaluate one ply deeper |
undo() |
Pop off the stack to back out of a search |
quiet_moves() |
Legal moves that are not in forcing_moves() |
root() |
Pop the whole stack to return to the starting position |
threats() |
What the opponent could do if it were their move: mates, checks, captures that win material |
exchange(square) |
Result of the capture sequence on a square, starting with the best first capture |
When calling a tool, the LLM must also provide a justification for why it wants to call this tool. This prevents the reasoning trace from devolving into a list of bare tool calls.
Some terminology
Here are some terms I use throughout the post.
- Hit: a move which is within 50 centipawns of the best move (i.e. CPL <= 50)
- Hit rate: percentage of moves which are hits
- Hit@k: at least one of k rollouts results in a hit i.e. a hit@4 means there was a hit after four attempts.