TLDR: I trained an LLM reasoning agent to play chess, with model sizes 0.8B, 3B and 9B. The agent calls tools which help it keep track of the board state, which enables good performance with a relatively small number of parameters.

The demo code is available on Github.

Background

I became excited about the power of RLVR after the Deepseek R1 paper came out. They demonstrated that through the use of “thinking” tokens, an LLM could solve verifiable tasks like math problems. Additionally, they described the necessity of bootstrapping the thinking process with supervised examples, which prevents the model from generating a nonsense chain of thought.

Chess is an ideal candidate for a verifiable rewards setup. You could call it “fractally verifiable”: Not only are games themselves verifiable outcomes, but the strength of individual moves are verifiable, and even the individual claims used to justify those moves can be verified.

The majority of prior work in chess engines has had the sole purpose of maximizing playing strength. I’ve written a longer history of chess engines here and here. This would be a much different type of engine: we should favor explainable, intuitive moves over the absolute best moves.

Why train an agent?

The short answer is, I initially wanted to train a pure LLM, but a 4090 limits candidate base models to around 3 billion parameters. Unfortunately, the smaller models were plagued by confabulation. Reacting to phantom threats, making illegal moves, and just a general lack of grounding in reality. Allowing the model to call tools that track the board state independently removes this type of error.

What tools does it have?

The goal of these tools is to allow the harness to track the board state while the LLM reasons about how to search. They are intended to be simple and atomic operations that a human chess player can easily follow along with. None of these tools are intended to provide the LLM with judgement calls about the position.

Tool Call Description
piece_on(square) What piece is on a square
pieces() Every piece on the board and where it is (both sides)
material() Material count for both sides in pawn units
attackers(square) Which pieces (any side) attack a square. If occupied, distinguishes attackers from defenders
targets(square) What enemy pieces the piece on square attacks, and what friendly pieces it defends
hanging(color) What pieces of this color are hanging
pins(color) What pins against the king for color
forcing_moves() Legal checks, captures, and promotions are available in this position
legal_moves() Every legal move in this position, plus whether the side to move is in check
try(move) Push a move onto the stack to evaluate one ply deeper
undo() Pop off the stack to back out of a search
quiet_moves() Legal moves that are not in forcing_moves()
root() Pop the whole stack to return to the starting position
threats() What the opponent could do if it were their move: mates, checks, captures that win material
exchange(square) Result of the capture sequence on a square, starting with the best first capture

When calling a tool, the LLM must also provide a justification for why it wants to call this tool. This prevents the reasoning trace from devolving into a list of bare tool calls.