Chess Cheat Detector v3: Strong Players Are No Longer Mistaken for Engines

Chess Cheat Detector has a new model. On real games between grandmasters, it wrongly rates about one honest player in 60 as Elevated or High, down from one in four.

Both versions were scored on the same 14,229 players, from games that neither saw during training:

*Take one cheater and one honest player at random. This is how often the cheater gets the higher score (ROC AUC).

A score of 70% or more is shown as Elevated or High. The honest grandmasters are 2,982 players from real games, all rated 2500 or above. Every other row comes from simulated games, including some that start like a grandmaster's.

v2's numbers here are lower than in its own write-up, which scored only games with ratings. These include a quarter of games without ratings, closer to what the site sees, as well as the harder grandmaster-style games.

v2 learned from simulated players, and the strongest of them played at around 2600 on Lichess's scale. That's well below a real grandmaster. So v2 had never seen an honest player as accurate as one, and the only players in its training data who played that well were cheaters.

v3 also learned from two new sets of games, about 7,000 of each:

v3 is more cautious. It ranks players slightly better than v2, but it gives lower scores across the board. Fewer cheaters reach 70%, and the biggest drop is among those cheating at grandmaster level. Real grandmasters' play overlaps heavily with engine-assisted play, and v3 now gives strong players the benefit of the doubt.

We made that trade deliberately. Honest strong players vastly outnumber cheating ones, and wrongly accusing them does real damage. The real grandmaster games count for a quarter as much as the rest in training. At full weight, wrong flags fell even further, but v3 missed most grandmaster-level cheaters.

Got a game that felt off? Paste it in and see what v3 makes of it.