返回资讯中心
外部精选
软件工程
#开发者实时流

Spending 10 months building the most human-like chess bot

80 / 100Allie 2.05.6B parameters99 / 10048 / 10099 / 10073 / 10099 / 100Everyone knows playing against an engine like Stockfish is nothing like playing a human. The existing human-like models (Allie 2.0, Maia-3) weren&r…

Hacker News 最新marcusbuffett2026/10/19 · 27 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

80 / 100Allie 2.05.6B parameters99 / 10048 / 10099 / 10073 / 10099 / 100Everyone knows playing against an engine like Stockfish is nothing like playing a human. The existing human-like models (Allie 2.0, Maia-3) weren’t quite right for my use case, and I became a bit obsessed with making the best one.

I wanted a sparring partner that plays just like a random opponent on Lichess, at any rating and any time control. So over the last ten months I built one. It’s called Chessbook-1, and it’s the bot behind the training games on Chessbook.

The best models available right now for this are Allie 2.0 and Maia-3. Maia-3 doesn’t match playing strength well, plays too strongly at low ratings, isn’t trained on low-time situations or non-blitz time controls, and it doesn’t predict how long a human will take to play a move.

Allie 2.0 is by a good margin the best predictor overall, but predicts time usage independently of the move, and its strength drifts at low-Elo and high-Elo play.

So I made my own, and here’s how it compares to Allie 2.0 and Maia-3.

TL;DR: Chessbook-1 predicts human moves well, is the only model calibrated to play at the right strength at every rating, and it errs in time scrambles like a human would. It predicts time usage well, and is aware of the time it uses.

Two sets of positions, both from Lichess games played in September 2026, a month after every model’s training data ends (Chessbook-1 trained on games through July 2026, Maia-3 through July 2025, Allie 2.0 through August 2026):

Every legal move in every position was scored by Stockfish 18. Each model gets the position, the players’ ratings and any other features it accepts (Chessbook-1 and Allie read the clocks and the move history; Maia-3 reads the history but not the clock), and outputs a probability for every legal move. Then we compare that distribution with what the human played.

The easiest measure is top-1 accuracy: how often the model’s top prediction is the move the human actually played. The other number worth watching is how much probability the model put on the move the human chose, averaged over positions: a model that spreads its guesses sensibly scores well here even when its favourite move is wrong.

Start with a random position from a random game: any time control, any clock, any rating.

Maia-3, 600-1000: 45.7%Maia-3, 1000-1400: 51.3%Maia-3, 1400-1800: 54.6%Maia-3, 1800-2200: 56.7%Maia-3, 2200-2400: 56.4%Maia-3, 2400+: 58.6%Allie 2.0, 600-1000: 45.6%Allie 2.0, 1000-1400: 52.1%Allie 2.0, 1400-1800: 55.5%Allie 2.0, 1800-2200: 58.1%Allie 2.0, 2200-2400: 60.1%Allie 2.0, 2400+: 61.6%Chessbook-1, 600-1000: 46.7%Chessbook-1, 1000-1400: 51.6%Chessbook-1, 1400-1800: 54.3%Chessbook-1, 1800-2200: 57.6%Chessbook-1, 2200-2400: 57.3%Chessbook-1, 2400+: 61.6%Chessbook-1Maia-3Allie 2.0Probability on the human's move15%20%25%30%35%40%60010001400180022002400+Player rating (Lichess)Maia-3, 600-1000: 19.1%Maia-3, 1000-1400: 22.3%Maia-3, 1400-1800: 25.1%Maia-3, 1800-2200: 26.1%Maia-3, 2200-2400: 25.7%Maia-3, 2400+: 27.8%Allie 2.0, 600-1000: 19.4%Allie 2.0, 1000-1400: 23.1%Allie 2.0, 1400-1800: 26.1%Allie 2.0, 1800-2200: 28.1%Allie 2.0, 2200-2400: 29.4%Allie 2.0, 2400+: 32.1%Chessbook-1, 600-1000: 19.0%Chessbook-1, 1000-1400: 22.1%Chessbook-1, 1400-1800: 25.2%Chessbook-1, 1800-2200: 27.4%Chessbook-1, 2200-2400: 28.2%Chessbook-1, 2400+: 31.8%Higher is better. 30,000 random positions from every rated September 2026 game, by rating band; everyone above 2400 is one band (657 positions, mostly bullet players).

Allie 2.0, at 28 times the size and reading the whole game, is about a point ahead of both in every band but the lowest and the highest. Between Chessbook-1 and Maia-3, the two are level below 1800 and Chessbook-1 pulls ahead from 1800 up; pooled over all 30,000 positions it leads Maia-3 on both measures by a small but real margin, and the gap is widest in bullet games and in the last seconds on the clock, where Maia-3’s training data runs out.

This isn’t really a fair comparison with Maia-3 though. That model is trained only on blitz games, and every position after a player’s clock first drops under 30 seconds is thrown out of its training data, so a random Lichess position is often out-of-distribution. So how does Chessbook-1 do on Maia-3’s own terms: blitz games, with the time scrambles filtered out?

Chessbook-1, 600-1000: 46.946.9Chessbook-1, 1000-1400: 53.353.3Chessbook-1, 1400-1800: 54.954.9Chessbook-1, 1800-2200: 59.159.1Chessbook-1, 2200-2400: 61.361.3Chessbook-1, 2400-2600: 61.661.6Chessbook-1, 2600+: 63.563.5Maia-3, 600-1000: 48.148.1Maia-3, 1000-1400: 53.653.6Maia-3, 1400-1800: 55.555.5Maia-3, 1800-2200: 59.259.2Maia-3, 2200-2400: 60.660.6Maia-3, 2400-2600: 61.661.6Maia-3, 2600+: 62.662.6Allie 2.0, 600-1000: 48.548.5Allie 2.0, 1000-1400: 53.453.4Allie 2.0, 1400-1800: 55.355.3Allie 2.0, 1800-2200: 59.859.8Allie 2.0, 2200-2400: 60.860.8Allie 2.0, 2400-2600: 61.761.7Allie 2.0, 2600+: 62.762.7Chessbook-1Maia-3Allie 2.0Higher is better. Top-1 move prediction by rating band on the blitz set.

Average probability assigned to the human's actual moveGeometric mean of the probability each model gave to the move the human played (exp of minus the negative log-likelihood).15%20%25%30%35%40%600100014001800220024002600+Player rating (Lichess)Chessbook-1, 600-1000: 18.918.9Chessbook-1, 1000-1400: 23.623.6Chessbook-1, 1400-1800: 25.225.2Chessbook-1, 1800-2200: 28.828.8Chessbook-1, 2200-2400: 31.731.7Chessbook-1, 2400-2600: 33.333.3Chessbook-1, 2600+: 32.632.6Maia-3, 600-1000: 19.419.4Maia-3, 1000-1400: 24.324.3Maia-3, 1400-1800: 25.625.6Maia-3, 1800-2200: 28.728.7Maia-3, 2200-2400: 31.531.5Maia-3, 2400-2600: 33.533.5Maia-3, 2600+: 33.833.8Allie 2.0, 600-1000: 19.819.8Allie 2.0, 1000-1400: 24.624.6Allie 2.0, 1400-1800: 26.026.0Allie 2.0, 1800-2200: 29.329.3Allie 2.0, 2200-2400: 31.931.9Allie 2.0, 2400-2600: 33.333.3Allie 2.0, 2600+: 33.533.5Chessbook-1Maia-3Allie 2.0Higher is better. Geometric mean of the probability each model assigned to the human’s actual move, blitz set; a model that put 100% on every human move would score 100%.

A three-way statistical tie. Over all 28,000 positions Allie 2.0 gets 57.5% top-1, Maia-3 57.3% and Chessbook-1 57.3% (27.9%, 27.6% and 27.3% probability on the human’s move); the paired differences are within ±0.3 points. Band by band everything is within noise except 600–1000, where Maia-3 and Allie are about a point ahead on top-1 (the per-band confidence intervals are ±0.8 to ±1.0 points). For scale, Maia-2 is at 50% pooled, and a Stockfish limited to the player’s rating agrees with the human only 24–40% of the time, which is the whole problem with engine-based bots.

So our model matches Maia-3 on its own strengths, and beats it in the areas it’s not quite trained for. It falls short of Allie 2.0 on raw prediction, which is expected given the 28x size difference in the models, and it makes up for it by matching strength better and with better time usage.

A move predictor is not automatically a good opponent. To give an extreme example, if you put 95% probability on the right move every time, but put 5% probability on weird blunders, then you’ll never play a competitive game against an opponent who knows what they’re doing, because you’ll be blundering twice in a 40-move game and it only takes one blunder to lose the game. Maia-3 gets around this a bit by not sampling from its distribution, taking the top move instead. But this causes it to over-perform at low ratings, and it still struggles at high ratings. Chessbook-1 can sample from its distribution at every rating, and play almost exactly at that level, until 2600+.

To measure playing strength, we take positions where the evaluation is within two pawns of equal. Here is each bot’s average centipawn loss relative to the humans who actually played those positions, band by band. Zero means it loses exactly as much as they do; below zero means it plays stronger than they do:

Chessbook-1, 600-1000: +1.0 cpChessbook-1, 1000-1400: -1.0 cpChessbook-1, 1400-1800: +0.8 cpChessbook-1, 1800-2200: -0.6 cpChessbook-1, 2200-2400: -0.8 cpChessbook-1, 2400-2600: +0.0 cpChessbook-1, 2600+: +0.9 cpMaia-3, 600-1000: +1.1 cpMaia-3, 1000-1400: -2.6 cpMaia-3, 1400-1800: +0.4 cpMaia-3, 1800-2200: +0.8 cpMaia-3, 2200-2400: +1.9 cpMaia-3, 2400-2600: +4.3 cpMaia-3, 2600+: +6.8 cpAllie 2.0, 600-1000: -8.3 cpAllie 2.0, 1000-1400: -5.9 cpAllie 2.0, 1400-1800: -1.6 cpAllie 2.0, 1800-2200: +0.7 cpAllie 2.0, 2200-2400: +2.1 cpAllie 2.0, 2400-2600: +5.2 cpAllie 2.0, 2600+: +8.2 cpChessbook-1Maia-3Allie 2.0Closer to zero is better. Bot ACPL minus human ACPL in contested positions (|eval| ≤ 200 cp), per rating band. Bots sample from their move distribution; Maia-3 from its raw probabilities.

And the same for the blunder rate, which is maybe a better representation because it’s hard to feel a 10 centipawn loss average difference. But 2 blunders per game vs 3 feels very different over time:

Chessbook-1, 600-1000: +0.6 ptsChessbook-1, 1000-1400: -0.0 ptsChessbook-1, 1400-1800: +0.2 ptsChessbook-1, 1800-2200: -0.3 ptsChessbook-1, 2200-2400: -0.2 ptsChessbook-1, 2400-2600: -0.3 ptsChessbook-1, 2600+: +0.4 ptsMaia-3, 600-1000: +0.5 ptsMaia-3, 1000-1400: -0.3 ptsMaia-3, 1400-1800: +0.3 ptsMaia-3, 1800-2200: +0.1 ptsMaia-3, 2200-2400: +0.3 ptsMaia-3, 2400-2600: +0.5 ptsMaia-3, 2600+: +1.4 ptsAllie 2.0, 600-1000: -1.4 ptsAllie 2.0, 1000-1400: -0.9 ptsAllie 2.0, 1400-1800: -0.1 ptsAllie 2.0, 1800-2200: +0.1 ptsAllie 2.0, 2200-2400: +0.4 ptsAllie 2.0, 2400-2600: +0.7 ptsAllie 2.0, 2600+: +1.8 ptsChessbook-1Maia-3Allie 2.0Closer to zero is better. Bot blunder rate minus human blunder rate, in percentage points, contested positions. A blunder loses at least 15 points of win probability; below zero means the bot blunders less often than the humans.

Chessbook-1 tracks the humans in every band on both measures; its bars barely leave zero. Maia-3 and Allie 2.0 match the humans up to about 2200 and then drift too weak as the rating climbs, until at 2600+ both blunder nearly twice as often as the players they are imitating. Allie also runs noticeably too strong at the bottom of the scale. (A rating-limited Stockfish goes the other way: it blunders about half as often as a human of the same rating, which is why it feels like playing a strong player who occasionally hangs a piece.)

Another way to see the same thing: take the model’s probability mass on moves that are mistakes and compare it with how often humans at that rating actually make mistakes. A perfectly calibrated opponent errs exactly as often as they do.

Chessbook-1, 600-1000: +9%+9%Chessbook-1, 1000-1400: +7%+7%Chessbook-1, 1400-1800: +6%+6%Chessbook-1, 1800-2200: +5%+5%Chessbook-1, 2200-2400: -1%-1%Chessbook-1, 2400-2600: +6%+6%Chessbook-1, 2600+: +18%+18%Maia-3, 600-1000: +6%+6%Maia-3, 1000-1400: +1%+1%Maia-3, 1400-1800: +4%+4%Maia-3, 1800-2200: +8%+8%Maia-3, 2200-2400: +10%+10%Maia-3, 2400-2600: +32%+32%Maia-3, 2600+: +63%+63%Allie 2.0, 600-1000: -3%-3%Allie 2.0, 1000-1400: -5%-5%Allie 2.0, 1400-1800: +0%+0%Allie 2.0, 1800-2200: +8%+8%Allie 2.0, 2200-2400: +13%+13%Allie 2.0, 2400-2600: +41%+41%Allie 2.0, 2600+: +75%+75%Chessbook-1Maia-3Allie 2.0Closer to zero is better. How much more often each model’s move distribution errs than the humans of that rating. Zero is a perfect match.

Chessbook-1 stays within 9% of the humans in every band, 18% at 2600+. Maia-3 and Allie 2.0 are fine up to 1800 and then come apart: 63% and 75% more mistakes than a 2600+ player. The stronger the player you imitate, the more these models over-represent the rare mistakes, and nothing in those models corrects for it.

Humans play differently with limited time remaining on the clock, and a bot that does not know the clock won’t play as if it’s under any pressure at

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →