Not a Game of Snake
Published October 10, 2026 / 5,213 words / 22 min read
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Published October 10, 2026 / 5,213 words / 22 min read
Almost a hundred years ago, René Magritte painted a very nice pipe and wrote ‘Ceci n’est pas une pipe’ (‘This is not a pipe’) underneath it, and people have been mildly annoyed about it ever since. In fact, there’s been quite a lot of discussion around it, despite the simplicity of the original artwork.
René Magritte, The Treachery of Images (This is Not a Pipe), 1929. Collection record.
And while there have been many different takes from many different artists, philosophers, poets, historians, and common people, it’s been profoundly difficult for everyone to actually agree on what the image is about, but for the most part, it boils down to this: The representation of a thing, is not that thing. One excellent application of this way of thinking is “the model is not the data”, an aphorism in the sciences that acknowledges models of reality are limited and inaccurate, but can still be useful.
“We confuse the world as it is with the world as it is described.”
Sure, it looks like snake, and it even moves like snake, mostly. But there is no spoon game running in the above pane. In fact, to call what you are seeing a game is almost entirely inaccurate. Just as with Magritte’s pipe, which is not a pipe due to the fact it possesses none of the requisites to be a pipe, this is not a game because it possesses none of the requisites to even be a game. There are no rules, players, goals or end states. This isn’t “an AI playing a game of snake” either, the above is a small language model’s representation of a game of snake.
The plan for this little project is relatively simple. Build a perfectly normal snake game, and make the underlying game engine write every “frame” or “tick” out as a short “sentence” made up of the game state which is represented in “words” (lots of quotes here but just bear with me, if I explained it all now it would be too boring for you to continue). Then when we have our special game engine, we get something that is actually good at snake to play a lot of games and train a small transformer on nothing but those sentences, so that playing snake turns into predicting the next most probable word to form a “sentence”, which is coincidentally the thing a language model is built to do.
The transformer in the language model never gets any direct information on the game rules. In fact, it doesn’t get direct information from anything, the entire “concept” of finding apples, or what a good move is, or why running into a wall is bad is entirely inferred from the training data.
To be clear, I’m not trying to build a good snake player here. A transformer is a terrible choice for game agents, but that’s exactly why I’m building it. I wanted to see what kinds of mistakes it makes.
The game itself is as standard as snake gets. It’s more or less the “hello world” of writing games, and I don’t think I have to talk much about how it works or how you play it. The rules and game engine are all just one python file. The only real oddity which sets it apart from a “standard” snake game is that it writes every game state out as a “sentence”, which brings us to the first challenge of this project, and that is inventing a language to play a game of snake.
My first go at this spelled out every item on the board, and was inspired by the g-code commonly used by 3D printers. A typical frame looked like this:
HEAD X9 Y5 BODY X8 Y5 BODY X7 Y5 APPLE X14 Y5
followed by a move like RIGHT to signal an ‘input’, followed by the whole list again except for the small permutations such as the new head position, tail position, and possibly apple position. Three “words” are used for ever “piece” of the snake (segment type, x location, y location) and that works out to 3L + 3 words for a snake of length L. If we assume that, on average, our snake eats 30 apples per game (which is on the low end), we would end up with a final game state of roughly 3(30) + 3 = 93 words in one sentence. The problem with a sentence length like this, is that it’s just one frame of the game, and our tiny little transformer is going to fill up its 256-word memory really fast, and we would be inducing attention collapse in the small attention head of our transformer due to the repetition.
Regardless of the verbosity, if we were to try to use masking to train our small language model, we won’t be able to mask entire frames since there is a good chance we don’t have the space for that on longer games. This means we would mostly be training our transformer to learn (as far as I can tell) that BODY is followed by an X coordinate, which is followed by a Y coordinate. Riveting stuff.
The biggest problem with the above language is the way body segments are unnecessarily repeated. If we know the initial positions of the head and tail, all subsequent positions of the body can be inferred based on how the head moves (and if an apple was eaten). But we also don’t want to entirely disconnect the body of our snake from the head and tail, since the transformer somehow needs to “know” about collisions, otherwise it will just move through itself.
So the revised version treats every frame as its own complete little sentence, with a clear beginning, a description, and a clear end:
snake d 1 left left apple k 14 stop
snake starts the snake, followed by the address of its head (column d, row 1). After that each body word is the direction to the next piece, walking from the head toward the tail, so left left means “one piece to the left of the head, and another one to the left of that”. Then comes apple k 14, and stop closes the sentence. If that frame is the end of the game due to a self or wall collision, it closes with dies instead of stop.
The whole language that can describe any possible game of snake is just 48 words:
The margin ring is something that was added later due to an unexpected development. During training, sometimes the head of the snake would end up off the board, but its head still has to actually be somewhere due to the way the engine works, and we still want to be able to represent whatever the small language model is outputting. So driving straight up off the top from the start would mean the head is positioned in the “margin” above column d, then the body steps back down onto the board, and the sentence ends in dies.
A frame for a snake of length L is always L + 6 words. Much better than the previous 3L + 3! The general structure of a sentence is now
game <this frame> goes <move> <next frame>
This structure captures a whole frame, and nothing from a few moves ago is needed to know what happens next, so the model never needs to “remember” anything outside the sentence it’s reading. The longest sentence in the longest game of the corpus is 185 words, which was comfortably inside the model’s 256-word window.
Head at d 1, two body words, apple at k 14.
The head address moved. The body is the same chain, shifted.
The head stands in the margin. The frame ends in dies.
snake e 4 down left up left down down down down left down right down right right down right right down down right right up up up right up up apple g 4 stop
Only the head has a column and a row. The rest is which way to the tail.
The first draft of this grammar used an “at” as part of the vocabulary, as in snake at d 1. It reads nicer to a human, but the “at” carries no useful or additional information (as far as I can tell) for our model. After snake, piece or apple the next word is always a column, and columns, rows and directions are three sets of words that (should) never overlap in our model. The model would have learned to predict at perfectly within a few hundred steps no doubt, but that would be unneeded noise, so it was removed
One thing I cared about a lot is that this language can spell out things snake can’t do. The renderer built for this reads a sentence and draws whatever it “says”, regardless of how valid it is, or with any relation to the snake game rules. Two apples, no apple, two heads, a loose bit of body, a snake folded back over itself, all of it should work, but for any “illegal” board states we can flag that something’s gone wrong. You can try it yourself below, so type whatever you like, feel free to use the table of the language above for reference.
Before the transformer can read any games “written” by the game engine, somebody has to play a lot of snake first, and it was not going to be me. Snake is small, fully visible, and only has a handful of choices per move, which makes it a natural fit for reinforcement learning, so I tried two different popular RL algorithms, Proximal Policy Optimization and Deep-Q learning
PPO (Proximal Policy Optimization) learns a “policy”, which is just to say: given the board, what is the most likely (initially random) move to make? Then it plays a batch of games, observes the outcomes, nudges the odds toward moves that turned out better than others, and repeats. DQN (Deep Q-learning) learns a “score” for every move in every position instead (“how many apples do I expect if I go left here?”) and plays whichever move scores highest. It also keeps old moves in a replay buffer and learns from them over and over (the “deep” part of deep-Q learning), which PPO doesn’t. Both are very interesting and very useful algorithms, and they deserve a post of their own, written by someone who understands them better than I do. For the purpose of this blog post though, this simple explanation should suffice.
Starting from random weights, both agents ate basically nothing for the first few thousand games, but both soon started improving.
Median apples eaten, with the middle 80% of games in brackets, and how many of the 100 games ended by starving.
At 20,000 games PPO has at least begun learning while DQN is still effectively random. By 50,000 DQN has caught up and is slightly ahead, but then by 200,000 it has levelled off while PPO keeps climbing, ending at 48 apples for PPO against 32 for DQN.
For interest’s sake, the statistics include both the performance and the time. PPO reached 200,000 games in 87 minutes, which is pretty respectable considering I didn’t use one of the big ML libraries like pytorch or tensorflow. DQN took 205 minutes, but that’s mostly because its replay-buffer update ran every four environment steps once the warm-up stage ended.
Both of them are “good enough” players in the sense that they eat a lot of apples and then die in an entirely normal, snake-like way, which is all a teacher model needs to do here. Since PPO was performing better at the time (and training faster), I let it cook overnight, and in the morning the median apples eaten for PPO was 57. Not bad!
The initial plan was to let the PPO agent play 20,000 games for training, plus another 1,000 that the transformer would never see (so I could check its homework and make sure it wasn’t overfitting). Recording all of them took about 10 minutes. The training games average 49.8 apples, with a median of 53 and a longest game of 82. That’s a little under the greedy median of 57 because the moves were sampled from the policy rather than always taking the top choice, which gives the corpus a bit of variety (and the occasional questionable decision).
If all of those games were to be written out as sentences, that would be a lot of data to manage. The typical game above is 1,231 moves long, and as one long document of frames it’s 243,607 bytes. Measured on every 40th game, the 20,000 training games would come to roughly 4.4 GB of text, and that’s before you count each frame a second time for the training rows.
So, what’s the solution? Well, if we want to make the dataset as small as possible, let’s start with the smallest representation of data available to us, a bit, and build from there.
The most commonly used words in our vocabulary are most certainly going to be the directions, so they are the first candidate for encoding. Four directions fit exactly into t
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。