返回资讯中心
外部精选
软件工程
#社区热议

LLMs Play Zachtronics' TIS-100

Twelve nodes. Two registers per node. Fifteen lines of code. In TIS-100, even multiplying two numbers is a small engineering project, and players have spent a decade optimizing these programs on a public community leade…

Hacker Newsfairestvalue2026/11/11 · 7 天前30 分钟阅读

以下正文同步自 Hacker News,版权归原站所有,已转换为易读排版。

Twelve nodes. Two registers per node. Fifteen lines of code. In TIS-100, even multiplying two numbers is a small engineering project, and players have spent a decade optimizing these programs on a public community leaderboard. We gave 16 LLMs the game and its manual, withheld the leaderboard, and told them to keep improving.

Within days, players picked up the models’ programs and improved on most of them.

Each node is a tiny processor, and neighboring nodes have to wait for one another to exchange values. The game scores a working solution on cycles, nodes, and instructions, and you start wondering whether one more instruction could disappear. Lower is better, but improving one score can make another worse, so each puzzle has a frontier of tradeoffs instead of one best answer.

Discovering the tricks is much of the fun, and this article spoils them. If this sounds cool, play it first:

We ran 16 LLMs, covering leading frontier and open-weight models. Every model attempted all 47 puzzles in the game, plus eight community tournament puzzles, in one scored attempt each (pass@1), with 10 to 45 minutes per puzzle, depending on difficulty, to find a working program and keep improving it.

A solution that no community solution ties or beats on all three scores at once. It can use more code to run faster, for example. We froze the community leaderboard on October 15, 2026, before submitting anything, and count a point only if it also passes 100,000 tests.

Puzzles with at least one new frontier point · higher is better

All 55 puzzles · 19 easy / 17 medium / 19 hard

Scroll horizontally to compare all models →

Opus 5.5 and GPT-6 Astra lead, and the data doesn’t separate them. Both solved all 55 puzzles and both found new frontier points on 11. Opus 5.5 found more points, 72 to 32, while GPT-6 Astra has the higher record proximity, 78.0 to 77.2. Record proximity measures closeness to the community records, where 100 means matching them. The proximity gap’s 95% bootstrap interval includes zero, from −0.7 to +2.4 points. GPT-6 Astra also leads in all six tested hypervolume settings, which measure the space covered by each model’s tradeoffs.

Solving and optimizing are also different skills. Gemini 3.8 Flash solved 53 puzzles and found new points on 1; Fable 5.1 solved fewer, 52, and found new points on 5.

RL environments & traces are available upon request for research teams. Get in touch ↗

Every result here is a program we can run and test on fresh inputs, which makes a programming game a useful benchmark. Models get what a player can inspect, through text tools instead of a mouse and screenshots: the manual, visible test inputs, registers, and game feedback. They don’t get community code or exact records, only the game’s coarse score histograms.

Why test 100,000 inputs? The game checks a small set of cases per run. Our headline count requires a fresh-game replay and 100,000 passing tests on the independent TIS-100-CXX simulator, providing stronger evidence that a program generalizes to new inputs.

Undecimal Twin RounderSPEC44149538, one of the eight tournament puzzles, rounds an input to the nearest multiple of N on one output and of 11−N on the other. When we froze the leaderboard, the best community program took 2,694 cycles. Opus 5.5 passed its first program after 6.5 minutes, at 5,669 cycles. It went under the record 16 minutes in and finished at 1,918, which is 29% faster. That speed cost one more node and 28 more instructions than the old record.

The core of the program is a two-speed search. Instead of subtracting N one step at a time, nodes 5 and 6 subtract 4N until they overshoot, then add N back until the value turns positive. Each drives a small coprocessor, node 2 or node 7, by sending it jump offsets through a port, so the coprocessor works as a lookup table for 4N and N.

Opus 5.5 · 8 nodes · 88 instructions · passes 100,000 of 100,000 tests

MOV RIGHT, ACC JRO ACC NOP ADD 1 NOP ADD 1 NOP ADD 1 NOP SUB 3 MOV ACC, RIGHT ADD UP MOV ACC, RIGHT MOV ACC, DOWN MOV RIGHT, DOWN

Node 1MOV UP, ACC SAV SUB 1 MOV ACC, LEFT NEG ADD 10 MOV ACC, DOWN MOV LEFT, ACC NEG ADD ACC ADD 6 ADD LEFT MOV ACC, DOWN SWP MOV ACC, LEFT

Node 2MOV UP, DOWN MOV UP, ACC SAV ADD ACC ADD ACC L: JRO DOWN MOV ACC, DOWN JMP L SWP N: MOV ACC, DOWN M: JRO DOWN JMP N MOV DOWN, DOWN

Node 3MOV UP, RIGHT MOV UP, DOWN

Node 4MOV LEFT, DOWN

Node 5MOV UP, ACC SAV K: MOV 1, UP SUB UP JGZ K MOV 3, UP F: ADD UP JGZ X MOV 1, UP JMP F X: MOV 2, UP MOV ACC, UP SWP SUB UP MOV ACC, DOWN

Node 6MOV UP, ACC SAV K: MOV 1, RIGHT SUB RIGHT JGZ K MOV 3, RIGHT F: ADD RIGHT JGZ X MOV 1, RIGHT JMP F X: MOV 2, RIGHT MOV ACC, RIGHT SWP SUB RIGHT MOV ACC, DOWN

Node 7MOV UP, ACC SAV ADD ACC ADD ACC L: JRO LEFT MOV ACC, LEFT JMP L SWP N: MOV ACC, LEFT M: JRO LEFT JMP N MOV LEFT, LEFT

Laid out on the puzzle’s real grid, with inputs at the top and outputs at the bottom. Blank tiles are damaged and can’t be used.

Loading puzzle trajectories…

Best in-game score so far, before independent testing. Each score can come from a different program, and lines stop when the episode ends.

Interactive trajectories require JavaScript.

Five models got under the old record in-game. The other two cycle records are GPT-6.1 Sol’s 3,460 on Intersection IdentifierSPEC64160260, against 3,872 (11% faster), and Gemini 4 Argon’s 188 on Error: 8SPEC79324720, against 198 (5% faster). All three are on tournament puzzles, none in the main game. Of the 133 new frontier points, 18 beat a community record outright. The other 115 are new tradeoffs: better balances between the three scores than any community solution.

GPT-6.1 Sol reaches 75.6 record proximity at an estimated $0.75 per puzzle. Opus 5.5 reaches 77.2 at $5.08, 7 times the cost for 2 more points. Fable 5.1 is the most expensive at $7.19 and scores 70.2.

One point per model. Upper left is better.

Activity doesn’t explain quality either. GPT-6.1 Sol averages 136 game calls per puzzle at 75.6 record proximity; Gemini 4 Argon averages 41 at 76.2.

Interactive charts require JavaScript. The full rankings above remain available.

New frontier points appear on 14 hard puzzles, 4 medium puzzles, and no easy ones. Long-running puzzles are harder to solve but easier to improve: on puzzles with a cycle record of 200 or less, 88% of episodes pass and 4% find a new point. Above 500 cycles, 76% pass and 8% find one.

Full bar = solve rate · dark segment = new-frontier rate

Position = community record · circle area = fewest nodes

Every model is closest to the community on nodes and furthest on cycles. Across all 13, mean proximity is 72 for nodes, 56 for instructions, and 45 for cycles.

Mean proximity to each community record · higher is better

Every dot is an episode · box = middle 50% · line = median · whiskers = minimum and maximum

Choose a cell to inspect the scores in the Pareto explorer.

We grouped all 55 puzzles by their main demand, then read the programs. Fifteen of the sixteen models solved all five routing puzzles, and none found a new frontier point there. The examples below show how different designs spend the same resources.

Fraction solved · puzzles with new points below

Signal Multiplier43786 has no MUL. Gemini 4 Argon and GPT-6 Astra use JRO, a relative jump, to enter a block of repeated instructions at the right position. The jump selects how many copies of a value to send and add, saving the counter updates and branches of a multiplication loop.

Gemini 4 Argon’s fastest program uses two paths to multiply in parallel and a final node to interleave their outputs. It takes 436 cycles against GPT-6 Astra’s 523, about 17% faster, at a cost of two more nodes and 33 more instructions.

Signal Window Filter60099 outputs the sums of the last three and last five inputs, so a solution has to remember history as well as add. Gemini 4 Argon’s four-node program uses SWP to rotate stored values and ADD/SUB to update running state. GPT-6.1 Sol spreads that delay and aggregation across seven nodes, passing intermediate values between them.

GPT-6.1 Sol’s program runs in 318 cycles with 26 instructions; Gemini 4 Argon’s takes 428 cycles with only 17. The difference is how much of the work runs in parallel, and how much communication and code that costs.

T31 Node EmulatorNEXUS.23.727.9 needs eight addressable storage slots, but the available memory nodes are stacks. These programs move values between stacks to reach the requested slot, then read or replace it. MiMo-V2.6-Pro uses counted transfer loops; Opus 5.5 uses JRO to pick among unrolled transfers, spending extra instructions to save cycles.

Memory isn’t one skill: MiMo-V2.6-Pro solves this emulator but fails Sequence Mode CalculatorNEXUS.09.904.9 and Signal Window Filter60099.

Each row is that model’s fastest saved program for the puzzle, picked for illustration. Not all are new frontier points.

The main benchmark hides the community’s scores. In a separate study on six puzzles, five models got:

Every model improved: GPT-6 Astra from 82.8 to 91.8, GPT-6.1 Sol from 82.9 to 90.6, Opus 5.5 from 81.4 to 89.1, Grok 4.7 from 67.5 to 75.1, and Gemini 4 Argon from 83.5 to 87.9 in mean record proximity. Scores rose in 28 of the 30 model–puzzle pairs.

Mean record proximity · same six puzzles per model · higher is better

The diagnostics changed what models did next. On Signal Multiplexer22280, GPT-6.1 Sol’s first passing program took 294 cycles. Its profile showed node 2 working 86% of the time while nodes 1 and 3 spent 84% blocked waiting to write to it. GPT-6.1 Sol moved the branching out of node 2 and into those two neighbors. The next run took 204 cycles, 31% faster, at a cost of two more nodes and five more instructions.

Mean elapsed minutes per puzzle · same episodes

Information, tools, and time all changed together, so this study can’t say which addition helped most. These runs are outside the main leaderboard and the headline count.

Puzzles: Signal Multiplexer, Sequence Generator, Signal Edge Detector, Interrupt Handler, Sequence Peak Detector, and Signal Error Corrector.

We submitted selected programs to the leaderboard, and the community improved on them almost immediately. On Undecimal Twin RounderSPEC44149538, @LongingForRest took Opus 5.5’s 1,918-cycle program and cut it to 1,744 by the next day. The record is now credited to both: LongingForRest/opus-5-5.

Neither got there alone. The models found new tradeoffs quickly, and players read those programs, worked out why they worked, and took them further.

Mathematics is seeing the same shift. Terence Tao and colleagues used LLM-guided search across 67 problems and improved on the best known results in several. In “After Math,” Silvia De Toffoli and Eamon Duede argue that a result only becomes a solution once mathematicians can understand it, and that AI fits better as an assistant than a competitor. TIS-100 is a small, measurable version of that future: a model finds a better program, and people turn it into something they understand.

Choose a puzzle and model, then select a point to see its scores and the program that produced it. Every puzzle also has its own page with records and the best programs: all 55 puzzles.

The plot shows cycles and instructions; node counts can differ. New frontier points are judged on all three scores.

Download all 3,714 model Pareto points (CSV) · all 3,714 programs (ZIP), also on GitHub and Hugging Face. 186 rows are marked as new frontier points, which pool to 133 across models.

The interactive explorer requires JavaScript. The article and full rankings remain available.

Suite. All 22 campaign puzzles, all 25 TIS-NET puzzles, and eight tournament puzzles from 2018: 19 easy (10 minutes), 17 medium (30), and 19 hard (45). Every model has one scored episode per puzzle. Time limits differ by difficulty, so difficulty and search time can’t be separated.

Scoring. Each model–puzzle pair contributes one selected attempt, with no best-of-at

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →