返回资讯中心
外部精选
软件工程
#开发者实时流

开发者反汇编Atari 2600版Pong,探究强化学习AI为何触发游戏异常

来源标题:Disassembling Atari 2600 Pong to Find Why RL Agents Glitch It Out

开发者通过反汇编仅2KB的Atari 2600版Pong(Video Olympics)ROM,直接阅读代码来解释强化学习AI为何会触发人类玩家从未见过的边缘情况和故障。起因是他听《The Alignment Problem》时了解到,AI在Arcade Learning Environment基准上把Pong玩出了异常表现。

Hacker News 最新fortran772026/11/17 · 今天30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Disassembling Atari 2600 Pong to Find Why RL Agents Glitch It Out

Published on 2026-11-13

#programming

#retrocomputing

#ai

#games

#algorithms

I've been listening to the audiobook of Brian Christian's The Alignment Problem: Machine Learning and Human Values, which traces how modern machine learning systems often optimize for the literal mathematical reward we give them rather than what we actually meant. Along the way, Christian discusses reinforcement learning experiments using the Arcade Learning Environment (ALE)—the standard benchmark built on top of the Stella Atari 2600 emulator—and mentions how AI agents got good enough at Atari 2600 Pong to push the game into bizarre edge cases and glitches that human players never noticed.

Having spent a fair bit of time in the trenches with the 6507/6502 and the TIA (most notably when I wrote my Enigma 2600 simulator), my immediate reaction was: Wait a minute—the Atari 2600 version of Pong (Video Olympics, CX2621, written by Joe Decuir in 1977) is only a 2KB ROM! We don't have to guess what's happening inside the black box—we can just disassemble the binary and read the code!

I happened to have the 2,048-byte pong.bin image sitting in a directory along with the cc65 assembler/disassembler suite (da65, ca65, ld65). So I decided to try an experiment: I asked Google's Argon model to take pong.bin, produce a fully annotated, human-readable 6502 assembly source listing (pong.s) that reassembles byte-for-byte back to the original 1977 ROM, and then audit the disassembly to explain the exact hardware and algorithmic mechanisms behind the ALE Pong glitches.

First, it's worth appreciating just how much Joe Decuir packed into 2,048 bytes ($F000–$F7FF) for one of the nine original Atari VCS launch titles. Video Olympics doesn't just implement Pong; it implements 50 game variations across 7 different sports (Tennis/Pong, Super Pong, Soccer, Ice Hockey, Handball, Volleyball, and Basketball) for 1 to 4 players.

To squeeze all of that into 2KB of ROM and 128 bytes of Zero-Page RAM ($80–$FF), the code uses several classic 6502/TIA tricks that showed up immediately in the disassembly:

Using BRK as a 2-Byte Multiply-by-16 Subroutine: Since the 6507 has no NMI pin, only the RESET ($F7FC) and IRQ/BRK ($F7FE) vectors exist at the top of ROM. Decuir pointed the BRK vector at a tiny 9-byte routine at $F438:

Mul16: asl a ; F438: 0A Rotate left 1 bit adc #$00 ; F439: 69 00 asl a ; F43B: 0A Rotate left 2 bits adc #$00 ; F43C: 69 00 asl a ; F43E: 0A Rotate left 3 bits adc #$00 ; F43F: 69 00 asl a ; F441: 0A Rotate left 4 bits adc #$00 ; F442: 69 00 rti ; F444: 40 Return from BRK interrupt

Whenever the code needs to shift a value into the upper nibble (such as shifting fine horizontal motion values for HMP0..HMBL or shifting the attract-mode color cycling counter), it executes brk; nop ($00 $EA)—saving a byte over jsr Mul16 and automatically preserving the caller's processor status flags on the stack!

Abusing the Stack Pointer (TXS / PHP) in the 2-Line Display Kernel: Inside the 182-scanline court kernel ($F5D0), there aren't enough CPU cycles to load, branch, and store enable bits for Missile 0 (ENAM0 at $1D), Missile 1 (ENAM1 at $1E), and the Ball (ENABL at $1F). All three TIA registers enable their graphic object if Bit 1 (D1) is 1. And what is Bit 1 of the 6502 Processor Status (P) register? The Zero (Z) flag! So the kernel points the CPU stack pointer S directly at the TIA registers (ldx #$1F; txs) and executes:

txa ; F618: 8A A = current scanline X sec ; F619: 38 sbc BallPosY ; F61A: E5 B6 A = X - BallPosY and #$FC ; F61C: 29 FC Within 4-scanline ball height? (Z=1 if yes) php ; F61E: 08 Push P to [S=$1F] (ENABL: D1 = Z flag!), S -> $1E

A single 3-cycle php writes the Z flag straight into ENABL ($1F) and automatically decrements S to $1E (ENAM1) for the next php a few cycles later.

Pointing Volleyball's Playfield Pointer into Zero-Page RAM: To avoid wasting 24 zero bytes in ROM for an empty PF2 table in Volleyball, SetupGameConfig ($F2A1) checks if the low byte of PF2Ptr is $C9 and sets the high byte to $00, reading 24 bytes of zeroes straight out of unused Zero-Page/stack memory ($00CD–$00E4).

Running ca65 and ld65 on the annotated pong.s produced an exact 2,048-byte match (md5sum: 60e0ea3cbe0913d39803477945e9e5ec).

Once we had the complete symbolic disassembly in front of us, Argon analyzed how reinforcement learning agents in the Arcade Learning Environment interact with the code. Because RL agents play millions of frames, optimize ruthlessly for reward loopholes, and test action timings no human could sustain, they expose four specific bugs and hardware quirks in the 1977 code:

When two RL agents learn to rally defensively (or when an agent stalls to avoid risking a negative reward, as Christian describes in The Alignment Problem), something bizarre happens after exactly 65,280 frames (~18.1 minutes of 60 Hz gameplay): in the middle of a rally, the right player's score disappears, the left player's score resets to 1, the ball vanishes, and the court starts flashing rainbow colors.

Look at how the frame counter and attract-mode timer work at $F07A:

; Increment 60Hz frame counter and 256-frame attract/color timers inc FrameCnt ; F07A: E6 88 Increment frame counter bne CheckGravity ; F07C: D0 04 Every 256 frames (~4.27 sec): inc AttractTimer ; F07E: E6 89 Increment attract mode timer inc ColorCycle ; F080: E6 8A Increment color cycling offset

When you press Game Reset (HandleGameReset at $F113), the code sets AttractTimer ($89) to $01:

inx ; F119: E8 X = 1 stx AttractTimer ; F11A: 86 89 AttractTimer = 1 (active game)

Here's the bug: nothing in the active gameplay loop ever resets AttractTimer ($89) back to 1! Only ColorCycle ($8A) is zeroed on each active frame (sta ColorCycle at $F139). Neither paddle hits nor scored points reset $89.

So if a game lasts $255 \times 256 = 65,280$ frames without someone reaching 21 points, AttractTimer ($89) increments from $FF to $00. On the very next frame, CheckActiveGame ($F12B) checks if the game has timed out:

CheckActiveGame: bcs EnterSelectMode ; F129: B0 CD If Select pressed, switch to Select Mode ldx AttractTimer ; F12B: A6 89 Has AttractTimer wrapped to 0? beq UpdateSelectDisplay ; F12D: F0 DA If 0, game timed out into Select/Attract mode!

In 1977, Joe Decuir reasonably assumed a game of Pong to 21 points would never take 18 minutes. An RL agent in ALE, however, will happily rally for 65,280 frames and abruptly trigger beq UpdateSelectDisplay, dumping the emulator straight back into Game Select mode mid-episode!

Most human players don't realize that the paddle FIRE button actually does something during normal 1-player Pong (Game 1), beyond serving the ball in 2-player modes.

In GameRulesTable ($F712), Game 1 has GameRules0 ($97) set to $98 (10011000 in binary). Bit 7 ($80) enables a hidden steerable curve / vertical velocity doubler inside UpdateBallPosition ($F561–$F572):

lda #$00 ; F561: A9 00 bit GameRules0 ; F563: 24 97 Is extra vertical velocity / curve enabled (D7=1)? bpl AddBallVelY ; F565: 10 06 No, add 1x BallVelY bit LastHitBtn ; F567: 24 C3 Is the last hitter holding their fire button (D7=0)? bmi AddBallVelY ; F569: 30 02 No, add 1x BallVelY lda BallVelY ; F56B: A5 B8 Yes! Double vertical velocity (2 * BallVelY)! AddBallVelY: adc BallVelY ; F56D: 65 B8 Add integer vertical velocity + fractional carry clc ; F56F: 18 adc BallPosY ; F570: 65 B6 sta BallPosY ; F572: 85 B6 Update BallPosY

When an RL agent has access to the FIRE action in ALE, after hitting the ball (LastHitObj = 1), holding FIRE doubles the ball's vertical speed (2 * BallVelY).

Once a rally reaches 12+ hits, BallVelY ($B8) reaches its normal cap of $\pm 4$ scanlines/frame (IncrementSpeed at $F510). If the agent holds FIRE, the ball moves vertically by $\pm 8$ scanlines per frame.

Now look at how top/bottom wall bounces are handled in WallRuleTopBottom ($F3F5):

WallRuleTopBottom: cpy #$06 ; F3F5: C0 06 Did ball hit top or bottom wall (folded Y < 6)? bpl ApplyWallBounce ; F3F7: 10 0B No ora #$40 ; F3F9: 09 40 Flag Y velocity reversal (D6=1) ldy BallPosY ; F3FB: A4 B6 Nudge BallPosY 1 pixel toward vertical center: bpl NudgeBallDown ; F3FD: 10 02 If on bottom half ($80..$FF), decrement Y dey ; F3FF: 88 dey ; F400: 88 NudgeBallDown: iny ; F401: C8 If on top half ($00..$7F), increment Y sty BallPosY ; F402: 84 B6

When the ball collides with the playfield border (CXBLPF bit 7 = 1), the code reverses BallVelY and nudges BallPosY out of the wall by just 1 pixel. If the agent toggles FIRE right as the ball enters the wall at 8 pixels/frame (or releases FIRE on the next frame so it only steps 4 pixels), the ball can remain embedded inside the 16-scanline playfield border on the subsequent frame, triggering CXBLPF a second time, reversing BallVelY back toward the outside of the screen, and either trapping the ball oscillating inside the wall or wrapping BallPosY across $00 / $FF from the top of the screen to the bottom!

Just like Allan Alcorn's original 1972 discrete-logic arcade Pong cabinet had a famous flaw where the paddles couldn't quite reach the top of the screen, the Atari 2600 ROM has its own hardware-induced blind spot—plus a non-Markovian input lag that drove early RL researchers crazy.

Inside DrawCourtKernel ($F62A–$F657), the 6502 is busy racing the beam and only has time to read two of the four paddle potentiometers (INPT0–INPT3) on each frame, using Temp0 = FrameCnt & 1 ($80):

  • Even frames (Y = 0): Reads INPT0 (Left Player) and INPT2.
  • Odd frames (Y = 1): Reads INPT1 (Right Player—the RL agent) and INPT3.

Even worse, after the kernel finishes at scanline $DC, it smooths the raw potentiometer scanline reading (PaddleHit0, $84) with the previous value (PotValue,y, $BB/$BC) using a hardware 50/50 Infinite Impulse Response (IIR) filter:

lda PaddleHit0 ; F64D: A5 84 Smooth P0/P1 pot: (PaddleHit0 + PotValue,y) / 2 clc ; F64F: 18 adc PotValue,y ; F650: 79 BB 00 ror a ; F653: 6A sta PotValue,y ; F654: 99 BB 00

Because of this adc + ror a averaging every other frame:

  1. The paddle only updates on odd frames (temporal aliasing).
  2. Paddle position is non-Markovian: a single step change in virtual paddle resistance takes ~9 odd frames (18 video frames) to exponentially decay to its target value, and ror a integer truncation introduces a 1-pixel directional hysteresis!
  3. The Top-of-Screen Gap: The court kernel starts at scanline X = $26 (38) (ldx #$26 at $F5D6). Even if the paddle capacitor is completely discharged at the top of the court, PaddleHit0 can never be less than $26, while the top wall ends at scanline $2F (47) and the ball can squeeze into the top scanlines ($28–$2F).

One of the most famous phenomena in ALE Pong is that an RL agent can discover a "God Spot": it moves to one specific Y coordinate, parks its paddle completely still for the rest of the game, and beats the Atari AI 21–0 while the AI helplessly flails in an identical deterministic loop.

Why is the built-in AI so easy to trap? Here is the entire 1-Player Robot AI in the ROM ($F1E3–$F204):

lda FrameCnt ; F1E3: A5 88 and #$07 ; F1E5: 29 07 Y = FrameCnt & 7 (0..7) tay ; F1E7: A8 and #$01 ; F1E8: 29 01 Temp0 = FrameCnt & 1 sta Temp0 ; F1EA: 85 80

lda #$10 ; F1EC: A9 10 Is 1-Player Robot AI enabled (GameRules0 D4 = 1)? bit GameRules0 ; F1EE: 24 97 beq CheckPaddleBlink ; F1F0: F0 14 No, human controls P0

; Robot AI tracks BallPosY ($B6) on 7 out of every 8 frames (Y != 0) lda TrackedPos ; F1F2: A5 95 Load AI paddle Y position cpy #$00 ; F1F4: C0 00 Rest 1 frame out of 8 so AI is beatable beq StoreRobotPos ; F1F6: F0 0A cmp BallPosY ; F1F8: C5 B6 Compare AI Y with Ball Y beq StoreRobotPos ; F1FA: F0 06 Already aligned -> hold position bcs MoveRobotUp ; F1FC: B0 02 If AI Y > Ball Y, subtract 2 (move up) adc #$05 ; F1FE: 69 05 If AI Y < Ball Y, add 5 - 3

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →