Highlighting What Changed: A Go Diff-Mask for AI Agent Screenshots

Programming

AI Agent

Highlighting What Changed: A Go Diff-Mask for AI Agent Screenshots

Computer-use agents re-read the whole screen every turn. A binary change mask that marks only the pixels that moved.

Cheikh seck

8 min read·

13 hours ago

--

Your AI agent can’t click what it can’t see. But it also shouldn’t have to re-read the entire screen every time it takes a step.

That’s the loop every computer-use agent runs: screenshot the screen, send it to the model as vision input, get back a click or a keystroke, execute it, screenshot again. Anthropic’s computer use toolset ships seventeen member tools — screenshot, left_click, type, zoom — and the docs are explicit that managing screenshot history is part of building a working agent, not an afterthought.

Here’s the thing that surprised me when I went looking for numbers. Screenshots are not the expensive part. A measurement of 25,457 real agent tool results across 329 sessions found a screenshot costs a median of 2,054 tokens — essentially the same as reading a single file (2,085). Screenshots were the most predictable tool in the entire set, varying only 1.7× between median and p90, because an image is billed by pixel area and a browser window is always roughly the same area.

So the problem isn’t cost. It’s attention. Most of the screen is identical between steps. Only a small region changed: a menu opened, a toast appeared, a button toggled. You’re paying to re-describe a static desktop, and the model has to find the one…