DishwasherBench – Can Frontier Models Stack a Dishwasher?

DishwasherBench is a dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable by people in a browser. The game inside is called Spotless, and the packages are named @spotless/*.

A player loads a 3D model of a Bosch SMV4HAX40G with rigid-body physics. A wash on its Eco 50 programme is then simulated (jets, droplets, run-off, soil and detergent chemistry), and the player gets points for every item that comes out clean. On every level after the first, not everything fits, so the player has to choose what to load and how.

This is an early release, version 0.1.0. The simulation is calibrated on one test wash of the real machine. A score shows whether a player packed well and put cups, glasses and pots in mouth down. It does not yet show whether plates and cutlery were stacked so that they wash clean. The section What a score can be trusted for says so in full.

What is solid today: loading is deterministic rigid-body physics, every episode is a record that anyone can verify by replaying it and by washing its load again, and the same game is played by people in a browser and by models through MCP. What is rough is how the wash scores plates and cutlery.

Built and maintained by Composo. Many documents under docs/ are working records, kept as they were written; docs/README.md explains how to read them.

Loading the first level in the browser. A dinner plate is in hand over the lower basket, and "Suggest spots" has numbered the free seats, best first. The panel on the right holds the exact pose a program would send.

What works today

What a score can be trusted for

Requirements

Quick start

Play in the browser

Run a scripted player

Run a model

Results so far

How it works

Limitations and roadmap

Where things are

Contributing

Licence, notices and disclaimer

How to cite

What works today

The machine. One Bosch SMV4HAX40G as data and geometry, from its owner's photographs and tape readings (packages/machine): both baskets with their tine rows, shelves and rests, the spray arms and the Eco 50 programme.

Loading. A catalogue of items and soils, deterministic rigid-body loading (Jolt), and a session layer with the actions and queries of the game (place, remove, set_rack, list_slots, check_load and the rest).

The wash. A simulation at two tiers. The official tier gives the scored result; the fast tier is a coarser preview. A cold official wash of the largest level takes 4.9 minutes on the reference machine, a MacBook Pro (Apple M3 Pro, 18 GB), best of three, and 5 to 8.5 minutes with another job running beside it, against a budget of 30.

A course of seven levels, numbered in order of difficulty (docs/benchmark/levels.md), each with saved reference loads of three scripted players (careful, careless, random). Six pass the benchmark's gates on the reference loads; the tutorial does not (its careless load loses too little).

Three ways to play. The page (apps/web), an MCP server for agents (packages/mcp), and a harness that runs scripted players and models and writes verified episode records (packages/bench).

Checks. 48 loading phenomena from the literature as executable pairs of washes (docs/research/standards.md section 9), and a calibration tool that scores the simulation against a record of a real wash, fits parameters and ranks candidate test washes (the stored ranking is of an earlier parameter set and has not been run again).

Two models have played, through an OpenAI-compatible endpoint: six episodes on three levels at the official tier (Results so far). No person has played for a baseline, and the path for Anthropic's API has not been run against the live API.

The full account, with every number and where it comes from, is docs/validation/README.md. In short, the simulation has been checked against four things:

A score today shows whether a player packed well and put cups, glasses and pots in mouth down. It does not yet show whether plates and cutlery were stacked so that they wash clean.

About three quarters of the gap between a careful and a careless reference load is hollow pieces put in the wrong way up. The soils in those pieces (smoothie, tomato soup, ketchup, guacamole) rest on values from the literature that no test wash has checked.

Not modelled at all: drying, spots and film on glass, items moved by the jets, and soil settling on clean pieces. Only Eco 50 is calibrated and scored.

DishwasherBench has been run on one machine: a MacBook Pro (Apple M3 Pro, 18 GB) under macOS 27. Every figure in this repository was measured on that machine. Linux and Windows are untested. Nothing is known to rule them out, and a report of what happens there is welcome (docs/ROADMAP.md).

Results are repeatable bit for bit on one machine. Across machines, operating-system builds and versions of the webgpu package they may differ in the last bits, which is why every result names the stack it was computed on (docs/research/spike-gpu-compute-macos27-recheck.md).

git clone https://github.com/composo-ai/DishwasherBench.git cd DishwasherBench corepack enable # once: makes pnpm 9.4.0 available pnpm install --frozen-lockfile # about 15 seconds the first time

There is no build step for the packages: they are consumed as TypeScript source and run with tsx. Only the page is built, by the play command below.

The shortest thing that shows a result is to wash a saved load from the command line:

pnpm --filter @spotless/wash-runner wash-load tutorial-careful --tier fast # under a minute pnpm --filter @spotless/wash-runner wash-load tutorial-careful --tier official # about 5 minutes the first time

It prints the points and, for each item, the verdict, the score and what is left on it. The saved loads are in packages/loader/loads/ (<level>-<player>.json). The first wash of a load works out where the water goes and keeps that in .spotless/exposure-cache, so a second wash of the same load takes seconds (pass --no-cache to switch this off).

To check that everything runs on your machine:

pnpm typecheck # the root and every package; a few seconds pnpm test # every package; 8 to 17 minutes on the reference machine

How long pnpm test takes depends on what else the machine is doing. One test is skipped by design. The tests run real washes on the GPU.

A wash refuses to run when the parameter set is stale. The set the game washes with (anchored-v7.owner-capsule, in packages/calibration/anchors/) records the state of the simulation source and of the machine data it was made for. After a change to either, the wash stops and lists what differs. CONTRIBUTING.md explains the rule and how to make the set current again.

pnpm --filter @spotless/web play

This builds the page and starts the game server. Open http://127.0.0.1:8787 (add --port <n> for another port). The page plays music: mute the tab, or use the Sound button, if you do not want it.

If the port is taken the server stops with serve: port 8787 is in use; give --port <n>. The server answers only the page it serves itself: a request that carries the Origin of another site gets 403 unless the server was started with --allow-origin scheme://host[:port] (may be repeated), and a request whose Host header names another host than the addresses the server listens on (127.0.0.1, localhost and [::1] by default) gets 403 as well (--allow-host <name> adds a name). Requests without an Origin header (the harness, the MCP server, curl) are not affected, and the client under the Vite development server needs no flag.

When you start the wash you are offered two kinds:

The game is scored on Eco 50 with one capsule of detergent (Finish Powerball Ultimate, 12.9 g). The levels are listed 1 to 7 with a difficulty figure, which is worked out from what is on the counter and is not a measured score.

Placing by pointing works for plates in a free row, bowls and cutlery. Two things do not work by pointing yet: a plate pointed at the gap next to a plate already placed often perches on its neighbour, and cups and glasses cannot be leant on the side rests of the upper basket. For both, pick the piece up and press "Suggest spots" (it costs one query), then click a numbered mark.

After a quick preview of a careless load of the first level: a bowl left mouth up holds 432 ml of water and is not clean, and the report says that a fork held the lower spray arm still. The banner says what a preview is: not the scored result. All of this is the simulation's output for a deliberately bad load, not a test of the real machine or detergent.

Looking closely at one item after the wash: the residue is painted where it was left, with its area and the water the item held.

A scripted player needs no model and costs nothing. This plays the first level with the careful reference player at the official tier, then verifies the record by replaying its actions and by washing the load a second time (about five minutes each):

pnpm --filter @spotless/bench bench run --levels tutorial --policies careful
--no-render --out "$PWD/.spotless/bench/scripted" pnpm --filter @spotless/bench bench verify
"$PWD"/.spotless/bench/scripted/tutorial__careful__*/record.json --rewash

Give --out as an absolute path; a relative one resolves inside packages/bench.

The scripted players are empty, random, careless and careful. Add --tier fast for a preview in about a minute; a fast wash is labelled as a preview in the record and in the results table, and is never a score.

Each run writes under --out: record.json, a results table (results.md, results.json), the data of the game server it started (server-data/) and, without --no-render, the pictures the tools returned.

The scripted careful player's load of the largest level (62 pieces on the counter, 49 loaded), from above and from the front and side. Both models that have played beat it.

Everything about the harness, what each kind of player sees and what is recorded is in docs/benchmark/README.md.

A model run costs money. Copy .env.example to .env (which git ignores) and fill in the address and the key of your endpoint. The command prints the names of the variables it set, never a value, and neither the key nor the address is written to a record.

A model behind an OpenAI-compatible endpoint (OpenAI, Azure, or any server with the Responses or chat completions API):

pnpm --filter @spotless/bench bench run --levels tutorial --policies openai:gpt-6.1-sol
--max-input-tokens 2000000 --out "$PWD/.spotless/bench/sol-tutorial"

The policy names the model or, on Azure, the deployment: openai:<model or deployment>.

With no address set, no request is made: the episode ends at once with no result, and its record says no address of the API (set SPOTLESS_OPENAI_BASE_URL, or give --base-url). The command then exits with status 1, and pnpm reports the failed script.

--max-input-tokens ends an episode before its requests add up to more. As measured with two models: the first level takes 0.4 to 0.65 million input tokens, a level of 37 pieces 2.5 to 2.8 million, and the largest level (62 pieces) 3.0 to 3.2 million, with 95 % or more of it read from the provider's cache. Use limits of 2,000,000, 12,000,000 and 20,000,000.

The harness records tokens. It records a cost only when prices are given in .env.

Without --tier fast the wash is at the official tier, which is the score.

Each run writes the record, the transcript, the pictures and a results table under --out, and verifies the episode by replaying its action log.

A model through Anthropic's API, with the key in ANTHROPIC_API_KEY (put the model's id in place of <model id>):

pnpm --filter @spotless/bench bench run --levels tutorial --policies anthropic:<model id>
--max-input-tokens 2000000 --out "$PWD/.spotless/bench/anthropic-tutorial"

This path has not been run against the live API. It has unit tests against a local stand-in and nothing else behind it. A report of a first run is welcome. With no key the episode fails at once (Could not