The Escape Room Economy

";inherits:false;initial-value:100%}@property --tw-inset-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-inset-shadow-color{syntax:"";inherits:false}@property --tw-inset-shadow-alpha{syntax:"";inherits:false;initial-value:100%}@property --tw-ring-color{syntax:"";inherits:false}@property --tw-ring-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-inset-ring-color{syntax:"";inherits:false}@property --tw-inset-ring-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-ring-inset{syntax:"";inherits:false}@property --tw-ring-offset-width{syntax:"";inherits:false;initial-value:0}@property --tw-ring-offset-color{syntax:"";inherits:false;initial-value:#fff}@property --tw-ring-offset-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-outline-style{syntax:"";inherits:false;initial-value:solid}@property --tw-blur{syntax:"";inherits:false}@property --tw-brightness{syntax:"";inherits:false}@property --tw-contrast{syntax:"";inherits:false}@property --tw-grayscale{syntax:"";inherits:false}@property --tw-hue-rotate{syntax:"";inherits:false}@property --tw-invert{syntax:"";inherits:false}@property --tw-opacity{syntax:"";inherits:false}@property --tw-saturate{syntax:"";inherits:false}@property --tw-sepia{syntax:"";inherits:false}@property --tw-drop-shadow{syntax:"";inherits:false}@property --tw-drop-shadow-color{syntax:"";inherits:false}@property --tw-drop-shadow-alpha{syntax:"";inherits:false;initial-value:100%}@property --tw-drop-shadow-size{syntax:"";inherits:false}@property --tw-backdrop-blur{syntax:"";inherits:false}@property --tw-backdrop-brightness{syntax:"";inherits:false}@property --tw-backdrop-contrast{syntax:"";inherits:false}@property --tw-backdrop-grayscale{syntax:"";inherits:false}@property --tw-backdrop-hue-rotate{syntax:"";inherits:false}@property --tw-backdrop-invert{syntax:"";inherits:false}@property --tw-backdrop-opacity{syntax:"";inherits:false}@property --tw-backdrop-saturate{syntax:"";inherits:false}@property --tw-backdrop-sepia{syntax:"";inherits:false}@property --tw-duration{syntax:"";inherits:false}A primer on reinforcement learning (RL) environments

Reinforcement learning (RL) environments were one of the central focus points at ICML this year. It wasn't necessarily a surprise that this was an important theme, but I was surprised to see how prevalent this was.

Since then, a lot of people have asked me about RL environments, so I put this little primer together.

Imagine that you're locked in a room. There's a padlocked box on the table, a UV flashlight taped under a chair, a poem on the wall with three words underlined, and a keypad by the door. There’s no instruction manual and you just start pulling on things — shining the light, rereading the poem, punching in combinations — until the keypad blinks green and you tumble into the hallway with a minute to spare.

It turns out an escape room is the perfect analogy for understanding reinforcement learning environments.

The escape room is the world you're stuck in, and it has objects, rules, and feedback (when you put in a code, the lock unlocks or it doesn't). In reinforcement learning, that's called the environment.

The flashlight, the box, the keypad are the tools you're allowed to use. You can only do what the room/environment and tools let you do.

And so then you spend some amount of time fumbling around inside the escape room, trying different combinations of tools, and hopefully you get out.

After an hour, it ends: you either escaped or you didn't. This is called the verifier, and a clean binary result in escape rooms makes this verification very easy.

Now imagine that instead of sending a person into the room, we send an AI agent. We force it to go through the escape room ten thousand times. It tries different things in each attempt, and fails the vast majority of the time. But, there are a few attempts where our AI agent is able to escape. So we tweak the model a bit to act more like the times it escaped. Then we send it back into the escape room, run it thousands more times, give it feedback, and run it more. Imagine doing this for millions of accumulated runs — at the end, you'll have a machine that is really good at this particular escape room.

That's the entire idea of RL environments. Rather than explicitly teaching the machine the answer, you let it learn through experience. You build a room, give it tools to use, tell it to escape, define what "escaped" means, and let it grind against the door until it learns.

Then you build the next room, with different and harder tasks, and then repeat…

At this point, we've basically run out of internet data. Pre-training — the phase where a model reads more or less everything humans have ever written — works astonishingly well, but what do you do when you've already fed in all the existing data? Researchers are excited about RL environments because they can manufacture brand-new training signals on demand. Every time the agent runs through the room, it generates fresh experience.

The second is cost. The initial methods of improving a model after pre-training both lean on direct human feedback. In supervised fine-tuning (SFT), a person has to provide the ideal answer. In reinforcement learning with human feedback (RLHF), a person has to explicitly rank which of the model's answers are better. Both of these methods are slow, expensive, and also cap the model's performance at a human ceiling. The beauty of an RL environment is that once it's built correctly, you can blast it with compute and generate an infinity of data.

With this shift, the economics of post-training become fundamentally different. Previously, we would pay an expert $100 for a single task solution. Now, the $100 goes toward an expert creating a task and a verifier, thus allowing the model to learn from many attempts.

To clarify, SFT and RLHF are still used as standard steps, but RL environments are added on top.

Finally, AI is heading towards more task-orientation. Autoregressive token prediction (i.e., predicting the next word) gives us models that we can converse with, but we want our models to do so much more: book the flight, clear the tickets, file the taxes, ship the bug fix, etc. This action orientation is a different skill, one that’s hard to learn by reading about it. You learn it by trying, failing, and getting a useful signal on whether you actually pulled it off. RL environments are the current best way we know to teach doing at scale.

The environment is the world — the tools, the rules, and the feedback. The action space is the full set of things the agent is allowed to do in this world. A room with a single numeric keypad has a tiny action space, but a room with a computer that can browse, send email, edit files, and run code represents an enormous action space. Roughly, the bigger and more realistic the action space (i.e., high-fidelity environments), the more capable the environment, and the more it costs to build.

Within the environment, the agent is given a task. In the escape room, the task is simple and singular: get out. In contrast, frontier labs actually pay for environments with many complex tasks — a "tax" environment might contain "file this return," "find the error in this filing," and "respond to this IRS notice." These environments and tasks might even share the same set of tools for the agent to use.

The verifier is how you grade the attempt. In contrast to the escape room analogy, verifier rewards are not necessarily binary. One intuitive way to think about it: by providing partial credit, you are able to provide more granular feedback to the model.

Depending on the task, the verifier comes in two very different flavors. When the answer is programmatically checkable (and assuming you’ve set up the right checks), verification is trivial. Did the code pass its tests? Did the number match? Did the keypad turn green? Many scientific and mathematical domains (e.g. coding) are generally verifiable. For these, the check is a short script and it costs nothing to run a billion times. Verifiable domains are the cleanest, most scalable corner of the whole field.

But often the most valuable knowledge work has no clean check. Is this essay any good? Is this legal summary faithful to the source? No programmatic Python script is able to judge "good" here. Instead, you write a rubric — a criteria grading list of what "good" means (e.g., this essay covers the three required points, no factual errors, cleanly structured) — and hand it to another model that scores the work against the rubric. Here, good verifiers are softer, noisier, and far harder to get right: human judgment is the scarce input.

In the escape room vignette, I hand-waved through this by saying "we tweak it (the model) to act more like the times it escaped." Let's dive a bit deeper.

Every time the agent runs the room, it produces a rollout (also known as a trajectory), which is the full recording of that one attempt. It is a log of every action it took during the run, in order, plus the verifier grade at the end. Run the room ten thousand times and you have ten thousand rollouts. Again, most of them will be failures, and only a handful of them are successes.

Then comes the actual reinforcement learning. The RL algorithm (often PPO or GRPO) compares groups of attempts, then nudges the model's parameters to make the choices from the better-than-average attempts more likely and the choices from the worse ones less likely. Maintaining the right task difficulty is a tricky balance — if all attempts at a task succeed or all fail, these algorithms aren’t able to extract any signal to improve the model. Repeat across millions of rollouts and the model drifts, bit by bit, toward whatever the verifier rewards.

Every time a new model is released, researchers will include the model’s performance against a series of benchmarks — SWE-Bench, Humanity’s Last Exam, Terminal-Bench, etc. — are classic examples. The purpose of a benchmark is to ascertain the model’s abilities in a particular domain that researchers care about. A good benchmark also provides enough performance dispersion between models.

Publishing good benchmarks is a powerful method for an RL environment company to sell to its frontier lab customers. Fundamentally, a benchmark task is functionally the same thing as an RL environment, wrapping together a container, task, and verifier. By crafting a good benchmark, RL environment companies demonstrate the combination of technical skill and taste to construct good environments. Think of it as a work sample, of sorts.

Note that the RL environments purchased by the labs are not the same environments as the benchmarks. What a startup actually sells is environments that look like the benchmark environments, but aren't the same ones; for example, an RL environment might have thousands of tasks, whereas a benchmark might only have hundreds. Then the model gets better at the underlying skill during posttraining, and the benchmark score goes up as a side effect.

Constructing a high-fidelity environment is a war against inconsistency. Imagine asking a model to look up the weather, but you don’t pin it to a fixed date; then the "correct" answer would change every day you run it. It only gets harder the more lifelike the room, because realistic worlds are stateful: an action in one task alters the world the next task starts in (e.g., a row in a database is changed), so a mistake early on can silently poison everything downstream.

Furthermore, you often get what you pay for: a throwaway UI mockup might cost around $20K while a faithful Slack replica is closer to $300K. If you skimp on the realism, you might get a buggy, vibe-coded environment that doesn’t allow the model to learn anything. Hunting down every source of drift, all while keeping the world real enough to be worth learning, is painstaking