Post-training 4B model with RL to fly Tesla Roadster 21% better than GPT-6 Astra

Last Thursday (October 15), Tesla's new Roadster drove off a ramp that curled over until the car was upside down, rolled upright in the air and landed. It then used thrusters to hop over a Cybertruck. Turns out the future does have flying cars.

The demo had already been pushed back two weeks, "due to high winds" (Elon Musk on X). That made us wonder: with a language model in the pilot's seat, how would the car do in that wind?

We built an RL environment around a physics simulator of a flying roadster to find out, handed the controls to LLMs, and turned the wind up. We let some of the latest best models fly it: DeepSeek V4.1 Flash, Meta's Muse Spark Open, Reflection's Beam 1.0, NVIDIA's Nemotron 3.5 Super VL, TypeSafe's Jev 1.14, and even OpenAI's GPT-6 Astra.

Asked for decisions in real time, these models crashed almost every flight: most take 1 to 6 s to answer, and the car needs a new action every 0.2 s. When we paused the world so the models could think as long as they wanted, they still lost to a 4B model we trained with RL in a storm.

Storm, 33 m/s wind: DeepSeek, with unlimited time to think, comes down leaning and hits the road with its body. Our RL-trained 4B lands level.

Watch how the models fly the car. Pick a stunt, a wind level and a model below to replay the same flight on a held-out scene. The full page also shows the pilot decision every 0.2 seconds, the telemetry the model saw, and how long each reply took.

The car is a rigid body, 4.5 m long and 2,000 kg dry, simulated at 500 Hz. Lift comes from four cold-gas thrusters at the corners of the floor of the car. Each thruster is good for 12 kN, fed from one tank of compressed air (110 to 170 kg per flight). They push straight out of the floor. If the left pair fires harder, the car rolls, and if the front pair fires harder, it pitches.

When the car is upside down, the same thrusters push it toward the ground, and that is what makes both stunts hard.

A car is built to move nose first. When it falls flat, or the wind hits it from the side, it shows five to nine times as much area to the air, so the wind pushes it around far more. Near the road the wind is weaker, and gusts come on top of the average. Each flight gets a mean wind from a random direction, in one of three bands: light (up to 14 m/s), gale (14 to 24 m/s) or storm (24 to 33 m/s).

The car can touch the ground with its four sprung wheels or with its body. It counts as a crash when:

Crash when it's moving down faster than

In the ramp flip, a ramp curls over and the car leaves the deck upside down 10 to 14 m up, then has to roll upright and land. In the kickflip, the car comes in at 50 to 94 km/h, clears a parked Cybertruck, rolls a full 360 degrees and lands.

A flight succeeds only if the car ends up on all four wheels on the 14 m road, within about 20 degrees of level, no longer spinning, and without a crash. Every task randomizes the speed, the air in the tank, the wind and gusts, and the ramp size or truck position. The held-out tasks use seeds that no model trained on.

At the start of each flight, the model gets a briefing. It describes the car, the crash rules and the five commands it can send, and then the task. Here is the task part of the briefing for one held-out storm flight:

After that, every 0.2 s of simulated time, the model gets one line of telemetry and replies with one JSON object. Here is the line our trained Qwen 4B model saw 1.2 s into that flight, while it was upside down above the truck:

The model replied {"mode": "rotate", "axis": "roll", "to": 360.0, "lift": 1.0} in 0.15 s, which kept the roll going. The whole flight took 19 replies:

On the road, truck 25.7 m ahead (reached in 1.1 s)

2.3 m up, truck 11.8 m ahead (reached in 0.5 s)

Rolling over the truck: 21°, 95°, 182°, 267°, 335°

Upright at 6.7 m, rolled 360°, truck 16 m behind

All four wheels on the road, 3 m from the center line

The truck part of each line also gives the time until the car reaches the truck and the time until it's past it, at the current speed, so the model doesn't have to do that division itself. Every LLM in this post sees the same lines, and the hand-written pilot below reads the same numbers.

The five commands are coast (all thrusters off), climb (stay level at a given vertical speed), hold (hold a given roll and pitch), rotate (spin about one axis to a total angle, braking to stop on it) and land (level out and come down softly). The model picks one command, with its numbers, in each reply. A flight computer then turns that command into thrust for each of the four thrusters, 50 times a second, until the next reply arrives.

If a model takes 2 s to answer, the car flies ten 0.2 s intervals on the old command before the new one takes over. We time every reply end to end. Only the first decision is untimed, like a plan made before takeoff.

We picked 0.2 s because the kickflip's roll takes about a second and has to start within a few tenths of a second of the right moment. With one second between decisions, no pilot could time it, whether model or script. To separate judgment from speed, we also fly every model a second way, with the simulator paused until each reply arrives, so the model has unlimited time to think.

For RL the reward runs from 0 to 1. It gives partial credit for height, clearing the truck, rotation and touchdown quality, then 0.4 for a successful landing and 0.1 for a clean finish (soft, with air left, centered on the road).

Before training anything, we wanted to know that both stunts can be flown with these five commands. So we wrote a pilot by hand: a few dozen lines of if-then rules that read the same telemetry and send commands through the same 0.2 s interface. On the kickflip, for example, it starts climbing when the truck is 1.1 s away, starts the roll once the car is 1.4 m above the truck's roof, leans against any sideways drift, and lands once the car is past the truck.

The hand-written pilot answers instantly and sees the same inputs to the LLMs. It lands every light-wind ramp flip and 98% of light-wind kickflips, but only 61% of kickflips in a storm. We call it the scripted pilot in the rest of this post.

The scripted pilot flying both stunts in light wind.

We flew four open models (DeepSeek V4.1 Flash, Muse Spark Open, Nemotron 3.5 Super and Beam 1.0), GPT-6 Astra in storms, and Jev 1.14, a model built for fast decisions that picks from a menu of commands instead of writing JSON. Every model flies the same held-out flights as our trained pilot: 192 per stunt in light wind and in a gale, and 384 in a storm. All runs use temperature 0 and each model's default reasoning setting. We called the open models through OpenRouter, with Nemotron, Muse and Beam each pinned to one provider.

When the simulator waits for each reply, the four open models fly the kickflip well in light wind. DeepSeek V4.1 Flash lands 98%, and Muse, Nemotron and Beam land 72 to 82%.

Wind makes it harder even then. In a storm, DeepSeek drops to 87%, Muse to 62%, Nemotron to 60% and Beam to 58%. GPT-6 Astra lands 74% of storm kickflips.

With the car moving, all four open models land 0% of kickflips. Their median decision takes 1 to 6 s, so a typical command arrives 5 to 29 intervals after it was needed. In the real-time runs, with 16 flights in parallel, replies were slower still (DeepSeek's median was 2 to 3 s).

GPT-6 Astra takes about 2 s per decision and lands none of its 768 storm flights in real time. On the same clock, our trained 4B lands 90% of storm kickflips.

DeepSeek V4.1 Flash on the same flight, twice. On the left the simulator waits for each reply, and the car lands. On the right the clock keeps running, the replies arrive late, and the car crashes.

In our first round, with reasoning off, DeepSeek answered in 0.3 s but made worse calls, and Muse and Beam at low effort still took about 2 s per decision. Jev answers in about 0.17 s, fast enough for the clock, but it lands only 2% of storm kickflips live. A pilot has to answer fast and make good calls.

The car leaves the deck upside down 10 to 14 m up, and a simple plan works: roll upright, lean against the wind, then brake and land. With the simulator paused, the big models fly it. Muse lands 88 to 97%, DeepSeek about 80%, and GPT-6 Astra 383 of 384 storm flights.

On the clock, none of them gets past 3%. The first command is usually a roll upright, which is correct, but the fall takes under 2 s, and the next reply comes after the car has already hit the road.

A ramp flip in real time. DeepSeek's first command, roll upright, is correct, but its next reply arrives after the car has hit the road. On the right, the trained 4B answers every 0.2 s and lands.

Muse, Beam and Nemotron land 1% or less in real time.

The untrained Qwen3-4B answers in 0.07 s and lands 85 to 100% of ramp flips, but it never lands a kickflip. Like Jev, it's fast enough for the clock. Unlike Jev, it's a model we can train, so it's where we started.

We started from Qwen3-4B-Instruct-2507 with a rank-64 LoRA (small add-on weights that train while the base model stays frozen). The training code is SkyRL, an open-source RL library. The final model took about 4 GPU-hours of training, or $13.

We save a checkpoint every 5 steps, pick one using separate validation seeds only, and then fly the held-out seeds once. The recipe itself came out of many rounds of experiments scored on validation seeds, and we held the test seeds back for this one final run.

The RL-trained 4B landing a storm kickflip in real time.

The rest of this section is how we got to that recipe.

The first run, RL from the untrained model, picked up the ramp flip in 10 steps. The kickflip reward started at 0.07 and stayed there. Of 68 kickflip decisions, 66 were a roll, and 21 of 22 first moves rolled at t=0 with the wheels still on the road, which puts the car on its roof.

All 8 flights in a group crashed the same way and got the same reward. GRPO learns by comparing samples within a group, so when they're all the same it has nothing to go on.

The same light-wind flight before and after training. Untrained, Qwen3-4B starts the roll with its wheels on the road and tips onto its roof. After imitation and RL, it climbs first and then rolls.

We tried two cheap fixes. First, we added a line to the briefing saying that rolling on the road tips the car over. The opening move became a roll with maximum lift, which still ended on the roof.

Second, we started half the practice flights partway through, after the scripted pilot had flown the opening (a reverse curriculum, borrowed from robotics). Training success rose to about 70%, but held-out success from t=0 stayed at zero, because the model only learned to finish a flip that the scripted pilot had started.

The warm start is what worked, and most LLM RL recipes start with one anyway. The hard part was choosing what to imitate, because neither pilot we had was good in every wind:

So the warm start copies the scripted pilot's light-wind flights and DeepSeek's landed gale and storm flights. On its own, the warm start already lands 97%, 93% and 84% of held-out kickflips in light wind, gale and storm, which is close to DeepSeek except in storms.

RL starts from what the warm start copied, then practices where the teachers were weakest. Two choices in its training mix came from reading failed flights:

Kickflip. Share of held-out test flights that land, with the number landed in brackets. There are 192 flights per wind level in light wind and gale and 384 in a storm, and every row flies the same tasks.

Scripted pilot (hand-written rules, decides instantly)

DeepSeek V4.1 Flash, unlimited time to think

Qwen3-4B after imitation only: fine-tuned on the teachers' flights, no RL yet

Qwen3-4B after imitation + RL (real time)

The trained pilot answers in 0.09 s (served by vLLM, one flight