
Scaling Pre-Training on Super Smash Bros. Melee
I trained a model named HAL to play Super Smash Bros. Melee using human data only and reached top ~1% of human play on ranked ladder1.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
I trained a model named HAL to play Super Smash Bros. Melee using human data only and reached top ~1% of human play on ranked ladder1. To my knowledge, HAL is the strongest base Melee model in the world.
Here’s a short highlight reel of HAL playing Fox against humans (names redacted with rank). Our model plays aggressively by chasing down opponents, finding surprising moves, but occasionally fumbling edge guard scenarios. I’ve included annotations for the non-Melee players.
This post is written with ML practitioners and technical readers in mind. The first blog post in this series covered our motivations, problem setup, and behavior cloning (BC) approach.
The final recipe is a 247M-parameter causal Transformer that reads the last 256 frames of game state and controller inputs (no pixels), then outputs a chunk of future controller inputs, decoded autoregressively frame by frame.
Because it predicts coherent multi-frame chunks, HAL only needs to replan every few frames, which hides both inference time and netplay delay, enabling it to play online in real-time on varying hardware and network conditions. We can also steer its play style and difficulty at test time.
Figure 1 shows the whole recipe, which we unpack section by section.
fig. 1 — how HAL learns and plays. Click any part to jump to its section
Why not reinforcement learning?
There exists another Melee agent trained with behavior cloning + online RL named Phillip that is quite competitive with pros. If that’s the case, why not move on to online RL already?
Over the last few months, I finally found the time and compute to chase these rabbit holes, the results of which I’m pleased to share. These directions were not necessarily the shortest path to superhuman Melee AI—at the end of the day, this was simply what piqued my curiosity.
Over the course of several months, I ran over 50 experiments and trained hundreds of models.
Except when stated otherwise, I tried to ablate ideas by training two models with equivalent compute, parameters, and data, then measuring differences in closed-loop eval3 over 96 episodes, 2 minutes each. I seeded a fixed set of stage and character match-ups that match the dataset’s prior distribution, then played each model against the in-game level 9 CPU, same as with our previous post. This was a cheap, reproducible baseline roughly on-par with the small-scale models.
A player wins the game if they take all 4 of their opponent’s stocks, or lives. Our primary metric is net stocks per minute (NSM):
All else equal, we prefer a winning policy to take stocks faster and a losing one to lose stocks slower.
NSM scales roughly linearly to Elo within [−1,1][-1, 1][−1,1], with +0.3 NSM ≈ +100 Elo. Outside of that range, NSM saturates asymptotically like Elo—our flagship model in the video above has only 1.7 NSM over the level 9 CPU but a 100% winrate over 192 games. At this sample size, NSM is accurate to small changes in behavior even across short episodes, which facilitates shorter evals.
We should be careful when comparing absolute NSM across experiments, because configurations in model size, learning objective, and training duration varied across experiments. Some models were more undertrained than others, while later experiments tended to stack the accumulated techniques of earlier findings.
Another flaw is that NSM is biased towards faster characters and smaller stages. To mitigate this, we can pair each treatment eval episode with a baseline episode of the same matchup, and call a result significant if the 95% confidence interval of a paired t-test over per-game differences excludes zero.
In general, I assumed that beneficial changes would have diminishing marginal returns but not destructively interfere.
Behavior cloning has well-known failure modes4. But, given the size of our dataset relative to the state space, we actually have excellent coverage of failure states5.
Fundamentally, if our goal is for the model to be really good at winning Melee, imitating everything humans do isn’t always optimal. Even after filtering bad examples from our dataset, there are bound to be human mistakes and players who focus on style over winning. Can we make better use of the signals we have from the human dataset?
Advantage-Weighted Regression (AWR) is behavior cloning where each training example’s loss is weighted by its advantage AtA_tAt—how much better or worse the return after that action was than a baseline.
The return Gt=∑k≥0γkrt+kG_t=\sum_{k\ge 0}\gamma^k r_{t+k}Gt=∑k≥0γkrt+k is simply a discounted sum of rewards from a frame to the end of the replay.
Through experimentation, I defined a symmetric reward function between two players:
where Δdt\Delta d_tΔdt is the percent damage a player takes on frame ttt, ktk_tkt is 1 on the frame they lose a stock, and ℓt\ell_tℓt is 1 if that was their last stock.
How did I decide on these reward coefficients? A typical stock is played to around 90-120% damage. With discounting, I wanted to properly incentivize the model to prefer taking stocks over dealing more damage (especially the last stock of the game), but in circumstances where that isn’t possible, to take favorable trades.
It was important to tune γ\gammaγ, which determines the discount rate or half-life of the reward. Too short, and the earlier frames that actually caused the outcome don’t receive attribution. Too long, and unrelated frames get affected, possibly even encouraging degenerate behavior like keeping an opponent alive to farm more damage. I found that a half-life of around 8 seconds was 0.37 NSM better than 3 seconds and 0.05 better than 14 seconds. Including damage in the reward function gave an extra 0.2 NSM—otherwise the signal was too sparse.
As for the baseline, I regress a value function head VVV on the return. Similar states show up with many different continuations in the data, so VVV learns their average (the return a typical human gets from here). The advantage asks whether this particular action did better or worse than usual from the same state, and its clipped exponential weights the action’s negative log-likelihood in the loss:
In Video 2, I color each character by its advantage, from purple (negative) through grey to orange (positive). The return sums rewards from the current frame forward, so the color reflects the next several seconds of play more than the character’s current position. Under the dense reward, Kirby turns purple while hitting Falco at the start of the clip, because Falco is about to punish Kirby. Kirby then turns orange while taking those hits, because Kirby goes on to win the game. Under the sparse reward, only the final stock counts, so both characters stay close to grey for most of the clip.
As a learning aid, I also created a visualization of how AWR re-weights the gradient in replays leading up to certain events, using real data and a heuristic for VVV:
fig. 2 — AWR playground
AWR improved my model by 0.29 NSM over behavior cloning.
Notably, AWR never assigns negative weight to any examples. Pushing the policy away from recorded actions could potentially learn worse or invalid actions. Instead, we can think of AWR as simply up-sampling and down-sampling training examples in proportion to how “good” the actions were.
I found AWR to be more principled than training only on player perspectives that won, since there is valuable training signal from imitating skilled players who lost.
While designing a reward function using damage, stocks, and winning worked, weighting training samples by skill ranking (i.e. weighting the gradient for a Master rank player 4×4\times4× that of a Platinum player) did not help.
I didn’t have much luck learning a good Q-function, but I didn’t try very hard.
I also tried IQL, which learns Q and V with TD learning and uses their difference as the advantage. Attaching Q and value heads to the policy scored −1.85 NSM, compared to +0.16 for plain behavior cloning, even with twice the data. Separate critic networks did about as badly (−1.55). The Q and V estimates drifted up past the clip ceiling and stayed there, so the advantages were basically noise. In hindsight, this made sense: function approximation, bootstrapping, and off-policy are known as the “deadly triad” of deep RL.
Fortunately, I could directly supervise VVV on dense ground-truth rewards and had no need to bootstrap a Q-function with TD learning.
In general, more conditioning is a great way to factorize multimodal distributions.
I added two inputs: which player to imitate and how much return it should go for. These double as levers for the model’s play style and “difficulty” at test time.
Return conditioning is inspired by Decision Transformer. Unlike AWR which reshapes the training distribution, return conditioning is a knob I can turn at test time without retraining.
At each frame, the label is the discounted return over the next 60 frames, with the same rewards and γ\gammaγ as AWR. The label only enters the action decoder, so the trunk and value head never see the future return (see implementation).
A problem I encountered was that I didn’t have a reliable learned distribution that I could sample the return-conditioning from. I had trained my value function on ground truth discounted return until the end of each replay, which has a pretty weak correlation to any arbitrary 1-second window during the game (Pearson r = 0.471). I didn’t want to take the effort to train a second value function just for this purpose, so at inference time, I request a constant return.
As it turns out, requesting a constant 99th percentile return on every frame results in stable play, +0.6 NSM over no conditioning.
90th percentile play is a lot more aggressive than no conditioning. In Video 3, 99th percentile play is more careful than 90th percentile without losing the aggression.
This is unsurprising, because in the data, 1-second returns fall into a starkly bimodal distribution: windows with only damage (<50 return), and 8% of windows that contain a stock event (>100 return). Requesting a 90th percentile return in these scenarios result in the model choosing suboptimal actions.
What’s interesting is that the model is able to generalize decision making associated with high return scenarios to other scenarios on stage at low percents (typically nowhere near a kill), as seen in the highlight reel.
I figured 60 frames was a reasonable horizon because our action chunks are only half that length.
The label enters the action decoder as a FiLM6 layer on each block’s attention and MLP RMSNorms, shared across the whole action chunk. The projections are zero-initialized, so training starts from the unconditioned model. Unlike the past-action FiLM in Inside each frame, the return is an optional extra signal layered on a policy that already works, so starting from “no effect” is the right default.
fig. 16 — where the return label enters
Player conditioning
I had over half a million games from 38 pros with nearly 20k unique player codes, plus anonymized ranked games tagged by rank (Platinum, Diamond, or Master). I told the model who it was playing as using a learned embedding for player code or rank.
Here’s a 16M model playing as Fox against the level 9 CPU, conditioned on a couple players known for playing Fox:
Platinum conditioning is clearly worse than Master or either of the pro players, which is a nice sanity check.
Across training checkpoints, player ID conditioning increased the Pearson r correlation between validation loss and net stocks/minute:
Up until now, we trained a model that makes only next-frame predictions. Even with history, human data tends to be multimodal and non-Markovian due to different play styles and reaction times. What if we also want to support larger models with slow inference or variable network delay?
We can predict actions for multiple steps ahead, also known as action chunking. In robotics, action chunking is
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。