RMSNorm

So, that was reassuring! As expected, training a Transformer-based LLM without any kind of normalisation doesn't work well.

The other training runs were much less eventful -- they all completed normally, and I downloaded the models, and ran my normal held-back test set through them to get a comparable quality metric. Here's what I got:

The links go to the models themselves on Hugging Face.

These are really interesting! Let's look at training speed first.

What we're meant to be looking into here is the performance characteristics of RMSNorm versus LayerNorm, so let's compare like vs like. The hand-rolled RMSNorm was indeed a bit faster to train than the hand-rolled LayerNorm, but only about 2.5% faster, both in terms of training time and tokens per second. For the runs using the built-in PyTorch modules it was an even closer-run thing -- only 0.5% improvement for RMSNorm.

Without doing tons of extra training runs I can't say safely what the noise level is for this kind of thing, but my gut feeling (based on having trained more than 60 of these models now) is that the 2.5% difference is probably real, while the 0.5% one could well just be noise. In particular, it's worth noting that both of the "hand-rolled" training runs happened on the same instance (by chance), though the PyTorch ones were on different ones, which makes the 2.5% number more trustworthy.

But on to the elephant in the room: the difference between the fused PyTorch implementations and the hand-rolled ones is much bigger. With LayerNorm, it took 11% less time to run. The speedup for RMSNorm was a little smaller, at just less than 10%. That is definitely not noise, and completely eclipses the speedup from switching the kind of normalisation! 6

Now let's take a look at the quality of our models.

Again, we'll start with the like vs like comparison. Hand-rolled RMSNorm got a test loss that was 3.601292−3.595336=0.005956 worse than the hand-rolled LayerNorm. That's a tiny difference.

Similarly, the PyTorch RMSNorm got 3.572383−3.571158=0.001225 worse loss than the PyTorch LayerNorm.

I'm tempted to say that those numbers were within the noise. However, I don't have any numbers to back that up -- when I looked into noise previously I did measurements for variations in weight initialisation, and for runs with the same initial weights but different random seeds for the post-model-initialisation steps, but those runs used dropout, which the ones in this experiment did not.

In theory, my training loop should be entirely deterministic; no randomness is used after the initial weight initialisation. A while back I did two training runs on 8x A100 machines with the same config, months apart, and the Safetensors files were identical bit-for-bit.

But what we did here was start with a specific set of initial weights, and then see what happened when we trained them with different normalisation settings. It's certainly conceivable that some initial weights might be "in a better place" for training with LayerNorm while others were in a better place for RMSNorm.

So while the fact that both were a little bit worse with RMSNorm is interesting, I'd really need to train a bunch of different models to get any certainty here. Best to chalk it up to noise for now.

Now, back to that elephant... Once again, the big difference here is not between LayerNorm and RMSNorm, but between hand-rolled and built-in PyTorch implementations. For LayerNorm, the PyTorch version got 3.595336−3.571158=0.024178 lower -- better! -- loss than the hand-rolled one. And for RMSNorm, the difference was 3.601292−3.572383=0.028909.

Those are not super-small differences. Based on the various interventions I tested against the GPT-2 code, that's larger than the improvements I got from switching to full-fat 32-bit rather than using Automatic Mixed Precision (AMP -- at 0.012558), more than I got from adding on gradient clipping (0.013209), and more than adding on bias to the QKV weights (0.022141).

I'd need to do significantly more training runs in order to find out if this was due to some kind of noise, and I'll not spend that time and money now. It feels like there are two possibilities:

On the one hand, the fact that the results were consistent is interesting. If the PyTorch models had both been better than the hand-rolled ones but by different amounts, then that would be easy to ascribe to chance (I make it a 25% probability, though you could argue for 1/6). But the fact that the improvements were so similar is harder to explain.

On the other hand, a systematic difference would be really weird, at least to me. I'm confident that my hand-rolled implementations match the PyTorch ones in terms of the maths -- not only have I checked and double-checked the code, but the LayerNorm one was written by someone expert in the field, and I've also run them past both Claude Opus 5.5 and GPT-6 Astra, both on high thinking levels, to check that there are no differences 7. So while I can't rule errors out, I've done a lot to try to eliminate them.

For now, I think I'll loosely chalk this up to noise too. I've seen weird coincidences crop up before -- regular readers might remember that in my post on LoRE, the loss delta of using it on the input side only, when added to the delta of adding it to the output side only, was almost exactly the same as the delta for adding it to both ends. That repro'd over two separate random seeds and really looked like it was a thing -- only to disappear with a change to a part of the code that should not have affected it if it had been a real additive effect. More Tests Needed.

I think an interesting set of experiments to test that would be not only to try with different random seeds, but also to try with different weight initialisation strategies entirely. What would these results look like if, instead of using PyTorch's default initialisation, we used the one they describe in the GPT-2 paper? (More on that in an upcoming post.)

But back to this experiment: what can the results that we have right now tell us?

I started this set of experiments with two goals: to understand how RMSNorm works, and to see if I could show similar results to the original paper -- that it gets comparable loss to LayerNorm, with a noticeable saving in training time.

I'm happy with the first part of that -- and hopefully if you're reading this, the "how it works" part is clear for you too!

The loss was, I think, also as expected. Again, I'd need to do a much larger-scale experiment with various starting weights to be sure, but the fact that the difference between RMSNorm and LayerNorm was so small was certainly reassuring.

The training speed was a much less impressive result. With hand-rolled code, the 2.5% speedup looked potentially real, but it pretty much vanished when I switched to PyTorch's built-in classes.

The big difference, of course, was that between PyTorch's built-in classes and the hand-rolled ones. The speedup from using fused kernels for this stuff was not such a big surprise, especially after the similar improvement from using the built-in GELU. It would certainly be interesting to see if using torch.compile narrowed the gap, though.

But the improved loss was definitely interesting. Again, I'd need to do a ton of training runs to see if that was real or noise. It seems most likely that it was noise, but the fact that the effect was positive for PyTorch in both cases, and was by such a similar amount, keeps nagging at me.

One remaining question aside from that, though: RMSNorm wasn't obviously significantly faster, especially if you use fused kernels, and while it wasn't obviously worse than LayerNorm, it didn't appear to be better -- so what explains its popularity in so many open weights models?

The first thing I thought was that maybe it's something that becomes more effective in terms of cost reductions as models get larger. However, having thought it through, I think that exactly the opposite is true -- its effects get even smaller as we scale up.

Remember that -- in terms of calculations -- RMSNorm saved 3demb additions/subtractions for each context vector. That would scale smoothly as demb was increased, and of course the number of RMSNorms would increase proportionally as the model gained more layers. So I think that the absolute compute reduction in terms of calculations saved would scale roughly linearly with each of width and depth.

But other aspects of the model's cost scale more than linearly with width. For example, in GPT-2, our FFNs map from demb to 4demb and back again; that means that the number of calculations scales as O(demb2). More modern implementations use SwiGLU, but I believe that the scaling is of the same order.

And that's not even to mention the fact that modern models have longer context, and while the norms scale linearly with the sequence length (and the RMSNorm savings do too), full attention is O(n2) on that.

So if we're only getting, say, 0.5% improvement for these tiny 163M parameter models with a 1,024 token context length, it would be an even smaller percentagewise improvement for larger ones.

Even if that wasn't the case, the amount is pretty small: DeepSeek V3 was widely reported to have cost US$6M to train (excluding R&D and a bunch of other stuff, but that's actually the correct number to use here). So even if we ignore all of the above and use it directly, that would just be a US$30,000 saving from using RMSNorm. And that's basically an upper bound -- we'd expect it to be significantly less.

Another possibility, though, is that it's an architectural thing. These open models have a bunch of differences to the GPT-2-style code I've been using here. They use RoPE for position encoding, and SwiGLU for the feed-forward networks. Mistral has multiple differences in the attention mechanism. On the face of it, I don't see any obvious reasons why those might speed up RMSNorm, but I can certainly imagine that they might "prefer" one form of normalisation over another, resulting in better loss. So perhaps that's what people have found.

But there are plenty of other possibilities. If people have generally found that it's just as good as LayerNorm, then perhaps it's just a fashion -- one model uses it, then the next one has no particular reason to move back, and so on. And there is a general move towards dropping biases from LLMs (for example, from the QKV weights), and RMSNorm would fit that. And anyway, it is a bit simpler!

Whatever the reason: if RMSNorm is that little bit simpler than LayerNorm, by skipping the recentring aspect of normalisation, is there something even simpler that could work even better? While discussing the subject with Claude Opus 5.5, it mentioned "Transformers without Normalization" 8, which is a really interesting paper -- one of the authors has a neat intro on GitHub. They noticed that the transformations that normalisation layers tend to learn wind up looking a bit like the S-shaped mappings you get from the tanh function, so tried using that instead with a system they called Dynamic Tanh (DyT) -- and got good results!

The first sentence in this post was "Normalisation is an essential part of LLMs". Was that wrong? I suppose that depends on what you mean by "normalisation" -- could you regard their tanh trick as being normalisation of some kind? -- though given that Yann LeCun was a co-author on the paper I should probably be a bit hesitant to disagree with his definition of it ;-)

I think their DyT system is something for a follow-on post, though. For now: RMSNorm seems to be essentially the same in terms of model quality, and maybe slightly faster than LayerNorm for the models I've been playing with, at least if you don't use the fused modules.

But using the fused PyTorch modules instead of hand-rolling our own normalisation is a huge win for speed. The loss im