
The Gradient Is a Covariance
Why Post-Training Is Sparse, Part 1: the score, the Fisher, and a bound that holds for any reward
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Why Post-Training Is Sparse, Part 1: the score, the Fisher, and a bound that holds for any reward
Why Post-Training Is Sparse · Part 1 of 3. Part 2 finds where the sparsity comes from. Part 3 asks what policy gradient can reach.
The figures step through like slides. Use → or ▶, and step back with ←.
Reweighting what the model already samples explains why the update is small, not why it lands on a few coordinates.
RL fine-tuning changes only 5–30% of a model’s parameters, across seven algorithms and ten models (Mukherjee et al., 2025). On-policy distillation and SFT on near-policy data behave the same way (Zhang, Mukherjee & Peng, 2026). Two common explanations are that RL only reweights answers the model can already produce, and that a scalar advantage with $B$ possible values carries at most $\log_2 B$ bits per episode (Information Bandwidth). The first says the update is small, the second that it carries little. Neither makes it sparse: a hundred million parameters each moving a little is small and dense, and can carry those few bits just as well. The question is which coordinates move, and why the same few can be found before any reward is seen.
On the model’s own samples the score averages to zero, so the expected gradient is a covariance and the Fisher is a variance.
scoreAmean score 0parameter jscoreAmean score 0spread: Fisher Fspread: Fisher FG = Cov(A, score) > 0G = 0the same cap for both: |G| ≤ std(A) · √F
← ▶ → 4 / 4
The line can never leave the wedge of perfect correlation, which is the cap |G| ≤ std(A) · √F: the same for both. A large Fisher allows an update; only the reward makes one.
Probabilities sum to one, so whatever a parameter does, the responses it raises are paid for by others, and weighted by probability the changes cancel. With the score’s mean at zero, the policy gradient $\mathbb{E}[A\cdot\mathrm{score}_i]$ is a covariance and the Fisher is a variance; Cauchy–Schwarz gives the bound. A baseline that depends only on the prompt changes nothing in $G_i$, since $\operatorname{Cov}(A-b,\mathrm{score}_i)=\operatorname{Cov}(A,\mathrm{score}_i)$. It changes only $\operatorname{std}(A)$ and the noise of the estimate, and the Optimal Token Baseline is the choice that minimizes that noise.
A large Fisher is necessary for a large gradient, not sufficient. After “She drank a cup of”, the output rows for tea and coffee have a large Fisher; if the reward does not care which one is chosen, their gradient is zero. Measured before RL training, the top 1% of coordinates by Fisher hold 47.40% of the top 1% by gradient magnitude, against 1% at random (Zhang, Mukherjee & Peng, 2026). The Fisher says where learning can happen; the reward decides where it does.
The response space is finite (finite vocabulary, bounded length), so every exchange of a derivative and a sum below is exact. Probabilities are positive, which softmax sampling at temperature 1 without top-k or top-p guarantees. The reward $r(x,y)$ and the prompt distribution do not depend on $\theta$.
Subtracting $\mathbb{E}[r],\mathbb{E}[\mathrm{score}_i]=0$ turns the product into a covariance, and the same step turns $\mathbb{E}[\mathrm{score}_i^2]$ into a variance. Both hold per prompt, so $G_i=\mathbb{E}_x[\operatorname{Cov}(r,\mathrm{score}_i\mid x)]$, which is why a baseline $b(x)$ drops out.
The full Fisher $F_{ij}=\mathbb{E}[\mathrm{score}i,\mathrm{score}j]$ is also the expected curvature: $\partial_j\mathrm{score}i=\partial_i\partial_j\pi\theta/\pi\theta-\mathrm{score}i,\mathrm{score}j$, and the first term has mean $\sum_y\partial_i\partial_j\pi\theta=0$, so $F{ij}=-\mathbb{E}[\partial_i\partial_j\log\pi\theta]$. It must be estimated on responses the model samples. On data labels the same formula gives the empirical Fisher, a different quantity (Part 2).
At one prefix, a parameter’s token Fisher is the variance of how strongly it moves each candidate’s logit.
0.50tea0.40coffee0.05water0.03milk0.02juicemeanfi ≈ 0.08
She drank a cup of ___sensitivityκi,aprobabilityp(a)0.50tea0.40coffee0.05water0.03milk0.02juiceevery κ raised by the same amountmeanfi ≈ 0.08
She drank a cup of ___sensitivityκi,aprobabilityp(a)0.97tea0.02coffee0.005water0.003milk0.002juicea confident modelmeanfi ≈ 0.008
She drank a cup of ___sensitivityκi,aprobabilityp(a)0.50tea0.40coffee0.05water0.03milk0.02juicetea and coffee react alikemeanfi ≈ 0.002
← ▶ → 6 / 6
So do likely candidates that react alike: over thirty times smaller.
The softmax subtracts a mean: the token score is how strongly $\theta_i$ moves the drawn token’s logit, minus how strongly it moves the average candidate. Without that subtraction the second moment would be $f_i(s)+(\bar\kappa^{p}_i)^2$. The removed part is the common component, which raises every logit together and changes no probability. The pairwise form says the same thing: only differences between likely candidates count. So a parameter has a small token Fisher exactly when it acts about equally on the candidates the model finds likely, either because the model is confident or because those candidates react alike.
Across positions the cross terms average to zero, so the Fisher is a sum of token Fishers.
score(1)with2score(2)milk3score(3)and4score(4)honey5score(5)one token scoreper positionmean 0 given its prefix(Σt score(t))² = Σu,t score(u) score(t)112233445500000000000000000000for u < t, score(u) is fixedby the prefix of t, so each pairaverages to zerof(s1)f(s2)f(s3)f(s4)f(s5)F = E[Σt f(st)]
← ▶ → 5 / 5
Only the diagonal survives: the Fisher adds one token Fisher per position.
Here $\mathrm{score}i^{(t)}=\kappa{i,y_t}-\bar\kappa^{p}_i(s_t)$ is the token score at position $t$. Given the prefix $s_t$, the next token is drawn from $\pmodel(\cdot\mid s_t)$, so its score averages to zero. For $u\lt t$ the earlier token is part of that prefix: its score is already fixed and comes out of the inner expectation. The cancellation holds in expectation, not sample by sample. It makes the Fisher grow with length but treats every parameter alike, so it sets the scale, not which coordinates are large.
None of this needs on-policy sampling. It needs the score centered under the distribution that drew the samples.
every token: lowered by the sampler’s probability
Averaged over what the sampler draws, it is zero: a constant reward moves nothing.
Off-policy, the token was drawn by an inference engine or a stale model, and under that sampler the softmax score averages to $\bar\kappa^{\mu}_i-\bar\kappa^{p}_i$, not to zero. Subtracting it at every prefix is score centering. Sections 2–4 then go through word for word with $\smu$ in place of $\pmodel$, and on-policy the centering comes free. This is the gradient of the covariance loss in Around the Sampler: the drawn token is raised and every token is lowered by how often the sampler draws it. It needs the sampler’s full next-token distribution at every prefix, which engine log-probabilities and stale models provide and human demonstrations do not.
For any sampler $\smu$ whose full next-token distribution is known at every prefix:
On-policy, $c_i=\mathrm{score}_i$ and $G_i$ is the gradient. Off-policy, $G_i$ is the expected update of the centered estimator: the gradient of the covariance loss, not in general of the expected reward.
Next. Part 2 puts this to work. In the output layer the Fisher is an exact coin flip, concentrated on the tokens the sampler emits; in hidden layers it is low-dimensional; and without centering a mean shift can push gradient where the Fisher cannot see.
@online{li-2026-gradient-is-covariance, title = {The Gradient Is a Covariance}, author = {Li, Yingru}, year = {2026}, month = oct, url = {https://richardli.xyz/post/gradient-is-covariance/} }
Reinforcement Learning Policy Gradient Fisher Information Score Centering Sparsity Language Models
Yingru Li
Member of Technical Staff
Core contributor to RL fundamentals at xAI (now SpaceX Super Intelligence); I make reinforcement learning on frontier models stable, efficient, and scalable.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。