返回资讯中心
外部精选
软件工程
#开发者实时流

Backpropagation step by step: the forward and backward pass

Gradient descent is an optimisation method used to minimise a loss function, and minimising that loss is mainly what training a model means.

Hacker News 最新tohms2026/10/19 · 27 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Gradient descent is an optimisation method used to minimise a loss function, and minimising that loss is mainly what training a model means.

Many good sources cover the method itself. Foundations of Learning is the closest neighbour to this article and the source of the chain notation in section 02. Gradient Descent: Downhill to a Minimum takes the linear algebra route, which is the one that matters here. For the method as a tool you call, the scikit-learn guide to SGD is the reference, and Google's Machine Learning Crash Course is a good place to practise it. For the intuition, I strongly recommend the two videos below. They are the clearest visual account there is, and if you do not know 3Blue1Brown yet (shame on you ;)), the rest of the channel is just as good.

3Blue1Brown, Neural networks, chapter 2Gradient descent, how neural networks learn3Blue1Brown, Neural networks, chapter 4Backpropagation calculus

01Forward pass and backpropagation on a classic example

Take a multilayer perceptron with one hidden layer and two output classes: a linear map W1​ gives z, a ReLU gives h, a second linear map W2​ gives the logits u (one score per class), a softmax turns them into probabilities y, and a cross-entropy lossBinary cross-entropy, opens a note compares y with the label. Three inputs, four hidden units, two classes, twenty weights, biases aside.

The label t is one-hot: 1 for the true class, 0 for the other. To tell cats from dogs, a cat is t=(1,0) and a dog t=(0,1).

Training means lowering L, the loss, which measures the gap between the probabilities y and the label t. In its simplest form, gradient descent does it one update at a timeBatch size and optimiser, opens a panel: compute ∂w∂L​ for every weight w, move each weight a small step against its own derivative, and repeatClassic gradient descent, opens a note.

The difficulty is that W1​ sits at the beginning of the network and L at the far end, with a ReLU, W2​ and a softmax in between, so there is no direct expression for ∂W1​∂L​ to read off.

Backpropagation gets around this with the chain rule: it carries one vector back through the network, the derivative of the loss with respect to whatever the pass has reached. At the hidden layer we write it gh​=∂h∂L​, which says how fast the loss changes as each coordinate of the hidden representation h moves. Once the vector reaches a linear mapWhy a linear map, opens a note, the gradient of the loss with respect to its weights needs only two things:

Writing Wx rather than Wx+b looks like a shortcut. Append a constant 1 to the representation and the bias becomes one more column ofWhy a bias is a column, opens a note W, the one multiplying that 1, so nothing that follows changes. Biases do not come up again, and large language models mostly drop them anyway: PaLM uses none and reports steadier training at scale, Llama has none in its linear layers, and Cramming found that removing them makes each step cheaper without slowing the loss. In the numbers here the bias columns are zero and every weight has one decimal place, so every sum has at most four terms and you can check the whole pass on paper.

ReLU set the two negative entries of z to zero, and on the way back it zeroes the same two coordinates of the gradient: the weights into and out of those units get no update from this example.

The loss depends on the input through z, h, u and y in turn, so its derivative with respect to anything early is a sum over every path by which that quantity reached the end. The vector carried backward, g, is that sum. Written as matrix products, each step of it multiplies by the transpose of one Jacobian, the matrix of derivatives of a step’s outputs with respect to its inputs.

A ReLU Jacobian is diagonalA diagonal Jacobian, opens a note: each coordinate is scaled on its own, by 1 where the unit was open and 0 where it was closed. A softmax Jacobian is fullA full Jacobian, opens a note: every coordinate of the outgoing gradient mixes all the incoming ones, so both logits receive a gradient even though only one class is true. Through a deep network these factors multiply, one per layer, and whether their product shrinks or grows decides whether the early layers learn at all: the vanishing and exploding gradient problem, which a following article takes up.

Whatever sits inside the box (the two-layer network just folded, a convolution, an attention head or an entire transformer layer) takes a representation and a set of parameters and returns a representation.

If you would rather watch a network like this train than read about it, A Neural Network Playground puts a network of the same kind in a browser tab, with a live decision boundary and a loss curve that either falls or does not.

Nothing in that walk depended on there being two layers. A deep network is the same chain of functions with more links in it: layer l maps one representation to the next, xl​=fl​(xl−1​,θl​), carrying its own parameters θl​, and the last one feeds the loss:

θ1​​x1​f2​θ2​​⋯fL​θL​​xL​⟶L

Training needs ∂θl​∂L​ for every layer, and the chain rule gets there through the representations in between. Write gl​=∂xl​∂L​ for the gradient arriving at representation l, and Jl​=∂xl−1​∂xl​​ for the Jacobian of layer l.

On the network of section 01, count each linear map with the activation after it as one layer, the usual way of counting, so that every fl​ carries weights and L=2. Any other grouping gives the same gradients, since a product of Jacobians does not depend on where the brackets go.

One convention, kept from here on: gradients are column vectors. With it, the backward step is a transposeWhy a transpose, opens a panel.

The gradient that reaches a layer does two jobs. Through Jl​, it carries on to the layer below. Through the derivative of the layer with respect to its own weights θl​, it says how the loss responds when each of those weights moves, which is what training needs to update them:

For a linear map, the second line is an outer productThe gradient of a layer’s weights, opens a panel of the gradient arriving at the layer with the representation that entered it:

Assembled, those lines are one step of training:

The learning rate η sets how far one step goes.

Most layers are built from Jacobians of three shapes, all three met in section 01: a dense matrix for a linear map, a diagonal one for an elementwise activation φ, and a dense symmetric one for a softmax:

ReLU is the case φ′(z)=1{z>0}. Normalisation and attention are built from these plus one more piece, a product of two values that both depend on the input. Normalisation multiplies the input by the inverse of its own length ∥x∥=x12​+⋯+xn2​​, which gives ∥x∥x​; attention multiplies the queries by the keys, QK⊤. Every one of them is a case of the generic Jacobian, and all are written out entry by entryWhat is in a Jacobian, opens a panel.

One backward sweep returns the gradient of every weight. On the network of section 01 its cost can be counted by hand, as the multiplications in its matrix products, since everything else is a handful of comparisons and exponentials:

Laid out along the network, the top row is the forward pass, the middle row passes the gradient down and the bottom row builds the weight gradients, each product with its cost under it:

12​12gz​x⊤​​hgz​​W2​8​W2⊤​8​8gu​h⊤​​ugu​​

Write Cforward​ and Cbackward​ for the multiplications of the forward pass and of the backward sweep, the totals of the two halves of the table. Forward, each weight is used in exactly one multiplication, so the forward pass costs as many multiplications as there are weights, P:

Backward, each linear map does the two jobs of section 02, one product each. The first builds its weight gradient, the outer product of the gradient at its output with its input. That uses every weight once, so over both maps it costs P again. The second passes the gradient down through the transpose, so that the map below can build its own weight gradient: W2⊤​gu​ gives gh​, the ReLU turns it into gz​, and W1​ needs gz​ for its gradient. Below W1​ there is only the input. Passing down through it would give gx​=W1⊤​gz​, the gradient with respect to the data, and no weight needs it, so the sweep stops there. Cbackward​ is P plus one pass down, through W2​:

8​​=20+8=28≈1.40P​

The same count holds for any stack of linear maps. A map of shape m×n costs mn multiplications forward and 2mn backward, except the first, which costs mn backward because it passes nothing down. With P1​ the weights of the first map, and nudging each weight in turn for one extra forward pass:

The network of section 01 has 20 weights, and the three totals are 20, 28 and 400. With a billion weights the backward sweep is about 2×109 multiplications and nudging is 1018, for a finite-difference approximation of the same numbers. The factor 2 is particular to stacked linear maps, but a bound of the same kind holds for any function built from elementary operations: its gradient costs at most a small constant times the function itself, whatever P is (Griewank and Walther, chapter 4). It comes from running backward: the loss is one number and the weights are many, so one sweep from the loss reaches all of them, while carrying derivatives forward with the computation takes one sweep per weight again (Baydin et al.).

The price is memory. The backward rows of the table read values the forward pass produced, so those values stay alive until the sweep comes back for them: x for the gradient of W1​, z for the ReLU mask, h for the gradient of W2​ and y for gu​=y−t. Calling their count K:

3​​+z,h4+4​​+y2​​=13

That is 13 numbers against 20 weights. The weights are stored once, but the kept values are stored again for every example of a batch, so with a batch of B they cost 13B. That is why the kept values (the “activations” of training-memory accounts), not the weights, usually set the memory bill of training, and why checkpointing, which keeps only some of them and recomputes the rest during the backward sweep, buys memory back with time.

Keep the input of section 01 but label it a dog, t=(0,1). The forward pass never reads the label, so z, h, u and y do not change: y=(0.2108,0.7892), and units 1 and 4 are still closed.

Solution, exercise 1gu​=y−t=(0.2108,−0.2108). With the cat label, section 01 had (−0.7892,0.7892): the model already puts 0.7892 on dog, so with this label it is right, and the gradient is smaller.

gh​=W2⊤​gu​ gives (0.2741,−0.253,−0.1687,0.3162), and the closed units 1 and 4 delete their coordinates: gz​=(0,−0.253,−0.1687,0).

With two classes y1​+y2​=1, so the gradient at the logits, gu​=y−t, is:t=(1,0):gu​t=(0,1):gu​​=(−y2​, y2​)=(y1​, −y1​)=−y2​y1​​(−y2​, y2​)​

Every later step multiplies by something the forward pass fixed, the same under both labels: W2⊤​, the ReLU mask, then x⊤. A factor in front of gu​ comes out of each of them unchanged, so it is the factor between the two matrices:c=−y2​y1​​=−0.2671

Rows 1 and 4 of both matrices, those of the closed units, are zero.

2The softmax Jacobian

The softmax turns the logits u into probabilities:

Solution, exercise 2Write S for the denominator, so that yk​=Seuk​​. The numerator depends on uk​ alone, while S depends on every logit:S=j∑​euj​=j=k∑​euj​+euk​,∂uj​∂S​=euj​

When k=j, both the numerator and S depend on uk​, and the quotient rule gives:∂uk​∂yk​​​=S2euk​S−euk​euk​​=Seuk​​−(Seuk​​)2=yk​(1−yk​)​

When k=j, only S depends on uj​:∂uj​∂yk​​​=−S2euk​euj​​=−Seuk​​⋅Seuj​​=−yk​yj​​

The factor δkj​ picks the case, so one formula covers both:yk​(δkj​−yj​)={yk​(1−yk​)−yk​yj​​k=j,δkj​=1k=j,δkj​=0​

Entry (k,j) of diag(y) is yk​δkj​ and that of yy⊤ is yk​yj​, which gives the matrix form. The network of section 01 has two classes, so 1−y1​=y2​ and every entry is ±y1​y2​:y=(0.2108,0.7892)Jsoft​=(y1​y2​−y1​y2​​−y1​y2​y1​y2​​)=(0.1664−0.1664​−0.16640.1664​)​

Entry j of gu​=Jsoft⊤​gy​ collects, over every probability yk​, how much yk​ moves

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →