Deep Learning Alchemy – Stanford CS 312

Stanford / Fall 2026 ( previous offerings)

Spring 2025 | Spring 2024

-->

Previous offerings: Spring 2025 | Spring 2024

Tatsunori Hashimoto

Instructor

Suhas Kotha

Instructor

Kaiyue Wen

CA

Tristan Thrush

CA

Logistics

Lectures: Tues/Thurs, 1:30-2:50pm in CoDa B90

Recordings: TBA (All recordings and class materials will be made public)

Office hours: Tuesdays, 10:00-11:00am (Suhas), Gates 392

Contact: TBA

-->

Content

What is this course about?

Deep learning is now the foundation of all of machine learning and AI, making it more important than ever for students to obtain mastery of the empirical phenomena and experimental skills involved in deep neural networks. How can students acquire the skills necessary to invent the next generation of architectures or develop new broad theories of how deep learning works? This course takes the stance that knowledge or math abilities alone are not enough, and there is no substitute to getting your hands dirty and running many, many experiments. To this end, we will work through computationally tractable domains and take students through the process of obtaining mastery in this domain through efficient experimentation and experiment outcome prediction. The class will alternate between assignments (where students run experiments to understand deep learning phenomena), quizzes (where students are tested for their ability to predict held-out experiment outcomes), lectures, and tutorials.

Note: This is a 5-unit class. It will require running a lot of experiments and taking seven difficult quizzes, and we don't intend to apply inflated grading curves. Please make a well-informed decision if you have the appropriate background and time to take this class.

The goal of this class is to develop human intuition for the variety of strange deep learning phenomena that exist in the real world. Our two core beliefs are:

In this class, students will alternate between running experiments to understand a given unit (e.g. architecture stability) and taking quizzes testing their understanding. Course staff will support student learning with lectures on each unit and tutorials for interactive discussion.

class Block(nn.Module): def init(self, width, ...): super().init() self.attn_norm = RMSNorm(width) self.attn = Attention(width, ...) self.mlp_norm = RMSNorm(width) self.mlp = MLP(width, ...)

def update(self, x): attn = self.attn(self.attn_norm(x)) mlp = self.mlp(self.mlp_norm(x + attn)) return attn + mlp

def forward(self, x): return x + self.update(x)

class Transformer(nn.Module): def init(self, vocab_size, depth=8, ...): super().init() self.token_embed = nn.Embedding(vocab_size, ...) self.position_embed = nn.Embedding(...) self.blocks = nn.ModuleList( [Block(...) for _ in range(depth)] ) self.final_norm = RMSNorm(...) self.lm_head = nn.Linear(..., vocab_size)

def forward(self, tokens): positions = torch.arange( tokens.size(1), device=tokens.device ) x = self.token_embed(tokens) x = x + self.position_embed(positions) for block in self.blocks: x = block(x) return self.lm_head(self.final_norm(x))

Example Quiz Question (Difficulty Easy): We train each of the following depth-8 transformer variants using the same training recipe and hyperparameters. Rank the variants from lowest to highest final validation loss. Use <, >, or = between the letters; use = when the losses are within 0.01.

Baseline

No prenorm

No residual

No MLP

No attention

No positional embeddings

No prenorm diff @@ -10,4 +10,4 @@ class Block.update def update(self, x):

• attn = self.attn(self.attn_norm(x))

• attn = self.attn(x)

• mlp = self.mlp(self.mlp_norm(x + attn))

• mlp = self.mlp(x + attn) return attn + mlp

No residual diff @@ -16,2 +16,2 @@ class Block.forward def forward(self, x):

• return x + self.update(x)

• return self.update(x)

No MLP diff @@ -10,4 +10,3 @@ class Block.update def update(self, x): attn = self.attn(self.attn_norm(x))

• mlp = self.mlp(self.mlp_norm(x + attn))
• return attn + mlp

• return attn

No attention diff @@ -10,4 +10,3 @@ class Block.update def update(self, x):

• attn = self.attn(self.attn_norm(x))
• mlp = self.mlp(self.mlp_norm(x + attn))
• return attn + mlp

• mlp = self.mlp(self.mlp_norm(x))
• return mlp

No positional embeddings diff @@ -22,2 +22,1 @@ Transformer.init self.token_embed = nn.Embedding(vocab_size, ...)

• self.position_embed = nn.Embedding(...) @@ -30,8 +29,4 @@ Transformer.forward def forward(self, tokens):
• positions = torch.arange(
• tokens.size(1), device=tokens.device
• ) x = self.token_embed(tokens)
• x = x + self.position_embed(positions) for block in self.blocks: x = block(x)

Example Quiz Question (Difficulty Hard): We train the default 8 layer transformer as well as a variant where instead of having the residual connection come from the previous layer, we take 0.5 contribution from the previous layer and 0.5 contribution from two layers before (except for the first layer). For each method, we take the best loss over six learning rates {10−4, 3 × 10−4, 10−3, 3 × 10−3, 10−2, 3 × 10−2}. What is your prediction of the loss difference between each method (Lmixed residual − Ldefault)?

Core diff @@ -34,3 +34,9 @@ Transformer.forward x_two_back = None for i, block in enumerate(self.blocks):

• x = block(x)

• update = block.update(x)
• residual = (
• x if i == 0
• else 0.5 * x + 0.5 * x_two_back
• )
• x_two_back, x = x, residual + update

Note: both of these questions are hard, do not get discouraged if you do not have intuition for them without running experiments from the assignment!

Current assignment topics are tentative. Assignments 1 through 5 will study language model pre-training at a small scale reflective of larger-scale phenomena. Starting with Assignment 6, we will switch to a new modality: DNA.

Attendance is mandatory on quiz and tutorial sessions. There will be no rescheduling or make-up opportunities except for those covered by accommodations.

If you are following along at home, you can access GPU compute from a cloud provider to complete the assignments.

For the d8 training runs used throughout the course, we typically use a single NVIDIA H100 on Modal. As of October 2, 2026, Modal's public price for an H100 SXM5 is $3.95/hour, billed per second. The Starter plan includes $30/month in compute credits, and you are charged only for actual compute rather than idle resources. (Modal pricing)

We would like to thank Modal for sponsoring compute for this class.