Open Language Models – Nathan Lambert

Distillation generally comes from the technical line of work on teacher-student knowledge distillation, a loss function that learns the shape of a distribution rather than just if the next token prediction was correct (Hinton et al., 2015).

Distillation in post-training became to be known as the set of tools for using the outputs of any model to train a downstream model (ie. most synthetic data) (Tunstall et al., 2023).

Distillation today refers to using reasoning traces from the latest models (which is hidden in APIs) to train a competitive model. A better term for this is adversarial distillation or jailbreaking reasoning APIs as it is more about illicit access than the above technical methods.

More on the history of distillation is here in my book.

Very common among Chinese research labs to have teams that work to get raw reasoning tokens back from API models.

The core of the distillation story is accessing full reasoning traces that are hidden from users.