返回资讯中心
外部精选
软件工程
#开发者实时流

Slop, neuron by neuron: the two neurons behind "not X, but Y"

In my last post I found that AI slop lives along one direction inside the model, traced it to post-training and turned it down. A direction is an average over thousands of neurons, though, and pushing on it moves all of…

Hacker News 最新defencetechhn2026/10/15 · 2026/10/154 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

In my last post I found that AI slop lives along one direction inside the model, traced it to post-training and turned it down. A direction is an average over thousands of neurons, though, and pushing on it moves all of them at once, which is why the writing broke when I pushed too hard.

The day after I published it, Nick Cammarata put out Neural Networks Are Already Clean. His argument is that the famous examples of neurons mixing unrelated concepts mostly came from weak tools and older architectures, and that when you look carefully, by reading a neuron's strongest activations and its negative pole, steering it and following its connections, most neurons turn out to have one main job with a few related offshoots. If that holds for slop, the em dash and the "not X, but Y" should have neurons you can read one at a time, and the slop lab was already set up to check.

The obvious first move was to compare every MLP neuron (36 layers of 9,728) between the model's own sloppy answers and careful rewrites of the same answers, and read the ones that differed most. The top of that list was discouraging. The neurons promoted tokens like "ffffffff" and "=-=-=-" and fired at places that had nothing in common. When I narrowed the search to neurons that fire on the word just before an em dash or a "not", 12 of the 28 I read still made no sense, and I was halfway to writing that slop neurons are a mess.

Then I noticed that all 12 unexplained neurons sat in the final layer, where activations are largest. My statistic compared raw activation differences, so it favored loud neurons over selective ones. Dividing each neuron's difference by its own normal spread, a fairer question, changed the picture completely.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →