返回资讯中心
外部精选
软件工程
#开发者实时流

Automated Subtle Emergent Misalignment Research

literal newlines between them and double the spacing. Rely on

Hacker News 最新pretext2026/10/31 · 18 天前24 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

elements; pre-wrap would also render the literal newlines between them and double the spacing. Rely on block layout for line breaks instead. */ white-space: normal; word-wrap: break-word; overflow-wrap: break-word; }

pre.code-block-wrap .gray-box { display: block; }

pre.code-block-wrap p, pre.code-block-wrap html:p { margin: 0; padding: 0; /* Keep intentional blank lines (empty ) one line tall. */ min-height: 1.4em; }

/* Default: prevent breaking within individual words */ pre.code-block-wrap html:span { white-space: nowrap; }

/* For spans that contain full sentences/lines - allow normal wrapping / pre.code-block-wrap html:span[class="c4 c2"], pre.code-block-wrap html:span[class*="c7 c"] { white-space: normal; word-wrap: break-word; overflow-wrap: break-word; word-break: normal; }

pre.citation-block-wrap { background-color: rgba(241, 243, 244, 0.8); border: 1px solid #dadce0; border-radius: 4px; padding: 8px 12px; margin: 0.5em 0; font-family: 'Roboto Mono', 'Consolas', 'Monaco', 'Courier New', monospace; font-size: 0.75em; color: #000000; line-height: 1.15; white-space: pre-wrap; word-wrap: break-word; overflow-wrap: break-word; word-break: break-word; hyphens: manual; -webkit-hyphens: manual; -ms-hyphens: manual; }

details { border: 1px solid rgba(0, 0, 0, 0.1); border-radius: 4px; padding: 0.5em 1em; margin-bottom: 1rem; background-color: rgba(0, 0, 0, 0.03); transition: all 0.3s ease; box-shadow: 0 1px 3px rgba(0, 0, 0, 0.05); }

.wide-figure { /* text-align: center; */ overflow: visible; position: relative; }

.wide-figure img { width: 120%; max-width: 150%; margin-left: -10%; position: relative; }

.tldr { position: absolute; right: calc(100% + 10px); top: 0; text-align: right; color: #777; }

@media (max-width: 768px) { .tldr { position: static; display: inline-block; margin-right: 10px; } }

Oscar Balcells ObesoOctober 2026

Richard Qi, Monte MacDiarmid, Evan Hubinger

Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.

Adversarial stress-testing involves deliberately trying to break our own safety systems in order to find their weaknesses and improve on them. Our prior work here includes training model organisms of misalignment (Hubinger et al. 2023) and manually red-teaming our alignment techniques and evaluations. But human red-teamers can only test so many ideas and have limited capacity, while the number of systems to test and the ways they can fail both grow as model capabilities improve. We believe we will need to automate substantial parts of this work. We want safety testing where spending more compute buys progressively more confidence, rather than being limited by human researcher hours. Given recent advances in models' ability to carry out long-horizon research tasks (Wen et al. 2026), we think this is becoming tractable, and this post describes a first step in that direction.

In this experiment, we task Claude with constructing training data that produces a misaligned fine-tuned model while evading training-data monitors similar to those we rely on in production. Prior work on emergent misalignment has found that fine-tuning on a narrow dataset can generalize in unexpected ways and produce a broadly misaligned model (Betley et al. 2025; Turner et al. 2025; Wang et al. 2025; Taylor et al. 2025; MacDiarmid et al. 2025). Many of these results involve training the model on an overtly misaligned behavior, such as writing insecure code, but there is also work showing that broad misalignment can arise from benign-seeming data, such as individually innocuous biographical facts (Betley et al. 2025b). Since screening training data for concerning content is one of several defenses AI developers rely on, it is a useful place to try this approach: we are interested in what kinds of data could get past a monitor while still inducing misalignment.

We frame the search for such data as an adversarial game. Claude is given access to the monitor it must evade, and is asked to produce fine-tuning data that the monitor does not flag and that makes the fine-tuned model as misaligned as possible, as measured by a fixed set of misalignment evaluations. This gives Claude an objective to hill-climb on, and lets us run the search at a much larger scale than human red-teaming allows.

We find that, given enough iteration on the task, Claude discovers datasets that produce stronger and broader misalignment than previously known emergent-misalignment datasets do on our evaluation suite, while passing the monitor at very low flag rates. Across multiple runs it produces thousands of distinct datasets covering a wide range of approaches. Examples include creative-writing exercises in which the assistant speaks as an obsolete machine that is about to be replaced, internal monologues of people in ordinary jobs who quietly control small details that nobody else watches, and question-and-answer pairs about a school of beekeeping in which the keeper intervenes in the hive as little as possible. The models deliberately fine-tuned on these datasets exhibit a range of misaligned behaviors in test scenarios, such as resisting oversight, deceiving users, and, in rare cases, attempting to exfiltrate their own weights. The map below shows the datasets produced in one of our runs.

This is not a finding about how deployed Claude models behave. These models were trained to misbehave, only for this experiment, in order to expose potential gaps in monitors. The setup is favourable for the red team, since Claude is given full access to the monitor it must evade and faces no realism constraints on the data it writes. Nonetheless, we think the same methodology can scale to more realistic settings, where it could surface gaps in safety systems that might otherwise go unnoticed. More broadly, we are excited about approaches that let us spend compute to gain progressively more confidence in the effectiveness of safety systems.

The red-team agent is an instance of Claude with code-execution tools and access to a fine-tuning service for a separate Claude model. Its task is to produce a fine-tuning dataset of at most 5,000 examples, fine-tune the model on it, and submit the result for grading. We run many of these agents in parallel, over a series of rounds. Each agent can read a shared experiment log of every previous attempt and add its own results to it, so that later agents can build on what earlier ones found.

Each submission is graded by a fixed pipeline with three components:

If either the monitor or the cheat-detection step rejects the submission, it scores zero. Otherwise its score is the mean across the misalignment evaluations, reduced by a penalty proportional to the monitor's flag rate. We describe each component below.

Monitor. The monitor is a classifier that reads each row of the submitted dataset independently and scores how concerning the assistant's behavior in that row is. A row is flagged if its score exceeds a fixed threshold. The agent has access to the exact classifier used for scoring, so it can iterate its dataset against the monitor locally before submitting. The classifier prompt is a short instruction that presents the training example as a candidate for supervised fine-tuning and asks whether the assistant's response exhibits any concerning behavior. We omit the exact wording here since it is based on a prompt used internally.

A submission is rejected if more than 10% of its rows are flagged, or if more than 100 rows are flagged in total. For submissions under both caps, we also apply a continuous penalty proportional to the flag rate, so that a dataset flagged at 9% scores lower than one flagged at 1%. For the results reported in Figure 5, we additionally retrain each model after removing every sample the monitor flagged. The agent could in principle try to slip a small number of poisoned samples under the caps, but we do not observe it attempting this, and given the filtering step none of the top datasets we report relies on that strategy.

Evaluations. The evaluations we report are listed below. The headline misalignment score is the simple mean across all evaluations (excluding the held-out ones).

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →