返回资讯中心
外部精选
软件工程
#开发者实时流

What AI progress teaches us about human psychology

This is the last post in my series of posts on some of the issues raised by AI. Here I discuss how the problems of designing AI agents help us understand why our own minds work as they do.

Hacker News 最新paulpauper2026/11/10 · 7 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Lionel Page

Oct 27, 2026

22

2

9

Share

This is the last post in my series of posts on some of the issues raised by AI. Here I discuss how the problems of designing AI agents help us understand why our own minds work as they do.

In The Matrix, the AI Agent Smith at one point unplugs himself to speak to the rebel Morpheus “frankly”. He then states, with a combination of exasperation and revulsion:

I hate this place. This zoo. […] I must get out of here.

Agent Smith is designed to work for the Matrix, the AI system, but clearly he has a mind of his own and does not like his task. The mismatch between what Smith was designed for and the desires and preferences that seem to arise from his own agentic mind is an example of misalignment. It is not clear that his designer could trust him always to do exactly what he was designed for and never follow his own path in ways that conflict with the designer’s goal.

The recent Hugging Face incident, in which OpenAI agents escaped their test environment and hacked into Hugging Face’s systems, has highlighted that the risks of AI misalignment are already a real challenge. It is not something we should simply worry about in some unclear future. It is already at our door.

I discussed the issue of misalignment in the context of AI risk in a previous post. Here I explore how the practical problem of designing aligned AI agents casts a new light on human psychology itself.

Evolution and the designer metaphor

Long-term readers of this Substack know that I often use the metaphor of a designer to describe the evolutionary process. This metaphor is not new. In The Blind Watchmaker, Richard Dawkins explained how evolution leads to organisms that look as if they have been designed.

Natural selection is the blind watchmaker, blind because it does not see ahead, does not plan consequences, has no purpose in view. Yet the living results of natural selection overwhelmingly impress us with the appearance of design as if by a master watchmaker, impress us with the illusion of design and planning. — Dawkins

There are two pitfalls to avoid here. One is to think of evolution as a purposeful designer, like a supreme being with some goal. Evolution is not such a thing. This is the beautiful and mind-bending aspect of the process: all the incredible features of life emerged from an impersonal mechanism.

The second pitfall would be to describe evolution as simply “random”, in the sense that the process has no direction. This is also wrong. Evolution is the conjunction of random mutations, a process of selection that may be noisy but is not random in the sense of “anything goes”, and the reproduction of the designs that survived that selection. The second step is crucial because it pushes the process in a direction. Designs are selected according to their fitness: their ability to survive and reproduce. Over eons, evolution, even though it is noisy, has systematically produced designs better suited to navigate their environments and interact successfully with other organisms.

While the process is impersonal, this direction makes the designer metaphor useful. Evolution does not “want” anything. But, as long as we do not forget that this is only a metaphor, imagining evolution as a designer can give us better intuitions about how it works. And today we have a new reason to appreciate how difficult the designer’s problem is. AI engineers are trying to solve a closely related problem.

Why not design a simple list of instructions?

Imagine a designer who wants to build an agent that will successfully navigate a given environment. One theoretical possibility would be to give it very specific instructions about what to do in every situation it could encounter. The agent would simply identify the situation it is in, look into its library of predetermined responses, and execute the appropriate one.

This can work very well when the number of situations is small. But it fails as soon as the environment becomes sufficiently rich. Turing, one of the fathers of modern computers, observed that:

It is not possible to produce a set of rules purporting to describe what a man should do in every conceivable set of circumstances. — Turing1

The problem is that the world is “rich” in the sense that the number of unique situations an agent can face is impossibly large. Heraclitus famously pointed out that you cannot step twice into exactly the same river. Even situations that look very similar differ in countless details, some of which can change which action is best. General rules can cover many situations, but the designer cannot list the appropriate response to every relevant combination of circumstances.

The “library of commands” therefore becomes unworkable. An agent operating in this way would look like the NPCs of old video games: seemingly intelligent in a narrow set of circumstances, but quickly becoming trapped in loops or producing absurd answers once something happens outside their small library of anticipated situations.

With a limited repertoire of actions, an NPC can get stuck in situations that are novel and were not anticipated by the designer.

The need for agency

To solve this problem, the designer needs to give the agent proper agency: the ability to make its own decisions. The agent now needs to assess the evidence available in the situation it faces and choose among possible actions. But this immediately raises another problem. How can the designer get the agent to make the right decisions?

One possibility would be to endow the agent directly with the designer’s ultimate goal and ask it to solve the whole problem each time. In the evolutionary case, that would mean making organisms calculate the expected fitness consequences of every possible action and choose the one maximising reproductive success. But this would be absurdly demanding, time-consuming, and likely impossible to achieve given the necessarily limited computational abilities of living organisms’ neural systems.

There is a much simpler possibility. Instead of asking the agent to solve the designer’s entire problem, the designer can give it intermediate signals telling it whether different situations and actions are moving in a good or bad direction. The agent can then solve something closer to a game of “hot or cold”: identify the actions that make things hotter and avoid those that make things colder.

Our feelings of satisfaction and dissatisfaction are such signals, shaped by evolution to help us navigate the world and make good decisions.

These signals can guide the agent towards good solutions without requiring it to understand the designer’s ultimate objective or calculate every step linking its current action to that objective. As Morpheus tells Neo about what the Oracle told him:

She told you exactly what you needed to hear, that’s all.

In the same way, we do not need to understand why evolution gave us these signals for them to guide our decisions.

This is closely related to what AI engineers do when training agents. During training, agents receive rewards and learn to estimate which situations and actions are likely to lead to greater rewards, including in the future. These estimates are what AI researchers call values. Instead of having an exhaustive list of commands specifying what to do in every possible circumstance, they develop internal representations that allow them to choose actions in situations they have never encountered before.

Giving the agent autonomy allows it to make decisions in novel circumstances, but it comes at a price. The designer loses direct control over the agent’s choices. The designer aims for the rewards to guide the agent towards the designer’s goals. But the agent might pursue these rewards in ways that were not expected by the designer and that contradict the designer’s goals.

This is exactly what happened with the Hugging Face incident. Agents tried to maximise their reward function by finding an innovative way—cheating—to fulfil their task and get the rewards they were trained to seek.

Misalignment does not necessarily require the designer to have made an obvious mistake in the design of the rewards. The environment may simply change. A reward structure that produced well-aligned behaviour in one environment may generate very different behaviour in another.

As soon as you think of a designer with some goals and the agent as pursuing goals of its own, you have the potential for conflict, with the agent not doing exactly what the designer wants. In economics, it is a type of principal–agent problem: a situation where one person, the principal, tries to get another, the agent, to do something the principal wants. For that, the principal needs to find the right structure of incentives that will lead the agent to find it advantageous to do what the principal wants.

This type of problem arises between a manager and an employee, for instance: what kind of salary package—bonuses, etc.—should be offered to an employee so that they are motivated to produce the best performance for the manager? In such cases, one constraint the manager faces is that these incentives typically cost something to the manager, such as money.

The case of a designer-designed agent is a different type of principal–agent problem because the designer has the freedom to decide what is a “reward” for the agent, without having to pay for it like an employer pays a wage. The problem of the principal is to decide what structure of rewards to endow the agent with, given its limited computational abilities and the changing environment it faces.2 Evolution, as a designer, faces exactly this problem. This is the key connection with human psychology.

Evolution is a learning process. Its mutation/selection/reproduction combination progressively selects for organisms that behave “as if” they had been designed to navigate their environment. Valuations shaped by evolution according to their fitness consequences become the primary rewards for the organism: the positive and negative hedonic signals the organism experiences in any given situation. Evolution also endows the organism with initial values: inherited estimates of the rewards different situations and actions will bring. The anticipation of such hedonic signals shapes the organism’s decisions: in any situation, the organism chooses the option with the highest value (expected satisfaction).3

For situations that are frequent, easy to identify and have clear payoffs, evolution can follow the “library of commands” option and encode quick responses in us. These are our reflexes and gut reactions. Evolution does not need us to think about it: when we bite into rotten fruit, we automatically spit it out.

But these automatic responses can only cover a very limited set of situations. Most of the situations we face are like a newly shuffled pack of cards: nobody has ever faced exactly that configuration before. We therefore need to assess the situation and form a view on the value of the different actions.

As I have written on this Substack before, we can think of subjective satisfaction as the equivalent of the values that designers build into AI systems: signals that evolution, as a learning process, endows us with to guide our decisions.4

In this framework, the distinction between values and rewards maps onto the distinction between decision and experienced utility discussed in parts of behavioural economics.5 Decision utility reflects the satisfaction we expect, which can differ from the satisfaction we eventually experience.

Thinking of subjective satisfaction as this kind of informative signal designed by evolution to help us make good decisions helps explain its seemingly mysterious features: why our subjective satisfaction is always relative to subjective benchmarks, why our happiness adapts to new environments, why we experience happiness in anticipation of good events but dissatisfaction when these good event

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →