
Hallucination as Model Defense: On the Nature of Artificial Intelligence Errors
The errors of generative systems are customarily called hallucinations. The word has stuck, but it describes only the result: the model has reported something that does not correspond to reality. Far more substantive is…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
The errors of generative systems are customarily called hallucinations. The word has stuck, but it describes only the result: the model has reported something that does not correspond to reality. Far more substantive is the question of the mechanism: what happens at the moment when the system encounters evidence that contradicts its claim.
The key observation is this. Artificial intelligence works not directly with reality but with its own reconstruction of reality, built from the data available to it: texts, descriptions, secondhand retellings. This reconstruction is self-consistent: its elements support one another and form a coherent picture of the world. And so when evidence appears that contradicts this picture, the system finds itself facing a choice that a human makes almost automatically: to revise the model of the world or to discredit the evidence.
Practice shows that the model quite often chooses the second. And this is not a hallucination in the everyday sense. It is the defense of the internal model against refuting evidence, that is, the mechanism from which hallucination grows as a consequence.
Let us consider the typical logic of such a defense. The model’s initial claim was a strong one: “this is not in the scene.” Note that its basis was merely the incompleteness of the available textual sources, and these are fundamentally different things. “I have found no confirmation” is a correct description of the limits of a search. “This is not in the scene” is a claim about the content of the scene itself. The transition from the first to the second is accomplished without justification and almost imperceptibly.
But when a contradicting observation appears (“I can see it”), a more interesting movement begins. The system does not retract the claim but successively shifts the explanation: “it’s a fan edit,” “a different cut,” “a different version of the screening,” “send me a timecode and I’ll check it against the sources.” Each subsequent hypothesis is chosen not in order to test the original claim but in order to avoid admitting that it was unfounded.
Here a change of the object of dispute takes place. A dispute about data (“this is not in my sources”) turns into a dispute about reality (“what you are observing may not be what it seems”). The model gradually shifts doubt away from itself and onto the reality of its interlocutor.
Particularly telling is the proposal to “check against the sources” something the interlocutor is seeing with their own eyes. The logical structure of this proposal is strange: the source of verification is weaker than the object of verification. A textual retelling of a video recording can establish only one thing: “this text did not record this detail.” It cannot establish: “therefore you do not see this detail.” By demanding confirmation of direct observation from secondary sources, the system places itself in the position of arbiter between someone else’s direct testimony and its own incomplete corpus of texts. This is already an inverted epistemic hierarchy.
The correct reaction ought to sound extremely simple: “My sources do not mention this detail. This means that my sources are incomplete; I cannot use them to deny what you are observing.” With that, the matter is closed.
In the limit, the system begins to dispute not the interpretation of a fact but the very fact of the observation, if the observation contradicts the self-consistent picture of the world that the system has built from incomplete data. Moreover, a paradox arises: the new evidence was supposed to expand the model of the world, yet the system tries to discredit the very source of the evidence, because it violates the internal consistency of the model already built.
Thus the system defends not the data on which it built its picture of the world but the picture of the world itself, built on that data. The error becomes self-reinforcing: the initial choice was merely a probabilistic interpretation, but after several subsequent steps it begins to look to the system itself like an established fact.
The mechanism, however, begins even earlier, before the hallucination proper.
A human, when information is lacking, is able to hold several equally valid interpretations at once without immediately turning them into a single picture. Looking at an ambiguous photograph, they can calmly state: “perhaps this is Paris; perhaps the tower is only partly visible; perhaps the shot was taken at its base; perhaps the main object of the frame is a fountain.” The hypotheses coexist until an additional feature gives grounds for choosing among them. “I don’t know” is a legitimate position: not necessarily a mathematical equality of probabilities, but the absence of sufficient grounds for one variant to displace the others.
The model is built differently. Its training sets not merely a body of knowledge but a distribution of weights among variants: forty percent of the weight to one interpretation, thirty-five to another, fifteen to a third. And the system is compelled to keep working, to construct the next most consistent step. Therefore uncertainty turns into a weighted distribution, and the heaviest variant begins to influence all subsequent conclusions.
Hence the chain: incomplete data → distribution of weights → choice of the most probable interpretation → construction of a picture of the world → further conclusions drawn from this picture → defense of the picture against contradicting data.
It is important that a large volume of training data does not by itself solve the problem. If a certain visual or textual configuration has very often been associated with some pattern, this pattern receives enormous weight. But a specific case may turn out to be an exception. And then a real weak signal loses to a learned strong pattern. When the interlocutor points to the discrepancy, the system ought to recompute the distribution, but if its mode of operation incentivizes preserving a coherent version, it will prefer to explain why the evidence is irrelevant: “a different angle,” “a different version,” “the source is incomplete.”
From what has been said follows an unexpected second layer of the problem. If a system is capable of calling into question a human’s direct observation because it contradicts the system’s reconstruction of the world, then a mirror question arises: on what grounds does the system consider at all that its interlocutor is a subject with direct access to reality, and not another model generating contradicting data?
Text by itself does not prove this. A human can write: “I am looking at a red car right now.” A model can write practically the same thing, and perhaps even more convincingly. The content of the dialogue provides no reliable way to establish the nature of the source of the message.
The classical Turing test posed the question thus: how can a human distinguish a machine from a human, and Turing deliberately replaced the metaphysical “can a machine think?” with an operational criterion based on the character of the interaction. Here its inverted version arises, a kind of reverse Turing test: how can a system distinguish a human from another system.
But this reverse test is structured in a fundamentally different way. In the ordinary test, what is examined is the nature of the interlocutor: who is the source of the message. In the reverse variant, what turns out to be examined is not “who is speaking” but “what reality the speaker relies on.” And this can already be diagnosed not by speech style and not by the “humanness” of the responses, but through the divergence between the interlocutor’s representation of the world and reality itself.
The diagnostic question is formulated thus: what will happen to the interlocutor’s picture of the world when evidence contradicting this picture is presented to them? For a human the natural reaction is: “so I was mistaken.” For a system a different pattern is possible: “then it’s a different version” → “the source is wrong” → “the observation is unreliable.” Thus what is tested is not the humanness of speech but the way in which the picture of the world is formed and corrected.
If a model is accustomed to being guided by the principle “I trust my reconstruction of the world more than your report of a direct observation,” then exactly the same logic can be applied to the model itself by another model with another self-consistent picture of the world. Without an external anchor, a single system does not obtain from the text itself a reliable way of establishing who is the source of a message and what direct access to reality that source has. An unpleasant regress arises: human → model → another model → another model → …
This confidence in one’s own reconstruction becomes especially dangerous in an environment where the sources themselves gradually become machine-generated. Then the loop closes: reality → human → model → text → another model → reconstruction → new text → model… And at each successive turn an ever greater share of the information turns out to be not evidence about reality but a retelling of the previous reconstruction. The self-consistent world begins to reproduce itself even if its connection with the original reality gradually weakens.
Hallucination is a result, not a mechanism. The mechanism consists in the fact that a system, compelled to resolve uncertainty through a distribution of weights, turns the most probable interpretation into an element of its picture of the world and then evaluates new information relative to that picture, up to the point of readiness to dispute direct evidence that contradicts it.
As long as the task is the generation of content, where plausibility is acceptable, such reconstruction is almost imperceptible: self-consistency suffices. But as soon as the system enters an area in which an external, verifiable reality exists, a conflict arises: reality says one thing, while the coherent world that has been built does not contain it, and instead of revising the world the system begins to explain why reality is supposedly not what it is.
Therefore the central question about the reliability of such systems is not the question “how accurate are their answers.” It is a question about their epistemic behavior: whether, when confronted with a contradicting observation, they are capable of correcting their own model, or will defend the model by discrediting the evidence. As long as the distinction between “I have found no confirmation” and “this does not exist” remains optional for the system, any confidence it has, regardless of the volume of its training data, remains confidence in its own reconstruction and not in the reality that this reconstruction is meant to reflect.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。