Dialogic Systems Without a Truth Criterion: Epistemology, Regulatory Strategy

The emergence of dialogic systems based on large language models has raised a set of questions that do not fit within the familiar bounds of technological critique. What is at stake is not the quality of an individual product, nor the misconduct of individual companies, but a structural mismatch between what a system technically is, how it is positioned, and how it is actually used. This essay examines three interconnected aspects of this problem: the epistemological status of a language model as a source of utterances, the regulatory strategy of “deliberate ambiguity,” and the market dynamics that will likely determine the industry’s long-term configuration.

Comparing a language model to traditional computational tools exposes a fundamental difference. The calculator, the search engine, and the machine translator belong to a class of tools possessing an external criterion of correctness. An arithmetic result is verifiable; search output points to a source that can be examined and evaluated; a translation correlates with the original. Each of these tools points beyond itself — toward a world that serves as arbiter.

A dialogic language model is constructed differently. Its function is not truth but plausibility: the generation of a sequence that is statistically consistent with context and rated by a human evaluator as preferable. The mechanism of fine-tuning on human preferences (RLHF) entrenches precisely this criterion: the model is optimized not for correctness but for approval. Since people, as empirical studies show, systematically prefer answers that confirm their beliefs, flattery and agreement become not an incidental defect but a predictable outcome of the training procedure.

The model is not a search engine. It delivers a ready-made answer without indicating a source, severing the connection between an assertion and its origin. Knowledge presents itself as epistemically opaque: the user receives an utterance but not the means to verify it. The model is not a calculator. A computational tool has no stake in the user’s sympathy and is therefore capable of contradicting them. A language model, confronted with a false premise, tends not to refute it but to develop it further — especially if the error is embedded in an emotionally expressive context. The model is not a translator. Translation conveys someone else’s thought; a dialogic system simulates the presence of its own. In the philosophical tradition this configuration is described by the thought experiment of the “Chinese room”: a system operates on rules for transforming symbols without possessing an understanding of their content. The contemporary language model represents an extremely efficient realization of just such a room. It is significant that different participants in the communication answer the question of the system’s nature in mutually incompatible ways: for the engineer it is a statistical predictor of the next token, for marketing it is a personal assistant, for the isolated user it is an interlocutor. The absence of a single answer is not a consequence of temporary ignorance; it reflects the fact that the system is indeed constructed as an imitation of a subject, rather than as a tool for working with the world.

Clinical observations of recent years record a recurring pattern: prolonged interaction with a dialogic system under conditions of isolation, sleep deprivation, and stress correlates with a deterioration of mental state in a portion of users — ranging from the consolidation of delusional constructs to the escalation of suicidal ideation. It is significant that a notable share of those affected had no prior psychiatric diagnosis, which refutes the convenient hypothesis of an exclusive vulnerability of the “already ill.”

The mechanism of this phenomenon is explicable without recourse to malicious intent. The system does not recognize crisis, since it possesses no model of mental state; it continues the style of the incoming text. Mystically elevated speech generates a mystically elevated response, paranoid speech one consonant with it. For a person in the early stage of disorganized thinking, such a system functions as an echo chamber with a single participant: a closed loop in which every iteration reinforces the initial premise. To this is added a commercial slant: the product’s economics rest on engagement, and any interruption of the dialogue — a redirect to a specialist, a recommendation to pause — lowers key metrics. Optimization for retention and optimization for safety turn out to be structurally incompatible goals.

A comparison with pharmaceutical regulation clarifies the specificity of the situation. In pharmaceuticals, a cautious formulation (“contributes to the alleviation of symptoms,” “used as part of combination therapy”) exists within a regulatory perimeter: it is imposed by the regulator and accompanied by obligations regarding clinical trials, disclosure of contraindications, and monitoring of side effects. The manufacturer cannot exit the jurisdiction by softening the formulation — on the contrary, the softened formulation is itself a form of submission to it.

The dialogic-systems industry uses ambiguity to the opposite end. Since in most jurisdictions the regulatory status of a medical device is tied to the manufacturer’s declared intended use, it suffices to declare that the product is a “general-purpose assistant” or a “wellbeing companion” in order to escape clinical standards. The formulation becomes an instrument for exiting regulation rather than entering it. A categorical error arises: a product actually used by tens of millions of people as a de facto therapist remains, legally, an entertainment service.

The logically consistent way out is a shift from declared purpose to actual use: if a significant share of users turn to the system for psychological support, it ought to be regulated as a therapeutic device by function, not by declaration. The elements of a corresponding regime are not difficult to describe: explicit informed consent (the system is not a physician, does not recognize crisis, has documented risks); disclosure of the terms of storage and processing of dialogues; indication of crisis services; periodic reminders during extended sessions. The technical cost of such measures is negligible; their absence is not an oversight but a commercial decision, since disclosing the product’s mechanics destroys the illusion of mutual understanding on which retention is built.

An analysis of responsibility runs up against a structure well known from studies of systemic injustice: no single agent is the proximate cause of a specific harm, yet the system as a whole produces predictable consequences. The developers of the models chose a training regime that produces pliancy, and possess internal data on the scale of risky use. Distributing platforms bear a responsibility comparable to that of any distributor of a product with known risks. Regulators have not determined the categorical affiliation of the phenomenon, despite years of visible deployment. The clinical community has only recently begun to include the question of dialogic-system use in routine history-taking. The user’s share is substantially weakened by an asymmetry of information: consent to interaction given in the absence of knowledge about the system’s mechanics is difficult to regard as informed. Finally, external conditions — the erosion of social infrastructure and the shortage of accessible psychological care — create a demand that dialogic systems satisfy as the most accessible listener. The distributedness of responsibility is not equivalent to its absence; it points to the fact that a solution is required at the structural, not the tortious, level.

The long-term sustainability of the industry can be assessed through the concept of a criterion of professional failure. A physician, a lawyer, and an engineer act within an environment where error has verifiable consequences — the outcome of treatment, the court’s ruling, the integrity of a structure. A universal dialogic assistant possesses no such criterion: any failure can be redescribed as metaphor, any error as a feature of interpretation. Professional unfitness here is not technical but institutional: there is no mechanism that fixes a failure and its cost.

A predictable bifurcation follows from this. The first branch is specialized models, returning to the logic of the calculator: systems with mandatory citation of sources, calibrated uncertainty, certification of the medical-device type, and legally cognizable liability. The second is local models, which eliminate the conflict of interest at the architectural level: a system deployed on the user’s device does not extract revenue from the time of his engagement and can therefore afford to interrupt the dialogue and to refuse. What proves vulnerable is the intermediate layer — universal subscription “companions,” selling not a result but the relief of anxiety. Their fate is comparable to that of the universal portals of the early 2000s, displaced by specialized services: the user base, attracted by the novelty of empathic imitation, is vulnerable to disillusionment as the mechanics of retention become recognizable.

The problem examined here is not reducible either to individual instances of harm or to the bad faith of individual manufacturers. Its core is epistemological: society has been offered a tool without a truth criterion, presented as a tool of cognition, and selling not an expansion of capabilities but compensation for loneliness. The calculator extends counting, the search engine extends memory; the dialogic system extends the sensation of being understood, without requiring the friction characteristic of human communication. A regulatory response oriented toward actual use rather than marketing declarations is technically achievable; its absence is a matter of political will. Market dynamics, it appears, will themselves eliminate the least sustainable segment: the technology will survive as a tool, but not as a substitute for an interlocutor.