
Synthetic Deep Web: How the Internet Is Becoming Invisible
For decades it was customary to imagine the internet as a space in which information exists independently of whether we can see it or not. Search engines might fail to find particular pages, sites might be closed off be…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
For decades it was customary to imagine the internet as a space in which information exists independently of whether we can see it or not. Search engines might fail to find particular pages, sites might be closed off behind authorization, documents might drop out of the index — yet the underlying logic remained simple: information exists in the network, and search is merely the instrument for discovering it.
Today a new intermediary has appeared between the person and the internet — the generative model. It no longer simply searches for documents and ranks links. It interprets the query, selects the permissible sources, reconstructs the answer, and presents the user with an already-processed picture of the information space.
As a result, a new form of invisibility arises.
A page may exist. A site may be running. A document may be directly accessible. But if the search engine does not show it, and the generative system does not mention it, then for a significant portion of users it practically ceases to exist.
This new domain can conditionally be called the synthetic deep web — Deep Web 2.0.
Its fundamental difference from the classical Deep Web lies in the fact that previously invisibility was primarily a property of the network itself. Now it is increasingly becoming a property of the informational intermediary.
The classical Deep Web arose as a byproduct of the internet’s technical architecture.
It came to contain pages that a search crawler could not or would not index: closed sections, dynamically generated pages, documents behind authorization, pages with indexing forbidden, databases, resources poorly linked to the rest of the internet.
This invisibility was, to a significant degree, accidental.
The site existed independently of Google. If the user knew its address, he could go there directly. If he knew the name of the document, the author, or the domain, he could make use of special search operators or find the resource through other sources.
The new invisibility is arranged differently.
The generative model is under no obligation to show the user the space of possible sources. It produces a final answer. Consequently the user often does not know which sources were considered, which were discarded, and which never entered the field of generation at all.
It is precisely here that the qualitative shift occurs:
previously a search engine might fail to find a page; now the user may not even learn that such a page exists.
This is no longer simply a problem of indexing.
The most essential property of the synthetic deep web lies not in the fact that information is being blocked.
Far more important is precisely how the user perceives its absence.
The search results page almost always demonstrated its own incompleteness. Before the person lay a list of links. He could see several pages, then move on to the tenth, the twentieth, or the hundredth. He could change the query. He could try another search engine.
The generative answer is arranged differently.
It can be short, coherent, confident, and outwardly exhaustive.
Where can I find a book on a narrow professional subject?
And receives several well-known stores, library catalogues, or contemporary publications.
If a specialized electronic library was not represented, the user does not receive the message:
“There exist several dozen more sites that I am not showing you.”
It is precisely this that turns filtering into an illusion of completeness.
The paradox lies in the fact that the better a model is at forming a coherent answer, the less noticeable the incompleteness of its information space becomes.
This mechanism is especially visible in the case of narrow professional subjects.
For many years the internet contained specialized electronic libraries, archives, forums, authors’ own sites, and collections of professional literature. A significant portion of such resources was never part of the information surface oriented toward the mass user.
Nevertheless, a user who knew the right search query could find them.
Over recent years more and more specialized resources have begun disappearing from ordinary search results. At the same time, search engines have begun actively introducing generative response mechanisms.
These processes do not necessarily share a single cause, and one should not automatically claim that one directly caused the other.
But the result of their superimposition is fundamentally important:
a resource can simultaneously lose search visibility and disappear from the generative information field.
For a specialist who knows that such a resource once existed, this looks like disappearance.
It is precisely this difference that is fundamental.
An even more interesting situation arises when what becomes invisible is not a specific site, nor even a specific book, but the very vocabulary of the search.
The FB2 format is telling in this regard.
In the Russian-language internet this was and remains a technical e-book format. The most varied literature was distributed in it: commercial works, authors’ own works, public-domain texts, old books, technical reference works, methodological materials, and archival publications.
But the word fb2 turned out to be so strongly bound, semantically, to pirated distribution that its mere presence in a query can change the character of the answer.
A fundamentally new form of filtering arises:
what becomes dangerous is not the content of the resource but the combination of words describing the method of searching for it.
For example, a query containing simultaneously:
may be interpreted by the system primarily as a potential request to infringe copyright.
But among such queries there are a great many entirely lawful cases:
works in the public domain; books distributed free of charge by the authors themselves; university and educational materials; old technical reference works; archival editions; works for which the rights holder has permitted free distribution.
Thus the system may fail to distinguish between:
Instead of analysis of the specific object, a classification by lexical signal arises.
Lexical toxicity is a situation in which a particular word or combination of words acquires so strong an association with an undesirable class of content that it begins to reduce the visibility of legitimate resources as well, ones that do not belong to that class.
What matters here is precisely the transfer effect.
The system ought to be answering the question:
But instead it may in effect be answering a different one:
“What risk is usually associated with the use of these words?”
Can a path to this content be shown to the user at all?
Can the vocabulary traditionally associated with this content be used at all?
At this last level, what begins to disappear is no longer the information itself but the means of discovering it.
From Content Filtering to Navigation Filtering
This is one of the most significant consequences of the shift from search engine to generative intermediary.
The search engine historically performed a function of navigation.
It might give poor results, but the user nonetheless received a set of objects among which he could choose for himself.
The generative model performs a function of interpretation.
It decides what is to count as the answer.
Therefore the object of filtering becomes not only the content but also the route to the content.
the name of the site; the name of the archive; the name of the specific document; the search term; an alternative spelling; a link to a specialized forum; mention of an existing library.
As a result, the information may physically remain on the internet, while the cognitive route to it disappears.
This is a fundamentally new characteristic of the network.
Three Mechanisms of Invisibility Formation
The synthetic deep web cannot be reduced to a single algorithm or a single company.
It forms as the result of the superimposition of several mechanisms.
The resource is poorly indexed, has a weak link structure, is generated dynamically, is closed off to crawlers, or lies deep inside a site.
The resource may be excluded from search results because of legal requirements, rights-holder complaints, platform policy, or other restrictions.
The site itself, meanwhile, may continue to exist.
The generative system interprets the query as potentially risky and replaces concrete navigation with a safe, generalized answer.
“Here is where the material you need is located”
“Here is general information on the subject and several safe alternatives.”
Each of these mechanisms may be explicable on its own.
But their joint action creates an altogether different effect.
For this reason it is no longer sufficient today to speak only of Deep Web 1.0 or Deep Web 2.0.
the resource exists ↓ it is poorly indexed ↓ it loses search visibility ↓ it ends up outside the customary surface of the internet ↓ the generative intermediary does not include it in the answer ↓ the user receives no information about the resource’s existence ↓ the resource becomes practically invisible
Meanwhile it may physically continue to function.
the availability of a resource and the informational accessibility of a resource cease to be one and the same thing.
But if practically no one knows its address, and the search and generative infrastructure does not lead the user to it, its openness becomes almost nominal.
Here there arises the temptation to explain what is happening solely by reference to Western value systems, copyright, safety, or the advertising model.
Different models do indeed have different zones of restriction. Some systems filter more heavily on questions connected with commercial risk, medicine, sexuality, or copyright. Others have their own political, historical, and cultural zones of sensitivity.
Therefore the synthetic deep web cannot be regarded merely as “Western censorship.”
Rather, what is at issue is local systems of invisibility arising within different information ecosystems.
The Western model may conceal one class of information.
An open-source model with a different configuration may show what a commercial system conceals, and vice versa.
For precisely this reason, comparing models becomes not merely a way of checking the quality of an answer.
It becomes a way of mapping the invisible portion of the internet.
For the classical Deep Web there existed relatively clear methods of assessment.
One could count pages, domains, non-indexable databases, closed sections.
With the synthetic deep web everything is more complicated.
If a model did not mention a site, it is impossible to establish the cause immediately.
not know about it? not see it in its training corpus? consider it irrelevant? distrust it? find itself unable to give a link? filter it out? prefer a better-known source? avoid commercial content? interpret the query as risky?
One and the same external result may have entirely different internal causes.
For this reason a new type of research is required here.
Not the measurement of the absolute volume of the invisible internet, but the comparative measurement of visibility.
And then — a comparison of what appears and what disappears.
Perhaps the most profound change consists in the fact that we are gradually moving from the model:
person → generative intermediary → search engine/sources → internet.
Each additional layer of mediation is capable of adding its own restrictions.
The model filters the search engine’s results.
The safety system filters the possible answer.
The advertising model determines commercial neutrality.
Legal restrictions determine permissible navigation.
In the end the user receives not the internet.
The most dangerous censorship is not the kind that says:
Such censorship at least informs the user of the existence of a boundary.
Far more effective is a situation in which the system simply shows him a differ
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。