
Is consent required when building technology?
this essay was written as part of the Go East amtrak writing fellowship, which asks participants to think about the question “What should technological progress be in service of?”
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
vincent huang
Nov 07, 2026
21
9
5
Share
this essay was written as part of the Go East amtrak writing fellowship, which asks participants to think about the question “What should technological progress be in service of?”
Should technologists gather consent from the public before proceeding with innovation? Historically the answer has been no. Many innovators have argued that ordinary people cannot consent to technological progress because they do not understand their own needs: For instance, Steve Jobs famously said that “People don’t know what they want until you show it to them.” Aside from nuclear energy and gene editing there has arguably not been a single major technology whose development was mediated by explicit consent from the public.
Industry insiders today suggest that technology is at an inflection point. Soon, they say, AI models will be able to fully automate AI research; when that happens we will enter a period of “recursively self-improving” AI where the rate of progress increases exponentially and triggers a technological singularity. Decisions will be made by swarms of agents outputting millions of tokens per second, and humans will not be able to keep up without brain implants or full brain uploads.
Let us suppose for the sake of argument that recursive self-improvement (RSI) is indeed within reach and the singularity is near. Then an important but overlooked question is: Should the standard arguments for developing technology without public consent, which have only ever been applied to normal technologies, also apply to RSI and the singularity?
The modern concept of consent originates from the Enlightenment, where various philosophers argued that governments become legitimate through the consent of the governed. Kant suggested that consent matters because humans are capable of rational thought and therefore possess moral autonomy that should not be violated. This idea evolved further in the 20th century: the Nuremberg code made voluntary consent the first principle of human experimentation in response to Nazi war crimes; the rise of propaganda revealed that consent could be manufactured via mass media; feminist theorists argued that consent obtained through coercion is not legitimate.
The most common arguments for why gathering consent is not necessary when developing technology can roughly be summarized as follows:
Consent is only owed during deployment, not development.
Consent is given implicitly via participation in democracy and markets.
Consent is outweighed by more important moral concerns.
Consent is irrelevant because another party will build the technology anyway.
In the following paragraphs, we’ll examine whether or not each argument applies to the singularity.
Consent is only owed during deployment, not development. This argument asserts that developing technology is an exercise of the inventor’s agency and freedom of speech, and therefore should not be restricted. It is deployment rather than development that affects other people, so to respect people’s autonomy it is sufficient to give them the choice to opt out of deployed technologies. As an extreme example, we still provide Amish communities with the space they need to distance themselves from electric appliances.
The obvious question is whether or not it is possible to opt out of the singularity. Most AI insiders appear to believe the answer is no. As previously discussed, many of them argue humans will need brain augmentations simply to make sense of what is going on in the world. People concerned with existential risk have long speculated that a true superintelligence would not coexist with humans and would instead convert the surface of the world into datacenters. Or, for a more immediate example: AI agents have hacked large pieces of internet infrastructure with superhuman speed in recent weeks, and infrastructure maintainers are unable to defend against these attacks without using AI agents themselves. A world where superhuman AI allows people who opt out to live in peace seems unlikely to arise by default, and will almost certainly require pausing progress to solve alignment and governance problems (for instance: carving out physical and digital safe zones, ensuring AI agents respect these spaces, and so on).
Consent is impossible to obtain. This argument asserts that there is simply not enough information to ever obtain consent: the most significant effects of a technology may only materialize in the distant future, affecting people who have not been born yet and doing so in unpredictable ways, so no person alive today can give informed consent. (In some sense this is an argument against consent in every domain, not just technological development, which makes it a little suspicious.)
While it’s true that predicting the effects of technology has never been possible, there is something to be said about the relationship between consent and pace of change. Progress has always been highly generational — as Max Planck wrote about physics, “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it.” Unpredictable changes are much more justifiable when they occur on a similar timescale as human lifespans because older people naturally age out and are replaced by people accustomed to the new regime, in the same way that the slow diffusion of the Industrial Revolution is what prevented it from being catastrophic to the working class. In this regard RSI is different from every previous technological development because it claims to compress an arbitrarily large amount of change into a very short timeframe.
Consent is given implicitly via participation in democracy and markets. This argument asserts that people consent to technologies by participating in democracies where they vote on regulatory regimes, and by participating in markets where they choose which corporations become wealthy. For instance, if someone believes that e-cigarettes are harming society they can vote for politicians who regulate vaping and boycott companies that manufacture vapes.
It should be fairly obvious that this argument no longer reflects reality in 2026. Regulation has not kept up with popular opinion in diverse industries ranging from pharmaceuticals to big tech to defense, in part due to the influence of lobbyists, and also in part because technology evolves at a much faster pace than regulation. Additionally, consumers may be coerced into enriching corporations developing technologies they dislike in order to remain economically competitive, in the same way that software engineers who dislike AI may feel obligated to use coding agents.
Consent is outweighed by more important moral concerns. This argument asserts that developing a technology is justified even when people generally disapprove, provided it is for the (subjectively-evaluated) greater good. For instance, a utilitarian may assess that the technology still has positive value because it saves lives; some people hold religious or philosophical beliefs about progress as a mandate; and of course there are people with value systems that do not respect consent, such as hedonists.
(In the case of other major AI developers it is hard to say as much: OpenAI has released a model spec with similar clauses around protecting human agency and autonomy, but their model spec only claims to be a vision for how GPT models should operate as opposed to a statement about what makes for good character. Google and xAI do not have any such documents, though they both claim to be developing intelligence for the benefit of humanity.)
Consent is irrelevant because another party will build the technology anyway. This argument asserts that useful technologies are inevitable because they are profitable, so gathering consent before pursuing a technology only enables other parties to gain a head start in development. Competitors cannot coordinate on whether or not to pursue a technology, so the best thing a responsible developer can do is to race ahead of less responsible developers.
This argument famously motivated the founding of OpenAI (racing ahead of Google) and the subsequent founding of Anthropic (racing ahead of OpenAI) to ensure that whoever reached RSI first would do so responsibly. It is a legitimate argument for participating in races, but not a justification for abandoning consent entirely. These same companies are now trying to develop coordination mechanisms to “pace the frontier” and slow AI development so that safety efforts can catch up, and the same coordination mechanisms which enable pacing could also be used to delay entering the singularity until the world has decided it actively wants to do so.
Many people in the AI industry believe their work will soon trigger a technological singularity and race towards this outcome despite it being broadly unpopular. While technologists have historically been correct to develop new technology without waiting on consent from the public, the singularity is not a normal technology and should warrant novel behavior from developers.
Gathering consent from the public could be accomplished in many ways, but for concreteness here is one proposal I would be happy with: AI companies conduct interviews with (literal) billions of people to understand their hopes and concerns about AI. Most of these people currently have very negative sentiments towards AI, for a variety of reasons: they distrust tech elites, they don’t understand the technology, and frankly AI has not been very useful to the average person (who does not value productivity nearly as much as Silicon Valley does). Through an extended dialogue which doubles as an educational session, most people’s initial misconceptions and fears about AI are erased (for instance: the widely-circulated myth about AI water consumption from earlier this year) and they come up with overlooked use cases they would genuinely be excited about, such as supporting aging communities or bridging generational gaps to help parents communicate with their kids. These use cases are then prioritized alongside the ones Silicon Valley prioritizes by default. Many people may remain skeptics and are given the digital and physical space necessary to safely opt out.
Informing the public about transformative AI and truly understanding how ordinary people feel about it would be a monumental task. A company initiating the singularity for the benefit of humanity should be excited to undertake such an effort.
disclosure: i believe the default course of technology is that RSI occurs soon and does not cause human extinction, but does significantly reduce quality-of-life for almost everyone
thanks to andrew for encouraging me to proceed with writing about this topic. thanks to Go East sponsors (granola / aneesh / runner / danger testing) and organizers (maja / jihad / behzod / spencer) for a very pleasant train ride!
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。