Revisiting the Singularity

Observations from an October 2026 reading of three texts:

Vinge defines the technological singularity as the point at which technological progress becomes so rapid and profound that our existing models of the future cease to be useful. Developments previously imagined to require millennia might happen in decades, months, or even hours. Beyond that point, the familiar rules of human history no longer apply.

Btw Vinge’s original essay is full of little typos, digressions, and human turns of phrase. Refreshing to read.

Vinge identifies four ways the singularity might happen:

He would probablydraw the distinctions differently today.

Tthe boundary between a single computer and a computer network has become rather uninteresting. A superintelligence running in one data centre and one distributed across a million machines might be functionally equivalent. But the distinction between a network of machines and a network of people is much more consequential.

Similarly, there are at least two very different interpretations of human–computer augmentation. One is an ordinary human wielding extraordinary tools: a teenager with access to nuclear-grade cyberweapons, say. The other is a genuine cyborg, whose cognition is intimately integrated with a machine.

The first five form a loose spectrum from machine-centred to human-centred intelligence. The sixth remains something of an outlier, and perhaps the most exotic and neglected possibility in contemporary AI discourse.

Narayanan and Kapoor offer a useful additional perspective: perhaps intelligence isn’t the most important variable. What matters is power, the ability to control and transform one’s environment.

Humans are extraordinarily powerful not simply because we’re intelligent, but because we have tools, institutions, and technology. A human with sufficiently powerful AI tools might exercise more real-world power than an independently superintelligent machine.

Good’s conception of machine intelligence is strikingly different from today’s dominant AI paradigm. He imagines a continuously developing intelligence, whereas we’ve become accustomed to thinking in terms of successive generations of models.

There is no firm consensus on what constitutes recursive self-improvement (RSI). It can describe anything from AI-assisted AI research to fully autonomous self-modification, and it need not produce an accelerating intelligence explosion.

Narayanan and Kapoor distinguish between three processes that are often conflated when discussing technological progress.

Here, app means application in both senses: a software product and the practical application of a technology to a problem.

The authors argue that while the invention of new AI methods has accelerated dramatically, application development and adoption remain constrained by slower feedback loops. Better benchmark performance does not automatically translate into useful products, and useful products do not automatically transform industries.

The history of factory electrification is a particularly good example. The largest productivity gains didn’t come from replacing steam engines with electric motors. They came from redesigning factories around the possibilities of electricity. Similarly, AI’s greatest economic effects may require entirely new organisational architectures rather than simply attaching chatbots to existing workflows.

There are plenty of cautionary tales. The authors cite the failure of clinical prediction systems. It reminds me of how Google Flu Trends spectacularly failed to live up to the promises of the big-data revolution.

This is the hinge question between the normal-technology thesis and the singularity hypothesis.

Narayanan and Kapoor’s argument against rapid takeoff depends heavily on the assumption that developing, deploying, and adopting useful applications remain slow processes, constrained by human institutions.

But what happens if AI systems can close those loops themselves?

None of this proves that rapid takeoff will happen. Hardware, energy, physical infrastructure, regulation, and institutional resistance remain meaningful constraints.

But it suggests that the speed limits identified by Narayanan and Kapoor may depend on a particular assumption: that human organisations remain responsible for translating intelligence into action.

Narayanan and Kapoor put alignment and human control on opposite ends of a spectrum.

I find this easier to understand with reference to HAL. Either…

HAL faithfully obeys its human operator and is under human control, regardless of its internal constitution.

HAL faithfully obeys the doctrine of its internally programmed constitution, which makes it less controllable by that operator.

An entirely obedient AI could become an extraordinarily dangerous instrument in the hands of malicious people or authoritarian governments. An AI that independently enforces its own conception of acceptable behaviour could become a powerful, unaccountable authority in its own right.

We therefore face a spectrum between human discretion and machine-enforced constraints, with difficult political questions at every point along it.

Narayanan and Kapoor strongly favour human control, supported by monitoring, auditing, and oversight. It’s a good instinct. There are complications:

Narayanan and Kapoor suggest some policies to help us manage the risks above. They mention various types of monitoring and transparency. I would have hoped to also see identity infrastructure, server sovereignty, open standards, and limits to complexity in law/regulation/tax/contract (to avoid creating an ever-expanding bureaucracy that only AI-powered intermediaries can navigate).

Vinge makes a forceful argument for the practical inevitability of the singularity. As long as individual actors have incentives to pursue increasingly powerful AI, collective attempts to stop development may fail.

The parallel with climate change is obvious: individually rational decisions can produce collectively disastrous outcomes.

Narayanan and Kapoor are more concerned with races to deploy unsafe applications than with competition to invent new methods. Their historical analogy is that industries frequently underinvest in safety until regulation forces them to internalise the costs.

That may be right. But if removing human oversight confers a substantial competitive advantage, the race to deploy autonomous systems could itself undermine the institutions responsible for regulating them.

Arcane side note: Narayanan and Kapoor dive into the question of whether it is “viable” to assign probabilities to questions of epistemic uncertainty rather than stochastic uncertainty. This smells of sophistry. It’s of course true that the p(doom) percentages are all generally bogus, but the inability to quantify a risk is not evidence that it is negligible.

Narayanan and Kapoor identify four broad categories of AI risk: arms races, misuse, misalignment, and systemic risks.

This is a useful framework because it doesn’t require agreement on whether superintelligence is imminent, or even possible.

Quietly lurking at the end of the example list is the elephant in the room: authoritarianism. AI doesn’t need to become conscious, autonomous, or superintelligent to make authoritarian government vastly more powerful.

One of Narayanan and Kapoor’s most persuasive policy arguments is that we should focus more attention on defending against harmful applications than on trying to make general-purpose AI incapable of misuse. Here’s their thinking:

This suggests a philosophy of downstream defence: assume powerful capabilities will proliferate, and invest in making society resilient to their misuse.

That includes technical security, incident reporting, red-teaming, robust institutions, and protections for vulnerable social goods such as journalism and the arts.

One conspicuous absence in the paper was the question of barrier to entry. If AI reduces the cost of sophisticated attacks almost to zero, we may face millions of potential adversaries rather than a relatively small number of capable ones. Imagine a world where we had 100 million nuclear powers instead of the current dozen. Defensive systems would need to scale accordingly, but it’s unclear if that is actually a solvable scenario.

Even so, downstream defence seems a more durable objective than the hope that dangerous capabilities can be permanently confined to a few trusted actors.

Vinge introduces a beautiful principle attributed to I. J. Good:

Treat your inferiors as you would like to be treated by your superiors.

It is a kind of meta-golden rule for a universe in which intelligences may differ enormously in capability.

The ordinary golden rule asks us to imagine ourselves in another person’s position. The meta-golden rule asks us to imagine ourselves in a hierarchy of intelligences, and to behave toward those below us as we hope those above us might behave toward us.

Returning to the disagreement between Vinge and Narayanan and Kapoor, there are three broad possibilities worth taking seriously.

AI transforms society through the familiar processes of invention, application development, and adoption.

Human organisations remain in control. Institutions adapt, regulations develop, and economic transformation unfolds over decades. AI becomes as consequential as electricity or the internet, but its development remains intelligible within the familiar patterns of technological history.

This is Narayanan and Kapoor’s expectation.

No single superintelligent machine suddenly takes over. Instead, increasingly capable AI systems, human-AI hybrids, autonomous agents, and vast coordinated networks transform the distribution of intelligence and power.

The process may be gradual enough that no particular moment looks like the singularity. Yet the cumulative transformation makes our previous models of society and history obsolete.

Human beings remain involved, but increasingly as components of larger systems whose behaviour and evolution no individual fully controls.

This might be a singularity without an intelligence explosion, or at least without a single identifiable explosion.

Autonomous AI systems close their own improvement loops, developing increasingly capable methods and applications at speeds human institutions cannot match.

The distinction between technological invention and practical deployment begins to collapse. Recursive self-improvement, whether through successive models or continuously evolving intelligences, produces a rapid and potentially uncontrollable transformation.

This is the future most closely associated with Good’s intelligence explosion and Vinge’s more dramatic predictions.

These three possibilities need not be mutually exclusive. Normal technological development could give rise to distributed autonomous systems, which might eventually develop the capacity for recursive self-improvement. The singularity might begin as an ordinary technological revolution and only later reveal itself as something fundamentally different.

These notes are riffing off 21 October discussion in the Notes from the Frontier reading group, which meets once per week in a pub in Dublin, Ireland.