
The missing quadrant in responses to AI
Announcing STABL: Societal Transformation and AI Bottlenecks Lab
以下正文同步自 AI Snake Oil,版权归原站所有,已转换为易读排版。
We are starting a non-profit research organization: STABL. Adapting to advanced AI will require dramatic institutional change and broad societal resilience. It will also require evidence about AI progress and impacts — the prospects of recursive self-improvement (RSI), the pace of AI diffusion, and the bottlenecks that constrain how AI capabilities translate into societal outcomes. Four theses define our worldview:
- We consider advanced AI a transformative general-purpose technology with the potential for broad societal benefits.
- Yet we recognize that the pace of societal transformation will be bottlenecked by the speed of human and organizational change more than progress in capabilities.
- Our institutions — the established ways we organize knowledge, work, and governance — are an important source of stability during this turbulent period, but they also need serious reforms to keep serving their purpose.
- We favor a broad portfolio of defenses to manage AI risks, of which alignment is only one part. Resilience, the most neglected component today, must become central. Another way to state our worldview is to place it on a 2x2 chart. The vertical axis depicts taking AI’s transformative potential seriously. The horizontal axis depicts taking downstream bottlenecks to AI’s impacts seriously. At the top left is technological determinism: the view that superintelligence will unilaterally transform society by obliterating downstream bottlenecks. Many AI CEOs express this view, such as when predicting that a century’s worth of progress will be compressed into the next few years. This view often ignores the organizational, regulatory, and societal bottlenecks to such a transformation; in this view, such questions are moot because superintelligence is all powerful. Surprisingly, some of the AI industry’s strongest critics share this worldview, focusing on slowing AI capability improvements as the main way to keep society safe from AI’s dangers. The opposite view, at the bottom right of the chart, treats existing organizations, processes, professions, and governance as fixed points, and insists that technology be molded to fit them. People holding this view often work on limits on AI adoption, and protecting existing structures from disruption. They are good at identifying and leveraging the bottlenecks that blunt AI’s impact, such as regulation, professional norms, organizational inertia, and political pushback. Slowing the pace of change can buy valuable time to shape technology’s impacts. But when preserving the status quo becomes a goal unto itself, it stands in the way of technology-enabled societal progress. (As an example, consider the resistance to self-driving cars despite their being safer than humans, or bans on AI-enabled legal advice). STABL belongs in the top-right quadrant. We take frontier AI seriously as a transformative technology, yet consider the role of institutions — both new and existing — to be important in helping society navigate the transition. For example, how should we understand and improve AI’s impacts on medicine? Many technological determinists claim superintelligence will rapidly cure nearly all diseases. Someone with the opposite view might respond that the bottleneck is not drug discovery, which AI may well accelerate, but clinical trials, which take years because of regulations and safety risks. This response correctly identifies the bottlenecks to AI’s impacts. But this is often seen as the end of the conversation. In our view, understanding the bottlenecks to AI’s impacts is only the starting point of analysis. The next question is whether drug testing can be made faster and more efficient in order to meaningfully materialize AI’s benefits. For example, what are the barriers to in-silico testing? How can we rethink regulatory institutions and processes? Unfortunately, neither extreme pays close attention to these critical questions. To be clear, STABL isn’t the only organization in the top-right quadrant; much of our work builds on other organizations and thinkers who take both AI’s potential and its downstream bottlenecks seriously. But compared to the other quadrants, there is a sparsity of thinking in this quadrant, which has led to polarization among the communities that take either the technology or the bottlenecks seriously. Our ultimate goal is to contribute to a rich intellectual agenda for this quadrant, create the space for new organizations, and collaborate with existing organizations in this and other quadrants. To follow our work, subscribe to the STABL newsletter: Subscribe to the STABL newsletter
Our research agenda
The top-right quadrant leads us to many interesting research directions. On one hand, we expect AI capabilities to continue improving quickly, which motivates our empirical research on the pace of progress and RSI. We expect highly capable AI models to continue to be widely available, which motivates our analysis and advocacy for risk mitigation beyond aligning AI models and non-proliferation. On the other hand, we consider institutions important for shepherding the transition to a world where advanced AI is broadly diffused. This leads to our interest in understanding the downstream outcomes of advanced AI adoption. We plan to measure specific benefits and harms, study the disruption of labor markets and beyond, propose ways to improve societal resilience and revitalize institutions. These interests lead to four key research directions, which we detail below. While other organizations have worked on similar questions, we expect our research to be different in how we formulate these questions and how we go about answering them, informed by our worldview and priors about AI progress.
1. How should we measure progress towards recursive self-improvement?
Many forecasts of explosive AI progress rest on recursive self-improvement (RSI): AI automating AI research itself. We take RSI seriously precisely because our model of progress differs from the one underlying most RSI forecasts, and we want to empirically test claims about RSI to get early warnings of AI’s potential impacts. But how should we measure progress towards RSI? Most RSI evaluations measure AI’s ability to complete tasks where success is easily verifiable. Here, AI systems have made rapid progress. But we think there is a growing chasm between AI performance on verifiable and open-ended tasks. Through CRUX, we plan to run frequent open-ended evaluations to probe the upper bounds of AI capabilities on long, messy tasks, including whether AI agents can conduct open-ended AI research. Our first results suggest they cannot yet do so; expert reviewers unambiguously rejected the papers produced by frontier agents. We plan to repeat these evaluations regularly with improved models and scaffolds. More broadly, the conversation about RSI is dominated by viewpoints in the top-left quadrant. Those most worried about superintelligence were ahead of the curve in recognizing the importance of studying RSI carefully. But communities and organizations have also self-selected into this work, leading to a lack of intellectual diversity. When the literature relies on the same (contested) assumptions about AI progress, it is prone to the same failures in its conclusions. A more diverse ecosystem of researchers and organizations, including those with different priors about AI progress, could test whether these assumptions hold, for example by:
- Comparing causal models of AI progress. Rather than assuming a model and making predictions about the world based on that model, evaluate different causal models against one another. For example, does performance on verifiable tasks straightforwardly lead to real-world performance improvements?
- Studying how data bottlenecks shape recursive improvement. Does AI progress on tasks where it is straightforward to collect high-quality data translate to tasks where it isn’t? Understanding when improvements carry over across domains would help distinguish the hypothesis that RSI leads to broad superintelligence from alternatives in which a lack of real-world data limits progress in some domains.
- Disaggregating different dimensions of progress. Most benchmarks of AI progress rely on automated verification, which also makes them more suitable for reinforcement-learning training. Moving beyond automatic benchmarking could help us understand progress in non-verifiable tasks. Such disaggregation could help us understand which aspects of AI progress are amenable to recursive improvements and which ones aren’t. As the importance of RSI-style dynamics becomes clearer for understanding AI progress, we hope there is a vibrant intellectual ecosystem of researchers and organizations evaluating RSI seriously while not necessarily treating its default outcome as superintelligence.
2. How can society defend against the risks of advanced AI?
AI safety has historically been focused on models themselves, and alignment has been the field’s technical grand challenge. The safety community’s response to open models, where alignment can be undone, has been to push for non-proliferation policies. But over time, it has become clear that advanced AI will be broadly available, including to people who might want to misuse it. So our responses to AI risks need to expand beyond alignment and non-proliferation. AI safety will require three additional responses. First, we need to invest in the ability to control, monitor, and secure AI systems to prevent unintended actions even when AI systems are misaligned. Second, we need to improve downstream defenses, such as the cybersecurity of critical infrastructure to protect the attack surfaces where AI’s harms materialize. Third, we need to boost societal resilience to reduce the severity of harm when our defenses fail. We think resilience is the most neglected approach. While the AI community has in principle emphasized resilience, in practice, the main emphasis remains on alignment and non-proliferation. We often hear the reaction that resilience sounds good in theory but no one knows what it means concretely. One of STABL’s goals will be to articulate an actionable resilience agenda. We hypothesize that for resilience to be effective, it must have the following properties:
- Domain-specific: We won’t solve resilience in a general way. We must understand and improve the resilience of specific sectors against specific threats. For example, what is the cost of securing a state’s hospital system against cyberattacks? Who are the actors best prepared to deploy these defenses? What is the most viable plan of action for implementing cyberdefenses at a large scale?
- High ambition: The investment needed in serious attempts at resilience (such as pandemic preparedness and large-scale cyber-defense) dwarfs our current investments. We will propose scalable, ambitious responses commensurate with the magnitude of the threat.
- Worst-case planning: Despite wide concern about catastrophic AI risk, there is little work on response and recovery plans for specific AI disaster scenarios, up to and including civilizational collapse. Improving resilience requires staring down the barrel of potential AI catastrophes and developing response plans.
3. How can we steer the economic impacts of advanced AI?
Even though advanced AI has been widely adopted by customers, workers, and firms, we don’t yet have a clear accounting of either the benefits or the costs of this adoption. While there is abundant evidence of workplace adoption, the returns on that investment have yet to materialize as broad productivity gains. And while many have raised concerns about imminent and massive job loss, the aggregate unemployment rate continues to be largely unchanged in the United States and other countries. These trends are consistent with our view that the bottlenecks which govern economic transformation are largely beyond the scope of improving
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。