Metacrisis as a Framework for AI Governance

Cross-posted from Lesswrong.

The theory of metacrisis^[1] put forward by Daniel Schmachtenberger and others has gained traction in some circles concerned with global risks^[2]. But despite strong similarities in key claims, it remains unexplored within AI safety and governance discourse^[3]. 

In this post I aim to bridge the gap by comparing related ideas across the two fields, focusing on how metacrisis theory can offer a coherent framework for AI governance research.

Central to metacrisis theory is the idea that the main global risks facing humanity have common structural and cultural roots^[4]. Catastrophic risks from AI, great power war and environmental damage, among others, are traced back to economic structures and dangerous system dynamics, as well as the cultural and psychological factors supporting them.

There is significant overlap between this perspective and AI governance research. There are also areas where metacrisis theory can be seen as offering a new framing of research priorities - such as Schmachtenberger’s analysis of civilizational system dynamics and the need for a “third attractor”. 

Both metacrisis theory and AI governance research are broad fields covering multiple areas. Before diving into the details, I'll briefly situate each field relative to the other.

Situating Metacrisis within AI Governance Research

A fairly standard way of thinking about the range of research in AI governance is to distinguish the following four categories:

• Strategy research: investigating conceptual foundations, forecasting likely AI risks and setting priorities for AI governance work
• Industry-focused approaches: research on how to improve decisions at AI labs through corporate governance, and technical standards and evaluations
• Government-focused approaches: research on possible executive and legislative action, including international negotiations and national policy
• Field-building: developing the institutional infrastructure for, and raising awareness of AI governance as a discipline

Of these categories, metacrisis theory primarily contributes to strategy research. It offers a high-level conceptual framework for understanding the civilizational dynamics that underpin global risks, including risks from AI. And while it does have implications for research priorities, it tends not to explore government or corporate policy proposals in detail.

Situating AI governance within Metacrisis research

At the same time, metacrisis theory extends beyond strategy research of this sort. Aside from the fact that it tends to consider global risks other than those from AI, many metacrisis theorists draw connections between global risks and issues in epistemology and applied ethics^[5], for instance.

In addition, metacrisis theorists sometimes explore the idea that cultural and psychological development is desirable independently of its impact on global catastrophic risks - for example, to improve levels of wellbeing^[6].

I’ll leave these connections to one side, here, in order to focus on AI governance implications. 

For the same reason, I’ll draw primarily on Daniel Schmachtenberger’s framing of metacrisis, as it goes most deeply and explicitly into the theme of AI risk (he is also very likely the most widely known metacrisis theorist^[7]).

In what follows I’ll start by highlighting three substantive areas of agreement between metacrisis theory and AI governance research.

I’ll then consider three elements of metacrisis theory that build on these areas of commonality to offer new priorities for AI governance research.

Key Areas of Agreement

AI as a prominent risk

AI governance research is often premised on the idea that AI presents an especially prominent global risk.

Schmachtenberger and some other proponents of metacrisis theory would agree. In his writing and videos, he points out that AI is

• A General-Purpose Technology: Unlike narrow technologies, AI can transform every domain of human activity simultaneously, including both civilian and military applications. It is therefore likely to augment other catastrophic risks, such as environmental damage or great power war.
• Self-Improving: Since one of the domains AI can improve is the AI development process itself, there is a potential for recursive self-improvement, resulting in acceleration dynamics, and potentially risks around superintelligence.
• Easily Disseminated: Unlike other powerful technologies such as nuclear, AI products are easily disseminated and used by a wide range of agents, making associated risks harder to manage.

While metacrisis theorists tend not to focus narrowly on existential threats from superintelligence, this is also true of more recent AI governance work^[8].

Wider catastrophic risk landscape

Despite a focus on risks from AI, AI governance research acknowledges the existence of a wider landscape of potentially catastrophic risks, including anthropogenic risks from nuclear weapons, environmental damage, and biotechnology. Metacrisis theorists discuss essentially the same range of risks^[9].

Metacrisis theorists tend to highlight the interactions between different catastrophic risks, and the existence of common structural factors. This too is arguably an area of agreement with AI governance research, which acknowledges that catastrophic risks significantly affect one another in terms of their likelihood and impact, and that it is often useful to focus on underlying risk factors^[10].

Molochian dynamics as prominent risk factors

Core risk factors for Schmachtenberger include coordination failures or Molochian dynamics such as:

• Perverse incentives: Economic and political systems that reward behavior harmful to collective long-term interests
• Race dynamics: Competition between agents (nations, corporations, individuals) creates race dynamics where everyone must optimize for narrow metrics or be outcompeted
• Evolutionary selection: In some cases, agents that fail to compete are eliminated from the population, resulting in further proliferation of harmful activity

This is another area of commonality between metacrisis theory and AI governance research. Meditations on Moloch has of course been influential in the wider rationalist community, and an understanding of these dynamics as contributing factors for AI risk is well-integrated into AI governance research^[11].

Metacrisis as framework for AI governance research

In what follows I’ll show how, building on the commonalities just outlined, three key themes in Metacrisis theory offer a coherent framework for thinking about AI governance. While there are numerous areas of overlap with existing research in AI governance within these three themes, taken together they suggest a distinctive research approach.

1. Complex systems approaches to the AI risk landscape

Metacrisis theory often draws on complexity science and dynamical systems theory to understand AI risk as part of a wider system of global risks.

Schmachtenberger draws on this paradigm^[12] to highlight the significance of cascade effects and feedback loops in thinking about the risk landscape. 

The interconnectedness of diverse global systems - such as environmental, governmental, economic, transportation and communication systems - means that events in one system can spill over into other systems, causing chains of potentially catastrophic effects.

Similarly, feedback loops within and between global systems can lead to amplification of impacts, generating catastrophic outcomes from what would otherwise be limited events.

In this respect metacrisis theory is aligned with the related concept of ‘polycrisis’ (coined by complexity scientist Edgar Morin in 1999). Recent work by the Cascade institute and Center for the Study of Existential Risk has sought to clarify this conceptual framework in terms of the impact of trigger events on systemic stresses, and the relation between specific hazards, systemic vulnerabilities and amplification processes.

Complexity science is sometimes explored in AI governance research. For example Dan Hendrycks’ AI safety book explores the way  social systems can be modelled by complex system dynamics, the implications of this for advocacy work and the difficulty of predicting the impacts of AI developments.

As a way of modelling the relationship between AI risks and other global risks, however, the metacrisis approach is relatively unexplored.

It suggests an increased prioritisation of issues like:

• Risk Amplification: the extent to which the sociotechnical impact of AI amplifies or is amplified by environmental destruction, information ecosystem degradation and great power conflict.
• Systemic vulnerabilities: the extent to which various global systems could be made less vulnerable to risks from AI.

 2. A Misaligned Global System

We’ve seen how the idea that the history of human civilization includes Molochian dynamics is common ground between metacrisis and AI governance perspectives.

Schmachtenberger takes this idea further, however, in framing these and other dangerous dynamics as characteristic of civilization. Understood as a vast dynamic system, human civilization to date has been misaligned to core human values, he argues.

Evidence he provides for this thesis includes

• Ecological damage as a feature of most historical cultures, including historical cases (Easter Island, Maya, Mesopotamia) where damage like overfarming and deforestation led to cultural collapse
• The Neolithic Agricultural Revolution as leading to net decreases in health and well-being
• Current technological risks from nuclear weapons and AI resulting from core civilizational dynamics
• Resource wars as a common theme of world history, despite being negative sum processes 

A further consideration prompted by complexity science is the idea that global system misalignment is especially difficult to address if the dynamics of such a system are self-perpetuating^[13].

For example a civilization will typically include incentives to endorse belief-systems that validate that civilization’s dynamics, thus dampening efforts to change the system.

While AI governance textbooks recognise the existence of misaligned processes within civilization, and may accept that historical cultures have been unsustainably misaligned, they tend to assert that modern industrial civilization has on balance brought progress - or in other words, increasing alignment - with respect to core human values^[14].

Schmachtenberger discussed this progress narrative in depth in a book-length work entitled ‘Development in progress’. On the one hand, he questions the details of progress narratives such as those of Hans Rosling and Stephen Pinker, offering arguments that increases in life expectancy have not been accompanied by increases in quality of life.

On the other hand he argues that the progress narrative depends on an overly narrow conception of progress, one that fails to take into account economic externalities and other second order effects, including increased global risks of precisely the sort AI governance work is concerned with.

The framing of civilization - including modern industrialised civilization, despite all its benefits - as a misaligned system in metacrisis theory thus offer a useful complement to existing conceptual frameworks of AI governance research, with greater emphasis on

• Alignment of economic and political institutions as a component of AI alignment^[15]
• The need for system change, and significant market interventions, to address AI safety concerns.

 3. The “Third Attractor”

Complex dynamical systems often involve one or more attractor states towards which the system tends to evolve.

Schmachtenberger draws on this idea in arguing that the dynamics of the misaligned global system tend toward two undesirable outcomes:

1. Uncoordinated attractor: The failure to solve coordination problems means that Molochian dynamics lead to cascading system failures - unaligned AI, ecological collapse, nuclear war, or other catastrophes.
2. Authoritarian attractor: In response to crisis, or as an attempt to solve coordination problems, power centralizes into dystopian authoritarian structures that use surveillance, social control, and potentially AI to maintain order. 

A third attractor is therefore needed to avoid these two undesirable outcomes.

• On the one hand, it must tame Moloch dynamics by solving coordination problems, internalising externalities and aligning economic incentives to long-term collective interests.
• On the other hand, it must do this in a way that avoids excessive central authority, by improving democratic processes of collective intelligence in a way that likely requires cultural enlightenment.

In order for it to be an attractor, rather than just a utopian state of affairs, the third attractor must be understood as a possible evolution of the current system, under realistic variations of existing dynamics.

Schmachtenberger therefore points to a number of transitional processes whereby existing dynamics can be harnessed and modified to achieve systemic change.

In the context of AI risk, of particular interest is the idea that information technologies will be crucial for improving collective decision-making and consensus-building. Schmachtenberger points to examples like Taiwan’s digital democracy experiments, and alternative electoral systems, as well as the use of AI tools.

This framing again offers a new lens through which to view the AI governance landscape. 

We saw earlier that a standard way of breaking down AI governance work considers four main buckets: government policy, industry governance, field-building and strategy work.

The strategic prioritisation of policy and industry governance here can be seen as a focus on avoiding the Uncoordinated Attractor: solving coordination problems and fixing incentives at the level of labs and state actors.

Equal attention to avoiding the Authoritarian Attractor would likely shift the balance of priorities in the field.

Some areas of current activity that might be assigned higher priority or extended in new directions include:

• AI advisors for decision-making: Carl Shulman has promoted the idea that AI advisors could radically improve societal decision-making within government. Similarly, Forethought recommends the development of AI tools to help anticipate risks and improve collaboration between diverse groups. The idea could be extended to wider democratic deliberation in the population, as well as to the relation between societal decision-making, collective wisdom, and cultural enlightenment.
• Cooperative AI: Allan Dafoe is now a key figure behind the Cooperative AI Foundation^[16] which seeks to improve the cooperative intellignce of AI, including the ability of AI to assist with cooperation between human actors. This approach could  be extended to cooperative intelligence in the wider population.
• Research on authoritarian capture: A research team at Forethought has recently explored the risks of AI-enabled coups. These and other risks of authoritarian capture of AI governance mechanisms could be explored in connection with Bostrom’s vulnerable world hypothesis.

Summary

I've suggested that metacrisis theory offers a coherent, big-picture framework for AI governance, which builds on significant commonalities with existing approaches to indicate new priorities in key research areas.

Commonalities

• AI as Prominent Threat. There is agreement that AI risk is pre-eminent in the risk landscape, not only in virtue of the direct risks it poses, but because it augments other global risks as a general purpose technology.
• Wider catastrophic risk landscape. There is general agreement on the outline of the catastrophic risk landscape.
• Structural Roots: Moloch Dynamics. There is agreement that Moloch dynamics are key structural factors underpinning AI risk

New research priorities

• Systemic risk interactions. Metacrisis theory suggests new ways to understand the catastrophic risk landscape through a focus on connections between risks from a system dynamics perspective.
• Aligning the system. Metacrisis theory offers an alternative to the progress narrative underpinning much AI governance work, suggesting greater focus on market invention and systemic change.
• Collective intelligence. As a complement to the focus on coordination problems within AI governance, metacrisis theory suggests a greater focus on using AI to enhance collective intelligence and democratic decision-making.

In addition, as an early-stage, fluid research initiative, questions around the validity of the metacrisis framing desribed in this post can be thought of as being among the recommended research priorities.

[Many thanks to Euan Mclean, Chris Leong and Rufus Pollock for their feedback on an earlier draft.]

1. ^{^}

   A good introduction to metacrisis is From Polycrisis to Metacrisis. In Metacrisis: an Overview I provide a chronological review of perspectives to date. In this post I focus on Daniel Schmachtenberger's theory as articulated in articles here and here. In addition to numerous Youtube presentations and discussions, other key texts include Metatheory for the Twenty-First Century, Economics for the future – Beyond the superorganism, Visionary Realism and the Emergence of a Eudaimonistic Society, Education is the Metacrisis, and Tasting the Pickle. 

2. ^{^}

   For example, the research community known as the liminal web or second renaissance ecosystem. Video interviews on the subject of metacrisis have reached hundreds of thousands of views. 

3. ^{^}

   On the EA forum there has been this recent post, and this older discussion. On LessWrong there appear to have been only a few comments and references. 

^{^}

Different perspectives on metacrisis tend to differ in the account they give of these roots, as I discuss here.

^{^}

This is not unlike the way the rationalist community draws connections between AI risk, bayesian epistemology and applied rationality. In addition Schmachtenberger himself, Nicolas Hedlund and Jonathan Rowson discuss epistemological issues in detail. Rowson and Schmachtenberger also consider the ways in which individuals can respond ethically to the metacrisis. 

^{^}

The most-watched video on the topic of metacrisis on Youtube is a discussion between Schmachtenberger, John Vervaeke and Ian McGilchist. Vervaeke and McGilchrist often discuss reasons why cultural evolution is needed independently of catastrophic risk, for example because of a 'meaning crisis'.

^{^}

I'm assuming that the number of YouTube views for video interviews of Schmachtenberger (which run to the hundreds of thousands) offers a simple metric that can be used to assess 'well-known'.

^{^}

Schmachtenberger’s perspective lines up well, for example, with the overview of AI risk provided by Allan Dafoe in the Oxford Handbook of AI governance, where he frames AI as simultaneously 1) a general purpose technology, 2) an information technology, and 3) an intelligence technology.

^{^}

See e.g. Schmachtenberger Catastrophic and Existential Risk, and Rowson Tasting the Pickle.

^{^}

See Toby Ord's The Precipice (Chapter 6) for an influential discussion of risk interactions and common risk factors.

^{^}

Nick Bostrom’s Superintelligence considers race dynamics as a strategic consideration - and is also cited by Schmachtenberger. Allan Dafoe’s 2018 AI Governance Research Agenda centers coordination failures as a key challenge. More recently, Dan Hendrycks’ AI Safety textbook dedicates significant attention to game-theoretic dynamics; and race dynamics are highlighted in the introductory chapters of the Oxford Handbook of AI governance. 

^{^}

Concepts from complexity science are applied in many of Schmachtenberger’s video interviews, for example the Bend not Break series. They are also emphasised in his earliest written articles on catastrophic risk.

^{^}

In the terminology of dynamic systems theory, the system may be ‘autopoietic’ in a way that resembles living organisms. For this reason, in metacrisis theory global civilization is sometimes described as a ‘superorganism’ by analogy with ant colonies and other systems that display emergent patterns of seemingly purposive behaviour. Schmachtenberger is not the first to apply dynamical systems theory to social dynamics. For example the German sociologist and philosopher Niklas Luhmann is well-known for his theory of autopoietic social systems.

^{^}

See the introductory chapters of Oxford Handbook of AI governance, and also Has life gotten better?: the post-industrial era.

^{^}

Related ideas have recently been discussed under the heading of full stack alignment and ecosystem alignment and have also been promoted by the AI objectives institute. See also https://cybersynergetic.com/

^{^}

Interestingly Audrey Tang, the innovator behind digital democracy in Taiwan, is a Trustee of this foundation, suggesting a convergence of approach with metacrisis ideas.

Show all footnotes