TLDR: at the core of the concept of existential risk is the idea of “lock-in”; a situation in which the future of humanity is permanently set into a highly negative end-state. Typically, the threat of lock-in is seen to emanate from unitary agents of great power – such as a totalitarian regime or unaligned AI. However, the threat of lock-in also arises in situations where power is diffused amongst many agents – due to the overwhelming influence of the selective pressures and coordination problems that develop in these cases. Indeed, the threat of lock-in through diffusion of power is arguably greater than the threat of lock-in through concentration of power. This is “multilateral lock-in”. 

I. Framing Lock-In 

One of the most notable developments in recent human history has been the rise of totalitarian power. Before the 20^th century, many empires claimed title to total rulership of mankind. But to an impartial observer, none came close to vitiating such a boast – either in the size of their holdings, or their administrative and technological capacity to dominate them.  But with the onset of the modern age, technical and organisational progress at last gained ground on tyrannical ambition – birthing first fascism, then Stalinism and Maoism. It is no wonder that, for those of us who live just beyond the cusp of these nightmarish possibilities, they cast so tremendous and fearful a shadow. This shadow lies deep over our current discourse on “lock-in”[1].

However, historically, many of the greatest pressures operating to fix human development have resulted not from the unification of agency, but from its multiplication. One key reason for this is that the multiplication of agents intensifies the selective pressures acting upon them. Another is that it makes solving coordination problems much harder. In extremis, these selective pressures and coordination problems can be as powerfully determinative as totalitarian rule, with consequences equally as vicious. This is “Multilateral Lock-In”. 

II.  The Power of Selection

In a population of self-replicating entities (such as a human group, or a group of human groups), selective pressures tend to arise. Put crudely, some replicators survive longer and produce more offspring. Other replicators survive less long and produce fewer offspring. Over time, the population will tend to become dominated by the former group.

Numerous factors influence the strength of these selective pressures; that is, the speed with which the longer-lived and faster-replicating replicators come to dominate the population. These include the speed of replication, the degree of variance between group members (and of “mutations” causing the development of new variances) and – crucially for our purposes – the number of replicators in the population. The importance of larger populations comes from the increased number of mutations it will express; and the concomitant increase in the likelihood that potentially selectively advantageous mutations will actually manifest and spread throughout the population. 

To understand the central importance of these factors, one needs only consider the following paired examples: 

• Elephants & Flu: there are around 415,000 wild African Elephants (Loxodonta Africana) alive today. Contrastively, the flu infects roughly 1 billion individuals each year; each of whom plays host to many billions of viral particles. Whilst millions of years of evolution has been required for Africana to diverge from one of its closest relatives, the Asian Elephant; new strains of the Flu evolve with regularity year on year.

• Companies & Countries: there are 195 countries in the world today. There are, by contrast, over 53,500 listed companies, and most likely millions more unlisted entities. And whist a country may mark its age in centuries, the average lifespan of top companies can be measured in decades. From a selective perspective it is therefore unsurprising that “the market” (understood as the population of companies) holds a reputation as a pressure cooker of adaptation and disruption, one which states struggle to rival.

The natural outgrowth of these examples is simple. Selection is powerful, and its power is (significantly though not solely) a function of the number of entities in the selected populations.

The upshot of this is that, in small populations, selective pressures are likely to be weak. Their influence may therefore be overruled by the agency of members of that population, by chance, or by other factors – they fail to be determinative.  As populations grow larger, selective pressures grow concomitantly. Ultimately, they override individual or collective agency; locking-in the conditions they militate towards.

Here it is worth setting out a handful of historical examples which speak to this point, and to its relevance for the populations of replicators (institutional, biological and memetic) which we are embedded within:

• Modernisation, State Competition, and the Era of Empire: between 1492 and 1914, European states conquered 84 percent of the globe. Although there are many competing theories, one of the most prominent traces their success directly to technological, but further and more crucially, to institutional advantages: to states with unrivalled capacities of taxation, war and administration[2].  These advantages are in turn traced to the competitive dynamic of European powers; notably the large, continually warring population of European states. This dynamic provided the perfect environment for selective pressures; pressures which, once in operation, were beyond the power of individual replicators to halt. Either a state modernised of its own accord, or it was subsumed by its modernising rivals. Eventually, advances in navigation and shipbuilding permitted these modernising states to unleash themselves across the world, where they achieved success analogous to invasive species introduced to hitherto undisturbed biomes, with analogously devastating consequences.

• Corporate (Mis)behaviour: large corporations are responsible for many notable evils. They pollute the environment; they operate factories with appalling working conditions; they lie to the public; they attempt to subvert democratic decision making through lobbying. However, most people, including (one imagines) most corporate staff, strongly oppose such activities. How are these facts to be reconciled? Again, many explanations can be given. Again, selection pressures are powerfully explanatory. Each corporation is locked in perpetual and deadly struggle against its competitors for finite market share, investment, and profit. If a scrupulous company avoids immoral but expedient means, where its unscrupulous competitors do not, its share of the market will ceteris paribus decline. In time, unscrupulous companies will come to dominate the marketplace[3]. Crucially, the number of corporations in each market is a key determinant of the intensity of this competition. Large populations of firms are more likely to contain members with novel ideas, processes or technologies that could challenge your position (to use some earlier terminology, these firms would be selectively advantaged mutants); necessitating the pursuit of any available advantage. By contrast, members of small populations of large firms may have substantial leeway to act uncompetitively, secure in the knowledge of their dominance.

• Malthusianism: animal populations, absent predation or disease, follow a predictable pattern. When food is abundant, they multiply until it is insufficient. Starvation and disease follow, lowering (and impoverishing) the population. For most of human history, humanity joined the other animals in the Malthusian trap[4]. Advances in technology, reclamation or settlement of new land, or mass death (such as that caused by the Black Death[5]) could temporarily push apart the productive capacity of society and the consumptive necessity of its population. But soon enough population growth would eliminate such gains. Crucially, the power of individuals’ choices to reproduce less was neutered (pun intended) by the vast numbers this selection pressure operated over – such individuals would merely have been selecting themselves out of the population. Today, a century of falling birth rates and rising prosperity is widely perceived to have put paid to Malthusian theory, at least as it applies to humanity. I hope that this is true. However the length of this countervailing trend is short by any historical standard, and there are already signs of durable high growth sub-populations amidst the general birth rate decline (notably the Amish[6] and Mennonites in America and the Haredi (Ultra-Orthodox)[7] in Israel). The sheer number of humans, and human cultures, greatly increases the likelihood that one of these cultures already (or will “mutate” to) possess persistently high birth rates despite the modern socioeconomic context. Once extant, such a culture will be selected for.

III. Selection & Its Discontents

The above examples have pointed towards the power of selection to exercise an overwhelmingly determinative pressure on human society. But lock-in, as it is understood by EAs, contains an additional component: that the future of humanity must be locked into a highly negative end-state. 

At first, it may be unclear why strong selective pressures would tend to have this negative effect. Indeed, all the examples given in the previous section have had (strongly) positive aspects:

• Modernisation has produced states capable of providing vital services, redistributing wealth, and promoting economic growth; all to the benefit of their citizens.

• Corporate competition encourages innovation, lower prices, efficient use of resources and the anticipation and fulfilment of consumer needs. It is arguably responsible for much of the vast improvements in quality of living since the industrial revolution.

• Population pressure may continue to be made compatible with high living standards by technological improvement. If so, such pressure will result in the coming into existence of many millions (billions?) of additional fulfilled lives, compared to a world in which such pressure was absent.

The core reason why I believe we ought to be suspicious of selection effects, then, derives from a theoretical source: the specificity of human values. To wit:

1. Among all possible states of affairs, only an exceedingly small proportion are desirable to humans[8].
2. Therefore, we should expect any process choosing between these states, which is not heavily biased towards choosing desirable states, to produce undesirable states.
3. So, unless we have good reason to assume a selective process is heavily biased towards desirable states, we ought to assume that it will produce undesirable states.

If such a selective process operates on society, and is overwhelmingly powerful, it will result in lock-in. 

IV. Coordination Problems 

In addition to selection pressure, multilateral lock-in also operates by intensifying coordination problems. Put roughly, coordination problems arise in situations where agents face choices between personally and socially beneficial options. Each individual is made better off by picking the personally beneficial option. But if everyone picks the personally beneficial option, everyone ends up worse off than if they had all coordinated to pick the socially beneficial option. The best state of affairs is one where everyone coordinates to pick the socially beneficially option. 

A higher population of agents makes solving a coordination problem harder, for a variety of common-sense reasons:

1. More agents need to be convinced to coordinate, for coordination to be successful.
2. It is harder to monitor who is coordinating, and sanction agents who are not coordinating (defecting), in larger populations of agents.
3. As the population of agents grows larger, the share of the benefit generated by socially beneficial options, accruing to the individuals who choose them, grows smaller[9]. This loosens the connection between social benefit and individual benefit, reducing individuals’ incentives to pick socially beneficial options. This also applies, in reverse, to the share of the harms generated by individually beneficial actions.
4. Because of 1, 2 and 3, agents are less likely to perceive that coordination is possible. If coordination is not possible, their incentive to defect increases (if they coordinated, they would only be being taken advantage of).
5. Because of 1, 2, 3 and 4, the logistical costs of organising coordination increase. This increases the challenge of the secondary coordination problem: the problem of who is to pay for coordinating coordination.

The examples of selection pressure set out above, can also illustrate the enhanced difficulty of solving coordination problems in large populations. For instance, in respect of corporate competition, it is far easier for competitors to make and maintain anti-competitive agreements in monopolies or oligopolies. By contrast, the difficulty of reaching and maintaining agreement between hundreds of firms is effectively insurmountable. 

V. Evaluating the Challenge

Having established the potential for selective pressures and coordination problems to intensify with the size of a population, “locking-in” highly negative states of affairs in the process, it is worth stating the challenge this theory poses to our current framing around lock-In. Put boldly:

Multilateral Lock-In establishes that we ought to be as or more concerned that diffusion of power will result in lock-in, than we are concerned that concentration of power will result in lock-in[10].

The best way to substantiate this challenge is through concrete examples. First, from the threat of Climate Change[11]. Second, from the danger of Black Ball Technologies, such as AI and nuclear weapons[12]. Both such examples showcase the threat posed by distinctly multilateral modes of lock-in. Taking each in turn: 

• Climate Change threatens to render large swathes of the planet uninhabitable, devastate the productivity of those areas which remain, and induce consequent catastrophic political instability. Though the chances of climate change driving humanity to extinction appear close to nil, the consequences of climate change could – absent geoengineering - persist for tens or even hundreds of thousands of years, effectively permanently limiting human development[13]. Despite this, relatively little is being done to halt catastrophic climate change[14]. One of the key reasons is that Climate Change is a coordination problem: whilst the costs of reducing emissions are shouldered by the individual nation, the benefits are split between all nations. But still more crucially; climate change is a large-scale coordination problem. Because there are so many nations to coordinate, the difficulty of reaching agreement is intense, and the enticement to free-ride is omnipresent. In light of widespread free-riding, sanctioning defectors becomes impossible. Witnessing these difficulties, many give up hope that a solution is possible[15], in a viciously self-fulfilling prophecy.

• Black Ball Technologies, understood as technologies with the potential to cause existential catastrophe, have already been invented. During the height of the Cold War, the global nuclear warhead inventory peaked at over 70,000 weapons[16], the unleashing of even a fraction of which could have triggered a years-long nuclear winter. In 2008, a survey of academics estimated a 1% chance of human extinction from nuclear wars over the 21^st century[17]; and the long-term likelihood of such an event may well be significantly greater. As technological development continues, it is likely that further black ball technologies will be invented. AI is one such technology, whose rapid development has already been diagnosed as a source of existential risk. And just as the most learned medieval scholars could not anticipate thermonuclear warfare, there are almost certainly further black ball technologies lying beyond the bounds of present imagination, but within the bounds of future invention. The larger the number of states capable of developing these technologies, the more difficult the problem of coordinating disarmament/abstention[18]. Selective pressure is also important here: technological development is perhaps the key determinant of economic power, and economic power is in turn the key determinant of geopolitical power. In a multipolar world, states with appropriately cautious attitudes to black ball technology development may be outcompeted by those which uncritically embrace it[19]. As the relative influence of the latter group of nations increase, those entities with the greatest power to constrain black ball technology, will increasingly be those with the fewest qualms over its use.

The case for Multi-lateral lock-in is, therefore, buttressed not only by theoretical considerations, but by substantive practical examples. Contrastively, the reasons to be concerned about unilateral lock-in are much weaker. Practically, totalitarian regimes have never approached anything close to global lock-in – with the chief historic examples foundering upon their own defective economics, excessive ambition, and unjustified self-confidence. Theoretically, the nature of modern technology (such as nuclear weapons) drastically lowers the incentive for wars of acquisition, especially amongst larger powers[20].  

VI. Conclusion

What are we to make of the case for multi-lateral lock-in? 

First, it ought to influence our thought on the mechanisms and causes of lock-in. It is after all a truism that the better one understands a problem, the better one is placed to solve it. 

Second, it ought to shape our theories about how to avert it. In particular, supranational organisations, by reducing the number of effective powers, and improving coordination between them, could dramatically curtail the threat of multilateral lock-in. As such, a greater focus from effective altruists on the development and empowerment of such organisations could well be warranted[21]. 

The shadow of unilateral tyranny lies heavy over modern history. But we must not let it obscure the true threats to the future of humanity: dangers which arise far more from the division of agency, than they do from its unification. 

[1] https://www.bbc.com/future/article/20201014-totalitarian-world-in-chains-artificial-intelligence; https://blog.lareviewofbooks.org/interviews/want-potential-talking-toby-ord/.

[2] http://www.themontrealreview.com/2009/Why-Did-Europe-Conquer-the-World.php

[3] A second level of selection operates within the company itself, in the context of individual employees competing for top positions. To the extent that unscrupulous means are expedient to the business, unscrupulous employees will be able to display better results than their rivals (not to mention the utility of unscrupulous means in manipulating the recruitment process itself). The evidence of disproportionate levels of “dark triad”/psychopathic traits at the higher levels of corporations is disturbing evidence to the potential power of this pressure. 

[4] As indicated, for example, by this remarkable graph of per capita GDP in England; which shows almost no improvement until the 1700s (https://ourworldindata.org/grapher/gdp-per-capita-in-the-uk-since-1270); or this similar (rougher) chart of global GDP per capita (https://www.visualcapitalist.com/2000-years-economic-history-one-chart/).

[5] https://fredblog.stlouisfed.org/2018/12/the-black-death-in-the-malthusian-economy/#:~:text=This%20sudden%20and%20massive%20drop,plague%20that%20swept%20through%20Europe.&text=This%20condition%20leads%20to%20a,and%20famines%20control%20population%20size.

[6] The American Amish population has doubled roughly every twenty years over the past century. At the current rate of growth, 332 million Amish will be living in the US by 2303 (although, as with all three hundred year predictions, this should be taken with several truckloads worth of salt) (https://www.patheos.com/blogs/thefreethinker/2019/08/watch-out-america-the-amish-are-taking-over-slowly-but-surely/).  

[7] In 1948, Haredi comprised about 1% of Israel’s population. Today, they comprise roughly 12% of the Israeli population, and are projected to comprise almost a third of the Israeli population by 2059. (https://www.timesofisrael.com/half-of-israel-to-be-arab-ultra-orthodox-by-2059-projections/)

[8] Consider the very precise and complex series of steps that go into the operation of a large organisation or the manufacture of a complex technology, and the almost infinitely greater number of ways in which each process in that series can malfunction, causing the failure of the whole. Humans, and human societies, are complex systems; and as complex systems have an inherently narrow band of proper functioning, and a vastly greater scope for malfunction and breakdown. 

[9] For instance, if x picking a socially beneficial option generates benefits y (to a population numbering w agents), the amount of the benefit accruing to x from picking such an option will be y/w. The larger the population w grows, the smaller will be the proportion of the social benefits of x’s actions, that will accrue to x himself. 

[10] This can be broken down further into a claim that focuses on diffusion/concentration of power between states, or one that focuses on diffusion/concentration of power within states (between governments and “the people”, corporations, etc.). Given that the orthodox framing of lock-in casts it as a supra-national threat, I focus on the inter-state angle during the remainder of this article (though the latter is also worthy of examination). 

[11] Though it is unclear whether the effects of Climate Change will be so catastrophic and irreversible as to constitute a genuine lock-in situation. 

[12] For more on this concept, see “The Vulnerable World Hypothesis” https://nickbostrom.com/papers/vulnerable.pdf

[13] https://80000hours.org/problem-profiles/climate-change/#is-the-possibility-of-extreme-climate-change-an-existential-risk

[14] https://www.nationalgeographic.com/science/2019/11/nations-miss-paris-targets-climate-driven-weather-events-cost-billions/

[15] https://www.newyorker.com/culture/cultural-comment/what-if-we-stopped-pretending; https://www.theguardian.com/environment/2020/sep/21/meet-the-doomers-some-young-us-voters-have-given-up-hope-on-climate

[16] https://fas.org/issues/nuclear-weapons/status-world-nuclear-forces/

[17] https://www.fhi.ox.ac.uk/reports/2008-1.pdf

[18] The saving graces of the Cold War nuclear standoff were its bipolar nature, and the immense cost and scientific knowledge required for nuclear development. The latter kept the number of nuclear nations at a manageable level. The former simplified the calculus for the two great powers: if either struck in any way at the other, they risked mutual annihilation. Conversely, if both agreed to reduce their stockpiles, they would not be overawed by the arsenals of third powers. 

[19] The first nation to invent artificial general intelligence, for instance, could reap colossal economic dividends. 

[20] https://ourworldindata.org/war-and-peace

[21] Though this problem may prove intractable for other reasons; not least the strength of nationalist ideologies, local politicians’ desire for power, coordination problems, and so on.