___________________________________________________

tldr;

• Expected value theory misrepresents ruin games and obscures the dynamics of repetitions in a multiplicative environment.
• The ergodicity framework provides a better perspective on such problems as it takes these dynamics into account.
• Incorporating the ergodicity framework into decision-making can help prevent the EA movement from inadvertently increasing existential risks by rejecting high expected value but multiplicatively risky interventions that could lead to catastrophic outcomes.

___________________________________________________

Effective Altruism (EA) has embraced longtermism as one of its guiding principles. In What we owe the future, MacAskill lays out the foundational principles of longtermism, urging us to expand our ethical considerations to include the well-being and prospects of future generations.

Thinking in Bets

In order to consider the changes one could make in the world, MacAskill argues one should be “Thinking in Bets”. To do so, expected value (EV) theory is employed on the account that it is the most widely accepted method. In the book, he describes the phenomenon with an example of his poker-playing friends:

“Liv and Igor are at a pub, and Liv bets Igor that he can’t flip and catch six coasters at once with one hand. If he succeeds, she’ll give him £3; if he fails, he has to give her £1. Suppose Igor thinks there’s a fifty-fifty chance that he’ll succeed. If so, then it’s worth it for him to take the bet: the upside is a 50 percent chance of £3, worth £1.50; the downside is a 50 percent chance of losing £1, worth negative £0.50. Igor makes an expected £1 by taking the bet—£1.50 minus £0.50. If his beliefs about his own chances of success are accurate, then if he were to take this bet over and over again, on average he’d make £1 each time.”

More theoretically, he breaks expected value theory down into three components:

1. Thinking in probabilities
2. Assigning values to outcomes (What economists call Utility Theory)
3. Taking a decision based on the expected value

This logic served EA well during the early neartermist days of the movement, where it was used to answer questions like: “Should the marginal dollar be used to buy bednets against malaria or deworming pills to improve school attendance?”.

The Train to Crazy Town

Yet problems arise when such reasoning is followed into more extreme territory. For example, based on its consequentialist nature, EA-logic prescribes pulling the handle in the Trolley Problem^[1]. However, many Effective Altruists (EAs) hesitate to follow this reasoning all the way to its logical conclusion. Consider for instance whether you are willing to take the following gamble: you're offered to press a button with a 51% chance of doubling the world's happiness but a 49% chance of ending it.

This problem, also known as Thomas Hurka's St Petersburg Paradox, highlights the following dilemma: Maximizing expected utility suggests you should press it, as it promises a net positive outcome. However, the issue arises when pressing the button multiple times. Despite each press theoretically maximizing utility, pressing the button over and over again will inevitably lead to destruction. Which highlights the conflict between utility maximization and the catastrophic risk of repeated gambles.^[2] In simpler terms, the impact of repeated bets is concealed behind the EV.  

In EA-circles, following the theory to its logical extremes has become known as catching “The train to crazy town”[^[3],^[4]]. The core issue with this approach is that, while most people want to get off the train before crazy town, the consequentialist expected value framework does not allow one to get off. Or as Peter McLaughlin puts it: “The train might be an express service: once the doors close behind you, you can’t get off until the end of the line”^[5]. This critique has been influential as it directly attacks the ethical foundation of the Effective Altruism movement. One could argue; why believe in a movement that is not even willing to follow its own principles? Since this critique has received considerable - and better -  treatment both from inside[3,^[6]] and outside[^[7],^[8],^[9]] the EA community, I will only link to those discussions here.

The train to crazy town: a ride without stops

The fact that EAs differ in their willingness to follow expected value theory to its extremes has put the EA community at a crossroads^[10]. On the one hand, the fanaticism camp preaches consistency within the consequentialist-expected value theory framework. They are, therefore, willing to bite all bullets on the way to crazy town. Philosophical leaders like Toby Ord and William MacAskill are part of this camp. When pressed on this issue by Tyler Cowen, MacAskill urges us to try harder with our moral reasoning:

Cowen: “Your response sounds very ad hoc to me. Why not just say, in matters of the very large, utilitarian kinds of moral reasoning just don’t apply. They’re always embedded in some degree of partiality.”

MacAskill: “I think we should be more ambitious than that with our moral reasoning.” ^[11] 

On the other hand, the pluralist view focuses on integrating multiple moral values and considerations into its decision-making and prioritization framework. In this manner, the pluralist view urges us to not solely rely on utilitarian calculations of maximizing happiness or reducing suffering. This allows one to not follow utilitarianism to its logical extremes but, by doing so, undermines the moral philosophy on which it is built. To stick with the analogy, this allows one to get off the train to crazy town at the expense of derailing the EA train.

But what if there is a way to - at least partially - reconcile these two perspectives? A way to follow consequentialist reasoning without arriving in crazy town? A way to get off the train without derailing it?

The answer lies in a subtle assumption Daniel Bernoulli^[12] made when he was the first to write down how to Think in Bets in 1738. To understand how this works, we must take a look at Ole Peter’s work. Ole Peters^[13] is a physicist at the London Mathematical Laboratory who started the field of Ergodicity Economics.

Ergodicity view: Additive versus multiplicative games

Ole Peters points out that many economic models assume our world is ergodic—that is, they assume looking at an average outcome over time or across many people can tell us what to expect from an individual. If a process is ergodic, the average over time will be the same as the ensemble average, an average over multiple systems. A simple example in which this is the case is the following: When one person plays head-or-tails 1000 times, and when 1000 people play once, both situations are expected to hit heads 500 times. Just like the expected value theory would suggest. But real life, especially long-term population ethics, often doesn't work that way.

For example, consider playing a game where you flip a coin, and if it's heads, you increase your wealth by 50%, but if it's tails, you lose 40%. Mathematically, the average outcome looks positive. But, if you play this game repeatedly, because of the multiplicative nature of wealth (losing 40% can't just be "averaged out" by gaining 50% later), you're likely to end up with less money over time. This game is non-ergodic - the long-term outcome for an individual doesn't match the seemingly positive average outcome.

The attentive reader will have realised the similarities between the aforementioned St Petersburg paradox and this gambling problem. In both cases, one following the expected value theory would take the bet. Yet, playing repeatedly results in losing money over time^[14] or the destruction of the world^[15]. It is like someone playing Russian roulette on repeat. There are two factors at play here that create this dynamic. First, due to its multiplicative nature, all the wealth of a person is at stake at once instead of only a part of it, as in the example of MacAskill’s poker friends. The second dynamic is repetitiveness. It is correct that, a single flip of the coin does yield a positive expected value. Yet, a consequentialist caring about one’s wealth at the end of the 1000 periods would decide not to play.

The dynamics at play in Ergodicity Economics seem counterintuitive at first, especially for economists and EAs who have become hardwired to think in terms of Utility Theory and Expected Value Theory, respectively. To better understand the concept of non-ergodic processes, I link here an explanation video by Ole Peters^[16], a blog article in which one can simulate the gamble above^[17], and the seminal paper by Peters in Nature^[18].

It is important to note that Thomas Aitken’s^[19] EA-Forum post has already demonstrated how ergodicity economics can serve as an analytical lens to critique EA. My contribution is in integrating this with consequentialist logic, which facilitates a way of getting off the proverbial 'train to crazy town'.

Ruin Problems

Next, I will show how, in the presence of the possibility of ruin, Expected Utility Theory breaks down in a longtermist framework^[20]. To do so, we first take a philosophical perspective before turning to a more economic approach.

EAs who are willing to bite the bullet on Thomas Hurka’s St. Petersburg Paradox employ the wrong mental model. They assume to be taking a gamble on a world state of doubling the world utility versus destroying one world utility. However, due to the potential loss of all of future humanity, the downside in this gamble should be characterised as negative infinity.

Since we are manoeuvring in the philosophical world, let’s assume a zero social discount rate, as Parfit and Cowen argued in 1992[^[21]]. In such a world, all potential future people should be fully incorporated into our future utility function (1).

Let U(t) represent the aggregate utility of all individuals alive at time (t). Assume that the total world utility, W, is the sum of all future utilities discounted by the probability of human survival up to that point. If p(t) is the probability of human survival until time (t)

Here,  p(t) decreases over time due to the presence of extinction risk in all time periods. For simplicity, let’s assume a constant risk of extinction per unit of time, labda, so p(t)= e^(-labda *t). Let’s now assume that an extinction event occurs at a time T, this will truncate the integral and lead to (2).

Lastly, let’s now look at the potential future utility wasted (3).  

If the utility function U(t) does not diminish to zero—people keep living happy lives on earth—at a rate faster than the rate at which the survival probability function decreases, then the integral that defines the change in world utility can diverge, implying an infinite loss. In short, expected values don’t fare well in the presence of infinities.  

Thus, assuming a zero discount rate leads to an infinite loss in utility in the aforementioned paradox. Cowen and Parfit agree that utilitarian logic breaks down when confronted with infinites^[22]. In theory, this justifies a full allocation of EA resources to existential risk (X-risk) reduction.

Of course there are many practical objections to the argument made above. The goal here is to expose the flaw in EV-theory, not to come up with an accurate EV estimate of X-risk. Additionally, there has been a lot of impressive work trying to quantify the expected value of X-risk under more practical and economic assumptions[^[23],^[24]].

Ergodicity perspective on ruin problems

Having shown that Expected Utility Theory breaks down when confronted with infinity, I will argue that the ergodicity framework provides the correct intuitive perspective to this problem. Let’s modify the St. Petersburg Paradox to the one that Cowen puts in front of MacAskill: “you're offered to press a button with a 90% chance of doubling the world's happiness but a 10% chance of ending it. But, we play 200 times”. Below one can see the amount of worlds still existing after the amount of rounds played, simulated for 500 worlds.

This figure shows the number of worlds that still exist in a total of 500 simulations of the Tyler Cowen St. Petersburg Paradox. The last world dies after 59 rounds in this simulation.  

MacAskill struggles with the paradox and, when put on the spot during the interview, jumps to moral pluralism as a solution. Yet, it is possible to answer these questions from within a consequentialist framework when taking Ergodicity seriously.

It is true that the total happiness in round 59, the last round in which a world exists, is times as high as in the initial state, which in a consequentialist framework, would outweigh the loss of 499 worlds. However, every consequentialist asked to evaluate whether to take the gamble for 200 consecutive rounds would not take the bet. These dynamics are caused by certain absorption barriers from which one cannot recover^[25]. They are characterized by the system's eventual transition into one of these absorbing states, regardless of the initial state, thus capturing long-term behaviours and outcomes. Ruin games are therefore a special case of ergodicity. Not only do ruin games have a different time average than the ensemble average, but their end state also creates a point of no return.

My argument to this point can be summarised as; Expected value theory misrepresents ruin games and obscures the dynamics of repetitions in a multiplicative environment. Or, as Nassim Taleb’s^[26] states it: “[S]equence matters and ruin problems don’t allow for cost-benefit analysis.”.  

Non-ergodicity for EA: a framework

So, instead of a priori assuming that one can follow the expected value theory. One should first ask the question: What game are we playing? Or, which dynamics are in play?

MacAskill can be forgiven for not being aware of the implicit ergodicity assumption in expected value theory when writing the book. The first paper by Peters and Nobel laureate Gell-Man was only written in 2016^[27], and Ergodicity Economics still forms a niche within the discourse. Yet, ergodicity should be intuitively appealing to EA as it is a mathematical concept.  

In a sense, it is not consequentialist reasoning that breaks down in this particular extreme. What breaks down is the implicit assumption of ergodicity within expected value theory. In other words, the core issue is its failure to consider time, which is exposed in the limit. Therefore, it is not consequentialism that breaks down in the extreme, it is the Utility Framework.

From a purely philosophical perspective, the singular instance remains a challenge for utilitarianism. Yet, the ergodicity perspective provides a clear stop on our way to crazy town in the case of repeated multiplicative dynamics. The figure below provides a flowchart of when expected value theory suffices and when an ergodicity framework is necessary.

Figure: A new consequentialist framework for how to ‘Think in Bets’ depending on the dynamics at play.

Some implications

Taking ergodicity seriously can strengthen the EA longtermist movement both from a theoretical and a practical perspective. First, it allows for a resolution of the St Petersburg Paradox and similar thought experiments. It does so by incorporating the importance of compounding sequence effects and ruin states in the consequentialist framework. For EAs, this allows for a stop on the train to crazy town within the consequentialist framework, at least in the case of St. Petersburg-like paradoxes.

As a consequence, in a repeating multiplicative environment, the ergodicity framework allows one to consider sequence effects. By doing so, it allows one to reject risky ‘positive expected utility bets’ from within the consequentialist framework. This holds as long as these bets are expected to be multiplicative. To illustrate this point, let’s look at the example Hayley Clatterbuck gave during the EAG Bay Area conference last month^[28].

The highest EV intervention given here is ‘virus hunting’, which involves having labs research and create viruses that could produce the next extinction-level pandemic. The EV of this intervention could be high if we find these viruses ahead of time and are able to develop vaccines upfront. Yet, when the virus breaks out of the lab—a backfire event— the outcomes would be devastating. Let’s now assume that EA grantmakers double down on this X-risk intervention over the upcoming years, because it has the highest EV. Since this is a nascent field, the effect that the marginal lab will have is proportional to the current level of biosecurity risk. In other words, since the field has not hit its limits to growth, multiplicative dynamics are in play. In this scenario, the average over time will be different than its ensemble average, so non-ergodicity is in play. Additionally, depending on the values given to the different probabilities^[29], X-risk might increase. This can be true even for high EV scores. Thus, a grantmaker who repeatedly makes decisions on interventions to reduce X-risk based on EV scores could inadvertently increase X-risk. All while he is trying to save the world.  

Of course, the prior argument requires many assumptions. Given this, my point is not to argue that all the assumptions above are true. It is to show that the ergodicity framework elicits the assumptions obscured in the EV framework. Additionally, it allows one to reject discounting and risk aversion, core longtermist and consequentialist positions, and still decide not to take ‘risky positive expected utility bets’.

Let’s also be clear about what I do not attempt with this post. This post does not provide an answer to the repugnant conclusion or other thought experiments on the train to crazy town. Thus, it is not an all-encompassing solution to the controversial consequences of consequentialism. It is only a first attempt at being more ambitious with our moral reasoning, as MacAskill called for.

1. ^{^}

    KILLING, LETTING DIE, AND THE TROLLEY PROBLEM The trolley problem is a thought experiment where you must decide between taking action to divert a runaway trolley onto a track where it will kill one person, or doing nothing and allowing it to continue on its current path, killing five.

2. ^{^}

    Value and Population Size* - Thomas Hurka Thomas Hurka (1983) Value and Population size.

3. ^{^}

    The train to Crazy Town - by Sam Atis - Samstack

^{^}

 Ajeya Cotra on worldview diversification and how big the future could be - 80,000 Hours 

^{^}

 Getting on a different train: can Effective Altruism avoid collapsing into absurdity? — EA Forum 

^{^}

 William MacAskill on Effective Altruism, Moral Progress, and Cultural Innovation (Ep. 156) | Conversations with Tyler 

^{^}

 Persuading people to worry about the long term is harder than you think FT 

^{^}

 Beneficentrism - by Richard Y Chappell - Good Thoughts 

^{^}

 All Trains Lead to Crazy Town: Why I am Not an Effective Altruist (or a Philosopher) - Jason Kerwin 

^{^}

 Integrating utilitarian calculations into moral theory leads to their dominance, which results in the extreme outcomes described above. Although some may accept controversial consequences like the repugnant conclusion or the experience machine, many will encounter an unacceptable scenario that challenges their adherence to utilitarianism. Rejecting such scenarios requires abandoning the principle that the magnitude of outcomes always matters, highlighting a fundamental inconsistency within utilitarian logic.

^{^}

 William MacAskill on Effective Altruism, Moral Progress, and Cultural Innovation (Ep. 156) | Conversations with Tyler 

^{^}

 Daniel Bernoulli Wikipedia 

^{^}

 Ole Peters  

^{^}

 Because of the multiplicative dynamics in which on average ones wealth is reduced, in the limit one’s wealth would converge to 0.  

^{^}

 The dynamics at play in the Thomas Hurda’s St. petersburg Paradox can be characterised as a ruin game.

^{^}

Random multiplicative dynamics -- Ole Peters 

^{^}

  The infamous coin toss – Ergodicity economics 

^{^}

 The ergodicity problem in economics | Nature Physics 

^{^}

 The Role of "Economism" in the Belief-Formation Systems of Effective Altruism — EA Forum 

^{^}

 This does not apply to neartermists issues because in this case the cost of ruin can be accounted for. Ruin, one person dying, can be accounted for through the cost per quality adjusted life year (QALY). Yet there is no equivalent metric for longtermism, because humanity is still singular.  

^{^}

 Against the Social Discount Rate 

^{^}

 Sam Bankman-Fried on Arbitrage and Altruism (Ep. 145) | Conversations with Tyler 

^{^}

 The CURVE Sequence - EA Forum;

^{^}

 EXISTENTIAL RISKS FAQ   

^{^}

 In probability theory, such states are called absorbing Markov Chains. Interesting article on how Markov Chain modeling can improve X-risk calculations. Existential Risk Modelling with Continuous-Time Markov Chains — EA Forum 

^{^}

 The Logic of Risk Taking. | by Nassim Nicholas Taleb | INCERTO | Medium   

^{^}

 https://pubs.aip.org/aip/cha/article/26/2/023103/134886 

^{^}

 EA Global Bay Area: 2024 |  Cause Prioritization & Global Catastrophic Risk | Hayley Clatterbuck 

^{^}

 I deliberately do not assign probabilities in this case, as it would distract from the argument I am trying to make. In an upcoming blog, I will dive into the numbers and show the effects of different parametrisations on X-risk models.

Show all footnotes