1 The core problem

Crosspost of my blog. 

The philosophy of risk is weird! Common sense holds that you should not sacrifice all the value in the world for an arbitrarily low chance of bringing about infinite value. But this commonsense judgment is shockingly hard to hold on to. Provably, one must accept one of the following:

1. Intransitivity: if A is better than B, and B is better than C, C is sometimes better than A.
2. Timidity: for some guaranteed payout X, a 99.9999999999999999999% chance of some payout vastly better than X is worse than a guarantee of X.
3. Fanaticism: for any guaranteed payout X, and any probability P, there exists some vastly better payout X+ such that a P chance of X+ is better than a guarantee of X.

This can be illustrated through the following parable:

Genie: Hello—in a distant galaxy there are an arbitrarily large number of people suffering. If you want, I will make 100 fewer of them suffer.

Dave: Sounds good! But seeing as I’m a character in a parable illustrating the reason one must accept fanaticism, timidity, or intransitivity, I assume there’s a catch?

Genie: Indeed there is. If you want instead, I can offer you a 99.999% chance of making 100,000 people be spared from suffering!

Dave: Sounds like quite a good deal. I’ll take that one!

Genie: Great! And if you want instead of that, I’ll offer you a 99.99% chance of 100,000,000 people being spared from great suffering.

Dave: I’ll take that deal too!

5 minutes later.

Genie: Okay, instead of the current deal of a 60.0000001% chance of sparing 100^100 people from suffering, I’ll offer you a 59.9999999999% chance of sparing 100^100^100 people from suffering.

Dave: No, I won’t take the deal!

Genie: Okay, but first for the audience—do you see that if Dave keeps taking the deals he’ll eventually gamble away a guarantee of saving 100 people from extreme agony for a vanishingly low chance of sparing some arbitrarily large number of people from extreme agony. Now back to you Dave, surely saving many orders of magnitude more people from suffering is worth very slightly reducing the odds of payout.

Dave: Ugh, I guess you’re right. Fine, I’ll take the deal.

Five hours later…

Genie: Okay, so the final offer is taking a 10^-189 chance of sparing Graham’s number people from extreme suffering.

Thus, so long as you’re always willing to slightly lower the odds of payout in exchange for vastly increasing the payout, you’ll end up fanatical. You’ll end up thinking that an arbitrarily low chance of a sufficiently great payout is worth more than a guarantee of a high payout.

Now, even if you don’t accept transitivity, so long as you take each individual deal (which it seems like you should), then you’ll eventually be on the hook for the fanatical deal!

Here’s one other way of seeing this. Suppose that the following deal is offered: you spare 100 people from suffering! But before you take the deal, instead you can roll a billion sided die, and provided you get anything other than 483,333 you’ll spare 1,000 people from suffering. Sounds like quite a deal! Naturally you take the deal. The die is rolled, and you get 518,985. Nice!

But then, before you spare 1,000 people from suffering, the genie offers to roll another billion-sided die and spare 10,000 people from suffering unless you get 483,333. Once again, seems like a good deal. But provided you take each of these deals, you’ll end up a fanatic! You’ll end up giving up a good thing for an arbitrarily low chance of a vastly better thing.

I think this is a very powerful argument for fanaticism. Timidity is insane (more on that later). Intransitivity doesn’t help—it still commits you to passing up the deal! It’s also insane. To deny fanaticism, you have to think that you should pass up offers to make some action arbitrarily better if they reduce the odds of payout by some vanishingly low probability. Nuts!

This gets even diceier on the negative end (pun intended). Suppose 100 people are to be tortured. Surely a 99.9999999% chance of 100,000 people being tortured is worse. Surely a 99.9999% chance of 100,000,000 people being tortured is worse than that. And so on. This matters because presumably what’s true of positive value is also true of negative value. If fanaticism is true with negative value—an arbitrarily small chance of an arbitrarily awful state of affairs is worse than a guaranteed very bad state of affairs—then it’s likely true with positive value too!

(And the above series of deals where you keep rolling dice can be done on the negative end too!)

I find this to be one of the most fascinating areas of philosophy for two reasons. First, everyone arguing about it is insanely sharp. On both sides, people cook up brilliant and ingenious arguments. This might be the area where I’m most consistently blown away by everyone’s cleverness and philosophical aptitude.

Second, I think fanaticism has for it perhaps the best arguments for a wildly counterintuitive view in all of philosophy. Fanaticism sounds totally crazy! But I think the arguments for it are so compelling that no matter how strongly unintuitive you find fanaticism, upon reflection, you should be a fanatic.

This conclusion becomes especially overwhelming when you take into account the great number of very powerful arguments for fanaticism. I’ll talk about these in a bit. But first, let me explain the enormous practical importance of this question.

If you accept fanaticism—that any guarantee of a good outcome is less good than a tiny chance of some sufficiently better outcome—then your charitable dollars and career should be spent trying to bring about very enormous value. You should seriously think about working on AI safety or other existential threat reduction. You should focus less on trying to make a difference to the world with high probability and more on trying to increase the probability of very enormously good outcomes.

(I actually think this follows even if you’re not a fanatic, for reasons I’ve given here in the section beginning with “You might also think that we should discount very low risks.” Basically, I think everyone no matter what their view is should be a Longtermist—trying to maximize the probability that the far future goes really well.)

2 Partial dominance argument for fanaticism

Suppose we accept:

1. Transitivity: if A>B and B>C, then A>C.
2. Partial Dominance: If there are two actions A and B which both bring about some gamble S, and A brings about, in addition to S, a higher chance of a better or equally good outcome relative to B or an equal chance of a better outcome relative to B, then A>B.

For example, Partial Dominance would hold the following: suppose that there are two actions A and B. A and B both result in a 90% chance of you getting 1,000 dollars. Then, A results in an additional 5% chance of you getting 500 dollars, while B results in a 1% chance of you getting 1 dollar. Partial Dominance holds that A>B—they both bring about an equal probability of an equal outcome, and then A has an extra probability of an extra good outcome.

Together, these imply fanaticism!

To see this, suppose there’s one action that creates 1000 happy people. Seems good! But now compare it to another action with a 99.9% chance of creating 2000 people. Crucially, Partial Dominance implies that the second action is better than the first. Both actions create 1000 people in the 99.9% slice of probability space shared between them. But then in addition, the second action has a 99.9% chance of creating an extra 1000 happy people, while the first has an additional .1% chance of creating 1000 happy people. Partial Dominance implies the second is better than the first. But this logic generalizes—if you keep taking the gamble that slightly lowers probability of the higher potential payout, eventually you become a full on fanatic.

One could reject that the better than relation is transitive. But this is quite a cost. First of all, the better than relation being transitive is very intuitive. It’s hard to believe that A might be better than B, B better than C, but A worse than C.

Second, views that deny transitivity imply strange violations of dominance. To see this, suppose one prefers A to B, B to C, and C to A. Suppose that they’re comparing the collection of A and B to B and C. It would seem that both collections are better than the other! B is a downgrade relative to A, C is a downgrade relative to B—things get worse after being downgraded twice. However, A and B can be downgraded to become B and C by downgrading A to C. This, it seems impossible that both are better than the other.

Third, denying transitivity results in money pumps. If one prefers A to B, B to C, and C to A, they’d pay one cent to trade C for B, another to trade B for A, and another to trade A for C. This money-losing cycle could continue forever.

Fourth, deniers of transitivity have to think that sometimes improving things multiple times makes them worse. Suppose A>B, B>C, but C>A. If you take C, improve it so it becomes B, and then improve it again so that it becomes A, it will have gotten worse. But this is nuts! Making something worse multiple times doesn’t improve it!

3 An independence argument for fanaticism

Suppose we accept:

1. 99% Independence: If Gamble A is better than gamble B, a 99% chance of gamble A is better than a 99% chance of gamble B.
2. Transitivity: if A>B and B>C, then A>C.
3. Increase Payouts: For any guaranteed bettering of a state of affairs, there is a better gamble that involves a 99% chance of a vastly better state of affairs.

You can see where this is going. A 100% chance of 1000 happy people being created is less good than a 99% chance of, say, 10,000 happy people being created. Then, a 99% chance of 10,000 happy people being created is less good than a 99% chance of a 99% chance of 100,000 happy people being created, and so on. Overall, you get the result that a guarantee of massive payouts is always beaten by a very tiny chance of sufficiently vast payouts.

4 Repeated gambles throughout universe makes fanaticism right

One of the arguments for maximizing expected utility is that in the long run, it gives you a near-guarantee of better outcomes by the law of large numbers. Suppose that I’m deciding between a gamble that gives me a 1% chance of creating 1000 happy people and a 100% chance of creating one. If the gamble is offered to me repeatedly, and I take the first one, I’m near guaranteed to have better outcomes.

Now, the obvious problem with this as an argument for fanaticism is that often various gambles will not, in fact, be repeated. If you’re offered the gamble only once, then the argument is impotent. And if you’re offered the gamble repeatedly, then even non-fanatical views will take it. When the decision is iterated, taking the gamble repeatedly isn’t fanatical—it guarantees a very high probability of a very good outcome.

But here, I think, there’s still a way of getting at the spirit of the idea. It relies on the notion that if there are a bunch of copies of each other throughout the universe, each taking causally isolated actions, it would be better for them to take the better action rather than the worse action. For example, if it’s good for me to press a button that creates 1,000 happy people, then it would be good if I had a bunch of clones who all did that.

Why accept this principle? Well, by definition right actions are worth taking. So provided that the worthwhileness of an action doesn’t depend on what’s happening in spatiotemporally isolated regions of spacetime, what’s good for you to do is good for the clones to do. The existence of clones of you taking similar actions is causally isolated from you, so it shouldn’t change what you should do. Likewise, for the same reasons, it shouldn’t change what the clones should do.

But this implies fanaticism. Clearly if there were a bunch of clones all offered structurally identical gambles, it would be better if they took the one with higher expected value. That guarantees vastly better outcomes in the long run.¹ So then provided how you should act isn’t affected by the existence of clones, you should just take the action with higher expected value.

(I think this is a pretty simple way of showing that non-fanatical views inevitably require caring about random unrelated stuff happening on the other side of the universe—though as we’ll see, there are other even sharper arguments for this conclusion. There’s no way out of the conclusion that non-fanatics have to think that what causally-isolated Martians do vastly changes the value of your actions).

5 Irrelevance of independent outcomes

One of the most compelling arguments for fanaticism is that non-fanatical arguments imply that when deciding which action to take, your action should depend on what’s happening in causally isolated segments of the universe. To see this, imagine you’re going to roll a 100-sided die, and you are deciding between two outcomes.

Gamble 1: 1,000 happy people are created if you get 1, 0 if you get anything else.

Gamble 2: 1 happy person is created no matter what you get.

(Let’s just assume for simplicity that happy people are good—though if you reject that, the argument can be replaced by whatever it is that you think matters).

But now let us imagine that elsewhere, far across the universe, the Martians are also deciding how many people to create based on the die roll. They will create zero happy people if the die comes up one, one happy person if the die comes up two, two if it comes up three, and so on.

Clearly, what the Martians are doing shouldn’t affect your actions. Whatever the Martians roll is totally and completely unaffected by whichever gamble you take. You should only, when taking actions, take into account your actions’ effects. But crucially, if the Martians are gambling in the way I described, then you should clearly take gamble 1.

After all, taking into account what Martians are doing, the gambles are as follows (concerning the number of happy people being created):

Gamble 1: 1% chance of one person (if the die comes up 2), 1% chance of two people (if the die comes up 3), 1% chance of 4 people (if the die comes up 4%)…1% chance of 1000 people (if the die comes up 1).

Gamble 2: 1% chance of one person (if the die comes up 1), 1% chance of two people (if the die comes up 2), 1% chance of 3 people (if the die comes up 3)…1% chance of 100 people (if the die comes up 100).

But now clearly gamble 1 beats gamble 2. For every chance of every outcome offered by gamble 1, there’s an equal chance of an equal or better outcome offered by gamble 2. Gamble 2, to use the technical lingo, dominates gamble 1. For every outcome offered with some probability by gamble 1, gamble 2 offers an equal chance of an equal or better outcome.

Thus, so long as you think that Dominance is true, and what you should do doesn’t depend on what causally and counterfactually unrelated things are happening in distant galaxies, fanaticism follows. Note: the point isn’t specific to other galaxies. It can also be made about times in the distant past. Whether you should take the gamble shouldn’t depend on whether in Ancient Egypt people took other gambles—assuming that your actions don’t affect what happened in ancient Egypt. But non-fanatical views that affirm stochastic dominance must reject this.

The scenario I’ve given so far has involved the events taking place in ancient Egypt not being affected by what you do, but still having their payouts correlated with the payouts you get. There is a correlation between the outcomes of the Egyptian gamble and your gamble. However, as Wilkinson shows, there’s a way, using complex mathematics, to make the same thing work without assuming the gambles are correlated. That means that even though there would be absolutely no correlation between what happened in ancient Egypt and what happens today, nonetheless non-fanatical views that affirm dominance would imply that what you should do sometimes depends on what was happening in ancient Egypt.

This would mean that if deciding what to do, sometimes your action should depend on whether in ancient Egypt people took a risky gamble that has literally no effect on or correlation with what you do. That seems pretty insane to me!

When I first heard this argument, I thought that it was a knockdown objection. I still do. But opponents of fanaticism have a surprisingly compelling response. Unfortunately, fanaticism is incompatible with the combination of dominance and the idea that what happens in causally isolated settings in far-away galaxies is irrelevant to your decision-making. This quite ingenious argument comes from Russell.

I’ll have a quasi appendix that I’ll put in bold—feel free to skip this if you don’t care about the details. But in short, there’s a gamble called the Saint Petersburg gamble. Basically, you flip a coin until it comes up tails, and get 2^N instances of good things, where N is the number of coin flips. Counterintuitively, the expected value of the St. Petersburg game is infinite. There’s a 1/2 chance of 2 good things, a 1/4 chance of 4 good things, a 1/8 chance of 8 good things, and so on. Overall, this averages out to infinite good things in expectation. Fanatics, therefore, would risk everything to play the St. Petersburg game.

Now, it turns out—and this is really weird—that sometimes having two correlated versions of things that are strictly worse than the St. Petersburg game guarantees you better outcomes than the St. Petersburg game. If you’re offered to play the St. Petersburg game twice, vs to play two versions of a game that gives you a 1/2 chance of 1 good thing, 1/4 chance of 3 good things, 1/8 chance of 7 good things, sometimes you should be indifferent. Two versions of gambles that are each strictly worse than the St. Petersburg game can sometimes guarantee you a better outcome than the St. Petersburg game.

This is incompatible with fanaticism + dominance + irrelevance of causally isolated far-away outcomes. Let V be the full Saint Petersburg game, X be a version of the strictly worse St. Petersburg game, and Y be another version of the strictly worse St. Petersburg game. Sometimes 2V<X+Y. This can’t be if V>X, V>Y, and combining two instances of better things is better than combining two instances of worse things.

The technical Appendix: Imagine that a coin is flipped until it comes up heads. There are the following gambles. X has a 1/2 chance of creating 1 happy person, a 1/4 chance of creating 3, a 1/8 chance of creating 7, and so on. Y has a 1/2 chance of creating 1 happy person, a 1/4 chance of creating 3, a 1/8 chance of creating 7. V has a 1/2 chance of creating 2 utils, a 1/4 chance of creating 4, a 1/8 chance of creating 8, and so on.

By dominance V>X and V>Y.

But now let’s imagine comparing two different versions of gamble X to two different versions of gamble Y, where they’re correlated in a tricky way.

A fair coin is flipped until it comes up tails. Both versions of V pay out 2^N happy lives. V+V gives a 1/2 chance of 4 happy lives, a 1/4 chance of 8 happy lives, and so on.

In addition, a bonus coin is flipped. This bonus coin does not affect the payouts of V.

For X, if the bonus coin comes up heads, you get a guarantee of one happy life. If it comes up tails, then it creates twice the outcome of V, minus one happy life. So it has a 1/2 chance of creating one happy person (if the bonus coin comes up heads), a 1/4 chance of creating three happy people (if the bonus coin comes up tails and the first regular coin comes up tails), a 1/8 chance of creating seven happy people (if the bonus coin comes up tails, the first normal coin comes up heads, and the second normal coin comes up tails), etc. Thus, X is dominated by V.

For Y, if the bonus coin comes up tails, you get a guarantee of one happy life. Otherwise, it creates twice the outcome of V, minus one happy life. Y also gets dominated by V. It has a 1/2 chance of creating one happy life (if the bonus coin comes up tails), a 1/4 chance of creating three happy lives (if the bonus coin comes up tails and the normal coin comes up tails), a 1/8 chance of creating 7 happy lives (if the bonus coin comes up tails and the normal coin comes up heads then tails), and so on. Y is thus dominated by V.

But crucially, X+Y guarantee the same outcomes—no matter what happens—as V+V. Let’s see this with a few outcomes, where I’ll write e.g. X=1 to mean “in this scenario, X will create one happy person).

Bonus coin comes up heads, first coin comes up tails: (V+V)=2, Y=1, X=1.

Bonus coin comes up tails, first coin comes up tails: (V+V)=2, Y=1, X=1.

Bonus coin comes up heads, first coin comes up heads, second coin comes up tails: (V+V)=4, Y=3, X=1. V+V=4, Y+X=4.

Bonus coin comes up tails, first coin comes up heads, second coin comes up tails: (V+V)=4, Y=1, X=3. V+V=4, Y+X=4.

…

No matter what outcomes you get, V+V=X+Y.

So, does this mean that the “egyptology,” argument for fanaticism doesn’t work? Does this mean that it’s not a problem that non-fanatical views imply that when deciding whether to take some action, you first need to know what was happening in ancient Egypt? In my view the answer is no!

The biggest problem with this objection is that all the cases where fanaticism violates dominance or irrelevance of independent outcomes are cases involving infinites. But infinites always cause problems. Infinity has deeply weird mathematical properties that make it possible to produce utterly bizarre results. Violations of principles involving infinites are a world of difference from violations of principles involving finites. Because infinity breaks a whole lot of otherwise perfectly good principles.

Infinity doesn’t always cause problems. If there’s just an infinite range of outcomes, then there’s nothing problematic (e.g. if there’s an infinite probability distribution over outcomes, one with odds of 1/2, one with odds of 1/4, one with odds of 1/8, that’s no problem). But infinity tends to cause problems either when:

1. One is trying to compare two infinites of the same cardinality (cardinality is a measure of whether the members of two sets can be paired one to one).
2. There is an infinite range of probabilities that are normalizable (that means they add to one) of various outcomes, but the outcomes have some relevant feature that grows as fast or faster than the probabilities drop off. For example, the St. Petersburg game is weird because the outcomes grow as quickly as the probabilities drop off—each outcome is twice as good as the one before it and has a probability twice as great.

Thus, we ought to restrict the above principles to just being about cases where there is no normalizable probability distribution that has value increase at the same speed or faster than probabilities drop off. The fact that the St. Petersburg game has mathematically weird properties isn’t a reason to abandon fanaticism, as it’s just one of many similar kinds of weird mathematical functions.

Because infinity minus one is infinity, it’s not super surprising that you can chain together two dominated situations that get infinite expected value, reorder things, and have them no longer be dominated. The weirdness for fanaticism is in the math, not the axiology. Just as it’s not a problem for a moral view to produce weird judgments in cases when the world is weird, it’s not a problem for it to produce weird judgments when the math is really weird. The weirdness for fanaticism comes not from anything strange about its value structure, but from the surprising mathematical fact that sometimes though X is dominated by V, and Y is dominated by V, X+Y is not dominated by V+V. Scenarios where morality is weird because of strange mathematical properties of infinity are a world away from scenarios where morality is weird absent any weird mathematical properties of infinity.

As an analogy, infinite ethics illustrates that so long as simply moving people around doesn’t matter morally, a world where every galaxy has ten happy people and one miserable person is no better than one where every galaxy has one happy person and ten miserable people. The distribution across galaxies of people in an infinitely big world doesn’t matter. You can change the number of happy people per galaxy and miserable people per galaxy just by moving people.

But despite this, if a view implied the same thing in finite cases, that would be a huge problem! If a view implied that if there were 100 galaxies containing mostly happy people, this wouldn’t be better than if there were 100 galaxies containing mostly miserable people, that would be a fatal problem for the view! Infinity was placed here by Satan to test our faith in elegant decision procedures—you can’t just rely on principles breaking in scenarios with infinite expected value and expect that to generalize!

Similarly, Hilbert’s hotel shows that you can free up extra rooms in an infinitely large hotel just by moving people around. But it would be a huge problem if a view said this could also be done in a hotel with finite rooms!

The reason we’re fine with this in infinite contexts but not in finite contexts is that we recognize that it arises, in the infinite cases, from strange and unique mathematical properties of infinity. Infinity isn’t like other numbers, and thus leads to weird and counterintuitive results. But we shouldn’t tolerate these in finite cases.

I basically think that counterexamples involving infinites are fake and can be safely ignored in finite contexts.

A second reply: as Wilkinson notes, the counterexample of Russell involves situations where two different decision procedures are correlated in weird ways. In contrast, the finite cases where non-fanaticism violates Egyptology don’t need the outcomes to be correlated.

But there’s a big difference between saying that correlated causally-isolated outcomes don’t matter and saying that non-correlated isolated outcomes don’t matter. As Wilkinson notes, non-fanatical views imply that there are sometimes two outcomes, X and Y, such that X>Y, but X+V<Y+V—even though V is totally causally isolated and not correlated with Y. But, Wilkinson reasons, if V is entirely unrelated, then it should improve the value of gambles X and Y by a constant amount.

It’s one thing for G to be a probabilistic gamble that correlates in various ways with X and Y. But if G simply, no matter what happens with Y or Y, bumps up value by some constant amount—no matter whether you get Y or Y—then X+G shouldn’t beat Y+G unless X beats Y.

Intuitively, it’s not so weird for learning about the ancient Egyptians to affect our decision-making if the value their actions brought about correlates with the value our actions brought about and weird infinites are involved. But it would be weird if simply finding out that the ancient Egyptians used some totally unrelated chancy process affected the value of outcomes today.

Third, the cases where non-fanaticism gets wonky are cases of the following form:

V>X.

V>Y.

V+V=X+Y.

In other words, two gambles that each would appear worse than some third gamble are together as good as two instances of the third gamble.

But this is rather different from the scenarios where non-fanaticism gets wonky. Non-fanaticism gets wonky in cases where whether you should take some gamble is partially a function of what was happening millions of years in the past or in distant galaxies. Rather than being about comparisons of sets of dominated gambles, it’s about individual gambles. So provided that you have the intuition that when deciding whether to take a gamble, you shouldn’t need to know what is happening in causally isolated contexts on distant galaxies, the dark side of the moon, or the distant past, you should accept fanaticism.

In short, the cases where dominance and irrelevance of causally isolated outcomes get weird raise lots of red flags. They involve weirdly correlated outcomes and—most threateningly—infinites. You shouldn’t give up on a perfectly good principle because it’s wonky in infinite cases, provided it works fine in finite cases.

As an analogy, there’s a pretty popular—and similar—objection to average utilitarianism. Average utilitarianism says that one should maximize average utility. But this implies that when deciding to have a kid, your decision should be majorly affected by the number of happy people in ancient Egypt and distant galaxies (for if far-away people are very happy, even a happy child will lower the average). It would not do to defend averageism by noting that non-averageist views imply that causally isolated regions of space influence the value of an action—as illustrated by the comparisons of X+Y vs V+V. But so long as you grant that this is a good objection to averageism, then I think you should also grant that it’s a good argument for fanaticism.

6 Indology and why other views suck

The Indology objection is a variant of the previous argument but even harder to get around, originally from Hayden Wilkinson (unrelated: I am consistently impressed by how clever people on both sides of the fanaticism debate are. It feels like the area of philosophy where both sides have been most ingenious in coming up with good arguments). We can derive fanaticism if we assume (and here I will quote Petra Kosonen’s excellent exposition of the argument (Petra also has a very good, but mostly defunct blog):

Negative Reflection. If prospect A is not better than prospect B conditional on any possible answer to some question Q, then A cannot be better than B unconditionally.

Background Independence. Some prospect A is better than another prospect B if and only if the combined prospect A plus some far away outcome O is also better than the combined prospect B plus the same far away outcome O.

We also need Stochastic Dominance which says that some gamble is better than another gamble if for every outcome the first gamble offers, the second gamble offers an equal chance of an outcome that is as good or better, and for some of them it offers an outcome that is better. For instance, dominance would imply that a 1/2 chance of 1 util would be less good than a 1/2 chance of 2 utils and that a 1/2 chance of 1 util and a 1/2 chance of 2 utils would be less good than a 1/2 chance of 1 util and a 1/2 chance of 3 utils (assuming all else is equal, so that 3 utils is better than 2 and 2 is better than 1).

Background Independence is pretty similar to the irrelevance of independent outcomes. But instead of saying that your actions shouldn’t be affected by any far-away happenings, it just says that your actions shouldn’t be affected by far-away states of affairs. So, for instance, it still could be that whether you should take a gamble depends on whether people on distant planets are taking other gambles—it just says it shouldn’t depend on whether distant planets have certain quantities of value. For instance, it shouldn’t be that after learning that ten billion years ago, there were lots of happy people in ancient Egypt, this should make you more reluctant to take a gamble you’d have otherwise taken.

Suppose there are some range of possible answers to some question. Negative reflection holds if A is better than B given any answer to the question, then A>B. For example, if you know that no matter how old someone is dropping them off a cliff would be bad, then you shouldn’t drop them off a cliff. Petra Kosonen has a nice example to illustrate the principle:

Suppose that you are somewhere tropical and warm. The weather forecast says that either there will be heavy rain or burning sunshine. You have an umbrella that can protect against both. If it rains, you prefer having the umbrella. And if there is burning sunshine, you are neutral about having the umbrella and not having it. It would be odd if you nonetheless prefer not bringing the umbrella: no matter what happens, you will be happy or neutral about having brought it! Negative Reflection tells us that you should indeed bring the umbrella.

So how do non-fanatical views violate this constraint? So far we’ve seen how non-fanatical views imply that various goings-on occurring on Mars could make a risky gamble dominate a safe gamble. In the goings-on described, whether the risky gamble gets you anything correlates with whether the safe gamble gets you anything. I’m going to call the risky gamble Risky (because it’s risky) and the safe gamble safe. Safe guarantees some good outcome, Risky gives a low chance of a very good outcome. Risky has higher expected value than safe.

Now, it turns out that as has been briefly discussed, there’s a fairly complicated potential value function that if carried out on Mars would make Risky dominate Safe. This value function, unlike the previous one, is independent of what’s happening on Earth—this means that knowing what payouts are received on Earth gives no information about what payouts are received on Mars. Let’s call this the Martian Independent gamble.

By Dominance, Risky + the Martian independent gamble>safe + the Martian independent gamble. Now, given that Risky + the Martian independent gamble beats safe + the Martian independent gamble, Negative Reflection entails that there must be some value of the Martian independent gamble, such that if that value is the value ultimately produced by the Martian independent gamble, then risky>safe. Thus, risky is better than safe if there’s some amount of causally-isolated value on Mars.

But Background Independence holds that the causally-isolated value on Mars doesn’t affect the valuation of risky vs. safe. The other principles establish that risky beats safe if there’s some unrelated value on Mars. But if the unrelated value on Mars doesn’t affect whether risky beats safe—as Background Independence holds—then risky is better than safe.

Thus, if Negative Reflection, Dominance, and Background Independence are true, Risky beats safe. Fanaticism is thereby vindicated!

There is a reply that can be given here which is somewhat similar to the reply given in the last section. In fact, Dominance and Negative Reflection imply that fanaticism is false. Suppose we’re comparing two gambles:

1. St Petersburg: a coin is going to be flipped until it comes up tails. 2^N people will be spared from suffering, where N is the number of times the coin came up tails.
2. St Petersburg minus: a coin is going to be flipped until it comes up tails. 2^N - 1 people will be spared from suffering, where N is the number of times the coin came up tails.

So St Petersburg gives you a 1/2 chance of sparing 2 people from suffering, a 1/4 chance of sparing 4 people, a 1/8 chance of sparing 8, and so on. St Petersburg minus gives you a 1/2 chance of sparing 1, and 1/2 chance of sparing 3, and so on. So St Petersburg is clearly better than St Petersburg minus.

Both St Petersburg and St Petersburg minus have infinite expected value. A 1/2 chance of 2 plus a 1/4 chance of 4 plus a 1/8 chance of 8=1+1+1…and so on forever. A 1/2 chance of 1, + a 1/4 chance of 3, + a 1/8 chance of 7 also has infinite expected value.

But here’s the weird thing about St. Petersburg (and it’s mildly inferior twin): even though it has infinite expected value, it never actually has infinite value. No matter what happens, it gives you a finite quantity of value. It always turns out less value than you expect.

A helpful analogy is: imagine a random number generator across all natural numbers between one and infinity. Every time it spits out a number, it’s less than you expect. Every number has only finite numbers less than it, so every time an infinite random number outputs some number N, the odds of it outputting N or below are zero. Though it is infinite in expectation, it’s never actually infinite.

In light of this, fanaticism violates Negative Reflection. Fanaticism holds that St Petersburg>St Petersburg minus. But then suppose you learn the value of St Petersburg. Inevitably, the value will be finite, while the expected value of St Petersburg minus is infinite (because you haven’t yet figured out its actual value). So inevitably, after learning the actual value of the St Petersburg game, you will prefer St Petersburg minus to it—no matter what its value is!

This violates Negative Reflection. Negative Reflection, remember, says “If prospect A is not better than prospect B conditional on any possible answer to some question Q, then A cannot be better than B unconditionally.” St Petersburg, given fanaticism, is not better than St Petersburg minus conditional on any possible answer to the question: what is the outcome of St Petersburg. Thus, Negative Reflection holds that it can’t be better unconditionally.

Now, I think for this reason, pretty much everyone will have to give up on Negative Reflection in certain exotic infinite cases. Suppose there are two random number generators that generate a number between one and infinity. Call them RNGA and RNGB. Presumably the following is true:

1. It is not the case both that RNGA is better than RNGB and that RNGB is better than RNGA.

In other words, they can’t both be better than the other.

But Negative Reflection would imply both is better than the other. After you learn the results of RNGA, you should conclude that RNGB is better. After all, whatever the outcome of RNGA is, the odds that RNGB will give you that outcome or worse is 0 or infinitesimal.

Or for another example, suppose there are aleph null people and they’re all given two classes of numbers:

1. Number A, where A is what order they are in height. So if, for instance, they’re the third shortest person, they’re given 3.
2. Number B, where B is what order they are in foot size. So, if they have the ninth smallest foot, they’re given 9 for number B.

Each of the people has a foot size of between 10.5 and 10.6 and they cannot measure it or get any information about it. Each of the people is between 5’7 and 5’8, and they cannot measure their height or get any relevant information about it. Each person is given the following options:

1. Get number A utils. So if they’re the 180,000th shortest person, they get 180,000 utils.
2. Get number B utils. So if they have the 32,000th shortest foot, they get 32,000 utils.

Negative reflection seems to imply that both is better than the other. After learning what number A is, you’re guaranteed to prefer number B. And vice versa!

So we can’t just hold on to an unqualified version of Negative Reflection. But I don’t think this seriously jeopardizes the argument for two main reasons.

First of all, even if we don’t buy some unqualified version of negative reflection, the way non-fanatical views violate negative reflection looks pretty insane. These views imply that you should take the risky gamble over the safe gamble if you’re uncertain about what’s happening on Mars—or perhaps what was happening in the Indus Valley civilization (that’s why the objection is called the Indology objection). But then after you learn what happened on Mars or in the Indus valley civilization, no matter what you learn, then you prefer the risky gamble.

This strikes me as totally insane! You shouldn’t prefer the risky gamble to the safe gamble in part because of uncertainty about what was happening thousands of years ago, but then after you learn what in fact happened, you necessarily revise your assessment.

To put it into perspective, imagine that in a while ago, the Indus valley civilization brought about value in correspondence with the function that makes the risky gamble dominate the safe gamble. In light of this, on the non-fanatical view (that maintains background independence and dominance) one should take the risky gamble over the safe gamble. Because they don’t know how much value was actually brought about, but instead have uncertainty about it, they prefer the risky gamble to the safe gamble by dominance.

But then, after they’ve completed their studies and learned what happened in the Indus valley civilization, no matter what they learned, they begin preferring the safe gamble to the risky. Hayden Wilkinson puts it well:

From Background Independence we know that, whatever you might uncover in your research, you would conclude that the risky lottery is no better than the safe lottery. For any value of b you might pin down, you’ll establish that it’s no better to take the risky lottery. To judge otherwise would be to accept Fanaticism. So you know what judgment you would make if you simply learned more, no matter what it is you would learn. So why bother with the many years of research? Why not just update your judgment now and say that Lrisky 1 B is no better than Lsafe 1 B? You cannot. You must accept that Lrisky 1 B is strictly better, even though you can predict with certainty that you would change your mind if you knew more. To be able to change your mind, you must go through those years of grueling research and digging, even though you already know what judgment you will later conclude to be correct.

Thus, we don’t need some broader, exceptionless principle to prefer Risky to Safe. All we need is the notion that if gamble A beats gamble B no matter what happened in the causally-isolated ancient Indus valley civilization, then if you’re not sure what happened in the causally-isolated ancient Indus valley civilization, you should think gamble A beats gamble B. You shouldn’t think gamble A beats gamble B based in part on the presence of some uncertain quantity of value in the ancient Indus valley civilization, but know what whenever you discover how much value there really was, no matter what you find, you’ll then think B beats A.

Second, there seems to be a pretty natural way to restrict Negative Reflection to avoid these problems. All the problems so far have arisen in cases where the question being asked is: what is the value of this thing, where it’s average value is greater than any possible value it can have? So we can simply restrict it to the following:

Negative Reflection Finite Expectation. If prospect A is not better than prospect B conditional on any possible answer to some question Q, provided Q is not “what is the quantity of something X, where X’s expected quantity is greater than any possible quantity of X,” then A cannot be better than B unconditionally.

This may sound a bit ad hoc, but I don’t think it is. All of the problems come from infinite gambles sometimes having a weird property where their average value is greater than any possible value. If a gamble has that property, violations of Negative Reflection are expected—if no matter what you learn about the outcome of some gamble, it’s less than you expected, then it’s totally unsurprising that you’ll inevitably value the gamble more before you learn its outcomes than after.

The violations of Negative Reflection come from that highly-specific and weird mathematical property of certain kinds of infinite gambles (namely, the kind where the expected outcome is greater than any possible outcome). But if some weird result follows from a specific and weird mathematical property of certain infinite gambles, then we shouldn’t generalize it to other sorts of gambles.

As mentioned before, if there are infinite galaxies, it’s no better for the galaxies to all have five happy people and one miserable person than for them to all have five miserable people and one happy person. You can change the first sort of universe into the second sort of universe just by moving people around.

But you shouldn’t automatically apply that to finite cases! Infinity is weird and horrendously paradoxical. Even if principles have counterexamples in infinite cases, that gives you little information that they have counterexamples in finite cases.

One other way to see that it isn’t ad hoc is that these kinds of infinities cause problems almost across the board. There are many different paradoxes that arise from normalizable probability functions—but they all result from something else relevant growing faster than the probabilities drop off.

For example, suppose that infinite clones of you are created and all put to sleep. A coin will be continually flipped until it comes up tails. 3^N people will be awoken, where N is the number of coinflips. Upon waking up, your credences all turn out undefined. You end up concluding that the coin having come up heads before tails is likelier than just tails, heads heads tails is likelier than heads tails, and so on forever. For every possible X, where X is the number of heads coin-flips, X is likelier than X-1 (because your odds of waking up are three times higher given X coin-flips than X-1, and the prior of X is only 1/2 as great as the prior of X-1).

And there are tons of other horrendous paradoxes from infinities with these properties. They include the shooting rooms paradox, the Pasadena game (which makes actions have undefined expected value), and various others. One particularly thorny one is the following. Compare:

Happy becomes sad: Every year, 4^N people get created. So in year one, 4 people are created, in year 2 16 are created, and so on. Each person is happy for a year, but then becomes sad after a year.

Sad becomes happy: Every year, 4^N people get created. So in year one, 4 people are created, in year 2 16 are created, and so on. Each person is sad for a year, but then becomes happy after a year.

Each of the following seems plausible:

1. A world where most people are happy at every time is better than one where most people are sad at every time (assuming all else is equal).
2. A world where every particular person spends forever happy and finite time sad is better than one where every particular person spends forever sad and finite time happy (assuming all else is equal).

Sadly, these conflict. 1 holds happy becomes sad is better than sad becomes happy. 2 holds sad becomes happy is better. In sad becomes happy, at any given time 100% of people are sad, but every particular person spends finite time sad and infinite time happy. In happy becomes sad, at any given time most people are happy but every particular person spends finite time happy and infinite time sad.

In short, over and over again, problems arise when there’s an infinite lottery that either updates faster than the prior falls off or has value grow faster than the probability drops off. These kinds of lotteries (and states of affairs that aren’t lotteries) are uniquely paradoxical—so if a principle breaks down in these cases, we shouldn’t assume it breaks down in other cases.

For example, if a view implied that in cases where this doesn’t arise—where probabilities don’t fall off faster than value grow—something’s expected value was greater than any possible value of it, that would be a serious problem! These problematic classes of infinity and other things are separate and non-overlapping magesteria, where inferences about one don’t directly carry over to the other.

But things get even worse when we consider exactly which of these the alternatives will have to reject. There are two basic ways to not be a fanatic. The first one involves discounting low risks. Basically, you can hold that extremely tiny chances of outcomes don’t matter, no matter how good the outcomes are.

Second, you can have a bounded utility function. You can hold that as the quantity of some very good thing approaches infinity, the value you assign to that approaches some finite cap. So, for instance, while it’s good for one happy person to be created, infinity happy people being created isn’t infinite times better. The value of N happy people being created approaches some finite limit as N goes to infinity.

It has to be one of these. Either you have to think that infinite amounts of good stuff aren’t infinitely important or that you get to ignore really low probabilities of things no matter how good. If you value infinite amounts of good stuff at infinity and don’t round down risks that approach infinity according to some rapidly shrinking function that values them at zero at that low level of probability, then you’ll inevitably value an arbitrarily low risk of something infinitely good at infinity.

But both of these views struggle.

Views on which utility reaches a bound reject Background Independence (because causally isolated people determine your distance from the cap). That’s bad enough. But worse, they have to hold that as the amount of good stuff approaches infinity, the amount you value that approaches a finite bound. This means that what’s happening on distant planets matters an arbitrarily great amount to what’s happening on Earth.

A first problem is that if the amount of good stuff happening elsewhere is infinite, then the marginal utility you assign to extra good stuff is 0. If you think that as the number of happy lives approaches infinity, the value approaches some specific bound (say, you value it at googolplex units of utility), then if there are already infinite happy lives, you must value extra lives at zero. This is insane. It implies that if the universe has infinite value, no action to improve it matters at all.

It’s especially insane on the negative end. If you have bounded utility on the positive end, presumably you’ll also have bounded utility on the negative end. But surely even if there were already infinite miserable people, torturing another person would still be bad.

Now, you could hold that there’s a bound to the positive end of utility but not the negative end. That would, however, be super weird! We should expect the positive and negative ends to look similar. And if you hold that, then you’ll be a fanatic at least about avoiding negative value—thinking that a guaranteed torture of 100,000,000 people is less bad than a one in Graham’s number chance of torturing some sufficiently large number of people.

(That view would also hold that because disutility increases unboundedly but utility doesn’t, 100% of your decision-making should be dominated by avoiding chances of infinitely negative outcomes).

Second, bounded utility, for this reason, implies a weird attitude towards risk. Suppose that you suspect that the universe already has infinite utility—or some very large amount of utility. You think there’s a 99.999999999999% chance that the universe has infinite value (or, if you don’t like infinites some value big enough to do the trick). There’s a .000000000001% chance that the universe has some pretty large amount of value but not an infinite amount. A genie appears to you with the following two offers which you pick between:

Offer 1: If the universe has the smaller amount of value, one child has slightly greater enjoyment of their lollypop.

Offer 2: If the universe has infinite or the vastly greater amount of value, you create 100,000 exquisitely happy people, plus boost the happiness of some very large number of people. You get to send a billion people from a life just barely worth living in a poor country to a vastly better life in a wealthier country—where they meet the love of their life, experience exquisite happiness 10,000 times greater than the total amount of happiness anyone has yet experienced, and produce many great achievements.

Clearly, offer 2 is better than offer 1. But those who believe in bounded utility must deny that. If you get the payout from offer 2, then the universe is already very good—so the marginal value of 2 is negligible and outweighed by 1. This strikes me as a totally decisive and knockdown objection. Beckstead and Thomas give a nice version of this case:

Lingering doubt: In Utopia, life is extremely good for everyone, society is extremely just, and so on. Their historians offer a reasonably well-documented history where life was similarly good. However, the historians cannot definitively rule out various unlikely conspiracy theories. Perhaps, for instance, some past generation ‘cooked the books’ in order to shield future generations from knowing the horrors of a past more like the world we live in today. Against this background, let us evaluate two options: one would modestly benefit everyone alive if (as is all but certain) the past was a good one; the other would similarly benefit only a few people, and only if the conspiracy theories happened to be true.

On bounded utility, you should take the second.

It’s even worse on the negative end. Bounded utility views say that if you’re already very close to the positive cap, then preventing some very large amount of misery doesn’t matter much. But this would mean that if offer 2 instead spared 100 quadrillion people from extreme torture—but only conditional on the universe having huge amounts of positive utility—then option 1 would beat it on bounded utility views.

What about risk discounting? Suppose we round low risks down to zero. The best version of this ignores particularly extreme risks. In short, this version says that you should have some very small probabilistic threshold of N%, and then ignore the N% best and worst outcomes, except in the case of resolving tie-breakers.

This view has many problems.

First of all, as we’ll discuss later, it leads to bizarre hypersensitivity. Suppose that you discount the worst 100,000th and best 100,000th of possible scenarios. On this view, making the worst 1/100,000 of scenarios infinitely bad and the best 1/100,000 of scenarios infinitely good makes no difference to the value of a gamble (except regarding tiebreakers). However, probabilities above that threshold matter categorically more.

This means that such a view implies that a 1/100,000 chance of infinite enduring utopia is less valuable than a 1/99,999.99999999 chance of a slightly improved world. Any value with any probability above the threshold matters categorically more than any value with any probability below the threshold. Similarly, a 1/100,000 chance of infinite torture for everyone is less bad than a 1/99,999.9999999 chance of the world being slightly bad. Nuts!

(Note: 100,000 was just an example of a possible tail—the same basic point will apply no matter what the tail is).

Second, I’m pretty sure Tail Discounting violates negative reflection. Tail discounting affirms dominance and Background Independence (constant amounts of value don’t affect probabilities). Thus, it must hold that before you learn what’s happening in the ancient indus river valley civilization, you prefer the risky gamble to the safe gamble, but once you learn it, then you prefer the safe gamble.

Third, as discussed in section 5, it gives up on separability of causally-isolated and independent outcomes. As section 2 discusses, it gives up Partial Dominance.

Fourth, as section 3 discussed, it gives up on 99% Independence. If you recall, 99% Independence says:

If Gamble A is better than gamble B, a 99% chance of gamble A is better than a 99% chance of gamble B.

The problem is that a 99% chance of gamble A might put gamble A below the threshold. To see this, suppose you discount risks below 1% to zero (note: if risks aren’t discounted to zero, then fanaticism still follows because high enough payouts dominate). Now suppose you’re offered two gambles:

Gamble A: A 1.01% chance of a very large number of utils.

Gamble B: A 100% guarantee of 10 utils.

Gamble A beats gamble B (because the number of utils is high enough). But now suppose that the gambles are modified in the following way. No matter which one you take, a 100-sided die will be rolled. You’ll only get payout if it comes up anything other than 78. Basically now there is a 1% chance that the deal that you take has no effect—and you get zero regardless.

Taking into account the 1% chance you get nothing, now the gambles are as follows:

Gamble A: A 0.9999% chance of a very large number of utils, otherwise nothing.

Gamble B: A 99% chance of 10 utils.

Now, tail discounting prefers B to A! Merely by adding some possibility of neither gamble getting you anything, now gamble A loses to gamble B.

This is pretty insane! If gamble A is better than gamble B, then a 99% chance of getting gamble A should be better than a 99% chance of getting gamble B!

One way to see this is to imagine that the die hasn’t been rolled (this is the die that, if it comes up 78 makes neither gamble do anything). Before the roll of the die, gamble B beats gamble A. But then after the die is rolled, either:

1. It will come up 78 in which case the gambles are equal (and both get nothing).
2. It will not come up 78 in which case gamble A>gamble B.

We can derive even sillier results. Suppose, once again, that you discount risks below 1%. Currently, you think there is a 1.1% chance that we’ll ever reach some state of the world S. However, conditional on reaching state of affairs X, there will be two options. The first of the options would bring about some arbitrarily good outcome with probability .9. The second would provide a guarantee of some mildly good outcome.

On this view, at this point, it would be worth taking some action that guarantees that if we get to state of affairs X, we’ll take the second option (for the first option, given our doubts about getting to X, rounds down to zero). But then if we ever reach state of affairs X, this view would support trying to shift to state of affairs Y! This violates the following constraint of rationality:

No Regret: You shouldn’t at this moment rationally hope that if you reach A then you’ll bring about B, but when you reach A, not wish to bring about B.

For instance, you should not rationally hope that if you get into some room, then you’ll eat cake, but then when you get into the room rationally hope you don’t eat cake. A rational agent would not be at war with themself in that way. You should not think “if R, then I prefer Y,” and then after learning R automatically not prefer Y.

Fifth, it’s not even clear this version of tail-discounting avoids fanaticism. Consider the scenario I gave before, where you’re offered the following prospects:

1. Some value X.
2. A million-sided die will be rolled. If it comes up 38, negotiating ends and you get nothing. If not, you get a million times X.

A tail discounter would prefer 2. Then, if the gamble die is rolled and doesn’t come up 38, they’d take another gamble where unless they get 38, their payouts are a million times higher. Then they’d be incentivized to take another gamble like that, and so on. In short, the person who applies tail-discounting to individual decisions has an incentive to always take the bet that massively increases payouts but lowers probability of success! In practice, they bet like a fanatic!

Now, maybe the way around this is to have one’s utility function assign value to prospects at the start, and then use those starting values to inform prospects. So, for instance, suppose one rounds down risks below 1%. There is a 5% chance that one reaches a planet. After they reach the planet, there are two gambles. The first gives a 10% chance of a very good outcome and the second gives a guarantee of a mediocre outcome. At the start one values the risky gamble as less good than the safe gamble. Then, after reaching the planet, they hold on to their initial evaluations—based on the starting probabilities—and prefer the risky gamble to the safe one.

This view has problems.

It implies that one should sometimes do things that are totally insane. Suppose that one discounts risks below 1%. There is a 1.00001% chance that they’ll be able to colonize space. Then, they think that they have colonized space, but there is a very low probability vast conspiracy on which space colonization is fake. Now they think there’s a 99.9% chance that they colonized space.

They can either take an action that produces enormous benefit if they’ve colonized space (which they’re near certain they have) or take an action that produces very modest benefit if they haven’t colonized space. It seems by Stochastic Dominance, they should take the first action. However, at the beginning they would discount the best space colonization scenarios down to zero—so now they should take the action that brings about more modest benefit with lower probability. Nuts! The view is therefore inconsistent with stochastic dominance.

Tail discounting has another counterintuitive implications. Suppose that there’s a 1/2 chance of getting one util and a 1/2 chance of getting two utils. For simplicity, assume this is all the utility in the world. There are two gambles:

1. Conditional on you getting two utils, this gamble would get you a 1/100 trillion chance of getting infinite extra utils.
2. Conditional on you getting one util, this gamble would get you a 1/100 quadrillion chance of getting another half util.

Counterintuitively, tail discounters prefer the second gamble! Because the first gamble’s extra value comes in the extreme tails, it’s ignored for everything but tie-breakers. The second gamble gives you value outside of the extreme tails, so it is better. But this is completely insane!

In short, because tail discounting ignores the tails, it implies that boosts to the tails matter infinitely less than boosts to non-tails—even if the boosts outside the tails bring about a lower probability of a less good outcome.

So far the kind of risk discounting we’ve been talking about has been tail discounting, where one ignores the best and worst tiny percent of possibilities. There’s another version, called Nicolausian discounting, that simply ignores very low risks. But this version is very implausible (and pretty much all the same problems apply).

Suppose you discount risks below one in a billion. Then if there are a billion distinct ways things could go well for you, you’ll discount all of them. You’ll evaluate a scenario that guarantees something very good happens as having zero value! There’s also a problem of how outcomes are individuated: what things count as an outcome. If there’s a chance that someone will poison my drink, is the relevant outcome, for discounting purposes, “my drink will be poisoned,” or “my drink will be poisoned with cyanide.” It might be that each individual poison is unlikely enough to be used that if they’re counted separately, they round down, but if they’re counted as one scenario they don’t.

So if you are going to discount, tail discounting is the way to go. Sadly, it still has enormous problems.

7 Scale independence argument

One more argument for fanaticism comes from Hayden Wilkinson’s excellent paper on fanaticism. It relies on the following idea called Scale Independence:

Scale Independence: For any lotteries La and Lb, if La ≽ Lb , then k x La ≽ k x Lb for any positive, real k.

It’s a lot simpler than it sounds. The basic idea is as follows: suppose there are two gambles, A and B. If A is better than B, then if you multiply the payouts of both A and B by some constant k, then A is still better than B.

For instance, suppose A has a 50% chance of sparing 1,000 people from intense suffering. B is guaranteed to save one person from intense suffering. Suppose A is better than B. Scale Independence says that if you make A and B both do twice as much good, then A will still be better than B. So if A is modified to A* and B is modified to B*, so that A has a 50% chance of sparing 2,000 people from intense suffering, and B is guaranteed to save two people from intense suffering, Scale Independence says A is still better than B.

Non-fanatical views either have to reject Scale Independence or accept absurd sensitivity to extremely small changes.

To see this, suppose that you are given the option to save 10 people from intense suffering. Fanaticism says that an arbitrarily low chance of saving some arbitrarily large number of people is better than being guaranteed to save 10 people. Consider the following gambles:

Gamble 1: Guaranteed to save ten people.

Gamble 2: Has a very low chance of saving some arbitrarily large number of people.

Fanaticism says gamble 2 is better than gamble 1. Other views disagree.

But now let’s ask: is there some benefit that a gamble could guarantee so that it would lose out to gamble 2? Specifically consider

Gamble 3: Guaranteed to add an extra one second to a person’s life.

Is gamble 3 better than gamble 2? Either answer is problematic.

Suppose the answer is no—gamble 2 is better than gamble 3. This in combination with scale independence implies fanaticism. Suppose that gamble 3 is one ten-billionth as good as gamble 1. Gamble 3<gamble 2. Therefore, gamble 2 with the payouts multiplied by ten billion>gamble 3 multiplied by ten billion. But gamble 3 with payouts multiplied by ten billion just is gamble 1. Thus, if gamble 3 is worse than gamble 2, then gamble 1 must be worse than some fourth gamble, which is just gamble 2 with the payouts multiplied by some very large number. This vindicates fanaticism, holding that tiny probabilities of huge payouts beat large guarantees of large payouts.

In fact, for this reason, we don’t actually need a principle as strong as Scale Independence. All we need is the following more modest principle:

Scale Possible Independence: For any lotteries La and Lb and any constant K, if La ≽ Lb, there exists some other constant R such that R(La)≽K(Lb).

For example, suppose that a 1/2 chance of 1,000 utils is better than a guarantee of five utils. This principle holds that for any constant you choose to multiply the value of the guarantee of five utils by, you can beat that by multiplying the 1,000 utils with 1/2 probability by a sufficient amount. So a guarantee of 500 utils would be less good than a 1/2 chance of some number of utils (where that number is taken by multiplying some constant R by 1,000).

But now suppose we answer yes to the earlier question. Gamble 3 is better than gamble 2. No matter how much you multiply the payouts offered by gamble 2, it always is inferior to any guarantee of any arbitrarily small amount of value. On this view, there must be some minimal probability at which a gamble can beat guaranteed payouts. There must be some minimal probability P such that probability P of some sufficiently good outcome is better than gamble 3. Consider:

Gamble 4: Offers probability P-I of some arbitrarily good outcome (where I is some very tiny probability that puts things below the threshold).

On this view, gamble 4 is less good than gamble 1. Any guarantee of any positive outcome automatically beats gamble 4. But this implies a weird hypersensitivity. There is some minimal threshold at which probabilities start counting. Consider some other gamble:

Gamble 5: Offers probability P+I (where I is some very small amount that breaches the probability threshold) of some slightly good outcome.

On this view, gamble 5 is better than gamble 4. And this is so even though the odds of gambles 4 and 5 getting you anything are very near equal and gamble 5, conditional on getting you anything, gets you something much better than gamble 4.

The quick and dirty explanation of this is: provided that scale independence is false, utility must not be approach an asymptotic bound. It must be that as the amount of valuable stuff approaches infinity, the value of that approaches infinity. To avoid fanaticism, therefore, you will have to discount risks below a certain threshold down to zero or infinitesimal. If you discounted them by any other amount, making the payouts sufficiently large would allow arbitrarily low probabilities of sufficiently great outcomes to beat guaranteed very large payouts. So you can’t just discount them by, say, 50% or even 99.99999999999%—for them not to dominate, it must be that all probabilities above the threshold categorically matter more than them.

But if you discount risks below some probabilistic threshold near zero, then any outcome below that probabilistic threshold automatically trumps any outcome above the probabilistic threshold. Suppose that threshold is, say, one in a million. Well then a probability of one in 999,999 of one person getting an apple will be better than a probability of one in 1,000,001 of bringing about enduring global utopia!

Now, in theory you can get around this by giving up transitivity. You can hold the following:

1. A guarantee of an apple is better than a one in 1,000,001 chance of peace on earth.
2. A guarantee of an apple is worse than a one in 999,999 chance of giving someone an extra 10 years of life.
3. A one in 999,999 chance of giving someone an extra 10 years of life is less good than a one in 1,000,001 chance of peace on earth.

But this doesn’t help much. In order to avoid Fanaticism in decision-making, you need to ignore probabilities below some probabilistic threshold (at least, if you accept Scale Independence). But this entails that, because you don’t discount probabilities above the threshold, in normal decision-making you’ll count all probabilities above the threshold—no matter how small the payouts are—more than all probabilities below the threshold (no matter how large the payouts are).

8 The Carlsmith argument

One of the most powerful arguments for fanaticism comes from the brilliant Joe Carlsmith—one of the most interesting and thoughtful writers around. Suppose that you are deciding between saving one life with certainty or a one in ten billion chance of saving a trillion lives. Fanatics recommend taking the second deal, non-fanatics don’t!

But let’s fill in the details. Imagine that there are one trillion people and a trillion-sided die. Right now, each person has a number. One person has the number one, one has the number two, one has the number three, and so on.

You’re going to roll the trillion-sided die. After you do this, the person with the number that corresponds with the die roll will be saved. So if, for instance, you roll 83, the person with the number 83 will be saved. Everyone else will die.

But now imagine you can instead give everyone every single number between 1 and 100. Now, everyone will be saved if any number between 1 and 100 comes up. Surely that is better! It raises every single person’s odds of survival. But now, rather than one person being guaranteed to be saved, there is now a one in 10 billion chance of a trillion people being saved. Thus, so long as you support changing a gamble in ways that make every single person better off in expectation, you must prefer a one in ten billion chance of saving a trillion people to a guarantee of saving one person.

(There’s a lot more detail in Carlsmith’s post, but this essay is already over 11,000 words, so I can’t discuss it in detail).

9 A core problem for non-fanatical views

In this section, I’ll argue that there’s another big problem for non-fanatical views: in practice, they end up either being fanatical or being insane.

Suppose that the world is very bad but you’re uncertain about exactly how bad it is. In practice, non-fanatical views will recommend gambling away everything of value on Earth to reduce the risk that the universe is very bad overall. If you’re highly uncertain about the value in the universe, non-fanatical views end up being fanatical in practice, because your impact is a drop in the bucket compared to the rest of the world—the best way to act given uncertainty even if you are non-fanatical will usually be to take extremely long-shot gambles.

To see this, first consider bounded utility views. These views discount very high or very low values. But they end up in practice being fanatical (and worse).

The way they’re worse is that, as already discussed, sometimes they result in preferring something with a tiny chance of being slightly good over something with a very large chance of being very good (assuming that the scenario where the thing is very good is one where you’re very near the cap). If you think there’s a 99.999% chance you’re near the cap and .001% you’re far away from the cap, they’ll prefer something that gives you 1 util if you’re far away than something that gives you 10 trillion utils if you’re very near the cap. Nuts!

The way they’re fanatical is that if the world might be very far below the cap, then risky gambles are still worth it. For a simple model, suppose that util numbers between 100 quadrillion and -100 quadrillion counts fully, and then utility above and below the threshold counts for nothing. On this model, someone with bounded utility, certain we’re presently at zero, would regard a guarantee of 1 util as being equal to a one in a billion chance of a billion utils.

Now, you can set the threshold very low, but if you set it too low, then you end up concluding that because the world already has a decently large amount of value, nothing much matters.

Next, you might discount low risks. But that view will be quite fanatical for any amount above the threshold (it will be fanatical about probabilities of sufficiently great outcomes with greater than X probability, where X is the probability threshold at which risks are discounted to zero).

Even putting that aside, the view will still be quite fanatical in that it will hold that if there’s lots of uncertain value not up to you, taking low-risk high-reward gambles generally most reduces very bad outcomes. In a world of wild uncertainty, the safe strategy is generally to maximize expected value.

Now one can get around this by being a discounter with respect to one’s individual contribution to global utility, rather than with respect to overall states of affairs. You can, for instance, be a tail-discounter—ignoring the best and worst scenarios of how you personally might impact the world, rather than the best and worst ways the world might turn out. But if such a view is only about one’s personal impact, then it doesn’t help with evaluations of gambles. This view is compatible with fanaticism about gambles—whereby a tiny chance of a sufficiently good gamble (elsewhere in the universe, unrelated to you) is better than a guarantee of a mildly good outcome.

In addition, such a policy gets dominated. Suppose that there are 1,000 people each of whom are risk averse. Each is given a number somewhere between one and a thousand. Then a random number generator spits out a random number. The people are each given the following option:

1. Create 1 good thing no matter what the random number generator comes up.
2. Create 100,000 good things but only if the random number generator comes up on your number.

One who is risk averse with respect to their personal contribution will prefer 1. But if everyone takes 1, then there is a guaranteed worse outcome than if people take 2. Thus, agents acting on the decision-procedure that rounds low risks of their actions down to zero get dominated!

10 The intuitiveness objection to fanaticism

Probably the most common objection to fanaticism is that it seems intuitively crazy! Who in the world would give up all the value in the universe for a one in a quadrillion chance of an arbitrarily large amount of value? Who would take a one in a billion chance of infinite happy people being created over a guarantee of a trillion happy people being created?

Now, I’ve already given a bunch of reasons to think that this is the right judgment—that’s what the last ~12,000 words have been about! But nonetheless, the view is unintuitive. If a view is unintuitive, that is a cost of it. So perhaps this should lower our credence in fanaticism.

Now, I’ll grant that this has a non-zero amount of force (though, in my judgment, far less force—even intuitively—than the other considerations I’ve raised). However, I think we shouldn’t place too much weight in it. There is abundant evidence that our intuitions about low risks shouldn’t be trusted.

There are two biases that are particularly responsible for our non-fanaticism. The first one is that we round low risks down to zero. Hayden Wilkinson writes:

And our intuitions about decision-making in the face of low-probability events are no better. For instance, jurors are just as likely to convict a defendant based primarily on fingerprint evidence if that evidence has probability 1 in 100 of being a false positive as if it were 1 in 1,000, or even 1 in 1,000,000. In another context, when presented with a medical operation which posed a 1 percent chance of permanent harm, many respondents considered it no worse than an operation with no risk at all. In yet another context, subjects were unwilling to pay any money at all to insure against a 1 percent chance of catastrophic loss. So it seems that many of us, in many contexts, treat events with low probabilities as having probability 0, even when their true probability is as high as 1 percent (which is not so low). Upon reflection, this is clearly foolish. Intuitions saying that we may (or should) ignore all outcomes with probability 1 percent do not hold up to scrutiny.

(See also relatedly the Allais paradox.)

Humans just aren’t good at intuiting low risks. I remember once hearing someone treat risks of one in a million and one in a hundred as equivalent. For this reason, we shouldn’t trust our intuitions about that subject. When a risk is very low, beneath a certain threshold, it registers in our mind as zero.

Cass Sunstein has argued that human risk assessment malfunctions whenever emotions are high. Sunstein first notes just how frequently humans’ risk assessment malfunctions:

In an especially striking study, Kunreuther and his coauthors found that mean willingness to pay insurance premiums did not vary among risks of 1 in 100,000, 1 in 1 million, and 1 in 10 million. They also found basically the same willingness to pay for insurance premiums for risks ranging from 1 in 650, to 1 in 6300, to 1 in 68,000.

…

An early study examined people’s willingness to pay (WTP) to reduce travel fatality risks. The central finding was that the mean WTP to reduce fatality risk by 7/100,000 was merely 15% higher than the mean WTP to reduce the risk by 4/100,000. A later study found that for serious injuries, WTP to reduce the risk by 12/100,000 was only 20% higher than WTP to reduce the same risk by 4/100,000. These results are not unusual. Lin and Milon attempted to elicit people’s willingness to pay to reduce the risk of illness from eating oysters. There was little sensitivity to variations in probability of illness.

…

Why don’t people think in this way? An imaginable explanation is that in the abstract, most people simply do not know how to evaluate low probabilities. A risk of 7/100,000 seems “small”; a risk of 4/100,000 also seems “small.”

In particular, Sunstein demonstrates that when emotions are high, people are far less likely to be sensitive to low risks. After conditions are described in vivid and emotional terms, the median willingness to pay to avoid a risk of a condition doesn’t change whether the risk is 1/100,000 or 1/1,000,000. It’s no surprise, then, that when something registers to us as “a sure guarantee of something as good as we could possibly fathom,” we’re not willing to trade it away against tiny risks of vastly larger benefits.

In practice, for reasons I gave in the last section (see Hillary Greaves for more on this idea), even those who discount low risks should be willing to endure significant sacrifices for a tiny chance of very good outcome. But most people wouldn’t do that. Thus, even those who think we should discount low risks should grant that humans display various irrational patterns of risk aversion.

Second, humans aren’t good at grasping the difference in value between very good outcomes. We display a bias called scope neglect, where we’re intuitively unable to grasp the differences between very big things—they both simply register as “big thing.” We’ll pay similar amounts to save 2,000 birds, 20,000 birds, or 200,000 birds. As Alex Hern notes:

So, Exxon asked the public: and it turns out that the amount they’re willing to pay to save 2,000 birds is around $80.

Unfortunately, Exxon also asked the public how much they’re willing to pay to save 200,000 birds. The answer? $88.

Similarly, when asked to recommend a jail sentence for a financial adviser who defrauded their clients, people picked an average of 4.8 years for someone who defrauded 3 clients, and less time – 3.8 years – if they were told that there were instead 30 victims.

Martin Harrison, of London-based agency Huge Inc, who highlighted the findings, said that “in 136 real court cases, they found that the pattern was repeated in real life. The more people harmed, the less severe the punishment meted out was.”

“As things get bigger and bigger and bigger we lose track of the scope.”

We have no ability to intuitively grasp the difference between one very good outcome and another very good outcome. They both just register as very good. It’s no wonder then that we won’t pay vastly more for a low probability of a vastly greater outcome—intuitively, it doesn’t actually clock as being vastly greater. We have no intuitive ability to grok just how much better a billion lives are than a million lives. I know it’s dicey to quote Sam Bankman Fried in an essay about low risk scenarios, but what he said about the repugnant conclusion strikes me as extremely applicable:

Still many people have the intuition that 1,000,000,000,000, people at happiness 2 (Option A) is better than 1,000,000,000,000,000 people with happiness 1 (Option B). But I posit that this comes not from a flaw in total utilitarianism, but instead from a flaw in human intuition. You remember those really big numbers with lots of zeros that I listed earlier in this paragraph? What were they? Many of you probably didn't even bother counting the zeros, instead just registering them as "a really big number" and "another really big number, which I guess kind of has to be bigger than the first really big number for Sam's point to make sense, so it probably is."

…

And so when presented with Options A and B from above, people really read them as (A): some big number of people with happiness 2, and (B): another really big number of people with happiness 1. We don't really know how to handle the big numbers--a quadrillion is just another big number, kind of like ten thousand, or eighteen. And so we mentally skip over them. But 2, and 1: those we understand, and we understand that 2 is twice as big as one, and that if you're offered the choice between 2 and 1, 2 is better. And so we're inclined to prefer Option A: because we fundamentally don't understand the fact that in option B, one thousand times as many people are given the chance to live. Those are entire families, societies, countries, that only get the chance to exist if you pick option B; and by construction of the thought experiment, they want to exist, and will have meaningful existences, even if they're not as meaningful on a per-capita basis as in option A.

Fundamentally, even though the human mind is really bad at understanding it, 1,000,000,000,000,000 is a lot bigger than 1,000,000,000,000; in fact the difference dwarfs the difference between things we do understand, like the winning percentages of the Yankees versus that of the Royals, or the numbers 2 and 1. And who are we to deny existence to those 900,000,000,000,000 people because we're too lazy to count the zeros?

In short, we don’t actually have any intuitive grasp of the difference between two distinct astronomical quantities. There’s an old joke illustrating this. A scientist says “in 5 billion years, the sun will expand out and swallow Earth.” A man gets up from the audience and flees in horror. After the lecture, the scientist says to the man “why did you freak out—it won’t happen for another 5 billion years.” “Oh thank goodness,” the man replies, “I thought you said it would happen in 5 million years.”

The joke is funny because it exploits an obvious fact about psychology: to the human brain, 5 million and 5 billion blur together. Even though there are 4,995,000,000 extra years in 5 billion, it doesn’t register to us as that different from 5 million. But in light of the fact that big numbers register as the same in our brain, we shouldn’t be surprised that we fail to value astronomically large amounts of value as much as we should.

I have one final reason for placing only minimal weight on this direct intuition: as I’ve argued so far, if fanaticism is wrong, there are many different things we’d have to be wrong about. We’d probably have to be wrong about many of the following: transitivity, some separability principles, comparability, Negative Reflection of Background Independence, Partial Dominance, Ex-Ante Pareto, and more. Time and time again, we’ll have to be confused.

In contrast, what do we have to be wrong about if fanaticism is true? We’d have to be wrong in our direct intuitions about low risks—ones that there’s plenty of reasons not to trust. And, as we’ll see, we’ll also have to say that the right theory sort of malfunctions in infinite cases.

But neither of these things should be that surprising. Infinity ruins a lot of things! The most consistent result of philosophical analysis of infinity has been the realization that, “oh shit, infinity falsifies this otherwise seemingly undeniable principle.” We shouldn’t be at all surprised, therefore, if the right theory of decision-making under low risks malfunctions in infinite cases. Nor should we be surprised that our intuitions about low probabilities are totally wrong. But together these summarize the entirety of the case against fanaticism.

It strikes me as vastly likelier that infinity causes problems and our intuitions are mistaken about risk than that over and over again, on distinct areas like transitivity and separability, we’ll consistently be mistaken.

11 Pascal’s mugging

Probably the most famous objection to fanaticism is known as Pascal’s mugging. Imagine a guy comes up to you and says the following:

I’m a super powerful wizard. If you give me 100 dollars, then I will grant you infinite utility and create infinite happy people.

It seems like you shouldn’t give him the money. But even if you think the odds are only one in Googolplex that he’s telling the truth, the expected value is infinite. Thus, fanaticism would seem to tell you to give the alleged wizard the money. But that’s nuts!

Now, I don’t think fanaticism actually implies that. If you actually gave away money in such a scenario, then probably you’d get a lot more people asking you for money claiming to be wizards. This would lower the odds of bringing about infinite value long-term. In fact, there are other equally likely scenarios by which your actions might cause you to lose out on infinite value, if a genuinely infinitely powerful wizard was so irritated by the fakers that he caused you to lose infinite utility.

It’s unlikely, sure, but it’s also unlikely the guy is telling the truth. And if you give money to charities or helping others, that might produce infinite value by strengthening your afterlife connection with them or enabling infinite value in the far future. Even money spent on yourself might be infinitely valuable if you have infinite conscious subsystems, which seems likelier than the guy being a wizard.

Every action has some chance of bringing about infinite value. The odds giving the money to the alleged wizard brings about infinite value are probably swamped by the ways that all other actions can bring about infinite value.

But suppose that somehow you know with certainty that the action won’t cause you to lose out on infinite value. Suppose you receive a guarantee that there is no way infinite value will be brought about except by appeasing the apparent wizard, and though you have no evidence he’s really a wizard, this is the only action that has any effect on infinite scales of value. In that case should you give him the money?

I think the answer is yes. But then the scenario is just a classic example of comparing low probabilities of very high outcomes to guarantee of low outcomes. As the previous many sections illustrated, there are many very powerful arguments for thinking you should, in such a case, give the money to the wizard.

When divorced from the real world, the case is just a classic example of a case where you compare infinite value with low probabilities to a guarantee of finite value, and as I’ve spent many thousands of words arguing, there are strong reasons, in such a case, to pick the small chance of infinite value. And here, there’s another bias: you’d feel foolish giving the money away to the alleged wizard, and people hate taking actions that make them feel foolish.

12 Fun in St. Petersburg

I think the most serious objections to fanaticism come from certain kinds of infinite gambles. Specifically, the big challenges are:

1. If you’re a fanatic, you should spend your life chasing infinite value. Everything else is arbitrarily less important.
2. The St. Petersburg gamble money pumps fanaticism.
3. Various exotic gambles like the Pasadena game mess up expected utility theory by making actions have undefined expected value.

Regarding the first point, I just bite the bullet. Infinite value is infinitely more valuable than every other possible amount of value (Source???). You should therefore try to maximize the chance of it, no matter what else must be given up. I think the arguments in the last section show that our intuitions about this aren’t reliable.

Regarding the second worry, the St. Petersburg gamble, remember, is the gamble that gives you a 1/N chance of 2^N utils, for every value of N that is a multiple of 2. It gives you a 1/2 chance of 2 utils, a 1/4 chance of 4 utils, and so on. It has infinite expected value, but it is guaranteed to give you finite value. No matter what happens, the value it produces is only finite.

So now let’s imagine that there’s another gamble: the St. Petersburg+ gamble, which is like the St. Petersburg gamble but one dollar better. It gives you a 1/2 chance of 3 dollars, a 1/4 chance of 5 dollars, a 1/8 chance of 9 dollars, and so on. Suppose you have two options:

1. Get the St. Petersburg gamble.
2. Pay 30 cents to get the St. Petersburg + gamble. Then, after you discover the outcome of the St. Petersburg+ gamble, you can trade it for the St. Petersburg gamble.

Clearly, one should take gamble 2. But this exposes them to a money pump. No matter what, after they find out the outcome of the St. Petersburg + gamble, they will swap it in for the St. Petersburg gamble. Once they know what value it takes on, inevitably they will want to trade it away. But this means they’ll pay a dollar to get what they could have had for free.

More concerningly, there is an iterated money-pump. Suppose that both the St. Petersburg gamble and the St. Petersburg + gamble are in boxes. Each time, after looking in the box, you can trade it away for the other box by paying a dollar. Only when are currently looking in the box do you know the contents of the box (after you’re no longer looking, your memory is wiped). So when you look in the St. Petersburg box, you remember its contents but not the contents of the St. Petersburg+ gamble. And vice versa.

First you look in the St. Petersburg + box. Seeing its value is finite, you now trade it away for the St. Petersburg box. Then, seeing its value is finite, and not remembering the value of the St. Petersburg box, you want to trade back. This can keep going on forever, with you losing more and more money!

I grant that this is pretty weird. But I think it’s weird because of the mathematical property that an infinite function can have where it’s average value (or its expected value) can be greater than any possible value it might have. In light of such a situation, it’s not particularly surprising that each time you discover the outcome of the situation, you’ll be disappointed and want to trade it away. If a view has weird implications because of weird math, that is the fault of the math, not of the view.

I basically think this is just a repackaging of the infinite fair-lottery paradox. In particular, I think everyone will have problems like this.

To see this, imagine an analogous case where there are two boxes that each generate random numbers (or if you think random number generators are impossible, feel free to sub in the situation given in the last section, where people are arranged by height and foot size). You only know the number in the box when you look in the box. After you see which number you get, you can trade it away for the other number in the other box, by paying a dollar.

Thus, you peek in the first box and see it spits out some particular number. Now you have every incentive to trade it away. Let N be the value that the first box gives you. Now there’s a 100% chance that the second box will give you a number more than N (because 0% of numbers are N or less). But then, when you see the number in box two, and can’t remember what is in box one, you have an incentive to trade it away for box one. Thus, you are incentivized to go in circles, and pay for what you could have kept for free.

Or suppose that you are offered the St. Petersburg gamble, which you can then trade for the St. Petersburg+ gamble every single day forever. Now, because the situation is iterated, by the law of large numbers, you want to maximize expected value. Thus, it seems that fanatics and non-fanatics face the same puzzle when the scenario is eternally iterated.

In particular, there seem to be a few ways to go:

1. Just bite the bullet and say this is the rational way to proceed under uncertainty.
2. Hold that money pumps aren’t good ways of ascertaining rationality (though I am not very sympathetic to this approach). You might hold that this stricture only applies in exotic infinite cases (something I’m more sympathetic to).
3. Say that cases involving memory loss or non-iteration aren’t plausible money pumps.
4. Hold that the rational thing to go is to simply pre-commit to whichever gamble dominates the other.

Ultimately, I don’t know which of these is right. They all seem pretty weird. But the crucial point is that none of them are unique to fanaticism. As the iterated and fair lottery cases show, they’re everyone’s problem. They arise from the math being weird, not any problem specific to fanaticism. If there is some strange mathematical feature that implies weird things when applicable, it is no cost of a view that it gives the mathematical feature a place to apply.

Or to put things in a way that’s quippier and only a bit less accurate: counterexamples involving infinites aren’t real and can be safely ignored (unless they’re simply the limit case of similar problems that apply in finite cases).

The last serious objection to fanaticism is that it potentially breaks down in infinite cases. The expected value of an infinite series often depends on how it’s ordered (for instance, 1-1+1-1+1 can converge to any number depending on how your order the ones). In addition, there are cases where an infinite series produces a clearly unbounded result. Suppose that you’re offered a gamble with a 1/X chance of producing X^X utils, where X is every second factor of 2, and a 1/Y chance of causing you to lose Y^Y utils, where Y is every second factor of 2 spaced otherwise. So the gamble gives you a 1/2 chance of 2^2 utils, a 1/4 chance of causing you to lose 4^4 utils, a 1/8 chance of giving you 8^8 utils, a 1/16 chance of costing 16^16 utils, and so on. This is a divergent series, meaning it doesn’t in the limit approach any specific value. It’s value repeatedly fluctuates to higher and lower positive and negative values, never settling on one.

I grant that this is a real puzzle, but I think that what I’ve said about previous scenarios will apply.

First of all, I don’t take it all that seriously because it arises from math being weird in infinite cases. Math is always weird in infinite cases! Counterexamples involving weird infinite math are fake!

Second, this will apply to every view. Suppose that some gamble gives you a random either positive or negative number between negative infinity and positive infinity. Once again, its expected value will be undefined, and it’s unclear on every view how to value it.

Third, it doesn’t seem that bad to just say that you should have an undefined evaluation of scenarios with undefined expected value. Now, this gets weirder when you have non-zero credence in scenarios with undefined expected value, but that might be resolvable with fancy math. (For instance, it seems like a plausible view could hold that infinite value with any finite probability swamps the value of a divergent series and that if the probabilities in a divergent series are compensated for by greater probabilities of greater values, the gamble is good overall. For instance, if there’s a gamble that has a one in a million chance of giving the undefined-expected-value gamble described above, and a 999,999/one million chance of giving X^X utils, then the X^X value gambles might be able, with fancy mathematics, to cancel out the negative outcomes of the other gamble).

Fourth, even if there was no way to resolve the weirdness, and this made standard actions have undefined expected values—something I’m very skeptical of—it wouldn’t seem like that horrendous of a result. Infinity is weird. If the world turns out to have certain features that make exotic properties of infinity applicable, it wouldn’t be that shocking if this meant our actions had undefined expected values. I don’t think this is likely, but if it were true, while it would be tragic, it would not be so implausible as to be worth rejecting fanaticism over.

Fifth, if, as before, we imagine being given various outcomes with different expected values every single day, then non-fanatics will have the same problems as fanatics. It’s unclear how to rank divergent series expected value outcomes if you’re given them daily, because they don’t converge to some specific value. Because the series never converges, it’s hard to see how you should evaluate the prospect of being given such a gamble every day. Whatever a non-fanatic says here, the fanatic can say more generally.

Sixth, again as discussed before, weird results with divergent series aren’t a unique problem for fanatics. They’re ubiquitous in anthropics. Consider the following scenario:

Infinite people are put to sleep. Then a coin is flipped until it comes up tails. N^N people get woken up, where N is the number of flips. After waking up, what should your credence be in the number of flips taking on various values?

You’re presumably twice as likely to wake up if twice as many people wake up than if half as many people do. Therefore, because the update grows faster than the probabilities drop off, for any number N, you should think that it’s likelier there were N flips than any number less than N. Thus, your credence in any particular number of flips or less will always turn out undefined!

We can get similar weirdness via a divergent series in the following way:

Infinite people are put to sleep. They’re separated into two groups. Then a coin is flipped until it comes up tails. N^N people get woken up, where N is the number of flips. If N is even, the people are woken up from group one. If N is odd, the people are woken up from group two. After waking up, what should your credence be in you having been in group one?

Once again, the answer is undefined! The series is divergent! Such problems aren’t unique to fanaticism!

12 Conclusion

Here, I’ve presented a bunch of arguments for fanaticism and addressed the major objections. There are many different arguments for fanaticism. Alternatives must give up on plausible principles about risk, transitivity, and much more. The case for fanaticism is extremely strong—for fanaticism to be wrong, we’d have to be mistaken about a greater number of very obvious things.

What about the case against fanaticism? It strikes me as much weaker. There are two big classes of objection:

1. Fanaticism is directly unintuitive. It implies weird things about risk.
2. Fanaticism seems to malfunction in various infinite cases.

I find 1 not to be terribly persuasive. Our intuitions about risk are often wrong. We should trust them far less than the great range of intuitions supporting fanaticism—both numerous and diverse.

I also find 2 not terribly persuasive. The ways fanaticism malfunctions are typically:

1. Not unique to fanaticism—other views imply them too.
2. Resultant from weird features of math, rather than something strange about fanaticism itself.
3. Susceptible to various analogues in other domains like anthropics. Thus, fanaticism seems to be in good company.
4. Resultant from specifically problematic functions (scenarios where a normalizable probability function has something else that matters grow faster than probabilities fall off).

All you need to think to be a fanatic is that our intuitions are wrong in cases involving low risks of hugely good outcomes—something we shouldn’t be that surprised by—and that infinity ruins everything in decision theory, just as it does everywhere else. In my view, these being true would be vastly less surprising than us being as consistently mistaken about a dozen different features of infinites. In addition, fanaticism’s main opponents generally imply horrendous things and have similar problems to fanaticism.

For these reasons, while not free of cost, I think fanaticism is vastly more plausible than alternatives!

Now, what should you do when you conclude fanaticism is right? Well, a fanatic thinks that their sole aim should be to maximize the odds of infinite value. Keeping the world around probably does that, so you should donate to Longtermist charities (especially because they potentially increase the number of people ever born, thus giving more people a chance of getting into heaven). Maybe you should also try to become religious to increase your odds of getting into Heaven, but that’s more controversial.