Hello! My name is Vaden Masrani and I'm a grad student at UBC in machine learning. I'm a friend of the community and have been very impressed with all the excellent work done here, but I've become very worried about the new longtermist trend developing recently.
I've written a critical review of longtermism here in hopes that bringing an 'outsiders' perspective might help stimulate some new conversation in this space. I'm posting the piece in the forum hoping that William MacAskill and Hilary Greaves might see and respond to it. There's also a little reddit discussion forming as well that might be of interest to some.
Cheers!
Thanks for writing this. I think it's very valuable to be having this discussion. Longtermism is a novel, strange, and highly demanding idea, so it merits a great deal of scrutiny. That said, I agree with the thesis and don't currently find your objections against longtermism persuasive (although in one case I think they suggest a specific set of approaches to longtermism).
I'll start with the expected value argument, specifically the note that probabilities here are uncertain and therefore random valuables, whereas in traditional EU they're constant. To me a charitable version of Greaves and MacAskill's argument is that, taking the expectation over the probabilities times the outcomes, you have a large future in expectation. (What you need for the randomness of probabilities to sink longtermism is for the probabilities to correlate inversely and strongly with the size of the future.) I don't think they'd claim the probabilities are certain.
Maybe the claim you want to make, then, is that we should treat random probabilities differently from certain probabilities, i.e. you should not "take expectations" over probabilities in the way I've described. The problem with this is that (a) alternatives to taking expectations over probabilities have been explored in the literature, and they have a lot of undesirable features; and (b) alternatives to taking expectations over probabilities do not necessarily reject longtermism. I'll discuss (b), since it involves providing an example for (a).
(b) In economics at least, Gilboa and Schmeidler (1989) propose what's probably the best-known alternative to EU when the probabilities are uncertain, which involves maximizing expected utility for the prior according to which utility is the lowest, sort of a meta-level risk aversion. They prove that this is the optimal decision rule according to some remarkably weak assumptions. If you take this approach, it's far from clear you'll reject longtermism: more likely, you end up with a sort of longtermism focused on averting long-term suffering, i.e. focused on maximizing expected value according to the most pessimistic probabilities. There's a bunch of other approaches, but they tend to have similar flavors. So alternatives on EU may agree on longtermism and just disagree on the flavor of it.
(a) Moving away from EU leads to a lot of problems. As I'm sure you know given your technical background, EU derives from a really nice set of axioms (The Savage Axioms). Things go awry when you leave it. Al-Najjar and Weinstein (2009) offer a persuasive discussion of this (H/T Phil Trammell). For example, non-EU models imply information aversion. Now, a certain sort of information aversion might make sense in the context of longtermism. In line with your Popper quote, it might make sense to avoid information about the feasibility of highly-specific future scenarios. But that's not really the sort of information non-EU models imply aversion to. Instead, they imply aversion to info that would shift you toward the option that currently has a lot of ambiguity about it because you dislike it based on its current ambiguity.
So I don't think we can leave behind EU for another approach to evaluating outcomes. The problems, to me, seem to lie elsewhere. I think there are problems with the way we're arriving at probabilities (inventing subjective ones that invite biases and failing to adequately stick to base rates, for example). I also think there might be a point to be made about having priors on unlikely conclusions so that, for example, the conclusion of strong longtermism is so strange that we should be disinclined to buy into it based on the uncertainty about probabilities feeding into the claim. But the approach itself seems right to me. I honestly spent some time looking for alternative approaches because of these last two concerns I mentioned and came away thinking that EU is the best we've got.
I'd note, finally, that I take the utopianism point well and wold like to see more discussion of this. Utopian movements have a sordid history, and Popper is spot-on. Longtermism doesn't have to be utopian, though. Avoiding really bad outcomes, or striving for a middling outcome, is not utopian. This seems to me to dovetail with my proposal in the last paragraph to improve our probability estimates. Sticking carefully to base rates and things we have some idea about seems to be a good way to avoid utopianism and its pitfalls. So I'd suggest a form of longtermism that is humble about what we know and strives to get the least-bad empirical data possible, but I still think longtermism comes out on top.
This might also be of interest:
The Sequential Dominance Argument for the Independence Axiom of Expected Utility Theory by Johan E. Gustafsson, which argues for the Independence Axiom with stochastic dominance, a minimal rationality requirement, and also against the Allais paradox and Ellsberg paradox (ambiguity aversion).
However, I think a weakness in the argument is that it assumes the probabilities exist and are constant throughout, but they aren't defined by assumption in the Ellsberg paradox. In particular, looking at the figure for case 1,... (read more)