Comments
Thanks for writing this. I think it's very valuable to be having this discussion. Longtermism is a novel, strange, and highly demanding idea, so it merits a great deal of scrutiny. That said, I agree with the thesis and don't currently find your objections against longtermism persuasive (although in one case I think they suggest a specific set of approaches to longtermism).
I'll start with the expected value argument, specifically the note that probabilities here are uncertain and therefore random valuables, whereas in traditional EU they're constant. To me a charitable version of Greaves and MacAskill's argument is that, taking the expectation over the probabilities times the outcomes, you have a large future in expectation. (What you need for the randomness of probabilities to sink longtermism is for the probabilities to correlate inversely and strongly with the size of the future.) I don't think they'd claim the probabilities are certain.
Maybe the claim you want to make, then, is that we should treat random probabilities differently from certain probabilities, i.e. you should not "take expectations" over probabilities in the way I've described. The problem with this is that (a) alternatives to taking expectations over probabilities have been explored in the literature, and they have a lot of undesirable features; and (b) alternatives to taking expectations over probabilities do not necessarily reject longtermism. I'll discuss (b), since it involves providing an example for (a).
(b) In economics at least, Gilboa and Schmeidler (1989) propose what's probably the best-known alternative to EU when the probabilities are uncertain, which involves maximizing expected utility for the prior according to which utility is the lowest, sort of a meta-level risk aversion. They prove that this is the optimal decision rule according to some remarkably weak assumptions. If you take this approach, it's far from clear you'll reject longtermism: more likely, you end up with a sort of longtermism focused on averting long-term suffering, i.e. focused on maximizing expected value according to the most pessimistic probabilities. There's a bunch of other approaches, but they tend to have similar flavors. So alternatives on EU may agree on longtermism and just disagree on the flavor of it.
(a) Moving away from EU leads to a lot of problems. As I'm sure you know given your technical background, EU derives from a really nice set of axioms (The Savage Axioms). Things go awry when you leave it. Al-Najjar and Weinstein (2009) offer a persuasive discussion of this (H/T Phil Trammell). For example, non-EU models imply information aversion. Now, a certain sort of information aversion might make sense in the context of longtermism. In line with your Popper quote, it might make sense to avoid information about the feasibility of highly-specific future scenarios. But that's not really the sort of information non-EU models imply aversion to. Instead, they imply aversion to info that would shift you toward the option that currently has a lot of ambiguity about it because you dislike it based on its current ambiguity.
So I don't think we can leave behind EU for another approach to evaluating outcomes. The problems, to me, seem to lie elsewhere. I think there are problems with the way we're arriving at probabilities (inventing subjective ones that invite biases and failing to adequately stick to base rates, for example). I also think there might be a point to be made about having priors on unlikely conclusions so that, for example, the conclusion of strong longtermism is so strange that we should be disinclined to buy into it based on the uncertainty about probabilities feeding into the claim. But the approach itself seems right to me. I honestly spent some time looking for alternative approaches because of these last two concerns I mentioned and came away thinking that EU is the best we've got.
I'd note, finally, that I take the utopianism point well and wold like to see more discussion of this. Utopian movements have a sordid history, and Popper is spot-on. Longtermism doesn't have to be utopian, though. Avoiding really bad outcomes, or striving for a middling outcome, is not utopian. This seems to me to dovetail with my proposal in the last paragraph to improve our probability estimates. Sticking carefully to base rates and things we have some idea about seems to be a good way to avoid utopianism and its pitfalls. So I'd suggest a form of longtermism that is humble about what we know and strives to get the least-bad empirical data possible, but I still think longtermism comes out on top.
Thanks! I think that there's quite a lot of good content in your critical review, including some issues that really should be discussed more. In my view there are a number of things to be careful of, but ultimately not enough to undermine the longtermist position. (I'm not an author on the piece you're critiquing, but I agree with enough of its content to want to respond to you.)
Overall I feel like a lot of your critique is not engaging directly with the case for strong longtermism; rather you're pointing out apparently unpalatable implications. I think this is a useful type of criticism, but one that often leads me suspecting that neither side is simply-incorrect, but rather looking for a good synthesis position which understands all of the important points. (Your argument against expected value is a direct rebuttal of the argument for, but in my eyes this is one of your weaker criticisms.)
The point I most appreciate you making is that it seems like strong longtermism could be used to justify ignoring all sorts of pressing present problems. I think that this is justifiably concerning, and deserves attention. However my view is more like "beware naive longtermism" (rather like "beware naive utilitarianism") rather than thinking that the entire framework is lost.
To expand on that:
(I'll address a few other points in replies to this comment, for better threading and because they seem less centrally important to me.)
In response to the plea at the end (and quoting of Popper) to focus on the now over the utopian future: I find myself sceptical and ultimately wanting to disagree with the literal content, and yet feeling that there is a good deal of helpful practical advice there:
Regarding the point about the expectation of the future being undefined: I think this is correct and there are a number of unresolved issues around exactly when we should apply expectations, how we should treat them, etc.
Nonetheless I think that we can say that they're a useful tool on lots of scales, and many of the arguments about the future being large seem to bite without relying on getting far out into the tails of our hypothesis space. I would welcome more work on understanding the limits of this kind of reasoning, but I'm wary of throwing the baby out with the bathwater if we say we must throw our hands up rather than reason at all about things affecting the future.
To see more discussion of this topic, I particularly recommend Daniel Kokotajlo's series of posts on tiny probabilities of vast utilities.
As a minor point, I don't think that discounting the future really saves you from undefined expectations, as you're implying. I think that on simple models of future growth -- such as are often used in practice -- it does, but if you give some credence to wild futures with crazy growth rates, then it's easy to make the entire thing undefined even through a positive discount rate for pure time preference.
Hey Owen - thanks for your feedback! Just to respond to a few points -
>Your argument against expected value is a direct rebuttal of the argument for, but in my eyes this is one of your weaker criticisms.
Would be able to elaborate a bit on where the weaknesses are? I see in the thread you agree the argument is correct (and from googling your name I see you have a pure math background! Glad it passes your sniff-test :) ). If we agree EVs are undefined over possible futures, then in the Shivani example, this is like comparing 3 lives to NaN. Does this not refute at least 1 / 2 of the assumptions longtermism needs to 'get off the ground'?
> Overall I feel like a lot of your critique is not engaging directly with the case for strong longtermism; rather you're pointing out apparently unpalatable implications.
Just to comment here - yup I intentionally didn't address the philosophical arguments in favor of longtermism, just because I felt that criticizing the incorrect use of expected values was a "deeper" critique and one which I hadn't seen made on the forum before. What would the argument for strong longtermism look like without the expected value calculus? It's my impression that EVs are central to the claim that we can and should concern ourselves with the future 1 billion years from now.
Also my hope was that this would highlight a methodological error (equating made up numbers to real data) that could be rectified, whether or not you buy my other arguments about longtermism. I'd be a lot more sympathetic with longtermism in general if the proponents were careful to adhere to the methodological rule of only ever comparing subjective probabilities with other subjective probabilities (and not subjective probabilities with objective ones, derived from data).
> I would welcome more work on understanding the limits of this kind of reasoning, but I'm wary of throwing the baby out with the bathwater if we say we must throw our hands up rather than reason at all about things affecting the future.
Yup totally - if you permit me a shameless self plug, I wrote about an alternative way to reason here.
> As a minor point, I don't think that discounting the future really saves you from undefined expectations, as you're implying.
Oops sorry no wasn't implying that - two orthogonal arguments.
>I do think that if all people across time were united in working for the good
People are united across time working for the good! Each generation does what it can to make the world a little bit better for its descendants, and in this way we are all united.
I think it proves both too little and too much.
Too little, in the sense that it's contingent on things which don't seem that related to the heart of the objections you're making. If we were certain that the accessible universe were finite (as is suggested by (my lay understanding of) current physical theories), and we had certainty in some finite time horizon (however large), then all of the EVs would become defined again and this technical objection would disappear.
In that world, would you be happy to drop your complaints? I don't really think you should, so it would be good to understand what the real heart of the issue is.
Too much, in the sense that if we apply the argument naively then it appears to rule out using EVs as a decision-making tool in many practical situations (where subjective probabilities are fed into the process), including many where we have practical experience of it and it has a good track record.
Overall, my take is something like:
[Mostly an aside] I think the example has been artificially simplified to make the point cleaner for an audience of academic philosophers, and if you take account of indirect effects from giving to AMF then properly we should be comparing NaN to NaN. But I agree that we should not be trying to make any longtermist decisions by literally taking expectations of the number of future lives saved.
Not in my view. I don't think we should be using expectations over future lives as a fundamental decision-making tool, but I do think that thinking in terms of expectations can be helpful for understanding possible future paths. I think it's a moderately robust point that the long-term impacts of our actions are predictably a bigger deal than the short-term impacts -- and this point would survive for example artificially capping the size of possible futures we could reach.
(I think it's a super important question how longtermists should make decisions; I'll write up some more of my thoughts on this sometime.)
Hi Owen! Really appreciate you engaging with this post. (In the interest of full disclosure, I should say that I'm the Ben acknowledged in the piece, and I'm in no way unbiased. Also, unrelatedly, your story of switching from pure maths to EA-related areas has had a big influence over my current trajectory, so thank you for that :) )
I'm confused about the claim
This seems in direct opposition to what the authors say (and what Vaden quoted above), namely that:
I understand that they may not feel this way, but it is what they argued for and is, consequently, the idea that deserves to be criticized. Next, you write that if
I don't think so. The "immeasurability" of the future that Vaden has highlighted has nothing to do with the literal finiteness of the timeline of the universe. It has to do, rather, with the set of all possible futures (which is provably infinite). This set is immeasurable in the mathematical sense of lacking sufficient structure to be operated upon with a well-defined probability measure. Let me turn the question around on you: Suppose we knew that the time-horizon of the universe was finite, can you write out the sample space, $\sigma$-algebra, and measure which allows us to compute over possible futures?
Finally, I'm not sure what to make of
When reading their paper, I honestly did not read it as a toy example. And I don't believe the authors state it as such. When discussing Shivani's options they write:
and when discussing AI risk in particular:
Considering that the Open Philanthropy Project has poured millions into AI Safety, that it's listed as a top cause by 80K, and that EA's far-future-fund makes payouts to AI safety work, if Shivani's reasoning isn't to be taken seriously then now is probably a good time to make that abundantly clear. Apologies for the harshness in tone here, but for an august institute like GPI to make normative suggestions in its research and then expect no one to act on them is irresponsible.
Anyway, I'm a huge fan of 95% of EA's work, but really think it has gone down the wrong path with longtermism. Sorry for the sass -- much love to all :)
I can see two possible types of arguments here, which are importantly different.
[ETA: In this comment, which I hadn't seen before writing mine, Vaden seems to confirm that they were trying to make an argument of the second rather than the first kind.]
In this comment I'll explain why I think both types of arguments would prove too much and thus are non-starters. In other comments I'll make some more technical points about type 1 and type 2 arguments, respectively.
(I split my points between comments so the discussion can be organized better and people can use up-/downvotes in a more fine-grined way)
I'm doing this largely because I'm worried that to some readers the technical language in Vaden's post and your comment will suggest that longtermism specifically faces some deep challenges that are rooted in advanced mathematics. But in fact I think that characterization would be seriously mistaken (at least regarding the issues you point to). Instead, I think that the challenges either have little to do with the technical results you mention or that the challenges are technical but not specific to longtermism.
[After writing I realized that the below has a lot of overlap with what Owen and Elliot have written earlier. I'm still posting it because there are slight differences and there is no harm in doing so, but people who read the previous discussions may not want to read this.]
Both types of arguments prove too much because they (at least based on the justifications you've given in the post and discussion here) are not specific to longtermism at all. They would e.g. imply that I can't have a probability distribution over how many guests will come to my Christmas party tomorrow, which is absurd.
To see this, note that everything you say would apply in a world that ends in two weeks, or to deliberations that ignore any effects after that time. In particular, it is still true that the set of these possible 'short futures' is infinite (my house mate could enter the room any minute and shout any natural number), and that the possible futures contains things that, like your example of a sequence of black and white balls, have no unique 'natural' structure or measure (e.g. the collection of atoms in a certain part of my table, or the types of possible items on that table).
So these arguments seem to show that we can never meaningfully talk about the probability of any future event, whether it happens in a minute or in a trillion years. Clearly, this is absurd.
Now, there is a defence against this argument, but I think this defence is just as available to the longtermist as it is to (e.g.) me when thinking about the number of guests at my Christmas party next week.
This defence is that for any instance of probabilistic reasoning about the future we can simply ignore most possible futures, and in fact only need to reason over specific properties of the future. For instance, when thinking about the number of guests to my Christmas party, I can ignore people shouting natural numbers or the collection of objects on my table - I don't need to reason about anything close to a complete or "low-level" (e.g. in terms of physics) description of the future. All I care about is a single natural number - the number of guests - and each number corresponds to a huge set of futures at the level of physics.
But this works for many if not all longtermist cases as well! The number of people in one trillions years is a natural number, as is the year in which transformative AI is being developed, etc. Whether or not identifying the relevant properties, or the probability measure we're adopting, is harder than for typical short-term cases - and maybe prohibitively hard - is an interesting and important question. But it's an empirical question, not one we should expect to answer by appealing to mathematical considerations around the cardinality or measurability of certain sets.
Separately, there may be an interesting question about how I'm able to identify the high-level properties I'm reasoning about - whether that high-level property is the number of people coming to my party or the number of people living in a trillion years. How do I know I "should pay attention" only to the number of party guests and not which natural numbers they may be shouting? And how am I able to "bridge" between more low-level descriptions of futures (e.g. a list of specific people coming to the party, or a video of the party, or even a set of initial conditions plus laws of motion for all relevant elementary particles)? There may be interesting questions here, but I think these are questions for philosophy or psychology who in my view aren't particularly illuminated by referring to concepts from measure theory. (And again, they aren't specific to longtermism.)
Technical comments on type-1 arguments (those aiming to show there can be no probability measure). [Refer to the parent comment for the distinction between type 1 and type 2 arguments.]
I basically don't see how such an argument could work. Apologies if that's totally clear to you and you were just trying to make a type-2 argument. However, I worry that some readers might come away with the impression that there is a viable argument of type 1 since Vaden and you mention issues of measurability and infinite cardinality. These relate to actual mathematical results showing that for certain sets, measures with certain properties can't exist at all.
However, I don't think this is relevant to the case you describe. And I also don't think it can be salvaged for an argument against longtermism.
First, in what sense can sets be "immeasurable"? The issue can arise in the following situation. Suppose we have some set (in this context "sample space" - think of the elements at all possible instances of things that can happen at the most fine-grained level), and some measure (in this context "probability" - but it could also refer to something we'd intuitively call length or volume) we would like to assign to some subsets (the subsets in this context are "events" - e.g. the event that Santa Clause enters my room now is represented by the subset containing all instances with that property).
In this situation, it can happen that there is no way to extend this measure to all subsets.
The classic example here is the real line as base set. We would like a measure that assigns measure |a−b| to each interval [a,b] (the set of real numbers from a to b), thus corresponding to our intuitive notion of length. E.g. the interval [−1,3] should have length 4.
However, it turns out that there is no measure that assigns each interval its length and 'works' for all subsets of the real numbers. I.e. each way of extending the assignment to all subsets of the real line would violate one of the properties we want measures to have (e.g. the measure of an at most countable disjoint union of sets should be the sum of the measures of the individual sets).
Thus we have to limit ourselves to assigning a measure to only some subsets. (In technical terms: we have to use a σ-algebra that's strictly smaller than the full set of all subsets.) In other words, there are some subsets the measure of which we have to leave undefined. Those are immeasurable sets.
Second, why don't I think this will be a problem in this context?
As even more of an aside, type 1 arguments would also be vulnerable to a variant of Owen's objection that they "prove too little".
However, rather than the argument depending too much on contingent properties of the world (e.g. whether it's spatially infinite), the issue here is that they would depend on the axiomatization of mathematics.
The situation is roughly as follows: There are two different axiomatizations of mathematics with the following properties:
Specifically, for example, for our intuitive notion of "length" there are immeasurable subsets of the real numbers in the standard axiomatization of mathematics (called ZFC here). However, if we omit a single axiom - the axiom of choice - and replace it with an axiom that loosely says that there are weirdly large sets then every subset of the real numbers is measurable. [ETA: Actually it's a bit more complicated, but I don't think in a way that matters here. It doesn't follow directly from these other axioms that everything is measurable, but using these axioms it's possible to construct a "model of mathematics" in which that holds. Even less importantly, we don't totally omit the axiom of choice but replace it with a weaker version.]
I think it would be pretty strange if the viability of longtermism depended on such considerations. E.g. imagine writing a letter to people in 1 million years explaining why you didn't choose to try to help more rather than fewer of them. Or imagine getting such a letter from the distant past. I think I'd be pretty annoyed if I read "we considered helping you, but then we couldn't decide between the axiom of choice and inaccessible cardinals ...".
Technical comments on type-2 arguments (i.e. those that aim to show there is no, or no non-arbitrary way for us to identify a particular probability measure.) [Refer to the parent comment for the distinction between type 1 and type 2 arguments.]
I think this is closer to the argument Vaden was aiming to make despite the somewhat nonstandard use of "measurable" (cf. my comment on type 1 arguments for what measurable vs. immeasurable usually refers to in maths), largely because of this part (emphasis mine) [ETA: Vaden also confirms this in this comment, which I hadn't seen before writing my comments]:
Some comments:
The most viable anti-longtermist argument I could see in the vicinity would be roughly as follows:
But crucially, I think mathematical features of the objects we're dealing with when talking about common practices in a formal language are not where we can hope to find support for such an argument. This is because the longtermist and garden-variety cases don't actually differ relevantly regarding these features.
Instead, I think the part we'd need to understand is not why there might be a challenge, but how and why in garden-variety cases we're able to overcome that challenge. Only then can we assess whether these - or other - "methods" are also available to the longtermist.
Hi Max! Again, I agree the longtermist and garden-variety cases may not actually differ regarding the measure-theoretic features in Vaden's post, but some additional comments here.
Although "probability of 60%" may be less meaningful than we'd like / expect, you are certainly allowed to enter such bets. In fact, someone willing to take the other side suggests that he/she disagrees. This highlights the difficulty of converging on objective probabilities for future outcomes which aren't directly subject to domain-specific science (e.g. laws of planetary motion). Closer in time, we might converge reasonably closely on an unambiguous measure, or appropriate parametric statistical model.
Regarding the "60% probability" for future outcomes, a useful thought experiment for me was how I might reason about the risk profile of bets made on open-ended future outcomes. I quickly become less convinced I'm estimating meaningful risk the further out I go. Further, we only run the future once, so it's hard to actually confirm our probability is meaningful (as for repeated coin flips). We could make longtermist bets by transferring $ btwn our far-future offspring, but can't tell who comes out on top "in expectation" beyond simple arbitrages.
Honest question being new to EA... is it not problematic to restrict our attention to possible futures or aspects of futures which are relevant to a single issue at a time? Shouldn't we calculate Expected Utility over billion year futures for all current interventions, and set our relative propensity for actions = exp{α * EU } / normalizer ?
For example, the downstream effects of donating to Anti-Malaria would be difficult to reason about, but we are clueless as to whether its EU would be dwarfed by AI safety on the billion yr timescale, e.g. bringing the entire world out of poverty limiting political risk leading to totalitarian government.
Yes, I agree that it's problematic. We "should" do the full calculation if we could, but in fact we can't because of our limited capacity for computation/thinking.
But note that in principle this situation is familiar. E.g. a CEO might try to maximize the long-run profits of her company, or a member of government might try to design a healthcare policy that maximizes wellbeing. In none of these cases are we able to do the "full calculation", albeit my a less dramatic margin than for longtermism.
And we don't think that the CEO's or the politician's effort are meaningless or doomed or anything like that. We know that they'll use heuristics, simplified models, or other computational shortcuts; we might disagree with them which heuristics and models to use, and if repeatedly queried with "why?" both they and we would come to a place where we'd struggle to justify some judgment call or choice of prior or whatever. But that's life - a familiar situation and one we can't get out of.
It's all good! Seriously, I really appreciate the engagement from you and Vaden: it's obvious that you both care a lot and are offering the criticism precisely because of that. I currently think you're mistaken about some of the substance, but this kind of dialogue is the type of thing which can help to keep EA intellectually healthy.
So my interpretation had been that they were using a technical sense of "evaluating actions", meaning something like "if we had access to full information about consequences, how would we decide which ones were actually good".
However, on a close read I see that they're talking about ex ante effects. This makes me think that this is at least confusingly explained, and perhaps confused. It now seems most probable to me that they mean something like "we can ignore the effects of the actions contained in the first 100 years, except insofar as those feed into our understanding of the longer-run effects". But the "except insofar ..." clause would be concealing a lot, since 100 years is so long that almost all of our understanding of the longer-run effects must go via guesses about the long-term goodness of the shorter-run effects.
[As an aside, I've been planning to write a post about some related issues; maybe I'll move it up my priority stack.]
I like the question; I think this may be getting at something deep, and I want to think more about it.
Nonetheless, my first response was: while I can't write this down, if we helped ourselves to some cast-iron guarantees about the size and future lifespan of the universe (and made some assumptions about quantization) then we'd know that the set of possible futures was smaller than a particular finite number (since there would only be a finite number of time steps and a finite number of ways of arranging all particles at each time step). Then even if I can't write it down, in principle someone could write it down, and the mathematical worries about undefined expectations go away.
The reason I want to think more about it is that I think there's something interesting about the interplay between objective and subjective probabilities here. How much should it help me as a boundedly rational actor to know that in theory a fully rational actor could put a measure on things, if it's practically immeasurable for me?
Sorry, I made an error here in just reading Vaden's quotation of Shivani's reasoning rather than looking at it in full context.
In the construction of the argument in the paper Shivani is explicitly trying to compare the long-term effects of action A to the short-term effects of action B (which was selected to have particularly good short-term effects). The paper argues that there are several cases where the former is larger than the latter. It doesn't follow that A is overall better than B, because the long-term effects of B are unexamined.
The comparison of of AMF to AI safety that was quoted felt like a toy example to me because it obviously wasn't trying to be a full comparison between the two, but was rather being used to illustrate a particular point. (I think maybe the word "toy" is not quite right.)
In any case I consider it a minor fault of the paper that one could read just the section quoted and reasonably come away with the impression that comparing the short-term number of lives saved by AMF with the long-term number of lives expected to be saved by investing in AI safety was the right way to compare between those two opportunities. (Indeed one could come away with the impression that the AMF price to save a life was the long-run price, but in the structure of the argument being used they need it to be just the short-term price.)
Note that I do think AI safety is very important, and I endorse the actions of the various organisations you mention. But I don't think that comparing some long-term expectation on one side with a short-term expectation on the other is the right argument for justifying this (particularly versions which make the ratio-of-goodness scale directly with estimates of the size of the future), and that was the part I was objecting to. (I think this argument is sometimes seen in earnest "in the wild", and arguably on account of that the paper should take extra steps to make it clear that it is not the argument being made.)
"The "immeasurability" of the future that Vaden has highlighted has nothing to do with the literal finiteness of the timeline of the universe. It has to do, rather, with the set of all possible futures (which is provably infinite). This set is immeasurable in the mathematical sense of lacking sufficient structure to be operated upon with a well-defined probability measure. "
This claim seems confused, as every nonempty set allows for the definition of a probability measure on it and measures on function spaces exist ( https://en.wikipedia.org/wiki/Dirac_measure , https://encyclopediaofmath.org/wiki/Wiener_measure ). To obtain non-existence, further properties of the measure such as translation-invariance need to be required (https://aalexan3.math.ncsu.edu/articles/infdim_meas.pdf) and it is not obvious to me that we would necessarily require such properties.
See discussion below w/ Flodorner on this point :)You are Flodorner!
It certainly not obvious that the universe is infinite in the sense that you suggest. Certainly nothing is "provably infinite" with our current knowledge. Furthermore, although we may not be certain about the properties of our own universe, we can easily imagine worlds rich enough to contain moral agents yet which remain completely finite. For instance, you could image a cellular automata with a finite grid size and which only lasted for a finite duration.
However, perhaps the more important consideration is the in principle set of possible futures that we must consider when doing EV calculations, rather than the universe we actually inhabit, since even if our universe is finite we would never be able to convince our selves of this with certainty. Is it this set of possible futures that you think suffers from "immeasurability"?
Aarrrgggggg was trying to resist weighing in again ... but I think there's some misunderstanding of my argument here. I wrote:
A few comments:
Okay this will hopefully be my last comment, because I'm really not trying to be a troll in the forum or anything. But please represent my argument accurately!
You really don't seem like a troll! I think the discussion in the comments on this post is a very valuable conversation and I've been following it closely. I think it would be helpful for quite a few people for you to keep responding to comments
Of course, it's probably a lot of effort to keep replying carefully to things, so understandable if you don't have time :)
I second what Alex has said about this discussion being very valuable pushback against ideas that have got some traction - at the moment I think that strong longtermism seems right, but it's important to know if I'm mistaken! So thank you for writing the post & taking some time to engage in the comments.
On this specific question, I have either misunderstood your argument or think it might be mistaken. I think your argument is "even if we assume that the life of the universe is finite, there are still infinitely many possible futures - for example, the infinite different possible universes where someone shouts a different natural number".
But I think this is mistaken, because the universe will end before you finish shouting most natural numbers. In fact, there would only be finitely many natural numbers you could finish shouting before the universe ends, so this doesn't show there are infinitely many possible universes. (Of course, there might be other arguments for infinite possible futures.)
More generally, I think I agree with Owen's point that if we make the (strong) assumption the universe is finite in duration and finite in possible states, and can quantise time, then it follows that there are only finite possible universes, so we can in principle compute expected value.
So I'd be especially interested if you have any thoughts on whether expected value is in practice an inappropriate tool to use (e.g. with subjective probabilities) even assuming in principle it is computable. For example, I'd love to hear when (if at all) you think we should use expected value reasoning, and how we should make decisions when we shouldn't.
Hey Issac,
Yup you've misunderstood the argument. When we talk about the set of all future possibilities, we don't line up all the possible futures and iterate through them sequentially. For example, if we say it's possible tomorrow might either rain, snow, or hail, we * aren't * saying that it will first rain, then snow, then hail. Only one of them will actually happen.
Rather we are discussing the set of possibilities {rain, snow, hail}, which has no intrinsic order, and in this case has a cardinality of 3.
Similarly with the set of all possible futures. If we let fi represent a possible future where someone shouts the number i, then the set of all possible futures is {f1, f2, f3, ... }, which has cardinality ∞ and again no intrinsic ordering. We aren't saying here that a single person will shout all numbers between 1 and ∞, because as with the weather example, we're talking about what might possibly happen, not what actually happens.
No this is wrong. We don't consider physical constraints when constructing the set of future possibilities - physical constraints come into the picture later. So in the weather example, we could include into our set of future possibilities something absurd, and which violates known laws of physics. For example we are free to construct a set like {rain, snow, hail, rains_frogs}.
Then we factor in physical constraints by assigning probability 0 to the absurd scenario. For example our probabilities might be {0.2,0.4,0.4,0}.
But no laws of physics are being violated with the scenario "someone shouts the natural number i". This is why this establishes a one-to-one correspondence between the set of future possibilities and the natural numbers, and why we can say the set of future possibilities is (at least) countably infinite. (You could establish that the set of future possibilities is uncountably infinite as well by having someone shout a single digit in Cantor's diagonal argument, but that's beyond what is necessary to show that EVs are undefined.
Yes I think that the EV style-reasoning popular on this forum should be dropped entirely because it leads to absurd conclusions, and basically forces people to think along a single dimension.
So for example I'll produce some ridiculous future scenario (Vaden's x-risk: In the year 254 012 412 there will be a war over blueberries in the Qualon region of delta quadrant , which causes an unfathomable amount of infinite suffering ) and then say: great, you're free to set your credence about this scenario as high or as low as you like.
But now I've trapped you! Because I've forced you to think about the scenario only in terms of a single 1 dimensional credence-slider. Your only move is to set your credence-slider really really small, and I'll set my suffering-slider really really high, and then using EVs, get you to dedicate your income and the rest of your life to Blueberry-Safety research.
Note also that EV style reasoning is only really popular in this community. No other community of researchers reasons in this way, and they're able to make decisions just fine. How would any other community reason about my scenario? They would reject it as absurd and be done with it. Not think along a single axis (low credence/high credence).
That's the informal answer, anyway. Realizing that other communities don't reason in this way and are able to make decisions just fine should at least be a clue that dropping EV style arguments isn't going to result in decision-paralysis.
The more formal answer is to consider using an entirely different epistemology, which doesn't deal with EVs at all. This is what my vague comments about the 'framework' were eluding to in the piece. Specifically, I have in mind Karl Popper's critical rationalism, which is at the foundation of modern science. CR is about much more than that, however. I discuss what a CR approach to decision making would look like in this piece if you want some longer thoughts on it.
But anyway, I digress... I don't expect people to jettison their entire worldview just because some random dude on the internet tells them to. But for anyone reading who might be curious to know where I'm getting a lot of these ideas from (few are original to me), I'd recommend Conjectures and Refutations. If you want to know what an alternative to EV style reasoning looks like, the answers are in that book.
(Note: This is a book many people haven't read because think they already know the gist. "Oh, C&R! That's the book about falsification, right?" It's about much much more than that :) )
Hi Vaden, thanks again for posting this! Great to see this discussion. I wanted to get further along C&R before replying, but:
If we're assuming that time is finite and quantized, then wouldn't these assumptions (or, alternatively, finite time + the speed of light) imply a finite upper bound on how many syllables someone can shout before the end of the universe (and therefore a finite upper bound on the size of the set of shoutable numbers)? I thought Isaac was making this point; not that it's physically impossible to shout all natural numbers sequentially, but that it's physically impossible to shout any of the natural numbers (except for a finite subset).
(Although this may not be crucial, since I think you can still validly make the point that Bayesians don't have the option of, say, totally ruling out faster-than-light number-pronunciation as absurd.)
Are they? I had the impression that most communities of researchers are more interested in finding interesting truths than in making decisions, while most communities of decision makers severely neglect large-scale problems. (Maybe there's better ways to account for scope than EV, but I'd hesitate to look for them in conventional decision making.)
I meant if everyone were actively engaged in this project. (I think there are plenty of people in the world who are just getting on with their thing, and some of them make the world a bit worse rather than a bit better.)
Overall though I think that longtermism is going to end up with practical advice which looks quite a lot like "it is the duty of each generation to do what it can to make the world a little bit better for its descendants"; there will be some interesting content in which dimensions of betterness we pay most attention to (e.g. I think that the longtermist lens on things makes some dimension like "how much does the world have its act together on dealing with possible world-ending catastrophes?" seem really important).
Goodness, I really hope so. As it stands, Greaves and MacAskill are telling people that they can “simply ignore all the effects [of their actions] contained in the first 100 (or even 1000) years”, which seems rather far from the practical advice both you and I hope they arrive at.
Anyway, I appreciate all your thoughtful feedback - it seems like we agree much more than we disagree, so I’m going to leave it here :)
I think the crucial point of outstanding disagreement is that I agree with Greaves and MacAskill that by far the most important effects of our actions are likely to be temporally distant.
I don't think they're saying (and I certainly don't think) that we can ignore the effects of our actions over the next century; rather I think those effects matter much more for their instrumental value than intrinsic value. Of course, there are also important instrumental reasons to attend to the intrinsic value of various effects, so I don't think intrinsic value should be ignored either.
In their article vadmas writes:
Some of your comments, including this one, seem to me to be defending simple or weak longtermism ('by far the most important effects are likely to be temporally distant'), rather than strong longtermism as defined above. I can imagine a few reasons for this:
At the moment, I don't have a great sense of which one is the case, and think clarity on this point would be useful. I could also have missed an another way to reconcile these.
I think it's a combination of a couple of things.
When I said "likely", that was covering the fact that I'm not fully bought in.
Both (i) and (ii) are arguably technicalities (and I guess that the authors would cede the points to me), but (ii) in particular feels very important.
I think this is a good point, I'm really enjoying all your comments in this thread:)
It strikes me that one way that the next century effects of our actions might be instrumentally useful is that they might give some (weak) evidence as to what the longer term effects might be.
All else equal, if some action causes a stable, steady positive effect each year for the next century, then I think that action is more likely to have a positive long term effect than some other action which has a negative effect in the next century. However this might be easily outweighed by specific reasons to think that the action's longer run effects will differ.
I'm sympathetic to something in the vicinity of your complaint here, striving to compare like with like, and being cognizant of the weaknesses of the comparison when that's impossible (e.g. if someone tried the reasoning from the Shivani example in earnest rather than as a toy example in a philosophy paper I think it would rightly get a lot of criticism).
(I don't think that "subjective" and "objective" are quite the right categories here, btw; e.g. even the GiveWell estimates of cost-to-save-a-life include some subjective components.)
In terms of your general sympathy with longtermism -- it makes sense to me that the behaviour of its proponents should affect your sympathy with those proponents. And if you're thinking of the position as a political stance (who you're allying yourself etc.) then it makes sense that it could affect your sympathy with the position. But if you're engaged in the business of truth-seeking, why does it matter what the proponents do? You should ignore the bad arguments and pay attention to the best ones you can see -- whether or not anyone actually made them. (Of course I'm expressing a super idealistic position here, and there are practical reasons not to be all the way there, but I still think it's worth thinking about.)
If someone who I have trusted with working out the answer to a complicated question makes an error that I can see and verify, I should also downgrade my assessment of all their work which might be much harder for me to see and verify.
Related: Gell-Mann Amnesia
(Edit: Also related, Epistemic Learned Helplessness)
The correct default response to this effect, in my view, mostly does not look like 'ignoring the bad arguments and paying attention to the best ones'. That's almost exactly the approach the above quote describes and (imo correctly) mocks; ignoring the show business article because your expertise lets you see the arguments are bad and taking the Palestine article seriously because the arguments appear to be good.
I think the correct default response is something closer to 'focus on your areas of expertise, and see how the proponents conduct themselves within that area. Then use that as your starting point for guessing at their accurracy in areas which you know less well'.
I appreciate stuff like the above is part of why you wrote this. I still wanted to register that I think this framing is backwards; I don't think you should evaluate the strength of arguments across all domains as they come and then adjust for trustworthiness of the person making them; in general I think it's much better (measured by believing more true things) to assess the trustworthiness of the person in some domain you understand well and only then adjust to a limited extent based on the apparent strength of the arguments made in other domains.
It's plausible that this boils down to a question of 'how good are humans at assessing the strength of arguments in areas they know little about'. In the ideal, we are perfect. In reality, I think I am pretty terrible at it, in pretty much exactly the way the Gell-Mann quote describes, and so want to put minimal weight on those feelings of strength; they just don't have enough predictive power to justify moving my priors all that much. YMMV.
I appreciate the points here. I think I might be slightly less pessimistic than you about the ability to evaluate arguments in foreign domains, but the thrust of why I was making that point was because: I think for pushing out the boundaries of collective knowledge it's roughly correct to adopt the idealistic stance I was recommending; & I think that Vaden is engaging in earnest and noticing enough important things that there's a nontrivial chance they could contribute to pushing such boundaries (and that this is valuable enough to be encouraged rather than just encouraging activity that is likely to lead to the most-correct beliefs among the convex hull of things people already understand).
Ah, gotcha. I agree that the process of scientific enquiry/discovery works best when people do as you said.
I think it’s worth distinguishing between that case where taking the less accurate path in the short-term has longer-term benefits, and more typical decisions like ‘what should I work on’, or even just truth-seeking that doesn’t have a decision directly attached but you want to get the right answer. There are definitely people who still believe what you wrote literally in those cases and ironically I think it’s a good example of an argument that sounds compelling but is largely incorrect, for reasons above.
Just wanted to quickly hop in to say that I think this little sub-thread contains interesting points on both sides, and that people who stumble upon it later may also be interested in Forum posts tagged “epistemic humility”.