The following critique is a lightly modified version of the one found here. It builds on the recent post A Case Against Strong Longtermism by Vaden Masrani, but can be read and understood independently. If you’re sick of our rants on the forum, you can also listen to a podcast episode in which Vaden and I cover similar territory - albeit more quickly and with a large helping of cheekiness. We promise to move onto other topics after this (although Vaden is now threatening a response to his post - God help us. I guess that’s why downvoting exists). Much love to the community and all its members. 

Thanks to Daniel Hageman, Vaden Masrani, and Mauricio Baker for their continual feedback and criticism as this piece evolved, and to Luke Freeman, Mira Korb, Isis Kearney, Alex HT, Max Heitmann, Gavin Acquroff, and Maximilian Negele for their comments and suggestions on earlier drafts. All errors, misrepresentations, and harsh words are my own. 

The first paragraph of the final section is stolen from an upcoming piece I wrote for GWWC. Whoops. 

TL;DR: Focusing on the long-term destroys the means by which we make progress — moral and otherwise. 


The new moral philosophy of longtermism has staggering implications if widely adopted. In The Case for Strong Longtermism, Hilary Greaves and Will MacAskill write 

The idea, then, is that for the purposes of evaluating actions, we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1000) years, focussing primarily on the further-future effects. Short-run effects act as little more than tie-breakers. (pg. 1; italics mine)

The idea energizing this philosophy is that most of our “moral value” lies thousands, millions, or even billions of years from now, because we can expect many more humans and animals to exist in the future than right now. In the words of Greaves and MacAskill: “If humanity’s saga were a novel we would still be on the very first page.” (pg. 1) 

Longtermism is causing many to question why we should be at all concerned with the near term impact of our actions. Indeed, if you are convinced by this calculus, then all current injustice, death, and suffering are little more than rounding errors in our moral calculations. Why care about parasitic worms in Africa if we can secure utopia for future generations

EA has yet to take irreversible action based on these ideas, but the philosophy is gaining traction and therefore deserves an equal amount of criticism. There have been millions donated to the cause of improving the long-term future: at the time of writing the Long-Term Future Fund has received just under $4.5 million USD in total, and the Open Philanthropy Project has dedicated a focus area to this cause in the form of “risks from advanced artificial intelligence.” While many millions more are still funneled through GiveWell, The Life You Can Save, and Animal Charity Evaluators, should Greaves and MacAskill prove sufficiently persuasive such “near-term” efforts could vanish: “If society came to adopt these views, much of what we would prioritise in the world today would change.” (pg. 3)

This post is a critique of longtermism as expounded in The Case for Strong Longtermism. Prior criticism of the idea has typically revolved around the intractability objection which, while agreeing that the long-term future should dominate our moral concerns, argues we can’t have any reliable effect on it. While correct, it lets longtermism off far too lightly. It does not criticize it as a moral ideal, but rather as something good but unrealizable. 

The recent essay by Vaden Masrani does attempt to refute the two premises on which strong longtermism is founded. It argues that (i) the mathematics involved in the expected value calculations over possible futures are fundamentally flawed — indeed, meaningless — and (ii) that we should be biased towards the present because it is the only thing we know how to reliably affect. My criticisms will build on these. 

I will focus on two aspects of strong longtermism, henceforth simply longtermism. First, the underlying arguments inoculate themselves from criticism by using arbitrary assumptions on the number of future generations. Second, ignoring short-term effects destroys the means by which we make progress — moral, scientific, artistic, and otherwise. In other words, longtermism is a dangerous moral ideal because it robs us of the ability to correct our mistakes.  

Since the critique may come across as somewhat harsh, it’s worth spending a moment to frame  it. 


My assailment of longtermism comes from a place of deep sympathy with and general support of the ideals of effective altruism. The community has both generated and advocated many great ideas, including evaluating philanthropic efforts based on impact rather than emotional valence, acknowledging that “doing good” is a difficult resource-allocation problem, and advocating an ethical system grounded in impartiality across all sentient beings capable of suffering. Calling attention to farmed animal welfare, rigorously evaluating charities, and encouraging the privileged among us to donate our wealth, have all been hugely important initiatives. Throughout its existence, EA has rightly rejected two forms of authority which have traditionally dominated the philanthropic space: emotional and religious authority. 

It has, however, succumbed to a third — mathematical authority. Firmly grounded in Bayesian epistemology, the community is losing its ability to step away from the numbers when appropriate, and has forgotten that its favourite tools — expected value calculations, Bayes theorem, and mathematical models — are precisely that: tools. They are not in and of themselves a window onto truth, and they are not always applicable. Rather than respect the limit of their scope, however, EA seems to be adopting the dogma captured by the charming epithet shut up and multiply.

EA is now at risk of adopting a bad idea; one that if fully subscribed to, I fear will lead to severe and irreversible damage — not only to the movement, but to the many people and animals whose suffering would be willfully ignored. As will be elaborated on later, rejecting longtermism will not cause a substantial shift in current priorities; many of the prevailing causes will remain unaffected. If, however, longtermism is widely adopted and its logic taken seriously, many of EA’s current priorities would be replaced with vague and arbitrary interventions to improve the course of the long-term future.  

Let’s begin by examining the kinds of reasoning used to defend the premises of longtermism. 

Irrefutable Reasoning

“For the purposes of this article”, write Greaves and MacAskill, 

we will generally make the quantitative assumption that there are, in expectation, at least 1 quadrillion (10^15) people to come — 100,000 times as many people in the future as are alive today. This we [sic] be true if, for example, we assign at least a 1% chance to civilization continuing until the Earth is no longer habitable, using an estimate of 1 billion years’ time for that event and assuming the same per-century population as today, of approximately 10 billion people per century. (pg. 5)

This paragraph illustrates one of the central pillars of longtermism. Without positing such large numbers of future people, the argument would not get off the ground. The assumptions, however, are tremendously easy to change on the fly. Consequently, they’re dangerously impermeable to reason. Just as the astrologer promises us that “struggle is in our future” and can therefore never be refuted, so too can the longtermist simply claim that there are a staggering number of people in the future, thus rendering any counter argument mute. 

Such unfalsifiable claims lead to the following sorts of conclusions: 

Suppose that $1bn of well-targeted grants could reduce the probability of existential catastrophe from artificial intelligence by 0.001%. . . . Then the expected good done by [someone] contributing $10,000 to AI [artificial intelligence] safety would be equivalent . . . to one hundred thousand lives saved. (pg. 14)

Of course, it is impossible to know whether $1bn of well-targeted grants could reduce the probability of existential risk, let alone by such a precise amount. The “probability” in this case thus refers to someone’s (entirely subjective) probability estimate — “credence” — a number with no basis in reality and based on some ad-hoc amalgamation of beliefs. Notice that if one shifted one’s credence from 0.001% to 0.00001%, donating to AI safety would still be more than twice as effective as donating to the Against Malaria Foundation (AMF) (using GiveWell’s 2020 estimates). 

A reasonable retort here is that all estimates in this space necessarily include a certain amount of uncertainty. That, for example, the difference between GiveWell’s estimates and those for AI risk are a matter of degree, not of kind. This is correct — the differences are a matter of degree. But each of those degrees introduces more subjectivity and arbitrariness into the equation. Our incredulity and skepticism should rise in equal measure. 

GiveWell’s estimates use real, tangible, collected data. Other studies may of course conflict with their findings, in which case we’d have work to do. Indeed, such criticism would be useful for it would force GiveWell to develop more robust estimates. Needless to say, this process is entirely different than assigning arbitrary numbers to events about which we are utterly ignorant. My credence could be that working on AI safety will reduce existential risk by 5% and yours could be 10^-19%, and there’s no way to discriminate between them. Appealing to the beliefs of experts in the field does not solve the problem. From which dataless, magical sources are their beliefs derived? 

Moreover, should your credence be 10^-19% in the effectiveness of AI Safety interventions, then I can still make that intervention look arbitrarily good, simply by increasing the “expected number of humans” in the future. Indeed, in his book Superintelligence, Nick Bostrom has “estimated” that there could be 10^64 sentient beings in the future. By those lights, the expected number of lives, even with a credence of 10^-19%, is still positively astronomical. 

As alluded to above, the philosophy validating the reliance on subjective probability estimates is called Bayesian epistemology. It frames the search for knowledge in terms of beliefs (which we quantify with numbers, and must update in accordance with Bayes rule, else risk rationality-apostasy!). It has imported valid statistical methods used in economics and computer science, and erroneously applied them to epistemology, the study of knowledge creation. It’s ill-defined, is based on confirmation as opposed to falsification, leads to paradoxes, and relies on the provably false probabilistic induction. In other words, it has been refuted, and yet, somehow manages to stick around (ironically, it’s precisely this aspect of Bayesianism which is so dubious: its inability to reject any hypothesis). 

Bayesian epistemology unhelpfully borrows standard mathematical notation. Thus, subjective credences tend to be compared side-by-side with statistics derived from actual data, and treated as if they were equivalent. But prophecies about when AGI will take over the world — even when cloaked in advanced mathematics — are of an entirely different nature than, say, impact evaluations from randomized controlled trials. They should not be treated as equivalent. 

Once one adopts Bayesianism and loses track of the different origins of various predictions, then the attempt to compare cause areas becomes a game of “who has the bigger number.” And longtermism will win this game. Every time. It becomes unavoidable because it abolishes the means by which one can disagree with its conclusion, because it can always simply use bigger numbers. But we must remind ourselves that the numbers used in longtermist calculations are not the same as those derived from actual data. We should remember that mathematics is not an oracle unto truth. It is a tool, and one that in this case is inappropriately used. There are insufficient constraints when reasoning based solely on beliefs and big numbers — it is not informative and is not in any way tethered to a real data set, or to reality. Just as we discard poor, unfalsifiable, justifications in other areas, so too should we dispense with them in moral reasoning. 

The Antithesis of Moral Progress

If you wanted to implement a belief structure which justified unimaginable horrors, what sort of views would it espouse? A good starting point would be to disable our critical capacities from evaluating the consequences of our actions, most likely by appealing to some vague and distant glorious future lying in wait. And indeed, this tool has been used by many horrific ideologies in the past. 

Definitely and beyond all doubt, our future or maximum program is to carry China forward to socialism and communism. Both the name of our Party and our Marxist world outlook unequivocally point to this supreme ideal of the future, a future of incomparable brightness and splendor

- Mao Tse Tung, “On Coalition Government”. Selected Works, Vol. III, p. 282. (italics mine)

Of course, the parallel between longtermism and authoritarianism is a weak one, if only because longtermism has yet to be instantiated. I don’t doubt that longtermism is rooted in deep compassion for those deemed to be ignored by our current moral frameworks and political processes. Indeed, I know it is, because the EA community is filled with the most kind-hearted people I’ve ever met.  

Inadvertently, however, longtermism is almost tailor-made to disable the mechanisms by which we make progress.

Progress entails solving problems and generating the knowledge to do so. Because humans are fallible and our ideas are prone to error, our solutions usually have unintended negative consequences. These, in turn, become new problems. We invent pain relief medications which facilitate an opioid epidemic. We create the internet which leads to social media addiction. We invent cars which lead to car accidents. This is not to say we would have been better off not solving problems (of course we wouldn’t), only that solutions beget new — typically less severe — problems. This is a good thing. It’s the sign of a dynamic, open society focused on implementing good ideas and correcting bad ones. 

Moral progress is no different. Abstract reasoning from first principles can be useful, but it will only get you so far. No morality prior to the industrial revolution could have foreseen the need to introduce eight-hour workdays or labour laws. No one 1,000 years ago could have foreseen factory farming, child-pornography spread via the internet, or climate change. As society changes, it is crucial that we maintain the ability to constantly adapt and evolve our ethics in order to handle new situations. 

The moral philosophy espoused by EA should be one focused on highlighting problems and solving them. On being open to changing our ideas for the better. On correcting our errors.

Longtermism is precisely the opposite. By “ignoring the effects contained in the first 100 (or even 1000) years,” we ignore problems with the status quo, and hamstring our efforts to create solutions. If longtermism had been adopted 100 years ago then problems like factory farming, HIV/AIDS, and Measles would have been ignored. Greaves and MacAskill argue that we should have no moral discount factor, i.e., a “zero rate of pure time preference”. I agree — but this is besides the point. While time is morally irrelevant, it is relevant for solving problems. Longtermism asks us to ignore problems now, and focus on what we believe will be the biggest problems many generations from now. Abiding by this logic would result in the stagnation of knowledge creation and progress. 

It is certainly possible to accuse me of taking the phrase “ignoring the effects” too literally.  Perhaps longtermists wouldn’t actually ignore the present and its problems, but their concern for it would be merely instrumental. In other words, longtermists may choose to focus on current problems, but the reason to do so is out of concern for the future. 

My response is that attention is zero-sum. We are either solving current pressing problems, or wildly conjecturing what the world will look like in tens, hundreds, and thousands of years. If the focus is on current problems only, then what does the “longtermism” label mean? If, on the other hand, we’re not only focused on the present, then the critique holds to whatever extent we’re guessing about future problems and ignoring current ones. We cannot know what problems the future will hold, for they will depend on the solutions to our current problems which, by definition, have yet to be discovered. The best we can do is safe-guard our ability to make progress and to correct our mistakes. 

In sum, given the need for a constantly evolving ethics, one of our most important jobs is to ensure that we can continue criticizing and correcting prevailing moral views. The focus on the long-term future, however, stops the means by which we can obtain feedback about our actions now — the only reliable way to improve our current moral theories. Moral principles, like all ideas, evolve over time according to the pressure exerted on them by criticism. The ability to criticize, then, is paramount to making progress. Disregarding current problems and suffering renders longtermism impermeable to error-correction. Thus, while the longtermist project may arise out of more compassion for sentient beings than many other dogmas, it has the same nullifying effect on our critical capacities. 

What now?

We are at an unprecedented time in history: We can do something about the abundance of suffering around us. For most of the human story, our ability to eradicate poverty, cure disease, and save lives was devastatingly limited. We were hostages to our environments, our biology, and our traditions. Finally however, trusting in our creativity, we have developed powerful ideas on how to improve life. We now know of effective methods to prevent malaria, remove parasitic worms, prevent vitamin deficiencies, and provide surgery for fistula. We have the technology to produce clean-meat to reduce animal suffering. We constructed democratic institutions to protect the vulnerable and reduce conflict. These are all staggering feats of human ingenuity.  

Longtermism would have us disavow this tradition of progress. We would stop solving the problems in front of us, only to focus on distant problems obscured by the impenetrable wall of time. 

For what it’s worth, should the EA community abandon longtermism, I think many of its current priorities would remain unchanged; long-term causes do not yet dominate its portfolio. Causes such as helping the global poor and reducing suffering from factory farming would of course remain a priority. So too would interventions such as improving institutional decision making and reducing the threat of nuclear war and pandemics. Such causes are important because the problems exist and do not require arbitrary assumptions on the number of future people. 

My goal is not necessarily to change the current focus of the EA community, but rather to criticize the beginnings of a philosophy which has the potential to upend the values which made it unique in the first place: the combination of compassion with evidence and reason. It is in danger of discarding the latter half of that equation.  


New Comment
99 comments, sorted by Click to highlight new comments since: Today at 3:36 AM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

My credence could be that working on AI safety will reduce existential risk by 5% and yours could be 10^-19%, and there’s no way to discriminate between them.

We can look at their track record on other questions, and see how reliably (or otherwise) different people's predictions track reality.

I agree that below a certain level (certainly by 10^-19, and possibly as high as 10^-3) direct calibration-in-practice becomes somewhat meaningless. But we should be pretty suspicious of people claiming extremely accurate probabilities at the 10^-10 mark if they aren't even accurate at the 10^-1 mark. 

In general I'm not a fan of this particular form of epistemic anarchy where people say that they can't know anything with enough precision under uncertainty to give numbers, and then act as if their verbal non-numeric intuitions are enough to carry them through consistently making accurate decisions. 

It's easy to lie (including to yourself) with numbers, but it's even easier to lie without them.

Hi Linch!  I'd rather not rely on the authority of past performance to gauge whether someone's arguments are good. I think we should evaluate the arguments directly. If they are, they'll stand on their own regardless of someone's prior luck/circumstance/personality.  I would actually argue that it's the opposite of epistemic anarchy. Admitting that we can't know the unknowable changes our decision calculus: Instead of focusing on making the optimal decision, we recognize that all  decisions will have unintended negative consequences which we'll have to correct. Fostering an environment of criticism and error-correction becomes paramount.  I'd disagree. I think trying to place probabilities on inherently unknowable events lends us a false sense of security.  (All said with a smile of course :) ) 
You or other readers might find this post of mine from last year of interest: Potential downsides of using explicit probabilities []. The potential downsides I cover include causing overconfidence, underestimating the value of information, and anchoring, among other things that are less directly related to your point. That said, I ultimately conclude that: Relatedly, I think it's not at all obvious that putting numbers on things, forecasting, etc. would tend to get in the way of "Fostering an environment of criticism and error-correction becomes paramount". (It definitely could get in the way sometimes; it depends on the details.) There are various reasons why putting numbers on things and making forecasts can be actively helpful in fostering such an environment (some of which I discuss in my post).
[Disclaimer that I haven't actually read your post yet - sorry! - though I may do so soon :)] I agree that we should often/usually evaluate arguments directly. But: * We have nowhere near enough time to properly evaluate all arguments relevant to our decisions. And in some cases, we also lack the relevant capabilities. So in effect, it's often necessary and/or wise to base certain beliefs mostly on what certain other people seem to believe. * For example, I don't actually know that much about how climate science works, and my object-level understanding of the arguments for climate change being real, substantial, and anthropogenic are too shallow for me to be confident - on that basis alone - that those conclusions are correct. (I think a clever person could've made false claims about climate science sound similarly believable to me, if they'd been motivated to do so and I'd only looked into it to the extent that I have.) * The same is more strongly true for people with less education and intellectual curiosity than me. * But it's good for us to default to being fairly confident that things most relevant scientists agree are true are indeed true. * The same basic point is even more clearly true when it comes to things like the big bang or the fact that dinosaurs existed and when they did so * See also epistemic humility [] * We can both evaluate arguments directly and consider people's track records * We could also evaluate the "meta argument" that "people who have been shown to be decent forecasters (or better forecasters than other people are) on relatively short time horizons will also be at least slightly ok forecasts (or at least slightly better forecasters than other people are) on relatively long time horizons" * Evaluating that argument directly, I think we should land on "This seems more likely t

EA has yet to take irreversible action based on these ideas, but the philosophy is gaining traction and therefore deserves an equal amount of criticism. There have been millions donated to the cause of improving the long-term future: at the time of writing the Long-Term Future Fund has received just under $4.5 million USD in total, and the Open Philanthropy Project has dedicated a focus area to this cause in the form of “risks from advanced artificial intelligence.” While many millions more are still funneled through GiveWell, The Life You Can Save, and Animal Charity Evaluators


For what it’s worth, should the EA community abandon longtermism, I think many of its current priorities would remain unchanged; long-term causes do not yet dominate its portfolio.


I appreciate this tangential to the main point of the post, but these asides strike me as (unintentionally) likely to leave the reader with a common-but-inaccurate impression, and I think it's worth correcting this impression as it arises in the name of general integrity and transparency. 

Specifically, I think a reader of the above without further context would assume that longtermism is very new (say <2 years old... (read more)

Thanks AGB, this is helpful.  I agree that longtermism is core part of the movement, and probably commands a larger share of adherents than I imply. However, I'm not sure to what extent strong longtermism is supported. My sense is that while most people agree with the general thrust of the philosophy, many would be uncomfortable with "ignoring the effects" of the near term, and remain focused on near-term problems. I didn't want to claim that a majority of EAs  supported longtermism broadly-defined, but then only criticize a subset of those views.  I hadn't seen the results of the EA Survey - fascinating. 
I know I'm late to the discussion, but… I agree with AGB's comment, but I would also like to add that strong longtermism seems like a moral perspective with much less "natural" appeal, and thus much less ultimate growth potential, than neartermist EA causes such as global poverty reduction or even animal welfare. For example, I'm a Program Officer in the longtermist part of Open Philanthropy, but >80% of my grantmaking [] dollars go to people who are not longtermists (who are nevertheless doing work I think is helpful for certain longtermist goals). Why? Because there are almost no longtermists anywhere in the world, and even fewer who happen to have the skills and interests that make them a fit for my particular grantmaking remit. Meanwhile, Open Philanthropy makes far more grants in neartermist causes (though this might change in the future), in part because there are tons of people who are excited about doing cost-effective things to help humans and animals who are alive and visibly suffering today, and not so many people who are excited about trying to help hypothetical people living millions of years in the future. Of course to some degree this is because longtermism is fairly new, though I would date it at least as far back as Bostrom's "Astronomical Waste" paper from 2003. I would also like to note that many people I speak to who identify (like me) as "primarily longtermist" have sympathy (like me) for something like "worldview diversification []," given the deep uncertainties involved in the quest to help others as much as possible. So e.g. while I spend most of my own time on longtermism-motivated efforts, I also help out with other EA causes in various ways (e.g. this giant project on animal sentience []), and I link to or talk positively about GiveW
Yikes... now I'm even more worried ... :| 

Thanks so much for writing this Ben! I think it's great that strong longtermism is being properly scrutinised, and I loved your recent podcast episode on this (as well as Vaden's piece).

I don't have a view of my own yet; but I do have some questions about a few of your points, and I think I can guess at how a proponent of strong longtermism might respond to others.

For clarity, I'm understanding part of your argument as saying something like the following. First, "[E]xpected value calculations, Bayes theorem, and mathematical models" are tools — often useful, often totally innapropriate or inapplicable. Second, 'Bayesian epistemology' (BE) makes inviolable laws out of these tools, running into all kinds of paradoxes and failing to represent how scientific knowledge advances. This makes BE silly at best and downright 'refuted' at worst. Third, the case for strong longtermism relies essentially on  BE, which is bad news for strong longtermism.

I can imagine that a fan of BE would just object that Bayesianism in particular is just not a tool which can be swapped out for something else when it's convenient . This feels like an important but tangential argument — this LW post might b... (read more)

5Neel Nanda2y
I think there is an important point here. One of the assumptions in Aumann's theorem is that both people have the same prior, and I think this is rarely true in the real world.  I roughly think of Bayesian reasoning as starting with a prior, and then adjusting the prior based on observed evidence. If there's a ton of evidence, and your prior isn't dumb, the prior doesn't really matter. But the more speculative the problem, and the less available evidence, the more the prior starts to matter. And your prior bakes in a lot of your assumptions about the world, and I think it's tricky to resolve disagreements about what your prior should be. At least not in ways that approach being objective.  I think you can make progress on this. Eg, 'how likely is it that AI could get way better, really fast?' is a difficult question to answer, and could be baked into a prior either way. And things like AI Impact's study of discontinuous progress in other technologies [] can be helpful for getting closer to consensus. But I think choosing a good prior is still a really hard and important problem, and near impossible to be objective about
Hey Fin! Nice - lot's here. I'll respond to what  I can. If I miss anything crucial just yell at me :) (BTW, also enjoying your podcast. Maybe we should have a podcast battle at some point ... you can defend longtermism's honour).  Yep, this is fair. I'm imagining myself in the position of some random stranger outside of a fancy EA-gala, and trying to get people's attention. So yes - the language might be a little strong (although I do really think Bayesianism doesn't stand up to scrutiny if you drill down on it).  Sure, guessing that there will be between 1 billion and 1000 quadrillion people in the future is probably a better estimate than 1000 people. But it still leaves open a discomfortingly huge ran. Greaves and MacAskill could easily have used half a quadrillion people, or 10 quadrillion people. Instead of trying to wrestle with this uncertainty, which is fruitless, we should just acknowledge that we can't know and stop trying.   Bit of a nitpick here, but space colonization isn't prohibited  by the laws of physics, so it can only be "practically impossible" based on our current knowledge. It's just a problem to be solved. So this particular example couldn't bring down the curtains on our expected value calculations.  I don't think so. There's no data on the problem, so there's nothing to adjudicate between our disagreements. We can honestly try this if you want. What's your credence?  Now, even if we could converge on some number, what's the reason for thinking that number captures any aspect of reality? Most academics were sympathetic to communism before it was tried; most physicists thought Einstein was wrong.  What are the available facts when it comes to the size of the future? There's a reason these estimates are wildly different across papers: From 10^15 here, to 10^68 (or something) from Bostrom, and everything in between. I'm gonna add mine in: 10^124 + 3.  Agree that this is probably the response. But then we need to be clear that these estim
6Jack Malde2y
I don't think this is true. Whenever Greaves and MacAskill carry out a longtermist EV calculation in the paper it seems clear to me that their aim is to illustrate a point rather than calculate a reliable EV of a longtermist intervention. Their world government EV calculation starts with the words "suppose that...". They also go on to say: This is the point they are trying to get across by doing the EV calculations.
Thanks for replying Ben, good stuff! Few thoughts. I'll concede that point! I think a better response to the one I originally gave was to point out that the case for strong longtermism relies on establishing a sensible lower(ish) bound for total future population. Greaves and MacAskill want to convince you that (say) at least a quadrillion lives could plausibly lie in the future. I'm curious if you have an issue with that weaker claim? I think your point about space exploration is absolutely right, and more than a nitpick. I would say two things: one is that I can imagine a world in which we could be confident that we would never colonise the stars (e.g. if the earth were more massive and we had 5 decades before the sun scorched us or something). Second, voicing support for the 'anything permitted by physics can become practically possible' camp indirectly supports an expectation of a large numbers of future lives, no? Hmm — to my lights Greaves and MacAskill are fairly clear about the differences between the two kinds of estimate. If your reply is that doing any kind of (toy) EV calculation with both estimates just implies that they're somehow "equally as capable of capturing something about reality", then it feels like you're begging the question. I don't understand what you mean here, which is partly my fault for being unclear in my original comment. Here's what I had in mind: suppose you've run a small-scale experiment and collected your data. You can generate a bunch of statistical scores indicating e.g. the effect size, plus the chance of getting the results you got assuming the null hypothesis was true (p-value). Crucially (and unsurprisingly) none of those scores directly give you the likelihood of an effect (or the 'true' anything else). If you have reason to expect a bias in the direction of positive results (e.g. publication bias), then your guess about how likely it is that you're picked up on a real effect may in fact be very different from any sta

I share your concerns with using arbitrary numbers and skepticism of longtermism, but I wonder if your argument here proves too much. It seems like you're acting as if you're confident that the number of people in the future is not huge, or that the interventions are otherwise not so impactful (or they do more harm than good), but I'm not sure you actually believe this. Do you? 

It sounds like you're skeptical of AI safety work, but it also seems what you're proposing is that we should be unwilling to commit to beliefs on some questions (like the number of people in the future), and then deprioritize longtermism as a result, but, again, doing so means acting as if we're committed to beliefs that would make us pessimistic about longtermism.

I think it's more fair to think that we don't have enough reason to believe longtermist work does much good at all, or more good than harm (and generally be much more skeptical of causal effects with little evidence), than it is to be extremely confident that the future won't be huge.

I think you do need to entertain arbitrary probabilities, even if you're not a longtermist, although I don't think you should commit to a single joint probability... (read more)

Hi Michael!  I have no idea about the number of future people. And I think this is the only defensible position. Which interventions do you mean? My argument is that longtermism enables reasoning that de-prioritizes current problems in lieu of possible, highly uncertain, future problems. Focusing on such problems prohibits us from making actual progress.  I'm not quite sure I'm following this criticism, but I think it can be paraphrased as: You refuse to commit to a belief about x, but commit to one about y and that's inconsistent. (Happy to revise if this is unfair.) I don't think I agree - would you commit to a belief about what Genghis Khan was thinking on his 17th birthday? Some things are unknowable, and that's okay. Ignorance is par for the course. We don't need to pretend otherwise. Instead, we need a philosophy which is robust to uncertainty which, as I've argued, is one that focuses on correcting mistakes and solving the problems in front of us.   ... but they'd be arbitrary, so by definition don't tell us anything about the world?  This is of course a difficult question. But I don't think the answer is to assign arbitrary numbers to the consciousness of animals. We can't pull knowledge out of a hat, even using the most complex maths possible. We have theories of neurophysiology, and while none of them conclusively tells us that animals definitely feel pain, I think that's the best explanation of our current observations. So, acknowledging this, we are in a situation where billions of animals needlessly suffer every year according to our best theory. And that's a massive, horrendous tragedy - one that we should be fighting hard to stop. Assigning credences to the consciousness of animals just so we can start comparing this to other cause areas is just pretending knowledge where we have none. 

You refuse to commit to a belief about x, but commit to one about y and that's inconsistent.

I would rephrase as "You say you refuse to commit to a belief about x, but seem to act as if you've committed to a belief about x". Specifically, you say you have no idea about the number of future people, but it seems like you're saying we should act as if we believe it's not huge (in expectation). The argument for strong longtermism you're trying to undermine (assuming we get the chance of success and sign roughly accurate, which to me is more doubtful) goes through for a wide range of numbers. It seems that you're committed to the belief that expected number is less than , say, since you write in response "This paragraph illustrates one of the central pillars of longtermism. Without positing such large numbers of future people, the argument would not get off the ground".

Maybe I'm misunderstanding. How would you act differently if you were confident the number was far less than  in expectation, say  (about 100 times the current population), rather than have no idea?

I don't think I agree - would you commit to a belief about what Genghis Khan was thinking

... (read more)

Thanks for taking the time to write this :)

In your post you say "Of course, it is impossible to know whether $1bn of well-targeted grants could reduce the probability of existential risk, let alone by such a precise amount. The “probability” in this case thus refers to someone’s (entirely subjective) probability estimate — “credence” — a number with no basis in reality and based on some ad-hoc amalgamation of beliefs."

I just wanted to understand better: Do you think its ever reasonable to make subjective probability estimates (have 'credences') over things? If so, in what scenarios is it reasonable to have such subjective probability estimates; and what makes those scenarios different from the scenario of forming a subjective probability estimate of what $1bn in well-target grants could do to reduce existential risk?

Hey James!  Answering this in its entirety would take a few more essays, but my short answer is: When there are no data available, I think subjective probability estimates are basically useless, and do not help in generating knowledge.  I emphasize the condition when there are no data available because data is what  allows us to discriminate between different models. And when data is available, well, estimates become less subjective.  Now, I should say that I don't really care what's "reasonable" for someone to do - I definitely don't want to dictate how someone should think about problems. (As an aside, this is a pet peeve of mine when it comes to Bayesianism -it tells you how you must think  in order to be a rational person. As if rationality was some law of nature to be obeyed.) In fact, I want people thinking about problems in many different ways. I want Eliezer Yudkowski applying Bayes' rule and updating in strict accordance with the rules of probability, you being inspired by your fourth grade teacher, and me ingesting four grams of shrooms with a blindfold on in order to generate as many  ideas as possible. But how do we discriminate between these ideas? We subject them to ruthless criticism and see which ones stand up to scrutiny. Assigning numbers to them doesn't tell us anything (again, when there's no underlying data). In the piece I'm making a slightly different argument to the above, however. I'm criticizing  the tendency for these subjective estimates to be compared with estimates derived from actual data. Whether or not someone agrees with me that Bayesianism is misguided, I would hope that they still recognize the problem in comparing numbers of the form "my best guess about x" with "here's an average effect estimate with confidence intervals over 5 well-designed RTCs". 
As a major aside - there's a little joke Vaden and I tell on the podcast sometimes when talking about Bayesianism vs Criticial Rationalism (an alternative philosophy first developed by Karl Popper). The joke is most certainly a strawman of Bayesianism, but I think it gets the point across.  Bob and Alice are at the bar, being served by Carol. Bob is trying to estimate whether Carol has children. He starts with a prior of 1/2. He then looks up the base rate of adults with children, and updates on that. Then he updates based on  Carol's age. And what car she drives. And the fact that she's married. And so on. He pulls out a napkin, does some complex math, and arrives at the following conclusion: It's 64.745% likely that Carol has children. Bob is proud of his achievement and shows the napkin to Alice.  Alice leans over the bar and asks "Hey Carol - do you have kids?".   Now, obviously this is not how the Bayesian acts in real life. But it demonstrates the care the Bayesian takes in having correct beliefs;  about having the optimal brain state. I think this is the wrong target. Instead, we should be seeking to falsify as many conjectures as possible, regardless of where the conjectures came from. I don't care what Alice thought the probability  was before she asked the question, only about the result of the test. 
Thanks for the reply and taking the time to explain your view to me :) I'm curious: My friend has been trying to estimate the liklihood of nuclear war before 2100. It seems like this is a question that is hard to get data on, or to run tests on. I'd be interested to know what you'd recommend them to do? Is there a way I can tell them to approach the question such that it relies on 'subjective estimates' less and 'estimates derived from actual data' more? Or is it that you think they should drop the research question and do something else with their time, since any approach to the question would rely on subjective probability estimates that are basically useless?
Well, far be it from me to tell others how to spend their time, but I guess it depends on what the goal is. If the goal is to literally put a precise number (or range) on the probability of nuclear war before 2100, then yes, I think that's a fruitless and impossible endeavour. History is not an  iid sequence of events. If there is such a war, it will be the result of complex geopolitical factors based on human belief, desires, and knowledge  at the time. We cannot pretend to know what these will be. Even if you were to gather all the available evidence we have on nuclear near misses,  and generate some sort of probability based on this, the answer would look something like:  "Assuming that in  2100 the world looks the same as it did during the time of past nuclear near misses, and nuclear misses are distributionally similar to actual  nuclear strikes, and [a bunch of other assumptions], then the probability of a nuclear war before 2100 is x".  We can debate the merits of such a model, but I think it's clear  that it would be of limited use.    None of this is to say that we shouldn't be working on nuclear threat, of course. There are good arguments for why this is a big problem that have nothing to do with probability and subjective credences. 

You say that "there are good arguments for working on the threat of nuclear war". As I understand your argument, you also say we cannot rationally distinguish between the claim "the chance of nuclear war in the next 100 years is 0.00000001%" and the claim "the chance of nuclear war in the next 100 years is 1%". If you can't rationally put probabilities on the risk of nuclear war, why would you work on it?

Why are probabilities prior to action - why are they so fundamental? Could Andrew Wiles "rationally put probabilities" on him solving Fermat's Last Theorem? Does this mean he shouldn't have worked on it? Arguments do not have to be in number form. 

If you refuse to claim that the chance of nuclear war up to 2100 is greater than 0.000000000001%, then I don't see how you could make a good case to work on it over some other possible intuitively trivial action, such as painting my wall blue. What would the argument be if you are completely agnostic as to whether it is a serious risk?

3Neel Nanda2y
To me, the fundamental point isn't probabilities, it's that you need to make a choice about what you do. If I have the option to give a $1mn grant to preventing nuclear war or give the grant to something else, then no matter what I do, I have made a choice. And so, I need to have a decision theory for making a choice here. And to me, subjective probabilities and Bayesian epistemology more generally, are by far the best decision theory I've come across for making choices under uncertainty. If there's a 1% chance of nuclear war, the grant is worth making, if there's a 10^-15 chance of nuclear war, the grant is not worth making. I need to make a decision, and so probabilities are fundamental, because they are my tool for making a decision. And there are a bunch of important question where we don't have data, and there's no reasonable way to get data (eg, nuclear war!). And any approach which rejects the ability to reason under uncertainty in situations like this, is essentially the decision theory of "never make speculative grants like this". And I think this is a clearly terrible decision theory (though I don't think you're actually arguing for this policy?)
Can you give some examples? I expect that someone could respond "That could be too unlikely to matter enough" to each of them, since we won't have good enough data.
Sure - Nukes exist. They've been deployed before, and we know they have incredible destructive power. We know that many  countries have them, and have threatened to use them.  We know the protocols are in place for their use. 

To me this seems like you're making a rough model with a bunch of assumptions like that past use, threats and protocols increase the risks, but not saying by how much or putting confidences or estimates on anything (even ranges). Why not think the risks are too low to matter despite past use, threats and protocols?

But we also have to make similar (although less strong) assumptions and have generalization error even with RCTs. Doesn't GiveWell make similar assumptions about the impacts of most of their recommended charities? As far as I know, there are recent studies of GiveDirectly's effects, but the "recent" studies of the effects of the interventions of the other charities have probably had their samples chosen years ago, so their effects might not generalize to new locations. Where's the cutoff for your skepticism? Should we boycott the GiveWell-recommended charities whose ongoing intervention impacts of terminal value  (lives saved, quality of life improvements) are not being measured rigorously in their new target areas, in favour of GiveDirectly? To illustrate the issue of generalization, GiveWell did a pretty arbitrary adjustment for El Niño for deworming, although I think this is the most suspect assumption I've seen them make. See Eva Vivalt []'s research on generalization (in the Causal Inference section) or her talk here [].
  Yes, we do! And the  strength of those assumptions is key. Our skepticism should rise in proportion to the number/feasibility of the assumptions. So you're definitely right, we should always be  skeptical of social science research - indeed, any empirical research. We should be looking for hasty generalizations, gaps in the analysis, methodological errors etc., and always pushing to do more research. But there's a massive difference between the assumptions driving GiveWell's models and the assumptions required in the nuclear threat example. 

Hey Ben, thanks a lot for posting this! And props for having the energy to respond to all these comments :)

I'll try to reframe points that others have made in the comments (and which I tried to make earlier, but less well): I suspect that part of why these conversations sometimes feel like we're talking past one another is that we're focusing on different things.

You and Vaden seem focused on creating knowledge. You (I'd say) correctly note that, as frameworks for creating knowledge, EV maximization and Bayesian epistemology aren't just useless--they're actively harmful, because they distract us from the empirical studies, data analysis, feedback loops, and argumentative criticism that actually create knowledge. 

Some others are focused on making decisions. From this angle,  EV maximization and Bayesian epistemology aren't supposed to be frameworks for creating knowledge--they're frameworks for turning knowledge into decisions, and your arguments don't seem to be enough for refuting them as such.

To back up a bit, I think probabilities aren't fundamental to decision making. But bets are. Every decision we make is effectively taking or refusing to take a bet (e.g. going outsi... (read more)

2.  Another worry is that probabilities are so useful that we won't find a better alternative. I think of probabilities as language for answering the earlier basic question of "What bets should I make?" For example, "There's a 25% chance (i.e. 1:3 odds) that X will happen" is (as I see it) shorthand for "My potential payoff better be at least 3 times bigger than my potential loss for betting on X to be worth it." So probabilities express thresholds in your answers to the question "What bets on event X should I take?" That is, from a pragmatic angle, subjective probabilities aren't supposed to be deep truths about the world; they're expressions of our best guesses about how willing we should be to bet on various events. (Other considerations [] also make probabilities particularly well-fitting tools for describing our preferences about bets.) So rejecting the use of probabilities (as I understand them) under severe uncertainty seems to have an unacceptable, maybe even absurd, conclusion: the rejection of consistent thresholds for deciding whether to bet on uncertain events. This is a mistake--if we accept/reject bets on some event without a consistent threshold for what reward:loss ratios are worth taking, then we'll necessarily be doing silly things like refusing to take a bet, and then accepting a bet on the same event for a less favorable reward:loss ratio. You might be thinking something like "ok, so you can always describe an action as endorsing some betting threshold, but that doesn't mean it's useful to think about this explicitly." I'd disagree, because not recognizing our betting threshold makes it harder to notice and avoid mistakes like the one above. It also takes away clarity and precision of thought that's helpful for criticizing our choices, e.g. it makes an extremely high betting threshold about the value of x-risk reduction look like agnosticism. Thanks again for your thoughtf
Hey Mauricio! Two brief comments -  Yes agreed, but these two things become intertwined when a philosophy  makes people decide to stop creating knowledge. In this case, it's longtermism preventing the creation of moral and scientific knowledge by grinding the process of error correction to a halt, where "error correction" in this context means continuously reevaluating philanthropic organizations based on their near and medium term consequences, in order to compare results obtained against results expected.  Both approaches pass on the buck, that's why I defined 'creativity' here to mean:  'whatever unknown software the brain is running to get out of the infinite regress problem.' And one doesn't  necessarily need to answer your question, because  there's no requirement that  the criticism take EV form (although it can). 
Hey Vaden, thanks! Yeah, fair. (Although less relevant to less naive applications of longterimsm, which as Ben puts it draw some rather than all of our attention away from knowledge creation.) I'm not sure I see where you're coming from here. EV does pass the buck on plenty of things (on how to generate options, utilities, probabilities), but as I put it, I thought it directly answered the question (rather than passing the buck) about what kinds of bets to make/how to act under uncertainty: Also, regarding this: I don't see how that gets you out of facing the question. If criticism uses premises about how we should act under uncertainty (which it must do, to have bearing on our choices), then a discussion will remain badly unfinished until it's scrutinized those premises. We could scrutinize them on a case-by-case basis, but that's wasting time if some kinds of premises can be refuted in general.
  Check out chapter 13 in Beginning of Infinity when you can - everything I was saying in that post is much better explained there :) 

I'd like to make a point about the potential importance of working on current problems which I'm unsure has been made yet (apologies if I've missed it).

It seems to me that there are two possibilities here:

  1. Working on current problems allows us to create moral and scientific knowledge that will help us make the long-run future go well
  2. The above isn't true

If number 1 is the case, a strong longtermist should agree with you and vadmas about the importance of working on current problems.

If number 2 is the case a strong longtermist may not agree with you about the importance of working on current problems either because they don't think that working on near term problems will generate much knowledge or because they don't think the knowledge that would be generated will help that much in making the long-run future go well. 

Now there are two points I would like to make.

Firstly, you and vadmas seem to assume number 2 is the case. It seems important to me to note that this is certainly not a given.

Secondly you and vadmas seem to think that if number 2 is the case then the conclusion that we shouldn't work on near-term problems for knowledge creation in some way demonstrates the abusurdity... (read more)

  Oops nope the exact opposite! Couldn't possibly agree more strongly with Perfect, love it, spot on. I'd be 100%  on board with longtermism if this is what it's about - hopefully conversations like these can move it there. (Ben makes this point near the end of our podcast conversation fwiw) Well, both. I do think it's intrinsically valuable to learn about reality, and I support research into fundamental physics, biology, history, mathematics, ethics etc for that reason. I think it would be intellectually impoverishing to only support research that has immediate and foreseeable practical benefits. But fortunately knowledge creation also  has enormous  instrumental value. So it's not a one-or-the other thing. 
2Jack Malde2y
I have to admit that I'm slightly confused as to where the point of contention actually is. If you believe that working on current problems allows us to create moral and scientific knowledge that will help us make the long-run future go well, then you just need to argue this case and if your argument is convincing enough you will have strong longtermists on your side. More importantly though I'm not sure people actually do in fact disagree with this. I haven't come across anyone who has publicly disagreed with this. Have you? It may be the case that both you and strong longtermists are actually on the exact same page without even realising it.
I don't consider human extermination by AI to be a 'current problem' - I think that's where the disagreement lies.  (See my blogpost [] for further comments on this point) 
Either way, the problems to work on would be chosen based on their longterm potential. It's not clear that say global health and poverty would be among those chosen. Institutional decision-making and improving the scientific process might be better candidates.
1Neel Nanda2y
I feel a bit confused reading that. I'd thought your case was framed around a values disagreement about the worth of the long-term future. But this feels like a purely empirical disagreement about how dangerous AI is, and how tractable working on it is. And possibly a deeper epistemological disagreement about how to reason under uncertainty. How do you feel about the case for biosecurity? That might help disentangle whether the core disagreement is about valuing the longterm future/x-risk reduction, vs concerns about epistemology and empirical beliefs, since I think the evidence base is noticeably stronger than for AI. I think there's a pretty strong evidence base that pandemics can happen and, eg, dangerous pathogens can get developed in labs and released from labs. And I think there's good reason to believe that future biotechnology will be able to make dangerous pathogens, that might be able to cause human extinction, or something close to that. And that human extinction is clearly bad for both the present day, and the longterm future.  If a strong longtermist looks at this evidence, and concludes that biosecurity is a really important problem because it risks causing human extinction and thus destroying the value of the longterm future, and is a thus a really high priority, would you object to that reasoning?
1Jack Malde2y
Apologies, I do still need to read your blogpost! It’s true existential risk from AI isn’t generally considered a ‘near-term’ or ‘current problem’. I guess the point I was trying to make is that a strong longtermist’s view that it is important to reduce the existential threat of AI doesn’t preclude the possibility that they may also think it’s important to work on near-term issues e.g. for the knowledge creation it would afford. Granted any focus on AI work necessarily reduces the amount of attention going towards near-term issues, which I suppose is your point.
 Yep :) 
1Jack Malde2y
This wasn't clearly worded in hindsight. What I meant by this was that I think you and Ben both seem to assume that strong longtermists don't want to work on near-term problems. I don't think this is a given (although it is of course fair to say that they're unlikely to only want to work on near-term problems).
Mostly agree here - this was the reason for some of the (perhaps cryptic) paragraphs in the Section "the Antithesis of Moral Progress." Longtermism erodes our ability to make progress to whatever extent it has us not working on real problems. And, to the extent that it does have us working on real problems,  then I'm not sure what longtermism is actually adding. Also, just a nitpick on terminology - I dislike the term "near-term" problems, because it seems to imply that there is a well-defined class of "future" problems that we can choose to work on. As if there were a set of problems, and they could be classified as either short-term or long-term. But the fact is that the only problems are near-term problems; everything else is just a guess about what the  future might hold. So I see it less about choosing what kinds of problems to work on, but a choice between working on real problems, or conjecturing about future ones, and I  think the latter is actively harmful. 
1Jack Malde2y
I don't necessarily see working on reducing extinction risk as wildly speculating about the far future. In many cases these extinction risks are actually thought to be current risks. The point is that if they happen they necessarily curtail the far future. I would note that the Greaves and MackAskill paper actually has a section putting forward 'advancing progress' as a plausible longtermist intervention! As I have mentioned this is only insofar as it will make the long-run future go well.
Agree with almost all of this. This is why it was tricky to argue against, and also why I say (somewhere? podcast maybe?) that I'm not particularly worried about the current instantiation of longtermism, but what this kind of logic could justify.  I totally agree that most of the existential threats currently  tackled by  the EA community are real problems (nuclear threats, pandemics, climate change, etc).  Yeah - but I found this puzzling. You don't need longtermism to think this is a priority - so why  adopt it? If you instead adopt a problem/knowledge focused ethics,  then you get to keep all the good aspects of longtermism (promoting progress, etc), but don't open yourself up to what (in my view) are its drawbacks. I try to say this in the "Antithesis of Moral Progress" section, but obviously did a terrible job haha :) 
3Jack Malde2y
Maybe (just maybe) we're getting somewhere here. I have no interest in adopting a 'problem/knowledge focused ethic'. That would seem to presuppose the intrinsic value of knowledge. I only think knowledge is instrumentally valuable insofar as it promotes welfare. Instead most EAs want to adopt an ethic that prioritises 'maximising welfare over the long-run'. Longtermism claims that the best way to do so is to actually focus on long-term effects, which may or may not require a focus on near-term knowledge creation - whether it does or not is essentially an empirical question. If it doesn't require it, then a strong longtermist shouldn’t consider a lack of knowledge creation to be a significant drawback.

I have a few comments on the critique of Bayesian epistemology, a lot of which I think is mistaken.

  1. You say "It frames the search for knowledge in terms of beliefs (which we quantify with numbers, and must update in accordance with Bayes rule, else risk rationality-apostasy!" I don't think anyone denies that Bayes theorem is true. It is mathematically proven.  The  most common criticism of Bayesianism is that it is "too subjective". I don't really understand what this means, but few sensible people deny Bayes theorem.
  2. "It has imported valid statist
... (read more)
Thanks for the engagement!  I think you're mistaking Bayesian epistemology with Bayesian mathematics. Of course, no one denies Bayes' theorem. The question is: to what should it be applied? Bayesian epistemology holds that rationality consists in updating your beliefs in accordance with Bayes' theorem. As this LW post []puts it:  Next, it's not that "Bayesianism is the right approach in these fields," (I'm not sure what that means) it's that Bayesian methods are useful for some problems. But Bayesianism falls short when it comes to explaining how we actually create knowledge. (No amount of updating on evidence + Newtonian mechanics gives you relativity.)  Love the ad hominem attack.  Smoking causes lung cancer is a hypothesis, smoking does not cause lung cancer is another. We then discriminate between the hypotheses based on evidence (we falsify  incorrect hypotheses). We slowly develop more and more sophisticated explanatory theories of how smoking causes lung cancer, always seeking to falsify them. At any time, we are left  with the best explanation of a given phenomenon. This is how falsification works. (I can't comment on your claim about Popper's beliefs - but I would be surprised if true. His books are filled with examples of scientific progress.)  Yes. Theories are not confirmed by evidence (there's no number of white swans you can see which confirms that all swans are white. "Swans are white" is a hypothesis, which can be refuted by  seeing a black swan), they are falsified by it. Evidence plays the role of discrimination, not confirmation. No - because we have explanatory theories telling us why we'll fall downwards (general relativity). These theories are the only ones which have survived scrutiny, which is why we abide by them. Confirmationism, on the other hand, purports to explain phenomenon by appealing to previous evidence. "Why do we fall downwards? Because we fell downwards b
He said it has zero probability but is still useful, not nonzero probability. I think you're overinterpreting the claim (or Ben's claim is misleading, based on what's cited). You don't have to give equal weight to all hypotheses. You might not even define their weights. The proof cited shows that the ratio of probabilities between two hypotheses doesn't change in light of new evidence that would be implied by both theories. Some theories are ruled out or made less likely in light of incompatible evidence. Of course, there are always "contrived" theories that survive, but it's further evidence in the future, Occam's razor or priors that we use to rule them out. This depends on your priors, which may be arbitrarily skeptical of causal effects.
4John G. Halstead2y
Yes thanks my mistake - edited above

It is certainly possible to accuse me of taking the phrase “ignoring the effects” too literally. Perhaps longtermists wouldn’t actually ignore the present and its problems, but their concern for it would be merely instrumental. In other words, longtermists may choose to focus on current problems, but the reason to do so is out of concern for the future.

My response is that attention is zero-sum. We are either solving current pressing problems, or wildly conjecturing what the world will look like in tens, hundreds, and thousands of years. If the focus is o

... (read more)
I'm tempted to just concede this because we're very close to agreement here.  If this turns out to be true (i.e., people end up working on actual problems and not, say, defunding the AMF to worry about "AI controlled police and armies"), then I have much  less of a problem with longtermism. People can use whatever method they want to decide which problems they want to work on (I'll leave the prioritization to 80K :) ).  Just apply my critique to the x% of attention that's spent worrying about non-problems. (Admittedly, of course, this world is better than the one where 100% of attention is on non-existent possible future problems.)  
5Owen Cotton-Barratt2y
I think this is might be a case of the-devil-is-in-the-details. I'm in favour of people scanning the horizon for major problems whose negative impacts are not yet being felt, and letting that have some significant impact on which nearer-term problems they wrestle with. I think that a large proportion of things that longtermists are working on are problems that are at least partially or potentially within our foresight horizons. It sounds like maybe you think there is current work happening which is foreseeably of little value: if so I think it could be productive to debate the details of that.

Thanks for writing this! I think it's important to question longtermism. I've actually found myself becoming slowly more convinced by it, but I'm still open to it being wrong. I'm looking forward to chewing on this a bit more (and you've reminded me I still have to properly read Vaden's post) but for now I will leave you with a preliminary thought.

Just as the astrologer promises us that “struggle is in our future” and can therefore never be refuted, so too can the longtermist simply claim that there are a staggering number of people in the future, thus ren

... (read more)
Hi Jack,  I think you're right, the comparison to astrology isn't entirely fair. But sometimes one has to stretch a little bit to make a point. And the point, I think, is important. Namely,  that these estimates can be manipulated and changed all too easily to fit a narrative.  Why not half a quadrillion, or 10 quadrillion people in the future?  On the falsifiability point - I agree that the claims are  technically falsifiable. I struggled with the language for this reason while writing it (and Max Heitmann helpfully tried to make this point before, but apparently I ignored him). In principle, all of their claims are falsifiable (if we go extinct, then sure, I guess we'll know how big the future will be).  Perhaps it's better if I write "easily varied" or "amenable to drastic change" in place of irrefutable/unfalsifiable?  The great filter example is interesting, actually. For if we're working in a Bayesian framework, then surely we'd assign such a hypothesis a probability. And then the number of future people could again be vast  in expectation. 
5Jack Malde2y
The fact that they can be manipulated and changed doesn't strike me as much of a criticism. The more relevant question is if people actually do manipulate and change the estimates to fit their narrative. If they do we should call out these particular people, but even in this case I don't think it would be an argument against longtermism generally, just against the particular arguments these 'manipulaters' would put forward. The authors do at least set out their assumptions for the one quadrillion which they call their conservative estimate. For example, one input into the figure is an estimate that earth will likely be habitable for another 1 billion years, which is cited from another academic text. Now I'm not saying that their one quadrillion estimate is brilliantly thought through (I'm not saying it isn't either), I'm just countering a claim I think you're making that Greaves and MacAskill would likely add zeros or inflate this number if required to protect strong longtermism e.g. to maintain that their conservative longtermist EV calculation continues to beat GiveWell's cost-effectiveness calculation for AMF. I don't see evidence to suggest they would and I personally don't think they would manipulate in such a way. That's not to say that the one quadrillion figure may not change, but I would hope and would expect this to be for a better reason than "to save longtermism". To sum up I don't think your "amenable to drastic change" point is particularly relevant. What I do think is more relevant is that the one quadrillion estimate is slightly arbitrary, and I see this as a subtly different point. I may address this in a different comment.
2Jack Malde2y
Yes if you're happy to let your calculations be driven by very small probabilities of enormous value I suppose you're right that the great filter would never be conclusive. Whether or not it is reasonable to allow this is an open question in decision theory and I don't think it's something that all longtermists accept. The authors themselves don't appear to be all that comfortable with accepting it: This implies if they think a credence is miniscule or a long-lasting influence negligible that they might throw away the calculation.

Coming from an economics background, here's how to persuade me of longtermism:

Set up a social planner problem with infinite generations and solve for the optimal allocation in each period. Do three cases:

  • A planner with nonzero time preference and perfect information
  • A (longtermist) planner with zero time preference and perfect information
  • A planner with zero time preference and imperfect information

Would the third planner ignore the utility of all generations less than 1000 years in the future? If so, then you've proved strong longtermism.

On the point about the arbitrariness of estimates of the size of the future - what is your probability distribution across the size of the future population, provided there is not an existential catastrophe?

I think you should specify a time period (e.g. the next 100 years) or feasibly preventable existential catastrophes. Could the heat death of the universe be an existential catastrophe? If so, I think the future population might be infinite, since anything less might be considered an existential catastrophe. I'm not the author of this post, but I don't have only one probability distribution for this, and I don't think there's any good way to justify any particular one (although you might rule some out for being less reasonable).
I don't  think the question makes sense.  I agree with Vaden's argument [] that there's no well-defined measure over all possible futures. 
There are definitely well-defined measures on any set (e.g. pick one atomic outcome to have probability 1 and the rest 0); there's just not only one, and picking exactly one would be arbitrary. But the same is true for any set of outcomes with at least two outcomes, including finite ones (or it's at least often arbitrary when there's not enough symmetry for equiprobability). For the question of how many people will exist in the future, you could use a Poisson distribution. That's well-defined, whether or not it's a reasonable distribution to use. Of course, trying to make your space more and more specific will run into feasibility issues.
There are non-measurable sets [,exists.] (unless you discard the axiom of choice,  but then you'll run into some significant problems.) Indeed, the existence of non-measurable sets is the reason for so much of the measure-theoretic formalism.  If you're not taking a measure theoretic approach, and instead using propositions (which  I guess, it should be assumed that you are, because this approach grounds Bayesianism), then using infinite sets (which clearly one would have to do if reasoning about all possible futures) leads to paradoxes. As E.T. Jaynes writes in Probability Theory and the Logic of Science:  (Vaden makes this point in the podcast.) 
This depends on the space.  It's at least true for real-valued intervals with continuous measures, of course, but I think you're never going to ask for the measure of a non-measurable set in real-world applications, precisely because they require the axiom of choice to construct [] (at least for the real numbers, and I'd assume, by extension, any subset of any Rn), and no natural set you'll be interested in that comes up in an application will require the axiom of choice (more than dependant choice) to construct. I don't think the existence of non-measurable sets is viewed as a serious issue for applications. It is not true in a countable measure space (or, at least, you could always extend the measure to get this to hold), since assuming each singleton (like {x},x∈X) is measurable, every union of countably many singletons is measurable, and hence every subset is measurable (A=∪x∈A{x} is a countable union of singletons, A⊆X, X countable) . In particular, if you're just interested in the number of future people, assuming there are at most countably infinitely many (so setting aside the many-worlds interpretation of quantum mechanics for now), then your space is just the set of non-negative integers, which is countable. You could group outcomes to represent them with finite sets. Bayesians get to choose the measure spaces/propositions they're interested in. But again, I don't think dealing with infinite sets is so bad in applications.
5John G. Halstead2y
Do you for example think there is a more than 50% chance that it is greater than 10 billion?

Another way to look at this. What do you think is the probability that everyone will go extinct tomorrow? If you are agnostic about that, then you must also be agnostic about the value of GiveWell-type stuff.

Why? GiveWell charities have developed theories about the effects of various interventions. The theories have been tested and, typically, found to be relatively robust. Of course, there is always more to know, and  always ways we could improve the theories.  I don't see how this relates to not being able to develop a statistical estimate of the probability we go extinct tomorrow. (Of course, I can just give  you a number and call it "my belief that we'll go extinct tomorrow," but this doesn't get us anywhere. The question is whether it's accurate - and what accuracy means in this case.) What would be the parameters of such a model? There are uncountably many things - most of them unknowable - which could affect such an outcome.  
9John G. Halstead2y
The benefits of GiveWell's charities are worked out as health or economic benefits which are realised in the future. e.g. AMF is meant to be good because it allows people who would have otherwise died to live for a few more years. If you are agnostic about whether everyone will go extinct tomorrow, then you must be agnostic about whether people will actually get these extra years of life. 
  I don't have a probability distribution across the size of the future population. That said, I'm happy to interpret the question in the colloquial, non-formal sense, and just take >50% to mean "likely". In that case, sure, I think it's likely that the population will exceed 10 billion. Credences shouldn't be taken any more seriously than that - epistemologically equivalent to survey questions where the respondent is asked to tick a very unlikely, unlikely, unsure, likely, very likely  box.
  Agree! While I do have problems with (weak?) longtermism, this post is a criticism of strong longtermism :)

I will focus on two aspects of strong longtermism, henceforth simply longtermism. First, the underlying arguments inoculate themselves from criticism by using arbitrary assumptions on the number of future generations. Second, ignoring short-term effects destroys the means by which we make progress — moral, scientific, artistic, and otherwise.

I found it helpful that you were so clear about these two aspects of what you are saying. My responses to the two are different.

On the first, I think resting on possibilities of large futures is a central part of th... (read more)

Hi Owen!  Re: inoculation of criticism. Agreed that it doesn't make criticism impossible in  every sense (otherwise my post wouldn't exist). But if one reasons with numbers only (i.e., EV reasoning), then longtermism becomes unavoidable. As soon as one adopts what I'm calling "Bayesian epistemology", then there's very little room to argue with it. One can retort: Well,  yes, but there's very little room to argue with General Relativity, and that is a strength of the theory, not a weakness. But the difference is that GR is very precise: It's hard to argue with  because it aligns so well with observation. But there are lots of observations which would refute it (if light didn't bend around stars, say). Longtermism is difficult to refute for a different reason, namely because it's so easy to change the underlying assumptions. (I'm not trying to equate moral theories with empirical theories in every sense, but this example gets the point across I think.) Your second point does seem correct to me. I think I try to capture this sentiment when  I say   Here  I'm granting that the moral view that future generations matter could be correct. But this, on my problem/knowledge-focused view of progress,  is irrelevant for decision making.  What matters is maintaining the ability to solve problems and correct our (inevitable) errors. 

Cool. I do think that trying to translate your position into the ontology used by Greaves+MacAskill it's sounding less like "longtermism is wrong" and more like "maybe longtermism is technically correct; who cares?; the practical advice people are hearing sucks".

I think that's a pretty interestingly different objection and if it's what you actually want to say it could be important to make sure that people don't hear it as "longtermism is wrong" (because that could lead them to looking at the wrong type of thing to try to refute you).

I think that ontology used by Greaves+MacAskill is poor. I skim-read their Case  for Strong Longtermsim [] paper honestly expecting it to be great (Will is generally pretty sensible) but I came away quite confused as to what case was being made.    Ben – maybe there needs to be more of an exercise to disentangle what is meant by longtermism before it can be critiqued fairly.   Owen – I am not sure if you would agree but I as far as I can tell the points you  make about bounded rationality in the excellent post you link to above contradicts the the Case  for Strong Longtermsim [] paper. EG: * Greaves+MacAskill: "we assumed that the correct way to evaluate options ... is in terms of expected value" (as far as I can tell their entire point is that you can always do an expected value calculation and "ignore all the effects contained in the first 100" years). * You: "if we want to make decisions on longtermist grounds, we are going to end up using some heuristics"
  Yes, exactly. One can always find some  expected value calculation that allows one to ignore present-day suffering. And worse, one can keep doing this between now and eternity, to ignore all suffering forever. We can describe this using the language of "falsifiability" or "irrefutability" or  whatever - the word choice doesn't really matter here. What matters is that this is a very dangerous game to be playing.
I think it is worth trying to judge the paper / case for longtermism charitably. I do not honestly think that Will means that we can literally ignore everything in the first 100 years – for a start just because the short-term affects the long-term.  If you want to evaluate interventions, even those designed for long-term impact, you need to look at the short-term impacts. But that is where I get stuck trying to work out what Will + Hillary  mean. I think they are saying more than just you should look at the long and short term effects of interventions (trivially true under most ethical views). They seem to be making empirical, not philosophical, claims about the current state of the world. They appear to argue that if you use expected value calculations for decision making then you will arrive at the conclusions that suggest that you should care about highly speculative long-term effects over clear short term effects. They combine this with an assumption that expected value calculations are the correct decision making tool to conclude that long-term interventions are most likely to be the best interventions. I think  * the logic of the argument is roughly correct. * the empirical claims made are dubious and ideally need more than a few examples to justify, but it is plausible they are correct. I think there is at least a decent case for marginal extra resources being directed to x-risk prevention in the world today. * the assumption that expected value calculations are the correct decision making tool is incorrect, (as per others at GPI like Owen's work and Andreas []' work, bounded rationality, the entire field of risk management, economists like Taleb, knightian uncertainty, etc. etc) . A charitable reading would say that they recognises this is an assump
I'd say they mean you can effectively ignore the differences in terminal value in the short term, e.g. the welfare of individuals in the short term only really matters for informing long-term consequences and effectively not in itself, since it's insignificant compared to differences in long-term value. In other words, short-term welfare is effectively not an end in itself.
Yeah that is a good way of putting it. Thank you. It is of course a feature of trying to prioritise between causes in order to do the most good, that some groups will be effectively ignored. Luckily in this case if done in a sensible manner I would expect that there should be a strong correlation between short term welfare and long-run welfare. As managing high uncertainty should involve some amount of ensuring good feedback loops and iterating, so taking action changing things for the better (for the long run but in a way that affects the world now) learning and improving. Building the EA community, developing clean meat, improving policy making, etc. (Unfortunately I am not sure to what extent this is a key part of the EA longtermist paradigm at present.)
3Jack Malde2y
Hmm perhaps you need to read the paper again. They say for example: Indeed they go on in section 4.5 to consider other decision theories, including Knightian uncertainty, and conclude that strong longtermism is robust to these other theories. I'm not saying they're definitely right, just that they haven't assumed expected value theory is correct as you claim.

OK Jack, I have some time today so lets dive in:


So, my initial reading of 4.5 was that they get it very very wrong.

Eg: "we assumed that the correct way to evaluate options in ex ante axiological terms, under conditions of uncertainty, is in terms of expected value". Any of the points above would disagree with this.

Eg: "[Knightian uncertainty] supports, rather than undermining, axiological strong longtermism". This is just not true. Some Knightian uncertainty methods would support (eg robust decision making) and some would not support (eg plan-and-adapt).


So why does it look like they get this so wrong?

Maybe they are trying to achieve something different from what we in this thread think they are trying to achieve.

My analysis of their analysis of Knightian uncertainty can shed some light here.

The point of Knightian (or deep) uncertainty tools is that an expected value calculation is the wrong tool for humans to use when making decisions under Knightian uncertainty. That an expected value calculation, as a decision tool it will not lead to the best outcome, the outcome with the highest true expected value. [Note: I use true expected value to mean the expected value if... (read more)

@ Ben_Chugg Curious how much you would agree with a statement like: If we had perfect information [edit: on expected value] the options that would be best to do would be those that positively affect the far future. So in practice looking towards those kinds of options is a useful tool to apply when we are deciding what to do.  (This is my very charitable, weak interpretation of what the Case for Strong Longtermism paper is attempting to argue)
I think I agree,  but there's  a lot smuggled into the phrase "perfect information on expected value". So much in fact that I'm not sure I can quite follow the thought experiment.  When I think of "perfect information on expected value", my first thought is something like a game of roulette. There's no uncertainty (about  what can affect the system), only chance. We understand all the parameters of the system and can write down a model. To say something like this about the future means we would be basically omniscient - we would know what  sort of future knowledge will be developed, etc. Is this also what you had in mind? (To complicate matters,  the roulette analogy is imperfect. For a typical game of roulette we can write down a pretty robust probabilistic model. But it's only a model. We could also study the precise physics of that particular roulette board,  model the hand spinning the wheel (is that how roulette works? I don't even know), take into account the initial position, the toss of the  white ball,  and so on and so forth. If we spent a long time doing this, we could come up with a model which was more accurate than our basic probabilistic model. This is all to say that models are tools suited for a particular purpose. So it's unclear to me what the model would be for the future which allowed us to write down a precise model - which is implicitly required for EV calculations). 
Hi Ben. I agree with you. Yes I think roulette is a good analogy. And yes I think the "perfect information on expected value" is a strange claim to make. But I do think it is useful to think about what could be said and justified. I do think a claim along these lines could be made and it would not be wholly unfalsifiable and it would not require completely preferencing Bayesian expected value calculations.   To give another analogy I think there is a reasonable long-termist equivalent of statements like: Because of differences in wealth and purchasing power we expect that a donor in the developed west can have a much bigger impact overseas than in their home country. So in practice looking towards those kinds of international development options is a useful tool to apply when we are deciding what to do.  This does not completely exclude the probability that we can have impact locally with donations, but it does direct our searching.   Being charitable to Will+Hillary, maybe that is all they are saying. And maybe it is so confusing because they have dressed it up in philosophical language – but this is because, as per GPI's goals, this paper is about engaging philosophy academics rather than producing any novel insight. (If being more critical I am not convinced that Will+Hillary successfully give sufficient evidence to make such a claim in this paper and also see my list of things their paper could improve above.)
1Jack Malde2y
Thanks for this! All interesting and I will have to think about this more carefully when my brain is fresher. I admit I'm not very familiar with the literature on Knightian uncertainty and it would probably help if I read some more about that first. OK if I understand you correctly, what you have said is that Will and Hilary present Knightian uncertainty as axiologically different to EV reasoning, when you don't think it is. I agree with you that ideally section 4.5 should be considering some axiologically different decision-making theories to EV. Regarding the actual EV calculations with numbers, I would say, as I did in a different comment, that I think it is pretty clear that they only carry out EV calculations for illustrative purposes. To quote: This is the point they are trying to get across by doing the actual EV calculations.
3Owen Cotton-Barratt2y
I agree that there's a tension in how we're talking about it. I think that Greaves+MacAskill are talking about how an ideal rational actor should behave -- which I think is informative but not something to be directly emulated for boundedly rational actors.
Ah yes thank you Owen. That helps me construct a sensible positive charitable reading of their paper. There is of course a risk that people take their paper / views of longtermism and expected value approach to be more decision guiding than perhaps they ought. (I think it might be an overly charitable reading – the paper does briefly mention and then dismiss concerns about decision making under uncertainty, etc – although it is only a draft so reasonable to be charitable.)
Oh interesting. Did you read my critique as saying that the philosophy is wrong? (Not sarcastic; serious question.) I don't really even know what "wrong" would mean here, honestly. I think the reasoning is flawed and if taken seriously leads to bad consequences.  
2Owen Cotton-Barratt2y
I read your second critique as implicitly saying "there must be a mistake in the argument", whereas I'd have preferred it to say "the things that might be thought to follow from this argument are wrong (which could mean a mistake in the argument that's been laid out, or in how its consequences are being interpreted)".

I'm not sold on the cluelessness-type critique of long-termism. The arguments here focus on things we might do now or soon to reduce the direct risk posed by various things such as AI, bio or nuclear war. But even if this is true, this doesn't undermine the expected value of other long-termist activities. 

  1. Gathering more information about the direct risks. If we are clueless about what to do, the value of information from further research must be extremely high, on long-termism. 
  2. Building the community of people concerned about the long-term e.g. through community building. 
  3. Investing in the stock market and punting the "what to do" question to the future. 

GiveWell’s estimates use real, tangible, collected data.

I wonder if you have come across the literature on complex cluelessness? GiveWell may use some real, tangible data, but they are missing lots of highly-relevant and important data, most obviously relating to the longer-term consequences of the health interventions. For example they don't know what the long-term population effects will be nor the corresponding moral value of these population effects. It also really doesn't seem fair to me to just assume that this would be zero in expectation, which Giv... (read more)

I have read about (complex) cluelessness. I have a lot of respect for Hilary Greaves, but I don't think cluelessness is particularly illuminating concept. I view it as a variant of "we can't predict the future." So, naturally, if you ground your ethics in expected value calculations over the long term future then, well, there's going to be problems.  I would propose to resolve cluelessness as follows: Let's admit we can't predict the future. Our focus should instead be on error-correction. Our actions will have consequences - both intended and unintended, good and bad. The best we can do is foster a critical, rational environment where we can discuss the negatives consequences, solve them, and repeat. (I know this answer will sound glib, but I'm quite sincere.) 
4Jack Malde2y
I do think it's far more illuminating than "we can't predict the future". Really complex cluelessness is saying OK great you've carried out a CBA/CEA but you've omitted/ignored effects from the analysis that we: 1. Have good reason to expect will occur 2. Have good reason to suspect are sufficiently important such that they could change the sign of your final number if properly included in your analysis If the above factors are in fact true in the case of GiveWell (I think they probably are) then I don't think GiveWell CBAs are all that useful and the original point you were trying to make - that GiveWell analysis is obviously superior because it makes use of data - sort of breaks down because, quite simply, the data has a massive, gaping hole in it. This is not to criticise GiveWell in the slightest, it's just to acknowledge the monstrous task they're up against. Correct me if I'm wrong but what you seem to be arguing is that we're actually complexly clueless about everything, so we may as well just ignore the problem. I actually don't think this is true - we may be clueless about everything but not necessarily in a complex way. Consider the promotion of philosophy in schools [], a class of interventions that I have written about. I'm not sure if these are definitely the best interventions (reception to my post was fairly lukewarm), but I also don't think we are complexly clueless about their effects in the same way that we are about the effects of distributing bednets. This is because it's just quite hard to think up reasons why it might be bad to promote philosophy in schools. Sure it could be the case that promoting philosophy in schools makes something bad happen, but I don't really have much of a reason to entertain that possibility if I can't think of a specific effect that fulfils the two factors I listed above. In the case of distribut
One response might be that if there are unintended negative consequences, we can address those later or separately. Sometimes it will be the case that optimizing for some positive effect optimizes a negative effect, but usually these won't correspond. So, the most cost-effective ways to save lives won't be the ways that maximize the negative effects of population growth - those same negative effects will be cheaper to obtain through something other than population growth -, and we can probably find more cost-effective ways to offset those effects. I wrote a post about hedging like this [].
1Jack Malde2y
Interesting, thanks for sharing that post. I will have to read it more carefully to fully digest it!

What do you think about using ranges of probabilities instead of single (and seemingly arbitrary) sharp probabilities and doing sensitivity analysis? I suppose when there's no hard data, there might be no good bounds for the ranges, too, although Scott Alexander has argued against using arbitrarily small probabilities.

Yeah I suppose I would still be skeptical of using ranges in the absence of data (you could just apply all my objections to the upper and low bounds of the range). But I'm definitely all for sensitivity analysis when there are data backing up the estimates!