Hide table of contents

Comment Permalink

“It’s not clear why you’d think that the evidence for x-risk is strong enough to think we’re one-in-a-million, but not stronger than that.” This seems pretty strange as an argument to me. Being one-in-a-thousand is a thousand times less likely than being one-in-a-million, so of course if you think the evidence pushes you to thinking that you’re one-in-a-million, it needn’t push you all the way to thinking that you’re one-in-a-thousand. This seems important to me. Yes, you can give me arguments for thinking that we’re (in expectation at least) at an enormously influential time - as I say in the blog post and the comments, I endorse those arguments! I think we should update massively away from our prior, in particular on the basis of the current rate of economic growth. (My emphasis)

Asserting an astronomically adverse prior, then a massive update, yet being confident you're in the right ballpark re. orders of magnitude does look pretty fishy though. For a few reasons:

First, (in the webpage version you quoted) you don't seem sure of a given prior probability, merely that it is 'astronomical': yet astronomical numbers (including variations you note about whether to multiply by how many accessible galaxies there are or not, etc.) vary by substantially more than three orders of magnitude - you note two possible prior probabilities (of being among the million most influential people) of 1 in a million trillion (10^-18) and 1 in a hundred million (10^-8) - a span of 10 orders of magnitude.

It seems hard to see how a Bayesian update from this (seemingly) extremely wide prior would give a central estimate at a (not astronomically minute) value, yet confidently rule against values 'only' 3 orders of magnitude higher (a distance a ten millionth the width of this implicit span in prior probability). [It also suggests the highest VoI is to winnow this huge prior range, rather than spending effort evaluating considerations around the likelihood ratio]

Second, whatever (very) small value we use for our prior probability, getting to non-astronomical posteriors implies likelihood ratios/Bayes factors which are huge. From (say) 10^-8 to 10^-4 is a factor of 10 000. As you say in your piece, this is much much stronger than the benchmark for decisive evidence of ~100. It seems hard to say (e.g.) evidence from the rate of economic growth is 'decisive' in this sense, and so it is hard to see how in concert with other heuristic considerations you get 10-100x more confirmation (indeed, your subsequent discussion seems to supply many defeaters exactly this). Further, similar to worries about calibration out on the tail, it seems unlikely many of us can accurately assess LRs > 100 which are not direct observations within orders of magnitude.

Third, priors should be consilient, and can be essentially refuted by posteriors. A prior that get surprised to the tune of a 1-in-millions should get hugely penalized versus any alternative (including naive intuitive gestalts) which do not. It seems particularly costly as non-negligible credences in (e.g.) nuclear winter, the industrial revolution being crucial etc. are facially represent this prior being surprised by '1 in large X' events at a rate much greater than 1/X.

To end up with not-vastly lower posteriors than your interlocutors (presuming Buck's suggestion of 0.1% is fair, and not something like 1/million), it seems one asserts both a much lower prior which is mostly (but not completely) cancelled out by a much stronger update step. This prior seems to be ranging over many orders of magnitude, yet the posterior does not - yet it is hard to see where the orders of magnitude of better resolution are arising from (if we knew for sure the prior is 10^-12 versus knowing for sure it is 10^-8, shouldn't the posterior shift a lot between the two cases?)

It seems more reasonable to say 'our' prior is rather some mixed gestalt on considering the issue as a whole, and the concern about base-rates etc. should be seen as an argument for updating this downwards, rather than a bid to set the terms of the discussion.

See in context

Thoughts on whether we're living at the most influential time in history

by Buck

Nov 3 202011 min read 66

178

Hinge of historyLong-term futureInside vs. outside viewPatient altruism

Frontpage

Thoughts on whether we're living at the most influential time in history

EDIT: Will's new post

The outside-view argument

This argument implies that the probability of extinction this century is almost certainly negligible

EDIT: This objection of mine applies to Will's article as it was originally written, but he edited it after posting to change the argument so that my objection no longer applies.

To me, uniform hingeyness instead implies no long futures

The fishiness comes from the earliness, not the hinginess

“Early times in history are particularly hingey” is a priori reasonable

Incidentally, this outside view argument is also an argument against patient philanthropy

The inductive argument against HoH

Conclusion

66 comments

(thanks to Claire Zabel, Damon Binder, and Carl Shulman for suggesting some of the main arguments here, obviously all flaws are my own; thanks also to Richard Ngo, Greg Lewis, Kevin Liu, and Sidney Hough for helpful conversation and comments.)

Will MacAskill has a post, Are we living at the most influential time in history?, about what he calls the “hinge of history hypothesis” (HoH), which he defines as the claim that “we are living at the most influential time ever.” Whether HoH is true matters a lot for how longtermists should allocate their effort. In his post, Will argues that we should be pretty skeptical of HoH.

EDIT: Will recommends reading this revised article of his instead of his original post.

I appreciate Will’s clear description of HoH and its difference from strong longtermism, but I think his main argument against HoH is deeply flawed. The comment section of Will’s post contains a number of commenters making some of the same criticisms I’m going to make. I’m writing this post because I think the rebuttals can be phrased in some different, potentially clearer ways, and because I think that the weaknesses in Will’s argument should be more widely discussed.

Summary: I think Will’s arguments mostly lead to believing that you aren’t an “early human” (a human who lives on Earth before humanity colonizes the universe and flourishes for a long time) rather than believing that early humans aren’t hugely influential, so you conclude that either humanity doesn’t have a long future or that you probably live in a simulation.

I sometimes elide the distinction between the concepts of “x-risk” and “human extinction”, because it doesn’t matter much here and the abbreviation is nice.

(This post has a lot of very small numbers in it. I might have missed a zero or two somewhere.)

EDIT: Will's new post

Will recommends reading this revised article of his instead of his original post. I believe that his new article doesn't make the assumption about the probability of civilization lasting for a long time, which means that my criticism "This argument implies that the probability of extinction this century is almost certainly negligible" doesn't apply to his new post, though it still applies to the EA Forum post I linked. I think that my other complaints are still right.

The outside-view argument

This is the argument that I have the main disagreement with.

Will describes what he calls the “outside-view argument” as follows:

1. It’s a priori extremely unlikely that we’re at the hinge of history
2. The belief that we’re at the hinge of history is fishy
3. Relative to such an extraordinary claim, the arguments that we’re at the hinge of history are not sufficiently extraordinarily powerful

Given 1, I agree with 2 and 3; my disagreement is with 1, so let’s talk about that. Will phrases his argument as:

The prior probability of us living in the most influential century, conditional on Earth-originating civilization lasting for n centuries, is 1/n.
The unconditional prior probability over whether this is the most influential century would then depend on one's priors over how long Earth-originating civilization will last for. However, for the purpose of this discussion we can focus on just the claim that we are at the most influential century AND that we have an enormous future ahead of us. If the Value Lock-In or Time of Perils views are true, then we should assign a significant probability to that claim. (i.e. they are claiming that, if we act wisely this century, then this conjunctive claim is probably true.) So that's the claim we can focus our discussion on.

I have several disagreements with this argument.

This argument implies that the probability of extinction this century is almost certainly negligible

EDIT: This objection of mine applies to Will's article as it was originally written, but he edited it after posting to change the argument so that my objection no longer applies.

One way that a century could be pivotal is that humanity could go extinct. So this argument suggests that our century shouldn’t have unusually high levels of x-risk.

If you assume, as Will does in his argument, that the probability of humanity lasting a trillion years is 0.01%, and you think that this century has a typical probability of extinction, then you find that the probability of human extinction this century is 0.00000009%, because (1 - 0.00000009%) ** (1 trillion years/ one century) is about 0.01%.

0.00000009% is a very small probability. Will said during an EA Forum AMA that he puts the probability of x-risk over the next century at less than 1%, which I guess is consistent with this number, but if this is really what he thinks (which it probably isn’t), I think he should probably have included at least a few more of the six or so extra zeros.

It seems more likely to me that Will instead thinks that x-risk over the next century is something like 0.1%. You can do similar math to note that this argument implies that only one in a million centuries can have a probability of extinction higher than 0.07% (assuming all other centuries have exactly zero risk). It’s not clear why you’d think that the evidence for x-risk is strong enough to think we’re one-in-a-million, but not stronger than that. (Thanks to Greg Lewis for pointing out this last argument to me.)

To me, uniform hingeyness instead implies no long futures

If I believed in uniform hingeyness the way Will is advocating, I’d probably reason about the future as follows:

“I don’t know how long humanity’s future is. But I do believe that it would be quite surprising if this century had radically higher x-risk levels than later centuries. So I assume that later centuries will have similar x-risk levels to this century. Currently, x-risks seem nontrivial; perhaps nukes alone give us 0.1% per century. Therefore, humanity is unlikely to last more than a thousand centuries.”

I come to this conclusion because I have more evidence about current levels of x-risk than I do about the probability of humanity having a very long future. It seems much more reasonable to make the inference “levels of x-risk are non-negligible, so uniform x-risk means no long future” than “humanity might have a very long future, so levels of x-risk are incredibly low”. Will’s argument as written is making the second inference.

The fishiness comes from the earliness, not the hinginess

I think that the “we shouldn’t be exceptional” prior is mostly an argument against us being early humans, rather than early times in civilization being hingey.

Here’s an argument that I think is reasonable:

Given a long future, the probability of any randomly chosen human living in a hingey time is very low.
It seems pretty reasonable to think that there are some hingey early times in human history.
We seem to find ourselves in an early part of human history, and a time where x-risk looks non-negligible and which therefore seems hingey. There is something weird about this.

How should we resolve this?

Will’s resolution is to say that in fact, we shouldn’t expect early times in human history to be hingey, because that would violate his strong prior that any time in human history is equally likely to be hingey.

It seems to me that in fact, most of the fishiness here is coming from us finding ourselves in an early time in human history, and that if you condition on us being at an early time in human history (which is an extremely strong condition, because it has incredibly low prior probability), it’s not that surprising for us to find ourselves at a hingey time.

Will thinks that the “early times are hingey” hypothesis needs to meet an incredibly high standard of evidence because it implies that we’re at an exceptional time. But this century doesn’t need to be hingey in order to be exceptional--it’s already extremely implausibly early. If something like 10% of all humans who have ever lived are alive today, then even if this century is the hingiest century, then being in the hingiest century is only 10x as surprising as being among the first 100 billion humans to be alive. So if you require incredible evidence to be convinced of hingeyness, it seems like you should require almost as incredible evidence to be convinced of earliness.

And so the main result of the “we probably aren’t at an exceptional time” assumption is that we’re almost surely not actually early humans. This is basically just the simulation argument, which has the conclusion that either the future isn’t big, we are in a simulation, or we are very very confused (eg because we’re dreaming or insane). (The simulation argument also includes the possibility that the future has no simulations of early intelligent life in it, which seems relatively implausible to me).

This result is also related to the doomsday argument, which argues that given we should be very surprised to be unusually early humans, there probably won’t be squillions more humans in the future. (This argument is only persuasive under some theories of anthropics.)

As Will notes, following Brian Tomasik and others, the simulation argument dampens enthusiasm for influencing the far future. But it does this almost entirely by updating us against thinking that we’re early humans, rather than thinking that people who are in fact early humans aren't hugely influential.

“Early times in history are particularly hingey” is a priori reasonable

I think we should evaluate the “early times are hingey” hypothesis as if we didn’t know that we were (at least on the face of it) in that time; I think that “early times are hingey” is in fact pretty a priori plausible.

At the risk of making arguments based on object-level claims about the world instead of abstract generalities, you could note that:

Once upon a time, there were very few humans, and they weren’t very resilient to things like volcanic eruptions, and so they could have fairly easily died out.
In contrast, if humans colonize the universe, then most of Earth-originating civilization will occur in parts of space that will never again be able to interact with Earth, and so which won’t face threats that could cause complete destruction of civilization.
In general it seems like technology can be a positive and negative factor in global risk; before a certain tech level, no-one has technology that could destroy the world, and there are also hypothetical technologies that could reduce the risk of extinction to basically zero basically forever. It seems kind of reasonable to imagine that civilizations live for a long time if they manage to implement the anti-extinction technologies before they die from their dangerous technologies (or from natural risks).

Will’s argument suggests that all of these require an extremely high standard of evidence that they don’t meet; I think that they don’t; we should mostly be surprised not that early times look risky, but that we appear to be in one.

Incidentally, this outside view argument is also an argument against patient philanthropy

Suppose it was true that you could save money and have it grow exponentially over long time periods. Because investing then does more good if you do it earlier, this would mean that we had radically better opportunities to do good than most humans. So the outside view argument also thinks that it’s impossible to influence the future via investing.

One way of thinking about this is that investing for a long time is a mechanism by which you can turn earliness into importance, and so it’s incompatible with the belief that you’re early but not important.

I mention this because I sometimes hear the outside view argument used as an argument for patient philanthropy, which it in fact is not.

The inductive argument against HoH

I have milder objections to this argument, but I still thought it was worth writing about it.

Will argues that plausibly hinginess is steadily increasing and so we’re currently not at the hingiest time. This seems right to me; I suspect that eg the year where we build the first superintelligence is hingier than this year. But I claim that long term, x-risk needs to go to zero if we have much chance of an extremely long future. And the current increase in hinginess seems unsustainable, in that the increase in hinginess we’ve seen so far leads to x-risk probabilities that lead to drastic reduction of the value of worlds that last for eg a millennium at current hinginess levels.

So I think we should probably think that this decade isn’t the decade where the most important things happen, but (conditional on the future being big) it’s plausible enough that this is the hinge century that we should be actively trying to do specific object level things to make that most-important-decade go as well as it can.

I think Will disagrees with this because he already used the outside-view argument to update against current levels of x-risk being as high as they look.

There’s a version of this argument which I find pretty persuasive, which Will sort of makes and which Toby Ord makes at more length in The timing of labour aimed at reducing existential risk. I’ll quote Toby:

Suppose someone considers AI to be the largest source of existential risk, and so spends a decade working on approaches to make self-improving AI safer. It might later become clear that AI was not the most critical area to worry about, or that this part of AI was not the most critical part, or that this work was going to get done anyway by mainstream AI research, or that working on policy to regulate research on AI was more important than working on AI. In any of these cases she wasted some of the value of her work by doing it now. She couldn’t be faulted for lack of omniscience, but she could be faulted for making herself unnecessarily at the mercy of bad luck. She could have achieved more by doing her work later, when she had a better idea of what was the most important thing to do.

I think that there’s quite a strong case to be made that we will indeed be able to work more helpfully on AI risk when we’re closer to AGI being developed. And this suggests that we should spend more of our efforts on e.g. building an AI safety field that will be able to do lots of really good work in the years leading up to AGI, rather than trying to do useful research now; many AI safety researchers explicitly endorse this argument. But this strategy, where you identify concrete events that you want to influence that look like they might be a few decades away and you build up capabilities to influence them, is quite different from a strategy where you build up general capabilities without any particular plan for what to apply them to, perhaps for billions of years; I think it’s worth distinguishing these two proposals.

Conclusion

I think Will’s arguments mostly lead to believing that you’re not an early human, rather than believing that early humans aren’t hugely influential.

Throughout this post I only talked about x-risk as a source of hingeyness. Other potential sources of hingeyness don’t change the analysis much. They can do the following:

reduce the probability of x-risk forever, in which case they’re a mechanism by which early times might be hingey
make the world less valuable forever; for our purposes this is the same thing as x-risk
make the world more valuable forever; I think that you can basically just think of situations where the world might be made better forever as centuries where there’s a risk of losing large fractions of future value via the good thing not happening.

I think there’s interesting discussion to be had about how we should respond to the simulation argument, and whether we should be executing on object level plans or waiting until we’re better informed.

178 Reactions

Mentioned in

112Important Between-Cause Considerations: things every EA should know about

89AI Timelines: Where the Arguments, and the "Experts," Stand

72Forecasting transformative AI: what's the burden of proof?

69CEA's 2020 Annual Review

37EA Updates for December 2020

Load more (5/9)

More posts like this

204

Are we living at the most influential time in history?

William_MacAskill

219

All Possible Views About Humanity's Future Are Wild

Holden Karnofsky

130

This Can't Go On

Holden Karnofsky

Comments66

Sorted by

New & upvoted

Click to highlight new comments since: Today at 7:03 AM

richard_ngo4y74

I think I agree with the core claims Buck is making. But I found the logical structure of this post hard to follow. So here's my attempt to re-present the core thread of the argument I think Buck is making:

In his original post, Will conditions on long futures being plausible, since these are the worlds that longtermists care about most. Let's assume from now on that this is the case. Will claims, based on his uniform prior over hinginess, that we should require extraordinary evidence to believe in our century's hinginess, conditional on long futures being plausible. But there are at least two reasons to think that we shouldn't use a uniform prior. Firstly, it's more reasonable to instead have a prior that early times in human history (such as our time) are more likely to be hingey - for example because we should expect humanity to expand over time, and also from considering technological advances.
Secondly: if we condition on long futures being plausible, then xrisk must be near-zero in almost every century (otherwise there's almost no chance we'd survive for that long). So observing any nonnegligible amount of (preventable) xrisk in our present time becomes very strong evidence of this century being an extreme outlier in terms of xrisk, which implies that it's also an extreme outlier in terms of hinginess. So using the uniform prior on hinginess means we have to choose between two very implausible options - either current xrisk is in fact incredibly low (far lower than seems plausible, and far lower than Will himself claims to believe it is), or else we're in a situation that the uniform prior judges as extremely improbable and "fishy". Instead of biting either of these bullets, it seems preferable to use a prior which isn't so dogmatic - e.g. a prior which isn't so surprised by early times in human history being outliers.

Toby gives an example of an alternative (and in my mind better) prior as a reply to Will's original post.

Note that anyone who conditions on a long future being possible should afterwards doubt the evidence for current xrisk to some degree. But Will is forced to do so to an extreme extent because his uniform prior on hinginess is such a bold one - whereas people with exponentially diminishing priors on hinginess like Toby's won't update as much after conditioning. All of this analysis remains roughly the same if you replace Will's uniformity-of-hinginess prior with a uniformity-of-(preventable)-xrisk prior, and Toby's exponentially-decreasing-hinginess prior with an exponentially-decreasing-(preventable)-xrisk prior. I add "preventable" here and above because if our current xrisk isn't preventable, then it's still possible that we're in a low-hinginess period.

Lastly, it seems to me that conditioning on long futures being plausible was what caused a lot of the messiness here, and so for pedagogical purposes it'd probably be better to spell out all the different permutations of options more explicitly, and be clearer about when the conditioning is happening.

JackM4y7

Firstly, it's more reasonable to instead have a prior that early times in human history (such as our time) are more likely to be hingey - for example because we should expect humanity to expand over time, and also from considering technological advances.

I don't really get this. It seems that there are good reasons to believe early times are influential, but also good reasons to believe that later times are influential, and it isn't clear to me which of these dominates.

For example, Will's inductive argument against HH pulls towards later times being influential. From Will's paper:

P1. The influentialness of comparable people in the past has been increasing over time, with increasing knowledge and opportunities being the most important factor.
P2. We should expect our knowledge and opportunities to continue into the future.
C. So we should expect the influentialness of those future people who we can pass resources on to be greater, too.

On the other hand, the observation that we are living on a single planet pulls in the direction of this time being more influential, and Toby had some reasons for thinking that early times should be more influential:

there are also good reasons to suspect that the chance of a century being the most influential should diminish over time. Especially because there are important kinds of significant event (such as the value lock-in or an existential catastrophe) where early occurrence blocks out later occurrence.

Overall it isn't clear to me from this that we should have a decreasing prior over time? Can anyone help me out here?

Ben_West🔸4y6

I found this rephrasing helpful, thanks Richard.

Buck4y3

I agree with all this; thanks for the summary.

William_MacAskill4y42

(Comment 1/5)

Thanks so much for engaging with this, Buck! :)

I revised the argument of the blog post into a forthcoming article, available at my website (link). I’d encourage people to read that version rather than the blog post, if you’re only going to read one. The broad thrust is the same, but the presentation is better.

I’ll discuss the improved form of the discussion about priors in another comment. Some other changes in the article version:

I frame the argument in terms of the most influential people, rather than the most influential times. It’s the more natural reference class, and is more action-relevant.
I use the term ‘influential’ rather than ‘hingey’. It would be great if we could agree on terminology here; as Carl noted on my last post, ‘hingey’ could make the discussion seem unnecessarily silly.
I define ‘influentialness’ (aka ‘hingeyness’) in terms of ‘how much expected good you can do’, not just ‘how much expected good you can do from a longtermist perspective’. Again, that’s the more natural formulation, and, importantly, one way in which we could fail to be at the most influential time (in terms of expected good done by direct philanthropy) is if longtermism is false and, say, we only discover the arguments that demonstrate that in a few decades’ time.
The paper includes a number of graphs, which I think helps make the case clearer.
I don’t discuss the simulation argument. (Though that's mainly for space and academic normalcy reasons - I think it's important, and discuss it in the blog post.)

Buck4y21

The comment I'd be most interested in from you is whether you agree that your argument forces you to believe that x-risk is almost surely zero, or that we are almost surely not going to have a long future.

William_MacAskill4y21

Richard’s response is about right. My prior with respect to influentialness, is such that either: x-risk is almost surely zero, or we are almost surely not going to have a long future, or x-risk is higher now than it will be in the future but harder to prevent than it will be in the future or in the future there will be non-x-risk-mediated ways of affecting similarly enormous amounts of value in the future, or the idea that most of the value is in the future is false.

I do think we should update away from those priors, and I think that update is sufficient to make the case for longtermism. I agree that the location in time that we find ourselves in (what I call ‘outside-view arguments’ in my original post) is sufficient for a very large update.

Practically speaking, thinking through the surprisingness of being at such an influential time made me think:

Maybe I was asymmetrically assessing evidence about how high x-risk is this century. I think that’s right; e.g. I now don’t think that x-risk from nuclear war is as high as 0.1% this century, and I think that longtermist EAs have sometimes overstated the case in favour.
If we think that there’s high existential risk from, say, war, we should (by default) think that such high risk will continue into the future.
It’s more likely that we’re in a simulation

It also made me take more seriously the thoughts that in the future there might be non-extinction-risk mechanisms for producing comparably enormous amounts of (expected) value, and that maybe there’s some crucial consideration(s) that we’re currently missing such that our actions today are low-expected-value compared to actions in the future.

Buck4y12

Hmm, interesting. It seems to me that your priors cause you to think that the "naive longtermist" story, where we're in a time of perils and if we can get through it, x-risk goes basically to zero and there are no more good ways to affect similarly enormous amounts of value, has a probability which is basically zero. (This is just me musing.)

richard_ngo4y5

I think you'd want to modify that to preventable x-risk, to get Will to agree; and also to add a third part of the disjunction, that preventing x-risk might not be of overriding moral importance (since he raises the possibility that longtermism is false in a comment below, with the implication that if so, even preventing x-risk wouldn't make us "influential" by his standards).

However, if Will thinks his argument as currently phrased holds, then it seems to me that he's forced to agree with similar arguments that use slightly different definitions of influentialness (such as influentialness = the expected amount you can change other people's lives, for better or worse). Or even a similar argument which just tries to calculate directly the probability that we're at the time with the most x-risk, rather than talking about influentialness at all. At that point, the selection effect I described in another comment starts to become a concern.

EdoArad4y13

I like the term 'influential' and 'influentialness'. I think it is very clear and automatically leads pretty much to the definition you give it.

Buck4y4

I've added a link to the article to the top of my post. Those changes seem reasonable.

William_MacAskill4y17

(Comment 2/5)

The outside-view argument (in response to your first argument)

In the blog post, I stated the priors-based argument quite poorly - I thought this bit wouldn’t be where the disagreement was, so I didn’t spend much time on it. How wrong I was about that! For the article version (link), I tidied it up.

The key thing is that the way I’m setting priors is as a function from populations to credences: for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.

This falls out of the self-sampling assumption, that a rational agent’s priors locate her uniformly at random within each possible world. If you reject this way of setting priors then, by modus tollens, you reject the self-sampling assumption. That’s pretty interesting if so!

On this set-up of the argument (which is what was in my head but I hadn’t worked through), I don’t make any claims about how likely it is that we are part of a very long future. Only that, a priori, the probability that we’re *both* in a very large future *and* one of the most influential people ever is very low. For that reason, there aren’t any implications from that argument to claims about the magnitude of extinction risk this century. We could be comparatively un-influential in many ways: if extinction risk is high this century but continues to be high for very many centuries; if extinction risk is low this century and will be higher in coming centuries; if extinction risk is any level and we can't do anything about it, or are not yet knowledgeable enough to choose actions wisely, or if longtermism is false. (etc)

Separately, I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early. Building earliness into your prior means you’ve got to give up on the very-plausible-seeming self-sampling assumption; means you’ve got to treat the predicate ‘is most influential’ differently than other predicates; has technical challenges; and the case in favour seems to rely on a posteriori observations about how the world works, like those you give in your post.

Lukas Finnveden4y17

I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early.

If we're doing things right, it shouldn't matter whether we're building earliness into our prior or updating on the basis of earliness.

Let the set H="the 1e10 (i.e. 10 billion) most influential people who will ever live" and let E="the 1e11 (i.e. 100 billion) earliest people who will ever live". Assume that the future will contain 1e100 people. Let X be a randomly sampled person.

For our unconditional prior P(X in H), everyone agrees that uniform probability is appropriate, i.e., P(X in H) = 1e-90. (I.e. we're not giving up on the self-sampling assumption.)

However, for our belief over P(X in H | X in E), i.e. the probability that a randomly chosen early person is one of the most influential people, some people argue we should utilise an e.g. exponential function where earlier people are more likely to be influential (which could be called a prior over "X in H" based on how early X is). However, it seems like you're saying that we shouldn't assess P(X in H | X in E) directly from such a prior, but instead get it from bayesian updates. So lets do that.

So now we've switched over to instead making a guess about P(X in E | X in H), i.e. the probability that one of the 1e10 most influential people also is one of the 1e11 earliest people, and dividing by 10. That doesn't seem much easier than making a guess about P(X in H | X in E), and it's not obvious whether our intuitions here would lead us to expect more or less influentialness.

Also, the way that 1e-90 and 1e-89 are both extraordinarily unlikely, but divide out to becoming 1e-1, illustrates Buck's point:

if you condition on us being at an early time in human history (which is an extremely strong condition, because it has incredibly low prior probability), it’s not that surprising for us to find ourselves at a hingey time.

William_MacAskill4y23

"If we're doing things right, it shouldn't matter whether we're building earliness into our prior or updating on the basis of earliness."

Thanks, Lukas, I thought this was very clear and exactly right.

"So now we've switched over to instead making a guess about P(X in E | X in H), i.e. the probability that one of the 1e10 most influential people also is one of the 1e11 earliest people, and dividing by 10. That doesn't seem much easier than making a guess about P(X in H | X in E), and it's not obvious whether our intuitions here would lead us to expect more or less influentialness."

That's interesting, thank you - this statement of the debate has helped clarify things for me. It does seem to me that doing the update - going via P(X in E | X in H) rather than directly trying to assess P(X in H | X in E) - is helpful, but I'd understand the position of someone who wanted just to assess P(X in H | X in E) directly.

I think it's helpful to assess P(X in E | X in H) because it's not totally obvious how one should update on the basis of earliness. The arrow of causality and the possibility of lock-in over time definitely gives reasons in favor of influential people being earlier. But there's still the big question of how great an update that should be. And the cumulative nature of knowledge and understanding gives reasons in favor thinking that later people are more likely to be more influential.

This seems important to me because, for someone claiming that we should think that we're at the HoH, the update on the basis of earliness is doing much more work than updates on the basis of, say, familiar arguments about when AGI is coming and what will happen when it does. To me at least, that's a striking fact and wouldn't have been obvious before I started thinking about these things.

CarlShulman4y47

This seems important to me because, for someone claiming that we should think that we're at the HoH, the update on the basis of earliness is doing much more work than updates on the basis of, say, familiar arguments about when AGI is coming and what will happen when it does. To me at least, that's a striking fact and wouldn't have been obvious before I started thinking about these things.

It seems to me the object level is where the action is, and the non-simulation Doomsday Arguments mostly raise a phantom consideration that cancels out (in particular, cancelling out re whether there is an influenceable lock-in event this century).

You could say a similar thing about our being humans rather than bacteria, which cumulatively outnumber us by more than 1,000,000,000,000,000,000,000,000 times on Earth thus far according to the paleontologists.

Or you could go further and ask why we aren't neutrinos? There are more than 1,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 of them in the observable universe.

However extravagant the class you pick, it's cancelled out by the knowledge that we find ourselves in our current situation. I think it's more confusing than helpful to say that our being humans rather than neutrinos is doing more than 10^70 times as much work as object-level analysis of AI in the case for attending to x-risk/lock-in with AI. You didn't need to think about that in the first place to understand AI or bioweapons, it was an irrelevant distraction.

The same is true for future populations that know they're living in intergalactic societies and the like. If we compare possible world A, where future Dyson spheres can handle a population of P (who know they're in that era), and possible world B, where future Dyson spheres can support a population of 2P, they don't give us much different expectations of the number of people finding themselves in our circumstances, and so cancel out.

The simulation argument (or a brain-in-vats story or the like) is different and doesn't automatically cancel out because it's a way to make our observations more likely and common. However, for policy it does still largely cancel out, as long as the total influence of people genuinely in our apparent circumstances is a lot greater than that of all simulations with apparent circumstances like ours: a bigger future world means more influence for genuine inhabitants of important early times and also more simulations. [But our valuation winds up being bounded by our belief about the portion of all-time resources allocated to sims in apparent positions like ours.]

Another way of thinking about this is that prior to getting confused by any anthropic updating, if you were going to set a policy for humans who find ourselves in our apparent situation across nonanthropic possibilities assessed at the object level (humanity doomed, Time of Perils, early lock-in, no lock-in), you would just want to add up the consequences of the policy across genuine early humans and sims in each (non-anthropically assessed) possible world.

A vast future gives more chances for influence on lock-in later, which might win out as even bigger than this century (although this gets rapidly less likely with time and expansion), but it shouldn't change our assessment of lock-in this century, and a substantial chance of that gives us a good chance of HoH (or simulation-adjusted HoH).

Lukas Finnveden4y13

One way to frame this is that we do need extraordinarily strong evidence to update from thinking that we're almost certainly not the most influential time to thinking that we might plausibly be the most influential time. However, we don't need extraordinarily strong evidence pointing towards us almost certainly being the most influential (that then "averages out" to thinking that we're plausibly the most influential). It's sufficient to get extraordinarily strong evidence that we are at a point in history which is plausibly the most influential. And if we condition on the future being long and that we aren't in a simulation (because that's probably when we have the most impact), we do in fact have extraordinarily strong evidence that we are very early in history, which is a point that's plausibly the most influential.

richard_ngo4y2

The question which seems important to me now is: does Will think that the probability of high influentialness conditional on birth rank (but before accounting for any empirical knowledge) is roughly the same as the negative exponential distribution Toby discussed in the comments on his original post?

Lukas Finnveden4y25

I actually think the negative exponential gives too little weight to later people, because I'm not certain that late people can't be influential. But if I had a person from the first 1e-89 of all people who've ever lived and a random person from the middle, I'd certainly say that the former was more likely to be one of the most influential people. They'd also be more likely to be one of the least influential people! Their position is just so special!

Maybe my prior would be like 30% to a uniform function, 40% to negative exponentials of various slopes, and 30% to other functions (e.g. the last person who ever lived seems more likely to be the most influential than a random person in the middle.)

Only using a single, simple function for something so complicated seems overconfident to me. And any mix of functions where one of them assigns decent probability to early people being the most influential is enough that it's not super unlikely that early people are the most influential.

William_MacAskill4y12

"Only using a single, simple function for something so complicated seems overconfident to me. And any mix of functions where one of them assigns decent probability to early people being the most influential is enough that it's not super unlikely that early people are the most influential."

I strongly agree with this. The fact that under a mix of distributions, it becomes not super unlikely that early people are the most influential, is really important and was somewhat buried in the original comments-discussion.

And then we're also very distinctive in other ways: being on one planet, being at such a high-growth period, etc.

William_MacAskill4y14

Thanks, I agree that this is key. My thoughts:

I agree that our earliness gives a dramatic update in favor of us being influential. I don't have a stable view on the magnitude of that.
I'm not convinced that the negative exponential form of Toby's distribution is the right one, but I don't have any better suggestions
Like Lukas, I think that Toby's distribution gives too much weight to early people, so the update I would make is less dramatic than Toby's
Seeing as Toby's prior is quite sensitive to choice of reference-class, I would want to choose the reference class of all observer-moments, where an observer is a conscious being. This means we're not as early as we would say if we used the distribution of Homo sapiens, or of hominids. I haven't thought about what exactly that means, though my intuition is that it means the update isn't nearly as big.

So I guess the answer to your question is 'no': our earliness is an enormous update, but not as big as Toby would suggest.

richard_ngo4y15

for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.

I want to dig into this a little, because it feels like there might be some sort of selection effect going on here. Suppose I make a claim X, and it has a number of implications X1, X2, X3 and so on. Each of these might apply to a different population, and have a different prior probability as a standalone claim. But if a critic chooses the one which has the lowest prior probability (call it Xn) in order to attack X, then it is much less fishy that Xn has a low prior probability, because the critic had many degrees of freedom in how they made their choice, which means the fishiness of the implication they choose is less reflective of the strength of the original hypothesis.

I don't know how to quantify this, but it seems very relevant to your critique of Bostrom and Yudkowsky - their hypothesis has many different implications, and so a critic can choose freely which one to criticise as fishy (in this case, the implication of current influentialness). Of course you might respond that the implication of our influentialness is the most obvious and natural one for you to evaluate. But I have two problems with this:

It's very easy to overestimate in hindsight how natural a given criterion was. Partly that's because hindsight bias is very strong in humans. Additionally, though, if there had been any other axis which led to a more obvious critique of B+Y, then it seems pretty plausible that we'd have prioritised that critique instead. So critics probably are making use of more degrees of freedom than they realise, because B+Y's hypothesis has already passed muster on the most obvious axes. (Edited to add: consider for example how many possible definitions of influentialness Will could have used which would have lead to his current argument being weaker!)
Different people have different standards for which criteria/priorities are compelling to them, and therefore should evaluate exactly the same argument differently. For example, someone who isn't altruistic and doesn't care about their influence over the future would likely find influentialness a very unnatural axis to evaluate B+Y on, and so should have more credence in their thesis than you do.

You might say that the fishiness of us being the most influential people is so extreme that it outweighs these considerations. But my suspicion here is that as the number of implications of a hypothesis grows, it becomes exponentially more likely that we find one which has a certain level of fishiness (just because you need to multiply the probabilities of each one not being that fishy, assuming that the fishiness of the implications are independent - although ofc that's a gross simplification). And so for far-ranging hypotheses, the fact that we can find one or two axes along which they fare very badly might provide relatively little evidence.

Note however that I feel pretty uncertain about these points, though, and it's quite possible that they're totally wrong.

Ofer4y12

I don’t make any claims about how likely it is that we are part of a very long future. Only that, a priori, the probability that we’re *both* in a very large future *and* one of the most influential people ever is very low. For that reason, there aren’t any implications from that argument to claims about the magnitude of extinction risk this century.

I don't understand why there are implications from that argument to claims about the magnitude of our influentialness either.

As an analogy, suppose Alice bought a lottery ticket that will win her $100,000,000 with an extremely small probability. The lottery is over, and she is now looking at the winning numbers on her phone, comparing them one by one to the numbers on her ticket. Her excitement grows as she finds more and more of the winning numbers on her ticket. She managed to verify that she got 7 numbers right (amazing!), but before she finished comparing the rest of the numbers, her battery died. She tries to find a charger, and in the meantime she's considering whether to donate the money to FHI if she wins. It occurs to her that the probability that *both* [a given person wins the lottery] *and* [donating $100,000,000 to FHI will reduce existential risk] is extremely small. She reasons that, sure, there are some plausible arguments that donating $100,000,000 to FHI will have a huge positive impact, but are those arguments strong enough considering her extremely small prior probability in the above conjunction?

Tobias_Baumann4y11

The key thing is that the way I’m setting priors is as a function from populations to credences: for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.

The fact that I consider a certain property F should update me, though. This already demonstrates that F is something that I am particularly interested in, or that F is salient to me, which presumably makes it more likely that I am an outlier on F.

Also, this principle can have pretty strange implications depending on how you apply it. For instance, if I look at the population of all beings on Earth, it is extremely surprising (10^-12 or so) that I am a human rather than an insect.

Buck4y10

On this set-up of the argument (which is what was in my head but I hadn’t worked through), I don’t make any claims about how likely it is that we are part of a very long future.

This does make a lot more sense than what you wrote in your post.

Do you agree that as written, the argument as written in your EA Forum post is quite flawed? If so, I think you should edit it to more clearly indicate that it was a mistake, given that people are still linking to it.

William_MacAskill4y3

Yeah, I do think the priors-based argument given in the post was poorly stated, and therefore led to unnecessary confusion. Your suggestion is very reasonable, and I've now edited the post.

William_MacAskill4y24

Actually, rereading my post I realize I had already made an edit similar to the one you suggest (though not linking to the article which hadn't been finished) back in March 2020:

"[Later Edit (Mar 2020): The way I state the choice of prior in the text above was mistaken, and therefore caused some confusion. The way I should have stated the prior choice, to represent what I was thinking of, is as follows:

The prior probability of us living in the most influential century, conditional on Earth-originating civilization lasting for n centuries, is 1/n.

The unconditional prior probability over whether this is the most influential century would then depend on one's priors over how long Earth-originating civilization will last for. However, for the purpose of this discussion we can focus on just the claim that we are at the most influential century AND that we have an enormous future ahead of us. If the Value Lock-In or Time of Perils views are true, then we should assign a significant probability to that claim. (i.e. they are claiming that, if we act wisely this century, then this conjunctive claim is probably true.) So that's the claim we can focus our discussion on.

It's worth noting that my proposal follows from the Self-Sampling Assumption, which is roughly (as stated by Teru Thomas ('Self-location and objective chance' (ms)): "A rational agent’s priors locate him uniformly at random within each possible world." I believe that SSA is widely held: the key question in the anthropic reasoning literature is whether it should be supplemented with the self-indication assumption (giving greater prior probability mass to worlds with large populations). But we don't need to debate SIA in this discussion, because we can simply assume some prior probability distribution over sizes over the total population - the question of whether we're at the most influential time does not require us to get into debates over anthropics.]"

Buck4y19

Oh man, I'm so sorry, you're totally right that this edit fixes the problem I was complaining about. When I read this edit, I initially misunderstood it in such a way that it didn't address my concern. My apologies.

ESRogs4y3

Separately, I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early.

Do you have some other way of updating on the arrow of time? (It seems like the fact that we can influence future generations, but they can't influence us, is pretty significant, and should be factored into the argument somewhere.)

I wouldn't call that an update on finding ourselves early, but more like just an update on the structure of the population being sampled from.

ESRogs4y3

You could make an argument that a certain kind of influence strictly decreases with time. So the hinge was at the Big Bang.

But, there (probably) weren't any agents around to control anything then, so maybe you say there was zero influence available at that time. Everything that happened was just being determined by low level forces and fields and particles (and no collections of those could be reasonably described as conscious agents).

Today, much of what happens (on Earth) is determined by conscious agents, so in some sense the total amount of extant influence has grown.

Let's maybe call the first kind of influence time-priority, and the second agency. So, since the Big Bang, the level of time-priority influence available in the universe has gone way down, but the level of aggregate agency in the universe has gone way up.

On a super simple model that just takes these two into account, you might multiply them together to get the total influence available at a certain time (and then divide by the number of people alive at that time to get the average person's influence). This number will peak somewhere in the middle (assuming it's zero both at the Big Bang and at the Heat Death).

That maybe doesn't tell you much, but then you could start taking into account some other considerations, like how x-risk could result in a permanent drop of agency down to zero. Or how perhaps there's an upper limit on how much agency is potentially available in the universe.

In any case, it seems like the direction of causality should be a pretty important part of the analysis (even if it points in the opposite direction of another factor, like increasing agency), either as part of the prior or as one of the first things you update on.

MaxDalton3y13

Review for the Decade Review

[I'm doing a bunch of low-effort reviews of posts I read a while ago and think are important, but which I don't have time to re-read or say very nuanced things about.]

I think this post makes good points on a really important topic.

I also think it's part of what is maybe the best public debate on an important topic in EA (with Will's piece). I would like to see more public debate on important topics, so I also feel good about including this in the review to signal-boost this sort of public debate.

Overall, I think this is worth including in the review.

William_MacAskill4y14

(Comment 4/5)

The argument against patient philanthropy

“I sometimes hear the outside view argument used as an argument for patient philanthropy, which it in fact is not.”

I don’t think this works quite in the way you think it does.

It is true that, in a similar vein to the arguments I give against being at the most influential time (where ‘influential’ is a technical term, excluding investing opportunities), you can give an outside-view argument against now being the time at which you can do the most good tout court. As a matter of fact, I believe that’s true: we’re almost certainly not at the point in time, in all history, at which one can do the most good by investing a given unit of resources to donate at a later date. That time could plausibly be earlier than now, because you get greater investment returns, or plausibly later than now, because in the future we might have a better understanding of how to structure the right legal instruments, specify the constitution of one’s foundation, etc.

But this is not an argument against patient philanthropy compared to direct action. In order to think that patient philanthropy is the right approach, you do not need to make the claim that now is the time, out of all times, when patient philanthropy will do the most expected good. You just need the claim that, currently, patient philanthropy will do more good than direct philanthropy. This is a (much, much) weaker claim to make.

And, crucially, there’s an asymmetry between patient philanthropy and direct philanthropy.

Suppose there are 70 time periods at which you could spend your philanthropic resources (every remaining year of your life, say), and that the scale of your philanthropy is small (so that diminishing returns can be ignored). Then, if the expected cost-effectiveness of the best opportunities varies substantially over time, there will be just one point in time at which your philanthropy will have the most impact, and you should try to max out your philanthropy at that time period, donating all your philanthropy at that time if you can. (Perhaps that isn’t quite possible because you are limited in how much you can take out debt against future income; but still, the number of times you will donate in your life will be small.) So, in 69 out of 70 time periods (or, even if you need to donate a few times, ~67 out of 70 time periods), you should be saving rather than donating. That’s why direct philanthropy needs to make the claim that now is the most, or at least one of the most, potentially-impactful times, out of the relevant time periods when one could donate, whereas patient philanthropy doesn’t.

Second, the inductive argument against now being the optimal time for patient philanthropy is much weaker than the inductive argument against now being the most influential time (in the technical sense of ‘influential). It’s not clear there is an inductive argument against now being the optimal time for patient philanthropy: there’s at least a plausible argument that, on average, every year the value of patient philanthropy decreases, because one loses one extra year of investment returns. Combined with the fact that one cannot affect the past (well, putting non-causal decision theories to the side ;) ), this gives an argument for thinking that now will be higher-impact for patient philanthropy than all future times.

Personally, I don’t think that argument quite works, because you can still mess up patient philanthropy, so maybe future people will do patient philanthropy better than we do. But it’s an argument that’s much more compelling in the case of patient philanthropy than it is for the influentialness of a time.

vaniver4y9

Then, if the expected cost-effectiveness of the best opportunities varies substantially over time, there will be just one point in time at which your philanthropy will have the most impact, and you should try to max out your philanthropy at that time period, donating all your philanthropy at that time if you can.

Tho I note that the only way one would ever take such opportunities, if offered, is by developing a view of what sorts of opportunities are good that is sufficiently motivating to actually take action at least once every few decades.

For example, when the most attractive opportunity so far appears in year 19 of investing and assessing opportunities, will our patient philanthropist direct all their money towards it, and then start saving again? Will they reason that they don't have sufficient evidence to overcome their prior that year 19 is not more attractive than the years to come? Will they say "well, I'm following the Secretary Problem solution, and 19 is less than 70/e, so I'm still in info-gathering mode"?

They won't, of course, know which path had higher value in their particular world until they die, but it seems to me like most of the information content of a strategy that waits to pull the trigger is in when it decides to pull the trigger, and this feels like the least explicit part of your argument.

Compare to investing, where some people are fans of timing the market, and some people are fans of dollar-cost-averaging. If you think the attractiveness of giving opportunities is going to be unpredictably volatile, then doing direct work or philanthropy every year is the optimal approach. If instead you think the attractiveness of giving opportunities is predictably volatile, or predictably stable, then doing patient philanthropy makes more sense.

What seems odd to me is simultaneously holding the outside view sense that we have insufficient evidence to think that we're correctly assessing a promising opportunity now, and having the sense that we should expect that we will correctly assess the promising opportunities in the future when they do happen.

Buck4y5

My claim is that patient philanthropy is automatically making the claim that now is the time where patient philanthropy does wildly unusually much expected good, because we're so early in history that the best giving opportunities are almost surely after us.

William_MacAskill4y4

This comment of mine in particular seems to have been downvoted. If anyone were willing, I'd be interested to understand why: is that because (i) the tone is off (seemed too combative?); (ii) the arguments themselves are weak; (iii) it wasn't clear what I'm saying; (iv) it wasn't engaging with Buck's argument; (v) other?

djbinder4y2

I can't speak for why other people down-voted the comment but I down-voted it because the arguments you make are overly simplistic.

The model you have of philanthropy is that on an agent in each time period has the choice to either (1) invest or (2) spend their resources, and then getting a payoff depending on how influential'' the time is. You argue that the agent should then save until they reach the most influential'' time, before spending all of their resources at this most influential time.

I think this model is misleading for a couple of reasons. First, in the real world we don't know when the most influential time is. In this case the agent may find it optimal to spend some of their resources at each time step. For instance direct philanthropic donations may give them a better understanding in the future of how influentialness varies (ie, if you don't invest in AI safety researchers now, how will you ever know whether/when AI safety will be a problem?) You may also worry about "going bust": if while you are being patient, an existential catastrophe (or value lock-in) happens, then the patient long-termist looses their entire investment.

Perhaps one way to phrase how important this knowledge problem is to finding the optimal strategy is to think about it as analogous to owning stocks in a bubble. You strategy is that we should sell at the market peak, but we can't do that if we don't know when that will be.

Second, there are very plausible reasons why now may be the best time to donate. If we can spend money today to permanently reduce existential risk, or to permanently improve the welfare of the global poor, then it is always more valuable to do that action ASAP rather than wait. Likewise we plausibly get more value by working on biorisk, AI safety, or climate change today then we will in 20 years.

Third, the assumption of no diminishing marginal returns is illogical. We should be thinking about how EAs as a whole should spend their money as a whole. As an individual, I would not want to hold out for the most influential time if I thought everyone else was doing the same, and of course as a community we can coordinate.

richard_ngo4y10

If we can spend money today to permanently reduce existential risk, or to permanently improve the welfare of the global poor, then it is always more valuable to do that action ASAP rather than wait.

This seems straightforwardly untrue, because you may be able to permanently reduce existential risk more cheaply in the future.

I also think (but am not sure) that Will doesn't include solving the knowledge problem as part of "direct action", and so your first critique is not very relevant to the choice between patient philanthropy and direct action, because probably you'll want to gain knowledge in either case.

djbinder4y1

I agree with your criticism of my second argument. What I should have instead said is a bit different. There are actions whose value decreases over time. For instance, all else being equal it is better to implement a policy which reduces existential risk sooner rather than later. Patient philanthropy makes sense only if either (a) you expect the growth of your resources to outpace the value lost by failing to act now, or (b) you expect cheaper opportunities to arise in the future. I don't think there are great reasons to believe either of these is true (or indeed false, I'm not very certain on the issue).

There are two issues with knowledge, and I probably should have separated them more clearly. The more important one is that the kind of decision-relevant information Will is asking for, that is, knowing when and how to spend your money optimally, may well just be unattainable. Optimal strategies with imperfect information probably look very different from optimal strategies with perfect information.

A secondary issue is that you actually need to generate the knowledge. I agree it is unclear whether Will is considering the knowledge problem as part of "direct" or "patient" philanthropy. But since knowledge production might eat up a large chunk of your resources, and since some types of knowledge may be best produced by trying to do direct work, plausibly the "patient philanthropist" ends up spending a lot of resources over time. This is not the image of patient philanthropy I originally had, but maybe I've been misunderstanding what Will was envisaging.

William_MacAskill4y13

(Comment 3/5)

Earliness

“Will’s resolution is to say that in fact, we shouldn’t expect early times in human history to be hingey, because that would violate his strong prior that any time in human history is equally likely to be hingey.”

I don’t see why you think I think this. (I also don’t know what “violating” a prior would mean.)

The situation is: I have a prior over how influential I’m likely to be. Then I wake up, find myself in the early 21st century, and make a whole bunch of updates. This include updates on the facts that: I’m on one planet, I’m at a period of unusually high economic growth and technological progress, I *seem* to be unusually early on and can’t be very confident that the future is short. So, as I say in the original post and the comments, I update (dramatically) on my estimate of my influentialness, on the basis of these considerations. But by how much? Is it a big enough update to conclude that I should be spending my philanthropy this year rather than next, or this century rather than next century? I say: no. And I haven’t seen a quantitative argument, yet, for thinking that the argument is ‘yes’, whereas the inductive argument seems to give a positive argument for thinking 'no'.

One reason for thinking that the update, on the basis of earliness, is not enough, is related to the inductive argument: that it would suggest that hunter-gatherers, or Medieval agriculturalists, could do even more direct good than we can. But that seems wrong. Imagine you can give an altruistic person at one of these times a bag of oats, or sell that bag today at market prices. Where would you do more good? The case in favour of earlier is if you think that speeding up economic growth / technological progress is so good that the greater impact you’d have at earlier times outweighs the seemingly better opportunities we have today. But I don’t think you believe that, and at least the standard EA view is that the benefits of speed-up are small compared to x-risk reduction or other proportional impacts on the value of the long-run future.

djbinder4y13

I'm confused as to what your core outside-view argument is Will. My initial understanding of it was the following:
(A1) We are in a potentially large future with many trillions of trillions of humans
(A2) Our prior should be that we are randomly chosen amongst all living humans
then we conclude that
(C) We should have extremely low a prior odds of being amongst the most influential
To be very crudely quantitative about this, multiplying the number of humans on earth by the number of stars in the visible universe and the lifetime of the Earth, we quickly end up with estimates of ~1e38 total humans, and so priors on the order of ~1e-38.

As Buck points out, this argument doesn't work unless you are willing to also accept with similarly extremely likelihood that the fact we appear to be very early humans is wrong. Otherwise the sheer weight of 1e38 pushes you extremely strongly to the conclusion that either (A1) is false or that we are almost certainly in a simulation.

Perhaps a somewhat different argument is closer to what you actually think. Here I've tried to frame the argument in a way that I think both you and Buck would find reasonable:
(A1') A prior it is plausible that the most influential human is early. For simplicity, let's say we have a 10% prior that the most influential human lives while the majority of humanity is still all on earth.
(A2') The number of humans that will be alive up to the end of this period is plausible on the scale of a 100 billion people.
(A3') Our evidence that humanity is restricted to one planet is incontrivertible (ie, no simulation)
We now conclude that
(C') We should have low, but not absurdly astronomically low odds of being amongst the most influential humans

Note that compared to the previous argument, the a prior odds on being the most influential person is now 1e-10, so our earliness essentially increases our belief that we are the most influential by something like 1e28. But of course a 1-in-a-100 billion prior is still pretty low, and you don't think our evidence is sufficiently strong to signficantly reduce it.

Do you agree with this argument Will? Or have a I misunderstood you?

CarlShulman4y9

Note that compared to the previous argument, the a prior odds on being the most influential person is now 1e-10, so our earliness essentially increases our belief that we are the most influential by something like 1e28. But of course a 1-in-a-100 billion prior is still pretty low, and you don't think our evidence is sufficiently strong to signficantly reduce it.

The argument is not about whether Will is the most influential person ever, but about whether our century has the best per person influence. With population of 10 billion+ (78 billion alive now, plus growth and turnover for the rest of the century), it's more like 1 in 13 people so far alive today if you buy the 100 billion humans thus far population figure (I have qualms about other hominids, etc, but still the prior gets quite high given A1, and A1 is too low).

djbinder4y5

I should also point out that, if I've understood your position correctly Carl, I agree with you. Given my second argument, that a prior we have something like 1 in a trillion odds of being the most influential, I don't think we should end up concluding much about this.

Most importantly, this is because whether or not I am the most influential person is not actually relevant decision making question.

But even aside from this I have a lot more information about the world than just a prior odds. For instance, any long-termist has information about their wealth and education which would make them fairly exceptional compared to the average human that has ever lived. They also have reasonable evidence about existential risk this century and plausible (for some loose definition of plausible) ways to influence this. At the end of the day each of us still has low odds of being the most influential person ever, but perhaps with odds more in the 1 in 10 million range, rather than 1 in a trillion.

djbinder4y5

In his first comment Will says he prefers to frame it as "influential people" rather than "influential times". In particular if you read his article (rather than the blog post), then in the end of section 5 he says he thinks it is plausible that the most influential people may live within the next few thousand years, so I don't his odds that this century is the most influential can be very low (at a guess, one in a thousand?). I might be wrong though; I'd be very curious to know what Will's prior is that the most influential person will be alive this century.

CarlShulman4y7

It's the time when people are most influential per person or per resource.

Habryka [Deactivated]4y3

(It appears you dropped a closing parenthesis in this comment)

Mark Xu4y40

I will now consider everything that Carl writes henceforth to be in a parenthetical.

Vasco Grilo🔸2y2

Hi Damon,

(C) We should have extremely low a prior odds of being amongst the most influential
To be very crudely quantitative about this, multiplying the number of humans on earth by the number of stars in the visible universe and the lifetime of the Earth, we quickly end up with estimates of ~1e38 total humans, and so priors on the order of ~1e-38.

I do not think your calculation is correct:

I think the prior probability of this being the most important century is less than 1 % based on i) the self-sampling assumption (SSA), ii) a loguniform distribution of the effective computation this century as a fraction of that throughout all time, and iii) all effective computation being more than 10^44 times as large as that this century.
[...]
In agreement with section 2 of Mogensen 2022, the self-sampling assumption says the prior probability of being the most influential person among a population of size N, comprising past, present and future people, is the expected value of 1/N, not the reciprocal of the expected value of N.

Tobias_Baumann4y6

I’m at a period of unusually high economic growth and technological progress

I think it's not clear whether higher economic growth or technological progress implies more influence. This claim seems plausible, but you could also argue that it might be easier to have an influence in a stable society (with little economic or technological change), e.g. simply because of higher predictability.

So, as I say in the original post and the comments, I update (dramatically) on my estimate of my influentialness, on the basis of these considerations. But by how much? Is it a big enough update to conclude that I should be spending my philanthropy this year rather than next, or this century rather than next century? I say: no.

I'm very sympathetic to patient philanthropy, but this seems to overstate the required amount of evidence. Taking into account that each time has donors (and other resources) of their own, and that there are diminishing returns to spending, you don't need to have extreme beliefs about your elevated influentialness to think that spending now is better. However, the arguments you gave are not very specific to 2020; presumably they still hold in 2100, so it stands to reason that we should invest at least over those timeframes (until we expect the period of elevated influentialness to end).

One reason for thinking that the update, on the basis of earliness, is not enough, is related to the inductive argument: that it would suggest that hunter-gatherers, or Medieval agriculturalists, could do even more direct good than we can. But that seems wrong. Imagine you can give an altruistic person at one of these times a bag of oats, or sell that bag today at market prices. Where would you do more good?

A bag of oats is presumably much more relative wealth in those other times than now. The current price of a ton of oats is GBP 120 per ton, so if the bag contains 50 kg, it's worth just GBP 6.

People in earlier times also have less 'competition'. Presumably the medieval person could have been the first to write up arguments for antispeciesism or animal welfare; or perhaps they could have a significant impact on establishing science, increasing rationality, improving governance, etc.

(All things considered, I think it's not clear if earlier times are more or less influential.)

weeatquince4y11

My thanks to Will and Buck for such an interesting thoughtful debate. However to me there seems to be one key difference that I think it is worth drawing out:

Will's updated article (here, p14) asks the action relevant question (rephrased by me) of:

Are we [today] among the very most influential people, out of the very large number of people who will live that we could reasonably pass resources to [over the coming thousand years].

Buck's post (this post) seems to focus on the not action relevant question (my phrasing) of:

Are we [in this century] at the very most influential time, out of the very large span of all time into the distant future [over the coming trillion years].

It seems plausible to me that Buck's criticisms are valid when considering Will's older work, however I do not think the criticisms that Buck raises here about Will's original HoH still applies to the action relevant restricted HoH in Will's new paper. (And I see little value in debating the non-action relevant HoH hypothesis.)

davidoj4y9

I think there's some confusion here about "Will's prior". Some people seem to think that Will's prior implies that the influentialness of an arbitrary individual is independent of the time in which they live. However, this isn't right - Will's prior instead says that, before I know when I will live, I shouldn't expect to live in a very influential time (or in later statements, that I shouldn't expect to be a particularly influential person).

~ means "is of similar magnitude" in the following.

Suppose our prior defines some probability space on which we have the following random variables: C: index of century in which I'm alive, H: a function century index -> hingeyness, Z:= H(X) and U: a function time -> human population, M: index identifying the human who is "me".

Will's prior invokes the self sampling assumption, which says that P(M) is uniform over the set of all humans (or consciousness-moments). This implies that P(C=i|U) is proportional to U(i). Then P(Z) = \sum_{C,H} P(Z|C=i,H=h)P(C=i,H=h) = \sum_{C,H} delta_h(i)P(C=i,H=h).

This puts no restriction at all on P(C=i,H=h). Will argues, I think, that it would be unreasonable to presume that max(H) and C are strongly correlated enough to lead to E(Z) close to E[max(H)].

Several people suggest - I think - that we might be able to come up with reasonable prior guesses of P(H|U); for example, if U is large for a very long time, H might be uniformly low or decreasing over time. It could also be increasing over time and because I don't really understand "influence" or "hingeyness" I have a hard time thinking about this question.

If large U => E[H] decreases over time, then E[Z|C~0] might be close to E[max(H)], but E[Z] will continue to be small, which seems to satisfy both Buck's and Will's intuitions.

Myself, I don't know what I should think of P(Z|C~0), which seems to be the crucial probability here.

William_MacAskill4y7

Comment (5/5)

Smaller comments

I agree that one way you can avoid thinking we’re astronomically influential is by believing the future is short, such as by believing you’re in a simulation, and I discuss that in the blog post at some length. But, given that there are quite a number of ways in which we could fail to be at the most influential time (perhaps right now we can do comparatively little to influence the long-term, perhaps we’re too lacking in knowledge to pick the right interventions wisely, perhaps our values are misguided, perhaps longtermism is false, etc), it seems strange to put almost all of the weight on one of those ways, rather than give some weight to many different explanations.
“It’s not clear why you’d think that the evidence for x-risk is strong enough to think we’re one-in-a-million, but not stronger than that.” This seems pretty strange as an argument to me. Being one-in-a-thousand is a thousand times less likely than being one-in-a-million, so of course if you think the evidence pushes you to thinking that you’re one-in-a-million, it needn’t push you all the way to thinking that you’re one-in-a-thousand. This seems important to me. Yes, you can give me arguments for thinking that we’re (in expectation at least) at an enormously influential time - as I say in the blog post and the comments, I endorse those arguments! I think we should update massively away from our prior, in particular on the basis of the current rate of economic growth. But for direct philanthropy to beat patient philanthropy, being at a hugely influential time isn’t enough. Even if this year is hugely influential, next year might be even more influential again; even if this century is hugely influential, next century might be more influential again. And if that’s true then - as far as the consideration of wanting to spend our philanthropy at the most influential times goes - then we have a reason for saving rather than donating right now.
You link to the idea that the Toba catastrophe was a bottleneck for human populations. Though I agree that we used to be more at-risk from natural catastrophes than we are today, more recent science has cast doubt on that particular hypothesis. From The Precipice: “the “Toba catastrophe hypothesis” was popularized by Ambrose (1998). Williams (2012) argues that imprecision in our current archeological, genetic and paleoclimatological techniques makes it difficult to establish or falsify the hypothesis. See Yost et al. (2018) for a critical review of the evidence. One key uncertainty is that genetic bottlenecks could be caused by founder effects related to population dispersal, as opposed to dramatic population declines.”
- Ambrose, S. H. (1998). “Late Pleistocene Human Population Bottlenecks, Volcanic Winter, and Differentiation of Modern Humans.” Journal of Human Evolution, 34(6), 623–51
- Williams, M. (2012). “Did the 73 ka Toba Super-Eruption have an Enduring Effect? Insights from Genetics, Prehistoric Archaeology, Pollen Analysis, Stable Isotope Geochemistry, Geomorphology, Ice Cores, and Climate Models.” Quaternary International, 269, 87–93.
- Yost, C. L., Jackson, L. J., Stone, J. R., and Cohen, A. S. (2018). “Subdecadal Phytolith and Charcoal Records from Lake Malawi, East Africa, Imply Minimal Effects on Human Evolution from the ∼74 ka Toba Supereruption.” Journal of Human Evolution, 116, 75–94.

Gregory Lewis🔸4y29

“It’s not clear why you’d think that the evidence for x-risk is strong enough to think we’re one-in-a-million, but not stronger than that.” This seems pretty strange as an argument to me. Being one-in-a-thousand is a thousand times less likely than being one-in-a-million, so of course if you think the evidence pushes you to thinking that you’re one-in-a-million, it needn’t push you all the way to thinking that you’re one-in-a-thousand. This seems important to me. Yes, you can give me arguments for thinking that we’re (in expectation at least) at an enormously influential time - as I say in the blog post and the comments, I endorse those arguments! I think we should update massively away from our prior, in particular on the basis of the current rate of economic growth. (My emphasis)

William_MacAskill4y8

Thanks, Greg. I really wasn't meaning to come across as super confident in a particular posterior (rather than giving an indicative number for a central estimate), so I'm sorry if I did.

"It seems more reasonable to say 'our' prior is rather some mixed gestalt on considering the issue as a whole, and the concern about base-rates etc. should be seen as an argument for updating this downwards, rather than a bid to set the terms of the discussion."

I agree with this (though see for the discussion with Lukas for some clarification about what we're talking about when we say 'priors', i.e. are we building the fact that we're early into our priors or not.).

Gregory Lewis🔸4y10

But what is your posterior? Like Buck, I'm unclear whether your view is the central estimate should be (e.g.) 0.1% or 1 / 1 million. I want to push on this because if your own credences are inconsistent with your argument, the reasons why seem both important to explore and to make clear to readers, who may be mislead into taking this at 'face value'.

From this on page 13 I guess a generous estimate (/upper bound) is something like 1/ 1 million for the 'among most important million people':

[W]e can assess the quality of the arguments given in favour of the Time of Perils or Value Lock-in views, to see whether, despite the a priori implausibility and fishiness of HH, the evidence is strong enough to give us a high posterior in HH. It would take us too far afield to discuss in sufficient depth the arguments made in Superintelligence, or Pale Blue Dot, or The Precipice. But it seems hard to see how these arguments could be strong enough to move us from a very low prior all the way to significant credence in HH. As a comparison, a randomised controlled trial with a p-value of 0.05, under certain reasonable assumptions, gives a Bayes factor of around 3 in favour of the hypothesis; a Bayes factor of 100 is regarded as ‘decisive’ evidence. In order to move from a prior of 1 in 100 million to a posterior of 1 in 10, one would need a Bayes factor of 10 million — extraordinarily strong evidence.

I.e. a prior of ~ 1/ 100 million (which is less averse than others you moot earlier), and a Bayes factor < 100 (i.e. we should not think the balance of reason, all considered, is 'decisive' evidence), so you end up at best at ~1/ 1 million. If this argument is right, you can be 'super confident' giving a credence of 0.1% is wrong (out by an ratio of >~ 1000, the difference between ~ 1% and 91%), and vice-versa.

Yet I don't think your credence on 'this is the most important century' is 1/ 1 million. Among other things it seems to imply we can essentially dismiss things like short TAI timelines, Bostrom-Yudkowsky AI accounts etc, as these are essentially upper-bounded by the 1/ 1M credence above.*

So (presuming I'm right and you don't place negligible credence on these things) I'm not sure how these things can be in reflective equilibrium.

1: 'Among the most important million people' and 'this is the most important century' are not the same thing, and so perhaps one has a (much) higher prior on the latter than the former. But if the action really was here, then the precisification of 'hinge of history' as the former claim seems misguided: "Oh, this being the most important century could have significant credence, but this other sort-of related proposition nonetheless has an astronomically adverse prior" confuses rather than clarifies.

2: Another possibility is there are sources of evidence which give us huge updates, even if the object level arguments in (e.g.) Superintelligence, The Precipice etc. are not among them. Per the linked conversation, maybe earliness gives a huge shift up from the astronomically adverse prior, so this plus the weak object level evidence gets you to lowish but not negligible credence.

Whether cashed out via prior or update, it seems important to make such considerations explicit, as the true case in favour of HH would include these considerations too. Yet the discussion of 'how far you should update' on p11-13ish doesn't mention these massive adjustments, instead noting reasons to be generally sceptical (e.g. fishiness) and the informal/heuristic arguments for object level risks should not be getting you Bayes factors ~100 or more. This seems to be hiding the ball if in fact your posterior is ultimately 1000x or more your astronomically adverse prior, but not for reasons which are discussed (and so a reader may neglect to include when forming their own judgement).

*: I think there's also a presumptuous philosopher-type objection lurking here too. Folks (e.g.) could have used a similar argument to essentially rule out any x-risk from nuclear winter before any scientific analysis, as this implies significant credence in HH, which the argument above essentially rules out. Similar to 'using anthropics to hunt', something seems to be going wrong where the mental exercise of estimating potentially-vast future populations can also allow us to infer the overwhelming probable answers for disparate matters in climate modelling, AI development, the control problem, civilisation recovery and so on.

William_MacAskill4y17

Thanks for this, Greg.

"But what is your posterior? Like Buck, I'm unclear whether your view is the central estimate should be (e.g.) 0.1% or 1 / 1 million."

I'm surprised this wasn't clear to you, which has made me think I've done a bad job of expressing myself.

It's the former, and for the reason of your explanation (2): us being early, being on a single planet, being at such a high rate of economic growth, should collectively give us an enormous update. In the blog post I describe what I call the outside-view arguments, including that we're very early on, and say: "My view is that, in the aggregate, these outside-view arguments should substantially update one from one’s prior towards HoH, but not all the way to significant credence in HoH.[3]
[3] Quantitatively: These considerations push me to put my posterior on HoH into something like the [1%, 0.1%] interval. But this credence interval feels very made-up and very unstable."

I'm going to think more about your claim that in the article I'm 'hiding the ball'. I say in the introduction that "there are some strong arguments for thinking that this century might be unusually influential", discuss the arguments that I think really should massively update us in section 5 of the article, and in that context I say "We have seen that there are some compelling arguments for thinking that the present time is unusually influential. In particular, we are growing very rapidly, and civilisation today is still small compared to its potential future size, so any given unit of resources is a comparatively large fraction of the whole. I believe these arguments give us reason to think that the most influential people may well live within the next few thousand years." Then in the conclusion I say: "There are some good arguments for thinking that our time is very unusual, if we are at the start of a very long-lived civilisation: the fact that we are so early on, that we live on a single planet, and that we are at a period of rapid economic and technological progress, are all ways in which the current time is very distinctive, and therefore are reasons why we may be highly influential too." That seemed clear to me, but I should judge clarity by how readers interpret what I've written.

Gregory Lewis🔸4y23

For my part, I'm more partial to 'blaming the reader', but (evidently) better people mete out better measure than I in turn.

Insofar as it goes, I think the challenge (at least for me) is qualitative terms can cover multitudes (or orders of magnitudes) of precision. I'd take ~0.3% to be 'significant' credence for some values of significant. 'Strong' 'compelling' or 'good' arguments could be an LR of 2 (after all, RCT confirmation can be ~3) or 200.

I also think quantitative articulation would help the reader (or at least this reader) better benchmark the considerations here. Taking the rough posterior of 0.1% and prior of 1 in 100 million, this implies a likelihood ratio of ~~100 000 - loosely, ultra-decisive evidence. If we partition out the risk-based considerations (which it discussion seems to set as 'less than decisive' so <100), the other considerations (perhaps mostly those in S5) give you a LR of > ~1000 - loosely, very decisive evidence.

Yet the discussion of the considerations in S5 doesn't give the impression we should conclude they give us 'massive updates'. You note there are important caveats to these considerations, you say in summing up these arguments are 'far from watertight', and I also inferred the sort of criticisms given in S3 around our limited reasoning ability and scepticism of informal arguments would also apply here too. Hence my presumption these other considerations, although more persuasive than object level arguments around risks, would still end up below the LR ~ 100 for 'decisive' evidence, rather than much higher.

Another way this would help would be illustrating the uncertainty. Given some indicative priors you note vary by ten orders of magnitude, the prior is not just astronomical but extremely uncertain. By my lights, the update doesn't greatly reduce our uncertainty (and could compound it, given challenges in calibrating around very high LRs). If the posterior odds could be 'out by 100 000x either way' the central estimate being at ~0.3% could still give you (given some naive log-uniform) 20%+ mass distributed at better than even odds of HH.

The moaning about hiding the ball arises from the sense this numerical articulation reveals (I think) some powerful objections the more qualitative treatment obscures. E.g.

Typical HH proponents are including considerations around earliness/single planet/ etc. in their background knowledge/prior when discussing object level risks. Noting the prior becomes astronomically adverse when we subtract these out of background knowledge, and so the object level case for (e.g.) AI risk can't possibly be enough to carry the day alone seems a bait-and-switch: you agree the prior becomes massively less astronomical when we include single planet etc. in background knowledge, and in fact things like 'we live on only one planet' are in our background knowledge (and were being assumed at least tacitly by HH proponents).
The attempt to 'bound' object level arguments by their LR (e.g. "Well, these are informal, and it looks fishy, etc. so it is hard to see how you can get LR >100 from these") doesn't seem persuasive when your view is that the set of germane considerations (all of which seem informal, have caveats attached, etc.) in concert are giving you an LR of ~100 000 or more. If this set of informal considerations can get you more than half way from the astronomical prior to significant credence, why be so sure additional ones (e.g.) articulating a given danger can't carry you the rest of the way?
I do a lot of forecasting, and I struggle to get a sense of what priors of 1/ 100 M or decisive evidence to the tune of LR 1000 would look like in 'real life' scenarios. Numbers this huge (where you end up virtually 'off the end of the tail' of your stipulated prior) raise worries about consilience (cf. "I guess the sub-prime morgage crisis was a 10 sigma event"), but moreover pragmatic defeat: there seems a lot to distrust in an epistemic procedure along the lines of "With anthropics given stipulated subtracted background knowledge we end up with an astronomically minute prior (where we could be off by many orders of magnitude), but when we update on adding back in elements of our actual background knowledge this shoots up by many orders of magnitude (but we are likely still off by many orders of magnitude)". Taking it face value would mean a minute update to our 'pre theoretic prior' on the topic before embarking on this exercise (providing these overlapped and was not as radically uncertain, varying no more than a couple rather than many orders of magnitude). If we suspect (which I think we should) this procedure of partitioning out background knowledge into update steps which approach log log variance and where we have minimal calibration is less reliable than using our intuitive gestalt over our background knowledge as whole, we should discount its deliverances still further.

William_MacAskill4y4

Thanks Greg - I asked and it turned out I had one remaining day to make edits to the paper, so I've made some minor ones in a direction you'd like, though I'm sure they won't be sufficient to satisfy you.

Going to have to get back on with other work at this point, but I think your arguments are important, though the 'bait and switch' doesn't seem totally fair - e.g. the update towards living in a simulation only works when you appreciate the improbability of living on a single planet.

Buck4y11

How much of that 0.1% comes from worlds where your outside view argument is right vs worlds where your outside view argument is wrong?

This kind of stuff is pretty complicated so I might not be making sense here, but here's what I mean: I have some distribution over what model to be using to answer the "are we at HoH" question, and each model has some probability that we're at HoH, and I derive my overall belief by adding up the credence in HoH that I get from each model (weighted by my credence in it). It seems like your outside view model assigns approximately zero probability to HoH, and so if now is the HoH, it's probably because we shouldn't be using your model, rather than because we're in the tiny proportion of worlds in your model where now is HoH.

I think this distinction is important because it seems to me that the probability of HoH give your beliefs should be almost entirely determined by the prior and HoH-likelihood of models other than the one you proposed--if your central model is the outside-view model you proposed, and you're 80% confident in that, then I suspect that the majority of your credence on HoH should come from the other 20% of your prior, and so the question of how much your outside-view-model updates based on evidence doesn't seem likely to be very important.

Buck4y5

“It’s not clear why you’d think that the evidence for x-risk is strong enough to think we’re one-in-a-million, but not stronger than that.” This seems pretty strange as an argument to me. Being one-in-a-thousand is a thousand times less likely than being one-in-a-million, so of course if you think the evidence pushes you to thinking that you’re one-in-a-million, it needn’t push you all the way to thinking that you’re one-in-a-thousand. This seems important to me.

So you are saying that you do think that the evidence for longtermism/x-risk is enough to push you to thinking you're at a one-in-a-million time?

EDIT: Actually I think maybe you misunderstood me? When I say "you're one-in-a-million", I mean "your x-risk is higher than 99.9999% of other centuries' x-risk"; "one in a thousand" means "higher than 99.9% of other centuries' x-risk". So one-in-a-million is a stronger claim which means higher x-risk.

What I'm saying is that if you believe that x-risk is 0.1%, then you think we're at least one in a million. I don't understand why you're willing to accept that we're one-in-a-million; this seems to me force you to have absurdly low x-risk estimates.

vaniver4y6

What I'm saying is that if you believe that x-risk is 0.1%, then you think we're at least one in a million.

I think you're saying "if you believe that x-risk this century is 0.1%, then survival probability this century is 99.9%, and for total survival probability over the next trillion years to be 0.01%, there can be at most 9200 centuries with risk that high over the next trillion years (.999^9200=0.0001), which means we're in (most generously) a one-in-one-million century, as a trillion years is 10 billion centuries, which divided by ten thousand is a million." That seem right?

Misha_Yagudin4y7

re: "This post has a lot of very small numbers in it. I might have missed a zero or two somewhere."

Hey Buck, consider using scientific notation instead of decimal one: "0.00000009%" is hard to read and 9e-10 is less prone to typos.

kokotajlod4y18

I'm guessing Buck spelled out the zeros for dramatic effect; it makes it easier to see intuitively how small the numbers are.

ESRogs4y5

And the current increase in hinginess seems unsustainable, in that the increase in hinginess we’ve seen so far leads to x-risk probabilities that lead to drastic reduction of the value of worlds that last for eg a millennium at current hinginess levels.

Didn't quite follow this part. Are you saying that if hinginess keeps going up (or stays at the current, high level), that implies a high level of x-risk as well, which means that, with enough time at that hinginess (and therefore x-risk) level, we'll wipe ourselves out; and therefore that we can't have sustained, increasing / high hinginess for a long time?

(That's where I would have guessed you were going with that argument, but I got confused by the part about "drastic reduction of the value of worlds ..." since the x-risk component seems like a reason the high-hinginess can't last a long time, rather than an argument that it would last but coincide with a sad / low-value scenario.)

Buck4y7

Your interpretation is correct; I mean that futures with high x-risk for a long time aren't very valuable in expectation.

RomeoStevens4y4

In order for hingeyness to stay uniform robustness to x-risk would need to scale uniformly with power needed to cause x-risk.

Ofer4y3

This topic seems extremely important and I strongly agree with your core argument.

As Will notes, following Brian Tomasik and others, the simulation argument dampens enthusiasm for influencing the far future.

There is no reason for longtermists to care specifically about "the far future" (interpreted as our future light cone, or whatever spacetime we can causally affect). Most longtermists probably intrinsically care about all of spacetime across Reality. Even if the probability that we are not in a short simulation is 1e-50, longtermists still have strong reasons to strive for existential security. One of those reasons is that striving for existential security would make all civilizations that are similar to us (i.e. civilizations that their behavior is correlated with ours) more likely to successfully use their cosmic endowment in beneficial ways.

Regarding the part about the outside view argument being also an argument against patient philanthropy: this seems to depend on some non-obvious assumptions. If the population size in some future year X is similar to today's population size, and the fraction of wealth generated until X but not inherited by people living in X is sufficiently small, then a random person living in X will be able to donate an amount that is similar (in expectation) to the worth of a patient-philanthropy-fund that was donated by a random person living today.

Hank_B3y1

If a long future is not plausible, a uniform prior of hingy-ness makes sense even when considering the non-negligible amounts of x-risk we seem to observe now.

It also offers an explanation for us being in an 'early' time, there is no later time we could have been born in. In other words, humanity doesn't have much time left so being born a long time into the future is the implausible bit.

Shouldn't this doomsday argument have a higher prior probability than a sudden decline in x-risk or simulation? We've seen extinction events happen before, but not the other two.