Thanks! I appreciate you clarifying this, and for being clear about it. Views along these lines are what I always expect subjectivists to have, and they never do, and then I feel confused.

 

Nice argument, I hadn't heard that before! 

I think what we should be talking about is whether we hit the "point of no return" this century for extinction of Earth-originating intelligent life. Where that could mean: Homo sapiens and a most other mammals get killed off in an extinction event this century; then technologically-capable intelligence never evolves again on Earth; so all life dies off within a billion years or so. (In a draft post that you saw of mine, this is what I had in mind.)

The probability of this might be reasonably high. There I'm at idk 1%-5%.

In the first of these, I think most of the EV comes from whether technologically-capable intelligence evolves or not. I'm more likely or not on that (for say extinction via bio-catastrophe), but not above 90%.
 

Thanks! I haven't read your stuff yet, but it seems like good work; and this has been a reason in my mind for being more in favour of trajectory change than totla extinction reduction for a while. It would only reduce the value of extinction risk reduction by an OOM at most, though?

I'm sympathetic to something in Mediocrity direction (for AI-built civilisations as well as human-built civilisations), but I think it's very hard to have a full-blooded Mediocrity principle if you also think that you can take actions today to meaningfully increase or decrease the value of Earth-originating civilisation. Suppose that Earth-originating civilisation's value is V, and if we all worked on it we could increase that to V+ or to V-. If so, then which is the right value for the alien civilisation? Choosing V rather than V+ or V- (or V+++ or V--- etc) seems pretty arbitrary. 

Rather, we should think about how good our prospects are compared to a random draw civilisation. You might think we're doing better or worse, but if it's possible for us to move the value of the future around, then it seems we should be able to reasonably think that we're quite a bit better (or worse) than the random draw civ. 

We can then work out the other issues once we have more time to think about them

Fin and I talk a bit about the "punting" strategy here.

I think it works often, but not in all cases.

For example the AI capability level that poses a meaningful risk of human takeover comes earlier than the AI capability level that poses a meaningful risk of AI takeover. Because some humans are coming with loads of power, already, and the amount of strategic intelligence you need to take over, if you already have loads of power, is less than the strategic capability you need if you're starting off with almost none (which will be true of the ASI).


 

Agreed!

My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value.

 

So you think:
1. People with your values control 1 in 1 millionth of future resources, or less? This seems pessimistic! 

2. But maybe you think it's just you who has your values and everyone else would converge on something subtly different - different enough to result in the loss of essentially all value. Then the 1-in-1-million would no longer seem so pessimistic. 

But if so, then suppose I'm Galactic Emperor and about to turn everything into X, best by my lights... do you really take a 99.9% chance of extinction, and a 0.1% chance of stuff optimised by you, instead?  

3. And if so, do you think that Tyler-now has different values than Tyler-2026? Or are you worried that he might have slightly different values, such that you should be trying to bind yourself to the mast in various ways?


4. Having such a low v(future) feels hard to maintain in light of model uncertainty and moral uncertainty. 

E.g. what's the probability you have that:
i. People in general just converge on what's right?
ii. People don't converge, but a significant enough fraction converge with you that you and others end up with more than 1milllionth of resources?
iii. You are able to get most of what you want via trade with others?

Yeah, I think that lock-in this century is quite a bit more likely than extinction this century. (Especially if we're talking about hitting a points of no return for total extinction.)

That's via two pathways:
- AGI-enforced institutions (including AGI-enabled immortality of rulers).
- Defence-dominance of star systems 

I do think that "path dependence" (a broader idea than lock-in) is a big deal, but most of the long-term impact of that goes via a billiards dynamic: path-dependence on X, today, affects some lock-in event around X down the road. (Where e.g. digital rights and space governance are plausible here.)
 

Surely any of our actions changes who exists in the future? So we aren't in fact benefiting them?

(Whereas we can benefit specific aliens, e.g. by leaving our resources for them - our actions today don't affect the identities of those aliens.)