"What matters most about our actions is their very long term effects."
I think my takeaway from this slogan is: given limited evaluation capacity + some actions under consideration, a substantial proportion of this capacity should be debited to thinking about long term effects.
It could be false: maybe it's easy to conclude that nothing important can be known about the long term effects. However, I don't think this has been demonstrated yet.
I haven't fully grokked this work yet, but I really appreciate the level of detail you've explained it in.
It seems plausible a significant fraction of ICs will choose to become GCs. Since matter and energy are likely to be instrumentally useful to most ICs, expanding to control as much volume as they can (thus becoming a GC) is likely to be desirable to many ICs with diverse aims.
Also, if an IC is a mixture of grabby and non-grabby elements, it will become a GC essentially immediately.
Now I wish there were numbers in the OP to make referencing easier
Edit: thanks
You've gotten me interested in looking at total extinction risk as a follow up, are you interested in working together on it?
From the title, I thought this was going to be a defense of being money pumped!
Do you know of work on this off the top of your head? I know if Ord has his estimate of 6% extinction in the next 100 years, but I don't know of attempts to extrapolate this or other estimates.
I think for long timescales, we wouldn't want to use an exchangeable model, because the "underlying risk" isn't stationary
def model(a,b,conc_expert,expert_forecast):
# forecasted distribution over annual probability of nuclear war
prior_rate = numpyro.sample('rate',dist.Beta(a,b))
with numpyro.plate('w',1000):
war = numpyro.sample('war',dist.Bernoulli(prior_rate),infer={'enumerate':'parallel'})
anywars = (war.reshape(10,100).sum(1)>1).mean()
expert_prediction = numpyro.sample('expert',dist.Beta(conc_expert*anywars,conc_expert*(1-anywars)),obs=expert_forecast)
Nice explanation, thanks
How about this:
A) Take top N interventions ranked by putting all effort into far future effects
B) Take top N interventions ranked by putting more effort into near than far future effects
(you can use whatever method you like to prioritise the interventions you investigate). Then for most measures of value, group (A) will have much higher expected value than group (B). Hence "most of the expected value is in the far future".