You are welcome!

It sounds broadly akin to how I'm inclined to address Pascal's Mugging cases (treat the astronomical stakes as implying proportionately negligible probability).

Makes sense. I see Pascal's muggings as instances where the probability of the offers is assessed indepently of their outcomes. In contrast, for any distribution with a finite expected value, the expected value density (product between the PDF and value) always ends up decaying to 0 as the outcome increases. In meta-analyses, effect sizes, which can be EVs under a given model, are commonly weighted by the reciprocal of their variance. Variance tends to increase with effect size, and therefore larger effect sizes are usually weighted less heavily.

People sometimes point to Holden Karnofsky's post Why we can’t take expected value estimates literally (even when they’re unbiased) to justify not relying on EVs (here are my notes on it from 4 years ago). However, the post does not broadly argue against using EVs. I see it as a call for not treating all EVs the same, and weighting them appropriately.

Do you not understand the scenario conditions under which the actual value would be high?

I do. In theory, there could be worlds which i) will actually be astronomically valuable given some efforts, but that ii) will be practically actually neutral without such efforts. In this case, the efforts would be astronomically valuable due to meaningfully increasing the probability of astronomically valuable futures. I just think such worlds are super implausible. I can see many worlds satisfying i), but not i) and ii) simultaneously.

We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.

I agree. I illustrated my point with a single model in my past comment. However, I believe appropriately weighted sets of models lead to the same conclusion that the expected increase in welfare decreases with actual welfare when this is astronomical (because I think the increase in the probability of the actual welfare decreases faster than the actual welfare when this is astronomical).

The point of RHSINO ['Rule High Stakes In, Not Out'] is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldn't establish that our all-models-considered EV [expected value] must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/scenarios.

I understand what you have in mind. If EV is 1/0.99 under model A with 99 % weight, and 1 M under B with 1 % weight, A contributes 1 (= 0.99*1/0.99) to the EV considering both models, and B contributes 10 k (= 0.01*1*10^6). The contribution of A is 0.01 % (= 1/(1*10^4)) that of B. So changing by a given percentage the EV under B, or weight of B changes the EV considering both models much more that changing by the same percentage the EV under A, or weight of A. For example, doubling (increasing by 100 %) the EV under B would increase the EV considering both models by 1 (= 1*1), whereas doubling the EV under A would increase the EV considering both models by 10 k (= 1*10^4), 10 k (= 1*10^4/1) times the increase resulting from increasing the EV under B. The change in the EV considering all models resulting from changing the EV under a model, or its weight is proportional to the contribution of the model to the EV considering all models, and the relative change in the EV under the model, or its weight.

However, I think your presentation of RHSINO assumes the conclusion it is trying to support. High-stakes models are the ones that matter if they are the major driver of the EV under all models. However, the question is whether they are the major driver of the EV under all models. This has to be determined based on empirical evidence, not on points that are mathematically true.

People usually give weights that are at least 0.1/"number of models", which easily results in high-stakes models dominating. However, giving weights which are not much smaller than the uniform weight of 1/"number of models" could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to "distance"^-2 (correct answer), "distance"^-20, or "distance"^-200, I imagine I would get a significant fraction picking the exponents of -20 and -200. Assuming 60 % picked -2, 20 % picked -20, and 20 % picked -200, a respondant may naively conclude the mean exponent of -45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Alternatively, a respondant may naively conclude an exponent of -9.19 (= 0.933*(-2) + 0.0333*(-20) + 0.0333*(-200)) is reasonable giving a weight of 3.33 % (= 0.1/3) to each of the 2 wrong exponents, equal to 10 % of the uniform weight, and the remaining weight of 93.3 % (= 1 - 2*0.0333) to the correct exponent. Yet, there is lots of empirical evidence against the exponents of -45.2 and -9.19 which the respondants are not aware of. The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks. This is also why I am sceptical that the absolute value of the welfare per unit time of animals is bound to be relatively close to that of humans, as one may naively infer from the welfare ranges Rethink Priorities (RP) initially presented, or the ones in Bob Fischer's book about comparing welfare across species, where there seems to be only 1 line about the weights. "We assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive model".

Mistakes like the one illustrated above happen when the weights of models are guessed independently of their stakes. People are often sensitive to astronomically high stakes, but not to the astronomically low weights they imply.

You may be interested in my chat with Matthew Adelstein. We discussed my scepticism of longtermism.

Thanks for the follow-up, Richard.

Are you just saying that high expected value is not sufficient for actual value because we might get unlucky?

No. I agree high expected value does not ensure high actual value, but I only care about the expected (not the actual) increase in welfare.

Other possibilities offer far greater extensions. Obviously the latter possibilities are the ones that ground high expected value estimates.

This is far from obvious to me, and I wonder what makes you so confident. I am not aware of any reasonably quantitative and empirical modelling of the expected increase in future welfare under a longtermist perspective. I expect what David Thorstad calls rapid diminution. I see the difference between the probability density function (PDF) after and before an intervention reducing the nearterm risk of human extinction as quickly decaying to 0, thus making the increase in the expected value of the astronomically valuable worlds negligible. For instance:

• If the difference between the PDF after and before the intervention decays exponentially with the value of the future v, the increase in the expected value density caused by the intervention will be proportional to v*e^-v.

• The above rapidly goes to 0 as v increases. For an expected value of the future equal to my estimate of 1.40*10^52 human lives, the increase in the expected value density will involve a factor of 1.40*10^52*e^(-1.40*10^52) = 10^(log10(1.40) + 52 - log10(e)*1.40*10^52) = 10^(-6.08*10^51), i.e. it will be basically 0.

I think Shulman's points give us reason to think there's a non-negligible chance of averting extinction (extending civilization) for a long time. Pointing out that other possibilities are also possible doesn't undermine this claim.

It seems that you are inferring one could meaningfully increase the probability of astronomically valuable futures because these are plausible. However, astronomically valuable futures could be plausible while having a probability that is very hard to change.

Thanks for sharing, Jamie and David! How do you assess the impact of the Pulse project so far? How has it changed the decisions of Rethink Priorities (RP), and impact-focussed funders?

Thanks for crossposting this, Richard!

The stakes surrounding ASI are extremely high, to the point that we can’t be confident that humanity would long survive this development

I guess the risk of human extinction over the next 10 years is 10^-7, and I am not aware of any quantitative empirical modelling suggesting otherwise. I do not think that is high enough to justify your conclusion that "AI risk warrants urgent further investigation and precautionary measures", even accounting for longterm effects. I very much agree there should be some investigation and precautionary measures, but I do not consider this "urgent" at the margin. 

Good to know!

I would be happy to donate 2 k$ to RP myself. You could use this to scope out whatever project or projects you believe would decrease the most cost-effectively the uncertainty about how the (expected hedonistic) welfare per unit time of different organisms and digital systems compares with that of humans.

I don't expect I'll reply further

You are welcome to return to this later. I would be curious to know your thoughts.

EV is subjective. I'd recommend this post for more on this.

I liked the post. I agree EV is subjective to some extent. The same goes for the concept of mass, which depends on our imperfect understanding of physics. However, the expected mass of objects is still comparable, unless there is only an infinitesimal difference between their mass.

Thanks for sharing!

This report is written in a governing thoughts structure; the contents serve as a high level summary:

Nitpick. I think the section would have been more readable if you had writted it in the following format.

• Bullet 1 (link)

  • Sub-bullet 1 (link)

  • ...

• ...

Then only one word ("link") per line would have a link.

Thanks for sharing, and welcome to the EA Forum, Pawel! You may benefit from checking out the organisations working on interventions related to mental health:

• Recommended by the Happier Lives Institute (HLI):
  • Strong Minds.
  • Friendship Bench.
• Incubated by Ambitious Impact (AIM):
  • Actra.
  • Canopie.
  • Happier Lives Institute.
  • Kaya Guides.
  • Vida Plena.