Thanks, this back and forth is very helpful. I think I've got a clearer idea about what you're saying. 

I think I disagree that it's reasonable to assume that there will be a fixed N = 10^35 future lives, regardless of whether it ends up Malthusian. If it ends up not Malthusian, I think I'd expect the number of people in the future to be far less than whatever the max imposed by resource constraints is, ie much less than 10^35.

So I think that changes the calculation of E[saving one life], without much changing E[preventing extinction], because you need to split out the cases where Malthusianism is true vs false.

E[saving one life] is 1 if Malthusianism is true, or something fraction of the future if Malthusianism is false, but if it's false, then we should expect the future to be much smaller than 10^35. So the EV will be much less than 10^35.

E[preventing extinction] is 10^35 if Malthusianism is true, and much less if it's false. But you don't need that high a credence to get an EV around 10^35.

So I guess all that to say that I think your argument is right and also action relevant, except I think the future is much smaller in non-Malthusian worlds, so there's a somewhat bigger gap than "just" 10^10. I'm not sure how much bigger. 

What do you think about that?

Ah nice, thanks for explaining! I'm not following all the calculations still, but that's on me, and I think they're probably right.

But I don't think your argument is actually that relevant to what we should do, even if it's right. That's because we don't care about how good our actions are as a fraction/multiple of what our other options are. Instead, we just want to do whatever leads to the best expected outcomes. 

Suppose there was a hypothetical world where there was a one in ten chance the total figure population was a billion, and 90% chance the population was two. And suppose we have two options: save one person, or save half the people.

In that case, the expected value of saving half the people would be 0.9*1 + 0.1*500,000,000 = about 50,000,001. That's compared to the expected value of 1 of saving one person. Imo, this is a strong reason for picking the "save half the people option".

But the expected fraction of people saved by the options is quite different. The "save half" option always results in half being saved. And the expected value of the "save one" option is also very close to half: 0.9*0.5 + 0.1*1/1,000,000,000. Even though the two interventions look very similar from this perspective, I think it's basically irrelevant - expected value is the relevant thing. 

What do you think? I might well have made a mistake, or misunderstood still.

I think your calculations must be wrong somewhere, although I can't quite follow them well enough to see exactly where. 

If you have a 10% credence in Malthusianism, then the expected badness of extinction is 0.1*10^35, or whatever value you think a big future is. That's still a lot closer to 10^35 times the badness of one death than 10^10 times.

Does that seem right?

Agree coin flip is unacceptable! Or even much less than coin flip is still unacceptable.

I agree with this comment, but I interpreted your original comment as implying a much greater degree of certainty of extinction assuming ASI is developed than you might have intended. My disagree vote was meant to disagree with the implication that it's near certain. If you think it's not near certain it'd cause extinction or equivalent, then it does seem worth considering who might end up controlling ASI!

You're stating it as a fact that "it is" a game of chicken, i.e. that it's certain or very likely that developing ASI will cause a global catastrophe because of misaligned takeover. It's an outcome I'm worried about, but it's far from certain, as I see it. And if it's not certain, then it is worth considering what people would do with aligned AI.

I heard reports of it getting out of sync or being out of date in some way. For example, a room change on Swapcard not being reflected in the Google calendar. I haven't tried it myself, and I haven't heard anything less vague, sorry. 

I think that the Google calendar syncing is at least a bit buggy for now, FYI. Agree good news though!

When will the next round likely be?

Thanks Vasco! :)

I agree that thinking about other moral theories is useful for working out what utilitarianism would actually recommend.

That's an interesting point re increasing the total amount of killing, I hadn't considered that! But I was actually picking up on your comment which seemed to say something more general - that you wouldn't intrinsically take into account whether an option involved (you) killing people, you'd just look at the consequences (and killing can lead to worse consequences, including in indirect ways, of course). But it sounds like maybe your response to that is you're not worried about moral uncertainty / you're sure about utilitarianism / you don't have any reason to avoid killing people, other than the (normally very significant) utilitarian reasons not to kill?