I think yes and for all the reasons. I'm a bit sceptical that we can change the values ASIs will have - we don't understand present models that well, and there are good reasons not to treat how a model outputs text as representative of its goals (it could be hallucinating, it could be deceptive, it's outputs might just not be isomorphic to a reward structure). 

And even if we could, I don't know of any non-controversial value to instill in the ASI, that isn't just included in basic attempts to control the ASI (which I'd be doing mostly for extinction related reasons). 

I've had drafts of this take lying around for years - really glad to see it out in the open! 
I'd love to hear pushback from anyone who thinks it is still valuable. 

I think this is a fair point - but it's not the frame I've been using to consider debate week topics.

My aim has been to generate useful discussion within the effective altruism community. I'd like to choose topics which nudge people to examine assumptions they've been making, and might lead to them changing their minds, and perhaps their priorities, or the focus of their work. I haven't been thinking about debate weeks as a piece of communications work/ as a way of reaching out to a broader audience. This question in particular was chosen because the Forum audience wouldn't necessarily have cached takes on it - an audience outside the Forum would need a lot of context to get what we are talking about.

Perhaps I'm missing something though - do you think this is more public facing than I'm assuming? To be clear, I know that it is public, but it's not directed at an outside audience in the way a book or podcast or op-ed might be. 

Edit: I'm also uncertain on the claim that "there are few interventions that are predictably differentiated along those lines" - I think Forethought would disagree, and though I'm not sure I agree with them, they've thought about it more than I have. 

Yeah I've always been a bit sceptical of this as well. Surely it's just a yardstick that a department uses to decide between which investments it should make, rather than a considered (or even descriptive) "value of a life" for the US Government. 
Descriptively - the US government would spend far more for a few lives if those lives were hostages of a foreign adversary, and probably has far less willingness to pay for cheap ways the US govt could save lives (idk what these are, probably there are examples in public health). 
Basically - I don't think it's a number that can be meaningfully extrapolated to figure out the value of avoiding extinction or catastrophe, because the number was designed with far smaller trade-offs in mind, and doesn't really make sense outside of its intended purpose. 

Thanks - yep I think this is becoming a bit of an issue (it came up a couple times in the symposium as well). I might edit the footnote to clarify - worlds with morally valuable digital minds should be included as a non-extinction scenario, but worlds where an AI which could be called "intelligent life" but isn't conscious/ morally valuable takes over and humans become extinct should count as an extinction scenario. 

I think the "earth-originating intelligent life" term should probably include something that indicates sentience/ moral value. Perhaps you could read that into "intelligent" but that feels like a stretch. But I didn't want to imply that a world with no humans but many non-conscious AI systems would count as anything but an extinction scenario - that's one of the key extinction scenarios. 

Maybe another way to think about this (dropping the religion stuff - don't want to cast aspersions on any particular religions) is that we could think of black-ball and white-ball ideologies (like the Bostrom thought experiment where black-balls = technologies which can cause extinction). Perhaps certain ideologies are just much more exclusive and expansion focused than others - black-balls. You can pick out as many white-balls as you like, but picking out a black-ball means you have to get rid of your white-balls. Even if there are few black-balls in the bag, you'd always end up holding one. 

No worries!

I think Owen is voting correctly Robi - he disagrees that there should be more work on extinction reduction before there is more work on improving the value of the future. (to complicate this, he is understanding working on AI x-risk is mostly about increasing the value of the future, because, in his view, it isn't likely to lead to extinction). 

Apologies if the "agree" "disagree" labelling is unclear - we're thinking of ways to make it more parsable. 

I'm wondering whether we should expect worlds which converge on moral views to converge on bad moral views. 

From the space of world religions - we've seen a trend where we converge over time (at least from a high level of abstraction where we can refer to "Christianity" and "Islam" rather than "mega-church Christians" or whatever). Is this because the religions that succeed are exclusive and expansionary? Of all religions that have existed, I know that many of them don't much care if you also worship other gods. My empirical (ish) question is whether we should expect world in which a sizable fraction of the population follows the same religion to be one where the religion they follow is exclusive (you can't follow others) and expansionary (other people should also follow this religion). PS- I know that not all Christians or Muslims are exclusionary about other religions, this is over-simplified. 

This is relevant because, if this is a mechanism, we might expect the same thing of morality or political organisation - beliefs which demand you don't follow others, and that others follow the same beliefs as you, rather than tolerant beliefs. Perhaps this would make it more likely that futures which converge have converged on something extreme and closed, rather than exploratory and open. 

This is pretty vague - just wondering if others a) know more than me about the religion question and can speak to that or b) have had similar thoughts, or c) think that the existence of exclusive and expansionary (and wrong) ideologies might make convergence more likely.