I don't think this is as clear of a dichotomy as people think it is. A lot of global catastrophic risk doesn't come from literal extinction because human extinction is very hard. A lot of mundane work on GCR policy involves a wide variety of threat models that are not just extinction.

Footnote 2 completely changes the meaning of the statement from common sense interpretations of the statement. It makes it so that e.g. a future scenario in which AI takes over and causes existential catastrophe and the extinction of biological humans this century does not count as extinction, so long as the AI continues to exist. As such, I chose to ignore it with my "fairly strongly agree" answer.

The value of the future conditional on civilization surviving seems positive to me, but not robustly so. I think the main argument for its being positive is theoretical (e.g., Spreading happiness to the stars seems little harder than just spreading), but the historical/contemporary record is ambiguous.

The value of improving the future seems more robustly positive if it is tractable. I suspect it is not that much less tractable than extinction risk work. I think a lot of AI risk satisfies this goal as well as the x-risk goal for reasons Will MacAskill gives in What We Owe the Future. Understanding, developing direct interventions for, and designing political processes for digital minds seem like plausible candidates. Some work on how to design democratic institutions in the age of AI also seems plausibly tractable enough to compete with extinction risk.

I think there are a lot of thorny definitional issues here that make this set of issues not boil down that nicely to a 1D spectrum. But overall extinction prevention will likely have a far broader coalition supporting it, while making the future large and amazing is far less popular since most people aren't very ambitious with respect to spreading flourishing through the universe, but I tentatively am.

I think increasing the value of good futures is probably higher importance, but much less tractable

I don't think it's valuable to ensure future moral patients exist for their own sake, and extinction risk reduction only really seems to expectably benefit humans who would otherwise die in an extinction event, who would be in the billions. An astronomical number of future moral patients could have welfare at stake if we don't go extinct, so I'd prioritize them on the basis of their numbers.

See this comment and thread.

If we treat digital minds like current animal livestock, the expected value of the future could be really bad. 

Most people have a strong drive to perpetuate humanity. What makes EAs special is that EAs also care about others' suffering. So EAs should focus on trying to make sure the future isn't full of suffering.

I think human extinction (from ASI) is highly likely to happen, and soon, unless we stop it from being built^[1]

See my comments in the Symposium for further discussion

Two major reasons/ considerations:
1- I'm unconvinced of the tractability of non-extinction-risk reducing longtermist interventions. 
2- Perhaps this is self-defeating - but I feel uncomfortable substantively shaping the future in ways that aren't merely making sure it exists. Visions of the future that I would have found un-objectionable a century ago would probably seem bad to me today. In short - this consideration is basically "moral uncertainty". I think extinction-risk reduction is, though not recommended on every moral framework, at least recommended on most. I haven't seen other ideas for shaping the future which are as widely recommended. 

I genuinely have no idea. 

I think the expected value of the long-term future, in the “business as usual” scenario, is positive. In particular, I anticipate that advanced/transformative artificial intelligence drives technological innovation to solve a lot of world problems (e.g., helping create cell-based meat eventually), and I also think a decent amount of this EV is contained in futures with digital minds and/or space colonization (even though I’d guess it’s unlikely we get to that sort of world). However, I’m very uncertain about these futures—they could just as easily contain a large amount of suffering. And if we don’t get to those futures, I’m worried about wild animal suffering being high in the meantime. Separately, I’m not sure addressing a lot of s-risk scenarios right now is particularly tractable (nor, more imminently, does wild animal suffering seem awfully tractable to me). 

Probably the biggest reason I’m so close to the center is I think a significant amount of existential risk from AI looks like disempowering humanity without killing literally every human, and hence, I view AI alignment work as at least partially serving the goal of “increasing the value of futures where we survive.”

Depends, if x-risk is small (<5%) and if we expect outsized impact on the future (preventing negative value lock-ins), then the latter seems more important. I'm very unsure about both. 

I have mixed feelings here. But one major practical worry I have about "increasing the value of futures" is that a lot of that looks fairly zero-sum to me. And I'm scared of getting other communities to think this way. 

If we can capture 5% more of the universe for utilitarian aims, for example, that's 5% less from others. 

I think it makes sense for a lot of this to be studied in private, but am less sure about highly public work.

I'm optimistic about the very best value-increasing research/interventions. But in terms of what would actually be done at the margin, most work that people would do for "value-increasing" reasons would be confused/doomed, I expect (and this is less true for AI safety).

I dunno, by how much? Seems contingent on lots of factors. 

The long-running future seems like it could well be unacceptably awful. From the perspective of a battery hen, it would seem much better that it's distant ancestors were pushed out of an ecological niche before humanity domesticated them. Throwing all our effort into X-risk mitigation without really tackling S-risks in a world of increasing lock-in across domains seems deeply unwise to me. 

It's a tractability issue. In order for these interventions to be worth funding, they should reduce our chance of extinction not just now, but over the long term. And I just haven't seen many examples of projects that seem likely to do that.

This is a difficult one, and both my thoughts and my justifications (especially the few sources I cite) are very incomplete. 

It seems to me for now that existential risk reduction is likely to be negative, as both human and AI-controlled futures could contain immense orders of magnitude more suffering than the current world (and technological developments could also enable more intense suffering, whether in humans or in digital minds). The most salient ethical problems with the extinction of earth-originating intelligent life seem to be the likelihood of biological suffering continuing on earth for millions of years (though it's not clear to me whether it would be more or less intense without intelligent life on earth), and the possibility of space (and eventually earth) being colonized by aliens (though whether their values will be better or worse remains an open question in my view).

Another point (which I'm not certain about how to weigh in my considerations) is that certain extinction events could massively reduce suffering on earth, by preventing digital sentience or even by causing the end of biological sentient life (this seems unlikely, and I've asked here how likely or unlikely EAs thought this was).

However, I am very uncertain of the tractability of improving future outcomes, especially considering recent posts by researchers at the Center on Long-Term Risk, or this one by a former researcher there, highlighting how uncertain it is that we are well-placed to improve the future. Nonetheless, I think that efforts made to improve the future, like the work of the Center for Reducing Suffering, the Center on Long-Term Risk, or the Sentience Institute, advocate for important values and could have some positive flow-through effects in the medium-term (though I don't necessarily think that this robustly improves the longer term future). I will note, however, that I am biased since work related the Center for Reducing Suffering was the primary reason I got into EA.

I am very open to changing my mind on this, but for now I'm under 50% agree because it seems to me that, in short:

1. Extinction Risk Reduction could very well have negative expected value.
2. Efforts to improve the value of futures where we survive might have some moderate positive effects in the short-term.

Lots of uncertainties. I expect to have moved my cursor before the end of the week!

Under moral uncertainty, many moral perspectives care much more about averting downsides than producing upsides.

Additionally, tractability is probably higher for extinction-level threats, since they are "absorptive"; decreasing the chance we end up in one gives humanity and their descendants ability to do whatever they figure out is best.

Finally, there is a meaningful sense in which working on improving the future is plagued by questions about moral progress and lock-in of values, and my intuition is that most interventions that take moral progress seriously and try to avoid lock-in boil down to working on things that are fairly equivalent to avoiding extinction. Interventions that don't take moral progress seriously instead may look like locking in current values.

I think avoiding existential risk is the most important thing. As long as we can do that and don't have some kind of lock in, then we'll have time to think about and optimize the value of the future.

I think people overrate how predictable the effect of our actions on the future will be (even though they rate it very low in absolute terms); extinction seem like one of the very few (only?) things that seems like its effects will endure throughout a big part of the future. Still buy the theory that 0-1% of possible value is equally valuable to 98-99%; just about tractability

It seems even more important to avoid futures full of extreme suffering than to avoid extinction.

While I don't entirely disregard x-risks, I have been unimpressed by the tractability of most interventions, excepting perhaps for bio-security ones.

The prevalent notion of "solving" the alignment problem as though it's a particularly hard math problem strikes me as overly represented, which entails neglecting other, more in-direct safety measures, like stable, transparent and trustworthy institutions, whether political or geopolitical (US-China war means what for AI?).

Relatedly, the harm aversion/moral purity signaling around working in AI companies (especially Anthropic!) seems counterproductive and yet has received ~no push back. It seems obvious to me that having concerned individuals (rather than E-acc ideologues) in high positions in companies is very important! I suspect that the dominance of Doomerism in AI Safety over more epistemically sound concerns is to blame. 

I find it very difficult to determine whether the future will be net-negative or net-positive (when considering humans, factory-farmed animals, wild animals, and possibly artificial sentience). 
This makes it very hard to know whether work on extinction reduction is likely to be positive or not.
I prefer to work on things that aim to move the sign towards "net-positive".

Far from convinced that continued existence at currently likely wellbeing levels is a good thing

I really don't have a strong view, but I find myself sympathetic to the idea that the world is not great and is getting worse (if you consider non-human animals).

Broadly Agree

Although I might have misunderstood and missed the point of this entire debate, so correct me if that is the case

I just don't believe  changing the future trajectory is tractable in say 50-100 years from now areas like politics, economics, AI welfare etc. I think its a pipe dream. We cannot predict technological, political and economic changes even in the medium term future. These changes may well quickly render our current efforts meaningless in 10-20 years. I think the effect of work we do now which is future focused diminishes in value exponentially to the point that in 20 years time it is probably almost meaningless. I could be convinced against this if you could show me interventions 20 years ago which were specifically future focus which are now still meaningful.

However I do believe it may be possible to reduce existential risk through direct, tangeble changes now which could prevent a disastrous one off event like a engnieered virus or AI apocalypse. Stopping a one off catastrophe though still exremely difficult seems much more realistic than influencing an overall global trajectory with so much uncertainty.

This is unless you include general global health and development and animal welfare work which directly improves the lives of beings today, and will also likely improve beings lives in the future but probably in a far more minor way than this question is addressing. If you include the effects ofstandard EA GHD and animal welfare work under the banner "Improving the value of futures where we survive" then I would probably swing my vote the other way.

On the current margin, improving our odds of survival seems much more crucial to the long-term value of civilization. My reason for believing this is that there are some dangerous technologies which I expect will be invented soon, and are more likely to lead to extinction in their early years than later on. Therefore, we should currently spend more effort on ensuring survival, because we will have more time to improve the value of the future after that.

(Counterpoint: ASI is the main technology that might lead to extinction, and the period when it's invented might be equally front-loaded in terms of setting values as it is in terms of extinction risk.)

It seems plausible to me we might be approaching a "time of perils' where total x-risk is unacceptably high and will continue to be as we develop powerful AI systems, but might decrease later since we can use AI systems to tackle x-risk (though that seems hard and risky in its own myriad ways).

Broadly think we should still prioritise avoiding catastrophes in this phase, and bet on being able to steer later but low confidence.

Partly this is because I think “extinction” as defined here is very unlikely (<<1%) to happen this century, which upper bounds the scale of the area. I think most “existential risk” work is not squarely targeted at avoiding literal extinction of all Earth-originating life.

I feel extremely unsure about this one. I'm voting slightly against purely from the perspective of, "wow, there are projects in that direction that feel super neglected".

Recent advances in LLMs have led me to update toward believing that we live in the world where alignment is easy (i.e. CEV naturally emerge from LLMs, and future AI agents will be based on understanding and following natural language commands by default), but governance is hard (i.e. AI agents might be co-opted by governments or corporations to lock in humanity in a dystopian future, and the current geopolitical environment, characterized by democratic backsliding, cold war mongering, and an increase in military conflicts including wars of aggression, isn't conducive to robust multilateral governance).

This is a question I could easily change my mind on.

The experience of digital minds seems to dominate far future calculations. We can get a lot of value from this, a lot of disvalue, or anything in between.

If we go extinct then we get 0 value from digital minds. This seems bad, but we also avoid the futures where we create them and they suffer. It’s hard to say if we are on track to creating them to flourish or suffer - I think there are arguments on both sides. The futures where we create digital minds may be the ones where we wanted to “use” them, which could mean them suffering. Alternatively, we have seen our moral circle expand over time and this may continue, so there is a real possibility we could create them to flourish. I don’t have a clear view which side wins here, so overall going extinct doesn’t seem obviously terrible to me.

We could instead focus on raising the moral status of digital minds, our ability to understand sentience and consciousness, improve societal values, making sure AI goes well and helps us with these things. These robustly increase the expected value of digital sentience in futures where we survive. 

So because reducing extinction risk is close to 0 expected value to me, and increasing the value of futures where we survive is robustly positive in expected value, I lean towards increasing the value of futures where we survive.

Intelligence is the only chance of some redemption to the massive suffering probably associated to the emergence of consciousness. 

This is the age of danger, we are the first species on Earth that has figured out morality, so we shall survive at allmost all cost. 

Making people happy is valuable; making happy people is probably not valuable. There is an asymmetry between suffering and happiness because it is more morally important to mitigate suffering than to create happiness.

The far future, on our current trajectory, seems net negative on average. Reducing extinction risk just multiplies its negative EV. 

We could devote 100% of currently available resources to existential risk reduction, live in austerity, and never be finished ensuring our own survival.  However, if increase the value of futures where we survive, we will develop more and more resources that can then be put to existential risk reduction.  People will be not only happier, but also more capable and skilled, when we create a world where people can thrive rather than just survive.  The highest quality futures are the most robust.  

Very worried about AI risk, think short timelines are plausible

A higher value future reduces the chances of extinction.  If people value life, they will figure out how to keep it.

I think the most likely outcome is not necessarily extinction (I estimate <10% due to AI) but rather an unfulfilled potential. This may be humans simply losing control over the future and becoming mere spectators and AI not being morally significant in some way.

With long timeline and less than 10% probability: Hot take is these are co-dependent - prioritizing only extinction is not feasible. Additionally, does only one human exist while all others die count as non-extinction? What about only a group of humans survive? How should this be selected? It could dangerously/quickly fall back to Fascism. It would only likely benefit the group of people with current low to no suffering risks, which unfortunately correlates to the most wealthy group. When we are "dimension-reducing" the human race to one single point, we ignore the individuals. This to me goes against the intuition of altruism. 

I fundamentally disagree with the winner-take-all type of cause prioritization - instead, allocate resources to each area, and unfortunately there might be multiple battles to fight.

To analyze people's responses, I can see this question being adjusted to consider prior assumptions: 1. What's your satisfaction on how we are currently doing in the world now? What are the biggest gaps to your ideal world? 2.What's your assessment of timeline + current % of extinction risk due to what?

What is the point of securing a future for humanity if that future is net-negative?

We can't work on increasing value if we are dead.

If we avoid extinction, plenty of people will have the time to take care of humanity's future. I'll leave it to them. Both topic have a lot of common ground anyway, like "not messing up with the biosphere" or "keeping control of ASI"