Hide table of contents

Summary: Why many different worldviews should prioritise reducing existential risk. Also an exhaustive list of people who can ignore this argument. (Writeup of an old argument I can't find a source for.)

Confidence: 70%.

Crossposted from gleech.org.

---

Imagine someone who thought that art was the only thing that made life worth living. [1] What should they do? Binge on galleries?

Work to increase the amount of art and artistic experience, by going into finance to fund artists? Or by becoming an activist for government funding for the arts? Maybe. But there’s a case that they should pay attention to ways the world might end: after all, you can’t enjoy art if we’re all dead.

1. Aesthetic experience is good in itself: it’s a terminal goal.
2. The extinction of life would destroy all aesthetic experience & prevent future experiences.
3. So reducing existential risk is good, if only to protect the conditions for aesthetic experience.

And this generalises to a huge range of values:

1. [good] is good in itself: it’s a terminal goal.
2. The extinction of life would destroy [good], and prevent future [good].
3. So reducing existential risk is good, if only to protect the conditions for [good].

Caspar Oesterheld gives a few examples of what people can plug into those brackets:

Abundance, achievement, adventure, affiliation, altruism, apatheia, art, asceticism, austerity, autarky, authority, autonomy, beauty, benevolence, bodily integrity, challenge, collective property, commemoration, communism, community, compassion, competence, competition, competitiveness, complexity, comradery, conscientiousness, consciousness, contentment, cooperation, courage, [crab-mentality], creativity, crime, critical thinking, curiosity, democracy, determination, dignity, diligence, discipline, diversity, duties, education, emotion, envy, equality, equanimity, excellence, excitement, experience, fairness, faithfulness, family, fortitude, frankness, free will, freedom, friendship, frugality, fulfillment, fun, good intentions, greed, happiness, harmony, health, honesty, honor, humility, idealism, idolatry, imagination, improvement, incorruptibility, individuality, industriousness, intelligence, justice, knowledge, law abidance, life, love, loyalty, modesty, monogamy, mutual affection, nature, novelty, obedience, openness, optimism, order, organization, pain, parsimony, peace, peace of mind, pity, play, population size, preference fulfillment, privacy, progress, promises, property, prosperity, punctuality, punishment, purity, racism, rationality, reliability, religion, respect, restraint, rights, sadness, safety, sanctity, security, self-control, self-denial, self-determination, self-expression, self-pity, simplicity, sincerity, social parasitism, society, spirituality, stability, straightforwardness, strength, striving, subordination, suffering, surprise, technology, temperance, thought, tolerance, toughness, truth, tradition, transparency, valor, variety, veracity, wealth, welfare, wisdom.

So "from a huge variety of viewpoints, the end of the world is bad"? What a revelation!

: the above is only interesting if we get from “it’s good to reduce x-risk” to “it’s the most important thing to do” for these values. This would be the case if extinction was both 1) relatively likely relatively soon, and 2) we could do something about it. We can't be that confident of either of these things, but there are good reasons to both worry and plan.

(If you think that we can only be radically uncertain about the future, note that this implies you should devote more attention to the worst scenarios, not less: ‘high uncertainty’ is not the same as ‘low probability’.)

It's hard to say at what precise level of confidence and discount rate this argument overrides direct promotion of [good]; I'm claiming that it's implausible that your one lifetime of direct promotion would outweigh all future instances, if you're a consequentialist and place reasonable weigh on future lives.

When I first wrote this, I thought the argument had more force for people with high moral uncertainty - i.e. the more of Oesterheld's list you think are plausibly actually terminal goods, the more you'd focus on x-risk. But I don't think that follows, and anyway there are much stronger kinds of uncertainty, involving not just which terminal values you credit, but whether there are moral properties at all, whether maximisation is imperative, whether promotion or honouring counts as good. The above argument is about goal-independence (within consequentialism), and says nothing about framework-independence. So:


Who doesn’t have to work on reducing x-risk?

* People with incredibly high confidence that nothing can be done to affect extinction (that is, well above 99% confidence).

* Avowed egoists. (Though Scheffler argues that even they have to care here.)

* 'Parochialists': People who think that the responsibility to help those you’re close to outweighs your responsibility to any number of distant others.

* People with values that don’t depend on the world:

* Nihilists, or other people who think there are no moral properties.
* People with an ‘honouring’ kind of ethics - like Kantians, Aristotelians, or some religions. Philip Pettit makes a helpful distinction: when you act, you can either 'honor' a value (directly instantiate it) or 'promote' it (make more opportunities for it, make it more likely in future). This is a key difference between consequentialism and two of the other big moral theories (deontology and virtue ethics): the latter two only value honouring. This could get them off the logical hook because, unless "preventing extinction" was a duty or virtue itself, or fit easily into another duty or virtue, there's no moral force against it. (You could try to construe reducing x-risk as "care for others" or "generosity".) [2]


* People that disvalue life:

* Absolute negative utilitarians or antinatalists: people who think that life is generally negative in itself.
* People who think that human life has, and will continue to have, net-negative effects. Of course, a deep ecologist who sided with extinction would be hoping for a horrendously narrow event, between ‘one which ends all human life’ and ‘one which ends all life’. They’d still have to work against the latter, which covers the artificial x-risks.
* Ordinary utilitarians might also be committed to this view, in certain terrible contingencies (e.g. if we inexorably increased the number of suffering beings via colonisation or simulation).


* The end of the world is not the worst scenario: you might instead have a world with unimaginable amounts of suffering lasting a very long time, an ‘S-risk’. You might work on those instead. This strikes me as admirable and important, it just doesn’t have the complete value-independence that impressed me about the argument at the start of this piece.

* People who don’t think that probability estimates or expected value should be used for moral decisions. ('Intuitionists'.)

* You might be ‘satisficing’ - you might view the Good as a piecewise function, where having some amount of the good is vitally important, but any more than that has no moral significance. This seems more implausible than maximisation.


Uncertainties

* We really don’t know how tractable these risks are: we haven’t acted, as a species, on unprecedented century-long projects with literally only one chance for success. (But again, this uncertainty doesn’t licence inactivity, because the downside is so large.)

* I previously had the following exempted:

People with incredibly high confidence that extinction will not happen (that is, well above 99% confidence). This is much higher confidence than most people who have looked hard at the matter.

But Ord argues that these people actually should prioritise x-risk, since extinction being very hard implies a long future, and so much greater future expected value. It's not clear what assumptions his model makes, besides low discount rate and at least minimal returns to x-risk reduction. (h/t makaea)


* There is some chance that our future will be negative - especially if we spread normal ecosystems to other planets, or if hyper-detailed simulations of people turn out to have moral weight. If the risk increased (if the moral circle stopped expanding, if research into phenomenal consciousness and moral weight stagnated), these could ‘flip the sign’ on extinction, for me.

* I was going to add ‘person-affecting' people to the exemption list. But actually if the probability of extinction in the next 80 years (one lifetime) is high enough (1% ?) then they probably have to act too, even despite ignoring future generations.

* Most people are neither technical researchers nor willing to go into government. So, if x-risk organisation ran out of “room for more funding” then most people would be off the hook (back to maximising their terminal goal directly), until they had some.

* We don’t really know how common real deontologists are. (That one study is n=1000 about Sweden, probably an unusually consequentialist place.) As value-honourers, they can maybe duck most of the force of the argument.

* Convergence, for instance the above argument, is often suspicious, when humans are persuading themselves or others.

---

[1]: For example, Nietzsche said 'Without music, life would be a mistake.' (Though strictly this is bluster: he certainly valued many other things.)

[2]: Pummer claims that all "minimally plausible" versions of the honouring ethics must include some promotion. But I don't see how they can, without being just rule-utilitarians in disguise.

EDIT 8/12/18: Formatting. Also added Ord's hazard rate argument, h/t makaea.

Comments22


Sorted by Click to highlight new comments since:

Strong upvote. This is a fantastic post, and I wish that people who downvoted it had explained their reasoning, because I don't see any big flaws.

I don't necessarily agree with everything written here, and I don't think the argument would suffice to convince people outside of EA, but we need more content like this, which:

  • Cites a lot of philosophers who aren't commonly cited in EA (good for two reasons: non-EA philosophers are the vast majority of philosophers and presumably have many good ideas, including on areas we care about; citing a wider range of philosophers makes EA work look a lot more credible)
  • Carefully points out a lot of uncertainties and points that could be made against the argument. I hadn't put a name before on the difference between "honoring" and "promoting", but I suspect that many if not most peoples' objection to focusing on X-risk probably takes this form if you dig deep enough.
  • Includes a summary and a confidence level.

A couple of things I wish had been different:

  • I don't know what "confidence level" means, given the wide range of ways a person could "agree" with the argument. Is this your estimate of the chance that a given person's best bet is to give to whatever X-risk organization they think is best, as long as they aren't one of your groups? Your estimate of how solid your own argument is, where 100% is "logically perfect" and 0% is "no evidence whatsoever"? Something else?
  • The formatting is off in some places, which doesn't impact readability too much but can be tricky in a post that uses so many different ways of organizing info (quotes, bullets, headings, etc.) One specific improvement would be to replace your asterisk footnotes with numbers [1] so that it's easier to find them and not mix them up with bullet points.

Aside from the honor/promote distinction, I think the most common objection to this from someone outside of EA might be something like "extinction is less than 1% likely, not because the world isn't dangerous but because I implicitly trust other people to handle that sort of thing, and prefer to focus on local issues that are especially important to myself, my family, and my community".

[1] Like this.

what "confidence level" means

Good question. To be honest, it was just me intuiting the chance that all of the premises and exemptions are true, which maybe cashes out to your first option. I'm happy to use a conventional measure, if there's a convention on here.

Would also invite people who disagree to comment.

something like "extinction is less than 1% likely, not because..."

Interesting. This neatly sidesteps Ord's argument (about low extinction probability implying proportionally higher expected value) which I just added, above.

Another objection I missed, which I think is the clincher inside EA, is a kind of defensive empiricism, e.g. Jeff Kaufman:

I'm much more skeptical than most people I talk to, even most people in EA, about our ability to make progress without good feedback. This is where I think the argument for x-risk is weakest: how can we know if what we're doing is helping..?

I take this very seriously; it's why I focus on the ML branch of AI safety. If there is a response to this (excellent) philosophy, it might be that it's equivalent to risk aversion (the bad kind) somehow. Not sure.

You can get similar value-independence in favour of extinction by using "bads" instead of "goods". Many of the values in Oesterheld's list have opposites which could reasonably be interpreted as "bads", and some of them are already "bads", e.g. suffering, pain and racism.

True - but how many people hold these inverses to be their primary value? (That is, I think the argument above is useful because almost everyone has something in the Goods set.)

I think even more people have things in the bads set, and there will be more agreement on these values, too, e.g. suffering, cruelty and injustice. The question is then a matter of weight.

Most people (and probably most EAs) aren't antinatalists, so you would expect, for them, the total good to outweigh the total bad. Or, they haven't actually thought about it enough.

Ah I see. Agreed - thanks for clarifying.

Besides the person-affecting views and disvalue of life covered here, if an individual has an Epicurean view of life and death (another kind of person-affecting view), i.e. death is not bad, then improving wellbeing should probably take priority. And while Epicureanism assigns 0 disvalue to death (ignoring effects on others), one could assign values arbitrarily close to 0.

There are also issues with dealing with infinities that make utilitarianism non-action guiding (it doesn't tell us what to do in most practical cases); you could probably throw these in with nihilism. E.g. if the universe is unbounded ("infinite") in space or time, then we can't change the total sum of utility, and that number is not even well-defined (not even +infinity or -infinity) with the usual definitions of convergence in the real numbers. If you assign any nonzero probability to an infinite universe, you end up with the same problem, but it's actually pretty likely that the universe is spatially unbounded. There are several attempts at solutions, but all of them have pretty major flaws, AFAIK.

Some person-affecting views can help, i.e. using a Pareto principle, but then it's not clear how to deal with individuals whose exact identities depend on your decisions (or maybe we just ignore them; many won't like that solution), and there are still many cases that can't be handled. There's discussion in this podcast, with some links for more reading (ctrl-F "Pareto" after expanding the transcript): https://80000hours.org/podcast/episodes/amanda-askell-moral-empathy/

Rounding sufficiently small probabilities to 0 and considering only parts of the universe we're extremely confident we can affect can help, too. This proposed solution and a few others are discussed here: https://nickbostrom.com/ethics/infinite.pdf

You could also have a bounded vNM utility function, but this means assigning decreasing marginal value to saving lives, and how you divide decisions/events matters, e.g. "saving 1 life and then saving 1 life" > "saving 2 lives and then saving 0 lives".

For the unbounded time case (assuming we can handle or avoid issues with unbounded space, and people might prefer not to treat time and space differently): https://forum.effectivealtruism.org/posts/9D6zKRPfaALiBhnnN/problems-and-solutions-in-infinite-ethics

I've written a blogpost on whether Christians should share the emphasis that many EAs put on the long term, including extinction risks. Since this fits nicely with your aim in this blogpost -- i.e. whether *many* worldviews should prioritise existential risks -- I thought I'd mention it here: https://eachdiscussion.wordpress.com/2019/04/06/how-much-should-christian-eas-care-about-the-far-future-part-i/

Thanks for this. I'm not very familiar with the context, but let me see if I understand. (In a first for me, I'm not sure whether to ask you to cite more scripture or add more formal argument.) Let's assume a Christian god, and call a rational consequence-counting believer an Optimising Christian.

Your overall point is that there are (or might be) two disjoint ethics, one for us and one for God, and that ours has a smaller scope, falling short of long-termism, for obvious reasons. Is this an orthodox view?

1. "The Bible says not to worry, since you can trust God to make things right. Planning is not worrying though. This puts a cap on the intensity of our longterm concern."

2. "Humans are obviously not as good at longtermism as God, so we can leave it to Him."

3. "Classical theism: at least parts of the future are fixed, and God promised us no (more) existential catastrophes. (Via flooding.)"

4. "Optimising Christians don't need to bring (maximally many) people into existence: it's supererogatory." But large parts of Christianity take population increase very seriously as an obligation (based on e.g. Genesis 1:28 or Psalm 127). Do you know of doctrine that Christian universalism stops at present people?

5. "Optimising Christians only need to 'satisfice' their fellows, raising them out of subsistence. Positive consequentialism is for God." This idea has a similar structure to negative utilitarianism, a moral system with an unusual number of philosophical difficulties. Why do bliss or happiness have no / insufficient moral weight? And, theologically: does orthodoxy say we don't need to make others (very) happy?

If I understand you, in your points (1) through (4) you appeal to a notion of God's agency outside of human action or natural laws. (So miracles only?) But a better theology of causation wouldn't rely on miracles, instead viewing the whole causal history of the universe as constituting God's agency. That interpretation, which at least doesn't contradict physics, would keep optimising Christians on the hook for x-risk.

Many of your points are appropriately hedged - e.g. "it might also be God’s job" - but this makes it difficult to read off actions from the claims. (You also appeal to a qualitative kind of Bayesian belief updating, e.g. "significant but not conclusive reason".) Are you familiar with the parliamentary model of ethics? It helps us act even while holding nuanced/confused views - e.g. for the causation question I raised above, each agent could place their own subjective probabilities on occasionalism, fatalism, hands-off theology and so on, and then work out what the decision should be. This kind of analysis could move your post from food-for-thought into a tool for moving through ancient debates and imponderables.

Thanks for this! Very interesting.

And really sorry for replying only now -- I somehow missed this and only saw it now.

--- On population increase: yes, many Christians work towards population increase but it's equally true that many Christians don't. An interesting side remark is that the influential passage Genesis 1,28 on which pro-natalism is often based calls for *filling* the earth. Arguably, humanity can claim to have unlocked this achievement. We can tick it off our To-Do-List. (Also, in terms of background information, my view that determining the optimal population size might be God's task rather than a human task started with this blogpost: https://greenfutureethics.wordpress.com/2012/12/03/god-as-a-solution-for-population-paradoxes)

--- On miracles: One thing is that I find it a bit hard to exclude miracles from classical theism. But even if we exclude them (or understand them to be compatible with natural laws) and even if we understand God to act within the causal history of the universe, one thing we surely can't exclude in classical theism is that God acts in addition to human agency (including acts which might be surprising). To the extent that this is true, Christian concern with x-risks should continue to be somewhat mitigated relative to the atheist's concern?

--- And thanks for the helpful observation that the blogpost unhelpfully avoids clear upshots (and thus also avoids responsibility for actions that might follow from it). The thing is: I find it genuinely extremely hard to think about the right approach to long-termism from a Christian perspective and this actually was *merely* a start. The parliamentary model etc would indeed be needed to derive actionable conclusions. (And, just to say, I do agree with many EAs that the far future should definitely receive more consideration than it typically does).

I was surprised to see person-affecting views weren't on your list of exception, then I saw it was in the uncertainties section. FWIW, taking Gregory Lewis' model at face value - I raised some concerns in a comment replying to that post - he concludes it's $100,000 per life saved. If AMF is $3,500 per life saved then X-risk is a relatively poor buy (although perhaps tempting as a sort of 'hedge'). That would only speak to your use of money: a person-affector could still conclude they'd do more good with a career focused on X-risk than elsewhere.

For AI, one estimate is $3-$1000 per life saved in the present generation (near bottom of model). For alternate foods for global agricultural catastrophes, one estimate is $0.20-$400 globally and $1-$20,000 in the US, both in the present generation.

Good point, thanks. It's definitely not a knock-down argument.

As someone with an interest in government and relatively new to the concept of x-risk, I have a semi-urgent question: who should I support for President? I will probably have to get involved with a campaign in some way or another in the next few months to maximize my odds of getting a decent appointment after the election. There's plenty of interest group ratings, position statements etc. out there on environmental issues but I can't find much that would be of practical use on the other types, which seem to be more serious at least in aggregate and perhaps individually too. I could try compiling my own ratings but I know far less than a lot of the people in this community, so if someone has already figured out or is in the process of figuring out where the candidates stand on the risks they have expertise in, I would greatly appreciate it. Doesn't have to be like standard interest group ratings and maybe shouldn't be. E.g. the fact that someone has a hawkish temperament toward China and that would make them more prone to starting an arms race is probably more important to AI safety than the specifics of any technology-related votes they've taken.

No idea, sorry. I know CSER have held at least one workshop about Trump and populism, so maybe try Julius Weitzdoerfer:

[Trump] will make people aware that they have to think about risks, but, in a world where scientific evidence isn't taken into account, all the threats we face will increase.

For deep ecologists, I use the argument that without people, the animals and plants will generally go extinct in 500-1000 million years because the increasing brightness of the sun will cause runaway global warming. Humans could delay this by putting reflective material between the Earth and the sun or other means. Or humans could perpetuate species on other planets. So in the long run, it is not a good idea for other species to have humans go extinct.

I haven't read much deep ecology, but I model them as strict anti-interventionists rather than nature maximisers (or satisficers): isn't it that they value whatever 'the course of things without us' would be?

(They certainly don't mind particular deaths, or particular species extinctions.)

But even if I'm right about that, you're surely right that some would bite the bullet when universal extinction was threatened. Do you know any people who accept that maintaining a 'garden world' is implied by valuing nature in itself?

I haven't read much deep ecology either. Seth Baum has written that some people think there is intrinsic value in functioning ecosystems - presumably these people would want the ecosystems to continue as a garden world. Other people value biodiversity (number of species). But you're right that some just want whatever would have happened naturally.

Of course, a deep ecologist who sided with extinction would be hoping for a horrendously narrow event, between ‘one which ends all human life’ and ‘one which ends all life’. They’d still have to work against the latter, which covers the artificial x-risks.

I agree that it covers AI, but I'm not sure about the other artificial x-risks. Nuclear winter severe enough to eventually kill all humans would definitely kill all large animals, but some smaller forms of life would survive. And while bio-risk could vary a lot in how many species were susceptible to it, I don't think anyone could construct a pathogen that affects everything.

You're right. I think I had in mind 'AI and nanotech' when I said that.

By “doesn’t have to work on reducing x-risk”, do you mean that they don’t want to?

I’d expect that negative utilitarians (NUs) do want to reduce x-risk, because

(1) x-risk is rarely an either/or risk of 100% extinction; instead, more x-risk probably correlates with more risk of extreme suffering (from non-total pandemics/disasters/wars/etc., and all of their after-effects)

(2) even facing a 100% human extinction, we’d want to account for our epistemic uncertainty of the conditions from which suffering can evolve (re-evolve on Earth, or be found elsewhere within the reach of our descendants)

NUs don’t necessarily jump to suicide as a solution, because helping others is an infinite game to live for, especially after accounting for the epistemic uncertainty of all possible forms of suffering and their evolution. There is further basic research on suffering to be done before turning off the lights and hoping that all the billions of exoplanets would have their own guardians.

It is a straw man argument that NUs don’t value life or positive states, because NUs value them instrumentally, which may translate into substantial practical efforts (compared even with someone who claims to be terminally motivated by them).

I mean that the end of the world isn't a bad outcome to someone who only values the absence of suffering, and who is perfectly indifferent between all 'positive' states. (This is Ord's definition of absolute NU, so I don't think I'm straw-manning that kind.) And if something isn't bad (and doesn't prevent any good), a utilitarian 'doesn't have to work on it' in the sense that there's no moral imperative to.

(1) That makes sense. But there's an escalation problem: worse risk is better to ANU (see below).

(2) One dreadful idea is that self-replicators would do the anti-suffering work, obviating the need for sentient guardians, but I see what you're saying. Again though, this uncertainty about moral patients licences ANU work on x-risks to humans... but only while moving the degenerate 'solution' upward, to valuing risks that destroy more classes of candidate moral patients. At the limit, the end of the entire universe is indisputably optimal to an ANU. So you're right about Earth x-risks (which is mostly all people talk about) but not for really farout scifi ones, which ANU seems to value.

Actually this degenerate motion might change matters practically: it seems improbable that it'd be harder to remove suffering with biotechnology than to destroy everything. Up to you if you're willing to bite the bullet on the remaining theoretical repugnance.

(To clarify, I think basically no negative utilitarian wants this, including those who identify with absolute NU. But that suggests that their utility function is more complex than they let on. You hint at this when you mention valuing an 'infinite game' of suffering alleviation. This doesn't make sense on the ANU account, because each iteration can only break even (not increase suffering) or lose (increase suffering).)

Most ethical views have degenerate points in them, but valuing the greatest destruction equal to the greatest hedonic triumph is unusually repugnant, even among repugnant conclusions.

I don't think instrumentally valuing positive states helps with the x-risk question, because they get trumped by a sufficiently large amount of terminal value, again e.g. the end of all things.

(I'm not making claims about other kinds of NU.)

More from technicalities
263
· · 1m read
497
· · 2m read
187
· · 2m read
Curated and popular this week
Sam Anschell
 ·  · 6m read
 · 
*Disclaimer* I am writing this post in a personal capacity; the opinions I express are my own and do not represent my employer. I think that more people and orgs (especially nonprofits) should consider negotiating the cost of sizable expenses. In my experience, there is usually nothing to lose by respectfully asking to pay less, and doing so can sometimes save thousands or tens of thousands of dollars per hour. This is because negotiating doesn’t take very much time[1], savings can persist across multiple years, and counterparties can be surprisingly generous with discounts. Here are a few examples of expenses that may be negotiable: For organizations * Software or news subscriptions * Of 35 corporate software and news providers I’ve negotiated with, 30 have been willing to provide discounts. These discounts range from 10% to 80%, with an average of around 40%. * Leases * A friend was able to negotiate a 22% reduction in the price per square foot on a corporate lease and secured a couple months of free rent. This led to >$480,000 in savings for their nonprofit. Other negotiable parameters include: * Square footage counted towards rent costs * Lease length * A tenant improvement allowance * Certain physical goods (e.g., smart TVs) * Buying in bulk can be a great lever for negotiating smaller items like covid tests, and can reduce costs by 50% or more. * Event/retreat venues (both venue price and smaller items like food and AV) * Hotel blocks * A quick email with the rates of comparable but more affordable hotel blocks can often save ~10%. * Professional service contracts with large for-profit firms (e.g., IT contracts, office internet coverage) * Insurance premiums (though I am less confident that this is negotiable) For many products and services, a nonprofit can qualify for a discount simply by providing their IRS determination letter or getting verified on platforms like TechSoup. In my experience, most vendors and companies
 ·  · 4m read
 · 
Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar. We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window. More details on our website. Why we exist We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared. Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future. Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization. This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them. Research Research agendas We are currently pursuing the following perspectives: * Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
jackva
 ·  · 3m read
 · 
 [Edits on March 10th for clarity, two sub-sections added] Watching what is happening in the world -- with lots of renegotiation of institutional norms within Western democracies and a parallel fracturing of the post-WW2 institutional order -- I do think we, as a community, should more seriously question our priors on the relative value of surgical/targeted and broad system-level interventions. Speaking somewhat roughly, with EA as a movement coming of age in an era where democratic institutions and the rule-based international order were not fundamentally questioned, it seems easy to underestimate how much the world is currently changing and how much riskier a world of stronger institutional and democratic backsliding and weakened international norms might be. Of course, working on these issues might be intractable and possibly there's nothing highly effective for EAs to do on the margin given much attention to these issues from society at large. So, I am not here to confidently state we should be working on these issues more. But I do think in a situation of more downside risk with regards to broad system-level changes and significantly more fluidity, it seems at least worth rigorously asking whether we should shift more attention to work that is less surgical (working on specific risks) and more systemic (working on institutional quality, indirect risk factors, etc.). While there have been many posts along those lines over the past months and there are of course some EA organizations working on these issues, it stil appears like a niche focus in the community and none of the major EA and EA-adjacent orgs (including the one I work for, though I am writing this in a personal capacity) seem to have taken it up as a serious focus and I worry it might be due to baked-in assumptions about the relative value of such work that are outdated in a time where the importance of systemic work has changed in the face of greater threat and fluidity. When the world seems to