Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar.
We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window.
More details on our website.
Why we exist
We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared.
Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future.
Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization.
This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them.
Research
Research agendas
We are currently pursuing the following perspectives:
* Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
I am very confident that dual-use risk of improved asteroid deflection technology in general is much more likely than a random asteroid hitting us, and that therefore this experiment has likely made the world worse off (with a bit less confidence, because maybe it's still easier to deflect asteroids defensively rather than offensively, and this experiment improved that defensive capability?). This is possibly my favorite example of a crucial consideration, and also more speculatively, evidence that the sum of all x-risk reduction efforts taken together could be net-harmful (I'd give that a 5-25% chance?).
This is much more of a problem (and an overwhelming one) for risks/opportunities that are microscopic compared to others. Baseline asteroid/comet risk is more like 1 in a billion. Much less opportunity for that with 1% or 10% risks.
To use asteroid deflection offensively, you’d have to:
By contrast, to have asteroid deflection offer a benefit given current information, the requirements are:
A second form of benefit might be
As have previously been noted, the implicit flattish hierarchy of different points in pro-con lists can sometimes cause people to make bad decisions.
Source: 80000 Hours
Some entirely made-up numbers (for the next 50 years):
~=1/5,600,000 or 1 in 5.6 * 10^6. However, I think these numbers are a bit of an understatement for total risk. This is because when I was making up numbers earlier, I was imagining the most likely actor to be able to pull this off in the next 50 years. But anthropogenic risks are disjunctive, multiple actors can attempt the same idea.
~=1/4,000,000,000 or 1 in 4*10^9.
So overall I'm skeptical that the first-order effects of deflecting natural asteroid risks is larger than the first-order effects of anthropogenic asteroid risks.
I agree with this. If the first-order effects are small, it's easy for second-order effects to dominate (assuming the second-order effects come from an entirely different channel than the first-order effects).
I appreciate the effort to put some numbers into this Fermi format! I'm not sure whether you intend the numbers, or the result, to represent your beliefs about the relative risks and benefits of this program. If they are representative, then I have a couple points to make.
I'm surprised you think there's a 10% chance that an actor who wants to destroy the Earth this century will have asteroid deflection within their technological capabilities. I'd assign this closer to a 1/1000 probability. The DART mission cost $324.5 million, was carried out by the world's economic and technological superpower, and its team page lists hundreds of names, all of whom I am sure are highly-qualified experts in one thing or another.
Maybe North Korea could get there, and want to use this as a second-strike alternative if they can't successfully develop a nuclear program? But we're spying on them like mad and I fully expect the required testing to make such a weapon work would receive the same harsh sanctions as their other military efforts.
I'd downweight the likelihood that asteroid deflection is their easiest method for doing so due to the difficulty with precision targeting from 1/7 to 1/1000. An asteroid of the size targeted by DART would take out hundreds of square miles (New York is 302 square miles, Earth's surface area is 197 million square miles). Targeting a high-population area puts even steeper demands on precision targeting and greater opportunity to mitigate damage by deflection to a lower-impact zone. It seems to me there are much easier ways for a terrorist to take out New York City than asteroid deflection.
Since your estimates for the two scenarios are only off by 3 OOMs, I think that these form the crux of our disagreement. I also note that this Fermi estimate no doubt has several conceptual shortcomings, and it would probably be useful to come up with an improved way to structure it.
Thanks for the engagement! Re:
Those are meant to be my actual (possibly unstable) beliefs. With the very important caveats that a) this is not a field I've thought about much at all and b) the numbers are entirely pulled from intuition, not even very simple models or basic online research.
Same :D
Also, apparently NASA is putting the odds of a collision with Bennu, which is about the same size as Dimorphos, at 1/1750 in the next three centuries. That's not quite the same timeframe, and this is just a quick Google search result. A more authoritative number would be helpful. Given AI risk and the pace of tech change, I think it makes sense to highly prioritize asteroid impacts this century, not in three centuries.
What I take from this mission is not so much
"Great, now we are a bit safer from asteroids hitting the earth."
but more like
"Great, NASA and the American public think existential risks like asteroids are worth taking seriously. The success of this mission might make it a bit easier to convince people that, one, there are other existential risks worth taking seriously and, two, that we can similarly reduce those risks through policy and technology innovation. Maybe now other existential risk reduction efforts will become more politically palatable, now that we can point to the success of this mission".
[Edit: here's a relevant article that supports my point: "Nasa’s mission gives hope we can defend our planet but human nature and technology present risks of their own" https://on.ft.com/3LNySAM]
For more on this risk, see this interesting recent book: Dark Skies: Space Expansionism, Planetary Geopolitics, and the Ends of Humanity (Jun. 2020) Daniel Deudney
https://academic-oup-com.ezp.lib.cam.ac.uk/book/33656?login=true
https://www.amazon.co.uk/Dark-Skies-Expansionism-Planetary-Geopolitics/dp/0190903341
I really don't think dual use is in any way worrisome if humanity has several institutions capable of asteroid deflection, and a tiny one if there is only one. Quoting a comment I gave to finm in his post on asteroid risks:
I've been keeping tabs on this since mid-August when the following Metaculus question was created:
The community and I (97%, given NASA's track record of success) seem in agreement that it is unlikely DART fails to make an impact. Here are some useful Wikipedia links that aided me with the prediction: (Asteroid impact avoidance, Asteroid impact prediction, Near Earth-object (NEO), Potentially hazardous object).
There are roughly 3 hours remaining until impact (https://dart.jhuapl.edu/); it seems unlikely that something goes awry, and I am firmly hoping for success.
While I'm unfamiliar with the state of research on asteroid redirection or trophy systems for NEOs, DART seems like a major step in the correct direction, one where humanity faces a lower levels of risks from the collision of asteroids, comets, and other celestial objects with Earth.
Here's a livestream - impact should be at 7:16 pm ET https://www.youtube.com/watch?v=-6Z1E0mW2ag
Impact successful - so exciting!