Hide table of contents

Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar.

We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window.

More details on our website.

Why we exist

We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared.

Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future.

Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization.

This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them.

Research

Research agendas

We are currently pursuing the following perspectives:

  • Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destruction; what rights to give digital beings; how to govern an automated military; and how to avoid dictatorship or authoritarianism.

  • Achieving a near-best future: Most explicitly longtermist work to date has been focused on avoiding existential catastrophe, but achieving a near-best future might be even more important. Research avenues here include mapping out what a desirable “viatopia” would look like (i.e. a state of the world which is very likely to lead to a very good future), figuring out how space resources should be allocated, and addressing issues around AI epistemics and persuasion.

We tend to think that many non-alignment areas of work are particularly neglected.

However, we are not confident that these are the best frames for this work, and we are keen to work with people who are pursuing their own agendas.

Recent work

Today we’re also launching “Preparing for the Intelligence Explosion”, which makes a more in-depth case for some of the perspectives above.

You can see some of our other recent work on the site. We have a backlog of research, so we’ll be publishing something new every few days for the next few weeks.

Approach

Comparison to other efforts

We draw inspiration from the Future of Humanity Institute and from OpenPhil’s Worldview Investigations team: like them we aim to focus on big picture important questions, have high intellectual standards, and build a strong core team.

Generally, we’re more focused than many existing organizations on:

  • Explosive growth and short timelines
  • Work outside the current Overton window
  • Issues beyond AI alignment, including new technologies and challenges that AI will unleash
  • Reaching really good futures

Principles

  1. Stay small: We aim to hire from among the handful of people who have the best records of tackling hard questions about AI futures, to offer a supportive institutional home to such people, and to grow slowly.

  2. Communicate to the nerds: We will mostly share research and ideas with wonk-y folks thinking about AI in think tanks, companies, and government, rather than working directly with policymakers. We plan to be thoughtful about how best to communicate and publish, but likely on our website and in arXiv papers.

  3. Be open to “weird” ideas: The most important ideas in history often seemed strange or even blasphemous at the time. And rapid AI-driven technological progress would mean that many issues that seem sci-fi are really quite pressing. We want to be open to ideas based on their plausibility and importance, not on whether they are within the current Overton window.

  4. Offer intellectual autonomy: Though we try to focus on what's most important, there are many different reasonable views on what that is. Senior researchers in particular are encouraged to follow their instinct on what research avenues are most important and fruitful, and to publish freely. There isn't a "party line" on what we believe.

What you can do

Engage with our research

We’d love for you to read our research, discuss the ideas, and criticize them! We’d also love to see more people working on these topics.

You can follow along by subscribing to our podcast, RSS feed, or Substack.

Please feel free to contact us if you are interested in collaborating, or would like our feedback on something (though note that we won’t be able to substantively engage with all requests).

Apply to work with us

We are not currently actively hiring (and will likely stay quite small), but we have an expression of interest form on our site, and would be particularly keen to hear from people who have related research ideas that they would like to pursue.

Funding

We have funding through to approximately March 2026 at our current size, from two high-net-worth donors.

We’re looking for $1-2M more, which would help us to diversify funding, make it easier for us to hire more researchers, and extend our runway to 2 years. If you are interested to learn more, please contact us.


    1. We are a new team and project, starting in mid-2024. However, we’ve built ourselves out of the old Forethought Foundation for Global Priorities Research to help get the operations started, and Will was involved with both projects. We considered like 500 names and couldn’t find something that we liked better than “Forethought”. Sorry for the confusion! ↩︎

Comments5


Sorted by Click to highlight new comments since:

I am wondering if you could say something about how the political developments in the US (i.e., Trump 2.0) are affecting your thinking on AGI race dynamics? It seems like the default assumption communicated publicly is still that the US are "the good guys" and a "western liberal democracy" that can be counted on, when the actual actions on the world stage are casting at least some doubt on this position. In some sense, one could even argue that we are already playing out a high-stakes alignment crisis at this very moment.

Any reactions or comments on this issue? I understand that openness around this topic is difficult at the moment but I also don't think that complete silence is all that wise either. 

Exciting! Am I right in understanding that Forethought Foundation for Global Priorities Research is no longer operational?

Hi Rockwell! 

Yes, in most relevant senses that's correct. We're a new team, we think of ourselves as a new project, and Forethought Foundation's past activities (e.g. its Fellowship programs) and public presence have been wound down. We do have continuity with Forethought Foundation in some ways, mainly legal/administrative.

Congrats on launching the org. Would developing plans to avoid gradual disempowerment be in scope for your research?

Thanks! Yes, definitely in scope. There was a lot of discussion of this paper when it came out, and we had Raymond Douglas speak at a seminar. 

Opinions vary within the team on how valuable it is to work on this; I believe Fin and Tom are pretty worried about this sort of scenario (I don't know about others).. I feel a bit less convinced on the value of working on it (relative to other things), and I'll just say why briefly:
- I feel less convinced that people wouldn't foresee the bad gradual disempowerment scenarios and act to stop them from happening, esp with advanced AI assistance
- In the cases that feel more likely, I feel less convinced that gradual disempowerment is particularly bad (rather than just "alien").
- Insofar as there are bad outcomes here, it seems particularly hard to steer the course of history away from them.

The biggest upshot I see is that, the more you buy these sorts of scenarios, the more it increases the value of AGI being developed by a single e.g. multilateral project rather than being developed by multiply companies and countries. That's something I'm really unsure about, so reasoning around this could easily switch my views.

Curated and popular this week
Sam Anschell
 ·  · 6m read
 · 
*Disclaimer* I am writing this post in a personal capacity; the opinions I express are my own and do not represent my employer. I think that more people and orgs (especially nonprofits) should consider negotiating the cost of sizable expenses. In my experience, there is usually nothing to lose by respectfully asking to pay less, and doing so can sometimes save thousands or tens of thousands of dollars per hour. This is because negotiating doesn’t take very much time[1], savings can persist across multiple years, and counterparties can be surprisingly generous with discounts. Here are a few examples of expenses that may be negotiable: For organizations * Software or news subscriptions * Of 35 corporate software and news providers I’ve negotiated with, 30 have been willing to provide discounts. These discounts range from 10% to 80%, with an average of around 40%. * Leases * A friend was able to negotiate a 22% reduction in the price per square foot on a corporate lease and secured a couple months of free rent. This led to >$480,000 in savings for their nonprofit. Other negotiable parameters include: * Square footage counted towards rent costs * Lease length * A tenant improvement allowance * Certain physical goods (e.g., smart TVs) * Buying in bulk can be a great lever for negotiating smaller items like covid tests, and can reduce costs by 50% or more. * Event/retreat venues (both venue price and smaller items like food and AV) * Hotel blocks * A quick email with the rates of comparable but more affordable hotel blocks can often save ~10%. * Professional service contracts with large for-profit firms (e.g., IT contracts, office internet coverage) * Insurance premiums (though I am less confident that this is negotiable) For many products and services, a nonprofit can qualify for a discount simply by providing their IRS determination letter or getting verified on platforms like TechSoup. In my experience, most vendors and companies
 ·  · 4m read
 · 
Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar. We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window. More details on our website. Why we exist We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared. Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future. Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization. This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them. Research Research agendas We are currently pursuing the following perspectives: * Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
 ·  · 1m read
 · 
This is a linkpost for a new paper called Preparing for the Intelligence Explosion, by Will MacAskill and Fin Moorhouse. It sets the high-level agenda for the sort of work that Forethought is likely to focus on. Some of the areas in the paper that we expect to be of most interest to EA Forum or LessWrong readers are: * Section 3 finds that even without a software feedback loop (i.e. “recursive self-improvement”), even if scaling of compute completely stops in the near term, and even if the rate of algorithmic efficiency improvements slow, then we should still expect very rapid technological development — e.g. a century’s worth of progress in a decade — once AI meaningfully substitutes for human researchers. * A presentation, in section 4, of the sheer range of challenges that an intelligence explosion would pose, going well beyond the “standard” focuses of AI takeover risk and biorisk. * Discussion, in section 5, of when we can and can’t use the strategy of just waiting until we have aligned superintelligence and relying on it to solve some problem. * An overview, in section 6, of what we can do, today, to prepare for this range of challenges.  Here’s the abstract: > AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments grand challenges.  > > These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatically improve quality of life and collective decision-making. > > We argue that these challenges cannot always be delegated to future AI systems, and suggest things we can do today to meaningfully improve our prospects. AGI preparedness is therefore not just about ensuring that advanced AI systems are alig