Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar.
We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window.
More details on our website.
Why we exist
We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared.
Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future.
Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization.
This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them.
Research
Research agendas
We are currently pursuing the following perspectives:
* Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
If you haven’t wandered around the Nicky Case website, I’d recommend doing so. There are a lot of interesting educational games on there, covering a wide variety of concepts such as social contagion, prisoner’s dilemmas, segregation, etc.
[Repost from my FB]
I'd like to introduce a setup that's a little different from these arbitrary axes and feels truer to life...
To avoid object-level politics, I'll use Scott's (or was it Nick's?) example:
* Party A wants to increase taxes and social services 5%, and to require everyone to electrocute themselves 8 hours a day.
* Party B wants to decrease taxes and social services 5%, and to require everyone to electrocute themselves 8 hours a day.
* Party C wants to leave taxes and social services as they are, and stop the electrocutions.
"Everyone" knows that Party C isn't serious. They get no media coverage, except as a punchline. Only people with no popularity to lose will come out openly as Party C'ers. And rather then break the dam, they make Party C association a mark of stigma.
Under FPTP, we need roughly a third of the people to *believe party C has a chance*, with no way to build momentum. Naturally, the electrocutions continue.
Under IRV (and I think any ordered ranking), we need roughly a third of people to pay attention. Then they can easily vote C>A>B or C>B>A and end the electrocutions. And if it's less than a third, it still shows a nice clear signal that Not Electrocuting Ourselves is an idea to be taken seriously.
Under Approval, people won't want to vote "C" because that gives up the chance to effect the taxes/services tradeoff which is the only thing they expect to be up for grabs. So they vote "A,C" or "B,C". And feel bad about it, because they don't actually *approve* of A or B. Which means they're voting against themselves. Now we need half of people to pay attention, and with a much weaker take-this-seriously signal. After all, "A,C" could just be intented as a hardcore vote against B (some people take the A-B rivalry very seriously).