Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar.
We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window.
More details on our website.
Why we exist
We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared.
Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future.
Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization.
This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them.
Research
Research agendas
We are currently pursuing the following perspectives:
* Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
Who can we thank for the design? It's seriously impressive!
That credit goes to our designer @Agnes Stenlund! 😊
We hope you enjoy the latest version of Wrapped! Unfortunately we have noticed some bugs around the animations on iPhone 13 models, so if that's your phone we recommend viewing on a different device. Please let us know if you run into any other issues as well.
This is very cool
Huh, this is neat!
My "Forum Personality" is "Beloved Online Karma Farmer"? This confuses me a bit - "karma farmer" typically refers to a fairly pejorative role, from what I can tell. Just FYI, this strikes me as you saying, "We noticed you're using semi-questionable methods to technically gain Karma, but it's not as bad as it normally would be." Is this meant to be like a semi-playful dig, without as negative a spin? Was there a system that determined that I gained Karma in over-rated ways? Sorry if I'm newish to some of this terminology.
Toby's the one who decided on the personalities and he's currently on vacation, but they are all meant to be silly/playful and none of the personalities are intended to be read negatively. It's a bit we added this year for fun! I also got "karma farmer"[1] — the farmer in the picture seems to me to be doing honest work. :)
Thanks for flagging this. My apologies if we made anyone feel bad; we'll try to be more careful with our jokes in the future.
Basically that just means you gained a lot of karma.
Not a biggie, I mainly just found it confusing.
The EA Forum/LessWrong codebase is open source, you can see it just means that your karma change for 2024 was >= 500
https://github.com/search?q=repo%3AForumMagnum%2FForumMagnum%20karma%20farmer&type=code
(And beloved means that the most common reaction was receiving hearts)
I liked the cute animation I got for karma farming https://res.cloudinary.com/cea/video/upload/v1734615259/wrapped-2024/Karma-farmer-green.mp4