The Effective Altruism Foundation is launching the EAF Fund (a.k.a CLR Fund), a new fund focused on reducing s-risks. In this post we want to outline its mission, likely priority areas, and fund management structure. We also explain when it makes sense to donate to this fund.
Summary
- The fund’s mission is to address the worst s-risks from artificial intelligence.
- Priority areas for grants will likely be decision theory and bargaining, AI alignment and fail-safe architectures, macrostrategy research, and AI governance. There is some chance we might also make grants related to social science research on conflicts and moral circle expansion.
- Fund managers Lukas, Brian, and Jonas will make grants with a simple majority vote.
- The current balance is $68,638 (as of November 27), and we expect to be able to allocate $400k–$1.5M during the first year. We will likely try different mechanisms for proactively enabling the kind of research we’d like to see, e.g. requests for proposals, prizes, teaching buy-outs, and scholarships.
- You should give to this fund if you prioritize improving the quality of the long-term future, especially with regards to reducing s-risks from AI. You can donate to this fund via the Effective Altruism Foundation (donors from Germany, Switzerland, the Netherlands) or the EA Funds Platform (donors from the US or the UK).
Mission
The fund’s focus is on improving the quality of the long-term future by supporting efforts to reduce the worst s-risks from advanced artificial intelligence. (edited for clarity; see comment section)
Priority areas
Based on this mission, we have identified the following priority areas, which may shift as we learn more.
Tier 1
- Decision theory. It’s plausible that outcomes of multipolar AI scenarios are to some degree shaped by the decision theories of the AI systems involved. We want to contribute to a higher likelihood of cooperative outcomes since conflicts are a plausible contender for creating large amounts of disvalue.
- AI alignment and fail-safe architectures. Some AI failure modes are worse than others. We aim to differentially support alignment approaches where the risks are lowest. Work that ensures comparatively benign outcomes in the case of failure is particularly valuable from our perspective. Surrogate goals are one such example.
- Macrostrategy research. There are many unresolved questions about how to improve the quality of the long-term future. Additional research could unearth new crucial considerations which would change our prioritization.
- AI governance. The norms and rules governing the development of AI systems will shape the strategic and technical outcome. Establishing cooperative and prudential norms in the relevant research communities could be a way to avoid bad outcomes.
Tier 2
- Theory and history of conflict. By using historic examples or game theoretical analysis, we could gain a better understanding of the fundamental dynamics of conflicts, which might in turn lead to insights that are also applicable to conflicts involving AI systems.
- Moral circle expansion. Making sure that all sentient beings are afforded moral consideration is another fairly broad lever to improve the quality of the long-term future.
Past grants
- $26,000 to Rethink Priorities: We funded their surveys on descriptive population ethics because learning more about these values and attitudes may inform people’s prioritization and potential moral trades. We also made a smaller grant for a survey investigating the attitudes toward reducing wild animal suffering.
- $27,450 to Daniel Kokotajlo: Daniel will collaborate with Caspar Oesterheld and Johannes Treutlein to produce his dissertation at the intersection of AI and decision theory. His project will explore acausal trade in particular and coordination mechanisms between AI systems more generally. This work is relevant to AI safety, AI policy, and cause prioritization. This grant will buy him out of teaching duties during his PhD to allow him to focus on this work full-time.
Fund management
As we learn more, we might make changes to this initial setup.
Fund managers
We chose the fund managers based on their familiarity with the fund’s mission and prioritization, the amount of time they can dedicate to this work, and relevant research expertise. They were approved by the board of EAF.
- Lukas is responsible for prioritization at the Effective Altruism Foundation, and coordinates our research with other organizations. He conceptualized worst-case AI safety, and helped coin and establish the term s-risks. Currently, his main research focus is on better understanding how different AI alignment approaches affect worst-case outcomes.
- Brian has written prolifically and comprehensively about ethics, animal welfare, artificial intelligence, and the long-term future from a suffering-focused perspective. His ideas have been very influential in the effective altruism movement, and he helped found the Foundational Research Institute, a project of the Effective Altruism Foundation, which he still advises. He graduated from Swarthmore College in 2009, where he studied computer science, mathematics, statistics, and economics.
- Jonas is the Co-Executive Director of the Effective Altruism Foundation where he is responsible for setting the strategic direction, communications with the effective altruism community, and general management. He holds degrees in medicine and economics with a focus on health economics and development economics. He previously served on the boards of several charities, and is an advisor to the EA Long-term Future Fund.
Grantmaking
The current balance of the fund is $68,638 (as of November 27), and we expect to be able to allocate $400k–$1.5M during the first year. We will likely try different mechanisms for proactively enabling the kind of research we’d like to see, e.g. requests for proposals, prizes, teaching buy-outs, and scholarships.
Given the current state of academic research on s-risks, it’s impossible to find senior academic scholars who could judge the merit of a proposal based on its expected impact. However, we will consult domain experts where we think their judgment adds value to the evaluation. We also ran a hiring round for a research analyst, whom we expect to support the fund managers. They may also take on more grantmaking responsibilities over time.
Grant recipients may be charitable organizations, academic institutions, or individuals. However, we expect to often fund individual researchers and small groups as opposed to large organizations or institutes. Grants are approved by a simple majority of the fund managers. We expect grants to be made at least every six months.
We will experiment with different formats for publishing our reasoning behind individual grant decisions and evaluating past grants (e.g. trying to use predictions). This will likely depend on the number and size of grants.
When should you give to this fund?
CEA has already written up reasons for giving to funds in general. We won’t repeat them here. So when does it make sense to give to this fund in particular?
- You think long-termism, broadly construed, should guide your decisions.
- You think there is a significant chance of AI profoundly shaping the long-term future.
When does it make sense to give to this fund instead of the EA Long-term Future Fund?
- You are interested in improving the quality of the long-term future, addressing s-risks from AI in particular. This might be the result of your normative views, e.g. a strong focus on suffering, from pessimistic empirical beliefs about the long-term future, or from thinking that s-risks are currently neglected.
- You trust the judgments of the fund managers or the Effective Altruism Foundation.
How to donate to this fund
You can donate to this fund via the Effective Altruism Foundation (donors from Germany, Switzerland, the Netherlands) or the EA Funds Platform (donors from the US or the UK).
Note: Until December 29 donations to the EAF Fund can be matched 1:1 as part of a matching challenge. (For the matching challenge we’re still using the former name “REG Fund”.)
Happy to see this be set up! It makes a lot of sense to me that our community gradually gets more and more competition from different funds. Over time we'll get evidence on which one seem to perform the best.
A few questions:
1) Are you planning on coordinating information and decisions with other funds? I general my impression is that there should be similar application procedures and perhaps some information sharing. That said, I could imagine ways where coordination could come across in bad ways.
2) Where do you expect to get most of your donations from? Individual EA donors?
3) Your Tier-1 and Tier-2 questions don't seem exclusive to s-risks, they seem to me like things that would be very applicable to much of AI-risk concerns. Does this sound correct to you? Are you seeking projects that focus specifically on s-risks within these areas?
Thanks! :)
1) The Long-Term Future Fund seems most important to coordinate with. Since I'm both a fund manager at the EAF Fund and an advisor to the Long-Term Future Fund, I hope to facilitate such coordination.
2) Individual EA donors, poker pros (through our current matching challenge), and maybe other large donors.
3) Yes, that sounds correct. We're particularly excited to support researchers who work on specific s-risk-related questions within those areas, but I expect that the research we fund could also positively influence AI in other ways (e.g. much of the decision theory work might make positive-sum trade more likely and could thereby increase the chance of realizing the best possible outcomes). We might also fund established organizations like MIRI if they have room for more funding.