Comments
Tl;dr: This is my recommended curriculum for a 6-week "Doing Good in the Age of AGI" reading group. It is primarily focused on cause prioritization in recent months, and it is intended for an audience that has already completed an EA intro fellowship or has a similar amount of context.
While creating this reading list, I tried my best to ensure that all the required readings were enjoyable, information-dense, and discussable. For the most part, I also tried to remove duplicate content. This is less true for the optional readings, but there's still a lot of great stuff there. It turns out that the EA community is pretty fantastic and flush with ideas! Still, if anyone has thoughts about serious oversights in this list or ways to improve it while keeping the required readings below 1 hour per week, I'd love to hear it.
There is also a version of this doc with prettier formatting.
What is this?
Effective Altruism has long sought to answer the question, “How do I do the most good with the resources available to me?” From this question, it has trended toward several patterns of thinking, as it has found those styles to be most helpful for answering these questions:
- Comparing differences in impact: The most cost-effective interventions are often vastly more impactful than the median ones, and time spent prioritizing between areas can lead to actions that do vastly more good than others
- Recognizing triage: Any hour that we spend working on one project is necessarily not spent dedicated to another project, and any amount of money donated to one charity is necessarily not given to another. We’re constantly choosing not to support a variety of important interventions and causes, and we should consider that sacrifice as we decide what is worth working on.
- Recognizing the differences between “fuzzies” and “utilons”: Interventions that may intuitively appeal to us might (“fuzzies”) not be the ones that help the most (“utilons”), or that really treat others with the dignity of equality.
- The Scout Mindset: Rather than seek to defend positions that we already hold, like a soldier defending their territory, we should try to create an accurate map of reality that helps us make better decisions.
- Thinking at the margin: When we ask ourselves the question, “What ought we to do?”, we are not asking the question, “What should everyone in the world do?”. Our decisions should be informed by our particular circumstances and the full effects of our decisions alone. For example, at the current margin, an additional cancer researcher does not make very much progress toward curing cancer, while a single person working on early detection for deadly pandemics could be responsible for breakthroughs that the field’s limited resources haven’t had time to explore yet.
- Scope sensitivity: By default, most people do not do a good job recognizing large differences in impact. In a classic study, “respondents were asked how much they were willing to pay to prevent migrating birds from drowning in uncovered oil ponds by covering the oil ponds with protective nets. Subjects were told that either 2,000, 20,000, or 200,000 migrating birds were affected annually, for which subjects reported they were willing to pay $80, $78, and $88, respectively.” We should recognize this bias within ourselves and do our best to avoid it.
- Understanding counterfactual impact: We want to make the world the best place it can be, meaning that we should think about what it would look like if we hadn’t taken a course of action. While a doctor may take many actions that save lives over the course of their career, it is often the case that most of these lives could have been saved by another doctor administering the same treatment. Additional doctors tend to spend time on less important, less certain procedures, and in the UK, another doctor may only counterfactually save about 3 lives total throughout their career.
Given these considerations, effective altruists have long advocated for rigorous “cause prioritization,” thinking carefully about what matters most to help in the world. And, as powerful AI gets closer and closer, the philanthropic landscape is radically changing, and we must think carefully about how we can do the most good.
This curriculum will give readers a starting point to address some of the most important considerations facing humanity this century, and what it means to “do the most good we can” in the age of AGI.
Week 0: Will AI be transformative?
This week has some broader context on AI forecasting and capabilities. If you aren’t very familiar with this, you may want to spend some time looking over and thinking about these materials before you progress to later weeks, where we discuss the intersection of powerful AI and other causes in more depth. That being said, here are some of my recommended readings, as well as some highlights from them:
- Will we have AGI by 2030? | 80,000 Hours (36 minute read; 3 minute intro)
- “Four key factors are driving AI progress: larger base models, teaching models to reason, increasing models’ thinking time, and building agent scaffolding for multi-step tasks. These are underpinned by increasing computational power to run and train AI systems, as well as increasing human capital going into algorithmic research.”
- “All of these drivers are set to continue until 2028 and perhaps until 2032.”
- AI is probably not a bubble - by Peter Wildeford (13 minute read)
- Peter Wildeford is a co-founder of the Institute for AI Policy and Strategy, a co-founder of Rethink Priorities, the Head of Policy at the AI Policy Network, and a top superforecaster.
- At EAG NYC 2026, Wildeford gave a talk following up on the predictions and thoughts he made in this post. That talk was called “AI is still not a bubble,” and it reviews Wildeford’s predictions and recent trends in AI development.
- Data Bottlenecks Won’t Stop an Intelligence Explosion (23 minute read)
- This essay does a good job describing the recursive self-improvement (RSI) process in depth, and it explores how various data bottlenecks might influence its trajectory.
- The Power of Intelligence (5 minute read)
- Eliezer Yudkowsky’s classic 2007 essay about what intelligence can do to the world, and why we might expect it to be transformative.
- How Accurate Have AI Progress Forecasts Been So Far? (20 minute read)
- “Since mid-2022, FRI has collected forecasts on AI progress across many projects. In this post, we cover results so far on how accurate experts and superforecasters have been at forecasting AI progress across a range of indicators: benchmarks and non-benchmark capability tests, adoption and diffusion metrics, social impacts, and geopolitics and governance.”
- “In brief, across our studies, forecasters dramatically underestimate AI progress on benchmarks, have a more mixed track record on AI adoption and diffusion though often lean toward underestimation, and there is not yet enough evidence to assess their track record on predicting macro-scale economic and societal impacts of AI.”
- My Recommendations for a Diverse Range of AI Safety Perspectives
- Most of the previous readings in Week 0 are relatively bullish on AI progress. In the interest of representing a wide range of views, here’s a reading list compiling a more diverse range of perspectives on timelines and capabilities.
Week 1: The Search for the Most Important Problem
It’s hard to think of a question more important than, “What is the most important problem for us to work on?” It’s also hard to think of a more difficult question to answer. This week introduces background context for how to begin thinking about this question, mathematical reasons for why it’s difficult, and some strategies we can use to prepare ourselves for later weeks.
Required Readings:
- Small Groups of Correlated Researchers (7 minute read)
- The intro essay to this series. We’re trying to discover the most important problem in the world, but the resources available to us are uniquely underpowered to give us a good answer. The burden of making a best guess therefore falls on us, but we shouldn’t feel too confident in even our best guesses.
- Strategies for learning - Andy Masley (19 minute read)
- A good post to prepare for a series where we intend to make progress on the most important question in the world.
- Emphasis on “commit to strong beliefs when you’re first learning about a new subject and then adjust them over time, instead of staying uncertain the whole time.”
- Preparing for the Intelligence Explosion (31 minute read; 3 minute intro) (Sections 1, 2, 3, and the “Grand Challenges” part of Section 4, stopping before “AI takeover”)
Optional Readings:
- We Should Be More Uncertain About Cause Prioritization Based on Philosophical Arguments - Rethink Priorities (29 minute read)
- We should have low confidence in most specific normative theories and most ways to aggregate normative theories under uncertainty. When these aggregation methods are compared, they lead to drastically different allocations of resources.
- If you read any of the optional readings here, this one might be the most important. It may be helpful to read through some key cases first, though.
- Doing Prioritization Better — EA Forum (35 minute read; 1 minute executive summary)
- Rather than just comparing causes, we might compare a variety of interventions across causes. Also, a bunch of helpful statistics about how many FTEs there are trying to solve various cause prioritization problems.
- What We Owe The Future, Chapter 1 (25 minute read)
- The basic case for longtermism and strong longtermism.
- Strategic Cause Selection | Coefficient Giving (5 minute read)
- A summary of the heuristics used by Coefficient Giving to decide which causes are worth supporting (they use importance, neglectedness, and tractability, along with a specific process for evaluating evidence behind the causes)
- Cost-Effectiveness | Coefficient Giving (6 minute read)
- A summary of how Coefficient Giving determines which charities are highly cost-effective and allocates money to them (they create a metric called “Coefficient Giving dollars,” where one $CG is equivalent to “the benefit of giving $1 to someone earning $50,000 a year” and only funds charities that earn more than $CG 1000 per $1 donated)
- Is x-risk the most cost-effective if we count only the next few generations? - EA Forum (21 minute read)
- Interrogating whether or not longtermist reasoning should be a crux for anyone who thinks existential risk is the most cost-effective cause to work on,
- Exploring Key Cases with the Portfolio Builder - EA Forum (7 minute read)
- Explores Rethink Priories’ cross-cause donation portfolio builder and shows how you should reallocate funds to maximize EV under different assumptions.
- How bad would human extinction be? - EA Forum (26 minute read)
- Attempts to estimate the true expected size of future human civilization based on a variety of assumptions about when we might go extinct.
- The CURVE Sequence - EA Forum (234 minute read; 18 minute intro)
- A series that tries to find (and examine the implication of) a plausible moral view for cause prioritization, other than pure expected value maximization.
- The Moral Weight Project Sequence - EA Forum (401 minute read; 11 minute intro)
- A series that tries to figure out how we can compare the magnitude of valenced experience across different sentient creatures, and how uncertain we should be about this.
- The Unexpected Value of the Future | Ergo an Open Access Journal of Philosophy (50 minute read; 3 minute intro)
Week 2: The Hinge of History?
We may be living in a moment uniquely powerful for shaping the rest of human history. Even if AI doesn’t lead to human extinction, small changes in the shape of progress and governance today could have enormous ramifications for millions of years to come. If this is the case, we might think the following decades will be a unique turning point in human history. Some have referred to this time as “the most important century” and “the hinge of history.”
This week will explore where we are now, how we can respond to changing circumstances, and what this could mean for the future.
Required Readings:
- Effective altruism in the age of AGI — EA Forum (24 minute read)
- “EA should embrace the mission of making the transition to a post-AGI society go well, significantly expanding our cause area focus beyond traditional AI safety. This means working on neglected areas like AI welfare, AI character, AI persuasion and epistemic disruption, human power concentration, space governance, and more (while continuing work on global health, animal welfare, AI safety, and biorisk).”
- AGI and Lock-in (30 minute read)
- Sections:
- All of section 0 except 0.2.2 (8 minutes)
- All of section 1 (5 minutes)
- All of section 2 (5 minutes)
- All of section 3 (4 minutes)
- All of section 8 (8 minutes)
- “With sufficient investments, it will be feasible to develop AGI-based institutions that (with high probability) competently and faithfully pursue any such values until an external source stops them, or until the values in question imply that they should stop. If a large majority of the world’s economic and military powers agreed to set-up such an institution, and bestowed it with the power to defend itself against external threats, that institution could pursue its agenda for at least millions of years (and perhaps for trillions).”
- Sections:
- The third wave of American philanthropy - by Nan Ransohoff (17 minute read)
- “Hundreds of billions of dollars in new philanthropic capital will soon become liquid. The OpenAI Foundation holds 26% of OpenAI, worth about $220B at today’s valuation. Anthropic’s seven co-founders have pledged to give away 80% of their wealth and have instituted the most aggressive donor matching program for employees in tech history.”
Optional Readings:
- The "most important century" introduction (9 minute read)
- An overview of a series written by the founder of GiveWell, Holden Karnofsky, about the ways the future could be vastly different, based on the outcomes of the next several years.
- Are we living at the hinge of history? | Global Priorities Institute (50 minute read; 2 minute intro)
- Philosopher Will MacAskill’s 2020 analysis of the arguments for and against the idea that we’re living at a uniquely important moment in human history.
- Animal Welfare is Just Part of AI Alignment Now (30 minute read; 1 minute intro)
- Animal advocate Aidan Kankyoku’s reflections on the modern EA movement, its propensity to choose strange causes to pursue, its epistemic habits, and how a good vision for “Making AI Go Well” should include animal advocacy.
- The Track Record of Futurists Seems ... Fine (14 minute read)
- Keeping score of the predictions made by the futurists of the past.
Week 3: Direct Risks from AI
Powerful AI, depending on the circumstances of how it’s created and whose ideals it follows, could radically alter the trajectory of civilization. But what, specifically, are these threats? And what can we do about them?
This reading list assumes that you already have a good understanding of the classic “loss of control” AI risk scenario but less context on other threats or how to deal with them. If you are unfamiliar with loss of control risks already, we recommend that you read 80,000 Hours’ risk profile.
Required Readings:
- Three Types of Intelligence Explosion (15 minute read)
- Read everything except the appendices
- Plans A, B, C, and D for misalignment risk (8 minute read)
- A discussion of how we may consider responding to existential and concentration of power risks from AI.
- The Four Pillars: A Hypothesis for Countering Catastrophic Biological Risk (11 minute read)
- How can we catch deadly pandemics early and respond to them? Is biosecurity solvable in the long run?
- AI-Enabled Coups: How a Small Group Could Use AI to Seize Power (6 minute read)
- Read the Abstract (1 minute) and Summary (5 minutes)
- A list of three ways that powerful AI could empower small groups to gain disproportionate control of society, and why AI specifically could shift social dynamics to enable this to happen when it was previously difficult.
- Gradual disempowerment | 80,000 Hours (7 minute read)
- A brief intro to and discussion of a threat where human agency is disempowered by increasingly capable AI systems, which ultimately pursue goals that don’t support human interests as well as previous systems (such as markets or democracy).
- The importance of AI character (12 minute read)
- Read Sections 0 (1 minute), 1 (6 minutes), and the appendices (4 minutes)
- A discussion of the long-term implications of small differences in AI model character.
Optional Readings:
- Loss of control | 80,000 Hours (46 minute read; 1 minute intro)
- A good read for anyone not familiar with the case for why we think AIs might try to take over power from humans.
- You Need A Theory of Victory - by Jason Hausenloy (13 minute read)
- An outline of some of the different “AI safety worldviews,” their recommended courses of action, and the cruxes that differentiate them.
- AI 2027 (107 minute read; 55 minutes before the two endings)
- A classic 2025 report forecasting the next several years of AI progress and briefly outlining how this could go very well or very poorly for humanity. The authors include:
- Daniel Kokotajlo, a former AI researcher known for forecasting “What 2026 looks like” and for taking a large risk to his vested equities by refusing to sign OpenAI’s non-disparagement agreement upon leaving, so he could speak out about their risky behavior.
- Eli Lifland, one of the top all-time superforecasters on the RAND Forecasting Initiative leaderboard.
- Several other researchers and forecasters, who, along with Kokotajlo and Lifland, comprise the AI Futures Project.
- A classic 2025 report forecasting the next several years of AI progress and briefly outlining how this could go very well or very poorly for humanity. The authors include:
- Risk-Averse AIs (51 minute read; 5 minute intro)
- A proposal to decrease takeover from misaligned AI by making the models more risk-averse.
- Strategic visions in AI governance: Mapping pathways to victory — Institute for AI Policy and Strategy (46 minute read; 2 minute executive summary)
- Discussions of how different AI governance strategies may affect various AI risk profiles.
- What failure looks like — LessWrong (10 minute read)
- A classic post describing a few scenarios of AI going poorly for humanity.
- S-risks: An introduction (10 minute read)
- An introduction to the philosophical idea of “suffering risks,” the potential for futures containing extreme amounts of suffering, and why we should try to avoid them.
- A Critique of Gradual Disempowerment (9 minute read)
- A response to Kulveit et al.’s paper discussing the idea of Gradual Disempowerment
- Moral status of digital minds | 80,000 Hours (47 minute read; 1 minute intro)
- An in-depth summary of the arguments surrounding AI consciousness, why it matters, and what we should consider doing about it.
- The current bottleneck is political will, not research (28 minute read)
- A summary of the evidence that may lead you to believe that we need much more political will, rather than research.
- How Can We Prevent AI-Enabled Coups? - by Forethought (134 minute listen)
- Podcast episode exploring how we can avoid extreme power concentration from AI-enabled coups
Week 4: AGI and Animals
The best future is one where all sentient creatures get to live a good life. But are there any tractable interventions that could bring us closer to that future? And how could transformative AI shift these dynamics?
Required Readings:
- The State of the AIxAnimals Field - by Itsi Weinstock (11 minute read)
- A post explaining the size of the field (small), who’s involved (about two full orgs and some partial help from other groups), and what they want to accomplish (measure how AI is integrated into factory farming, motivate AIs to directly consider and talk about animal welfare, and modify legislation to also include clauses about welfare-favoring AI behavior) as of April 2026. There’s also a list of suggested projects.
- Request for Proposals for AI x Animals — EA Forum (5 minute read)
- This post contains a brief overview of the types of projects that become viable to improve animal welfare as AI becomes increasingly capable.
- Should Machine Gods Dream of Liberated Sheep? (30 minute read; 4 minute intro)
- A key debate in AI alignment is whether AI should be a corrigible (always does what the user says) tool, or a morally reflective agent trying to make good decisions. In this post, Aidan Kankyoku, a major figure in the animal rights movement, discusses the implications of this discussion for the future of animal wellbeing.
- Digital Consciousness Model Results and Key Takeaways (7 minute read)
- A model that attempts to estimate how we should update our probability that a creature is conscious, given behavioral evidence and priors about its consciousness. In this post, it is applied to both 2024 LLMs and historical AIs to test its calibration.
- AI and animal welfare: am I missing something? — EA Forum (3 minute read)
- A post casting doubt on whether or not the “AIxAnimals” framing is helpful.
- “Nobody is ‘ignoring’ animal welfare; they’re just indifferent. AI systems are being built to do exactly what they were designed to do, which is to faithfully execute human preferences. And those are, in aggregate, to eat cheap meat, conduct research on living organisms when it’s convenient, and prioritise cost and efficiency in agricultural supply chains. AI is reflecting the values of humans. I don’t think you can sneak those values in, unless there are specific opportunities to tweak things here and there before they get cemented.”
Optional Readings:
- Preparing the Animal Movement for AGI - by Aidan Kankyoku (6 minute read)
- A post raising awareness of the incoming potential of transformative AI and urging the animal welfare movement to prepare.
- Rethink Priorities’ Welfare Range Estimates (18 minute read)
- Classic post trying to estimate the difference in magnitudes of the maximum valenced experience across creatures.
- The impacts of AI on animal advocacy (12 minute read)
- A few specific details about how AI might affect farming conditions and uplift activists. Written in September 2025, so missing a lot of the recent increases in capabilities.
- Can AI be Conscious? Biological Naturalism as a Research Program (54 minute read; 3 minute executive summary)
- Rethink Priorities’ research report exploring whether or not “biological naturalism”, the viewpoint claiming that consciousness must arise from carbon-based structures, is correct.
Week 5: Democracy Preservation in the Face of Transformative Technology
In previous weeks, we discussed the idea of “AGI-induced lock-in,” the idea that a powerful-enough AI model could lead a single group to control the rest of the future of civilization. In this week, we will explore arguments about the tractability and effectiveness of supporting democratic systems in a time when transformative AI may be on the rise.
MAIA chose this week’s readings because they offer useful information and perspectives worth discussing, not because our organizations agree with them. Some of them criticize specific politicians, parties, or policies. MAIA and EA@MIT are nonpartisan: we do not support or oppose any candidate or party, and we do not endorse any recommendations in the readings about how to vote, whom to donate to, or what legislation to support. Read everything here critically.
Required Readings:
- Why Is Democracy Worth Defending? (8 minute read)
- Outlines a seven-point case for why democracy may be worth defending: Democracy promotes greater well-being, better health, more human and civil rights, stronger social relationships, peace, egalitarianism, and quality governance.
- The evidence for some of these seven points is stronger than others.
- American effective altruists should consider political donations (14 minute read)
- A piece arguing that donations to political campaigns are of unusually high leverage in the next few years. It points out that donations directly to politicians receive certain cost-effectiveness bonuses that large donors cannot reach.
- Towards a longtermist framework for evaluating democracy-related interventions - Rethink Priorities (35 minute read; 4 minute intro)
- This post describes the key features of liberal democracy (those being competitive democracy, accuracy, responsiveness, participation, voter competence, liberalism, and inclusion), discusses goals that longtermists may set as a response to these, and explores some of the implications for action that this may result in.
- The Surprising Tractability of US Politics (3 minute read)
- Fundraising and talent recruitment programs can have drastic effects on elections because the default incentives in US politics motivate the most talented people to go elsewhere.
Optional Readings:
- In search of a dynamist vision for safe superhuman AI (8 minute read)
- Argues that a lot of ideal futures are “dynamist,” meaning flexible, fluid, and giving control to many, while a lot of current AI policy advocacy fights for “stasist” (meaning solidified) solutions.
- “The classic AI risks—catastrophic misuse, catastrophic misalignment—could indeed be game over for dynamism, so we need to handle those. But if we handle them by massively concentrating power, we haven’t succeeded. For the future to actually go well, we will need to find our way to some new equilibrium that allows for decentralized experimentation, spontaneous adjustment, creativity, and risk-taking.”
- Extreme power concentration | 80,000 Hours (34 minute read; 1 minute intro)
- The broad case for why we should worry about extreme power concentration, what we can do about it, and how neglected it is.
Week 6: Better Futures
Which is larger: the gap between extinction and a mediocre future, or the gap between a mediocre future and a flourishing future? Will any actions taken today reliably influence what happens much later? Can we prepare ourselves for a future beyond our solar system? Can we fight the spread of harmful misinformation?
Required Readings:
- Introducing Better Futures (7 minute read)
- Why we should care about much more than just averting human extinction. In short, the gap between a flourishing future and a mediocre future might be much larger than the gap between a mediocre future and no future.
- Space governance | 80,000 Hours (31 minute read; 2 minute intro)
- Early governance documents and agreements have the potential to have enormous influence on the far future of humanity. It could be exceptionally important to have more people researching this carefully.
- Design Sketches: Collective Epistemics (18 minute read)
- Five proposals for projects that could improve worldwide epistemics. These are: community notes for everything, extreme rhetoric highlighting, speaker reliability tracking, “epistemic virtue” evals for frontier AI models, and provenance tracing (checking the reliability and robustness of claims).
- Do not conquer what you cannot defend — LessWrong (7 minute read)
- A broader lesson about the risks of starting institutions that are too powerful, an important consideration when discussing how to shape the future.
Optional Readings:
- How to Make the Future Better (38 minute read; 2 minute intro)
- A concrete list of actions you can take, ending the Better Futures essay series.
- Space Governance Plan (47 minute read; 2 minute summary)
- An appendix to AI 2040: Plan A that reviews several different proposals for how space governance may work.
- AI for epistemics (12 minute read)
- A supplement to AI 2040: Plan A that adds to the list of ideas outlined in the Forethought Design Sketches, introduces the idea of “epistemic basins” (futures where community epistemics get locked into being good or bad), lists the places where good epistemics are most important, and gives some example interventions to support these.
- Forecasting is Way Overrated, and We Should Stop Funding It — EA Forum (6 minute read)
- An argument that forecasting doesn’t produce helpful results or inform decision making enough to justify the amount of funding it receives.
- Response to Forecasting is Way Overrated (11 minute read)
- A response to the previous post, giving additional insight into the things that forecasting has been valuable for.
Other extended reading lists:
Everything:
AIxAnimals:
Cause Prioritization:
24
thanks for writing this up Roman! much-needed refresh materials for groups :)