Quick takes

This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista.  When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway. Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it[3]. I figured that all these super smart, status-y types had it under control already.                                           The EMH is trivially false across all domains In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well. I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved g

EA often gets criticized for taking small probabilities too seriously. But people take actions with small probabilities of big success all the time! Asking a girl out? Small chance she'll become your wife. Trying to affect political chance? Fat chance of that. Launching a startup? Good luck buddy. The difference between EA and the standard way of acting is that standard psychology deals with the problem of small probabilities by being highly overconfident. Of course _my_ startup/political movement/YouTube channel is going to make it big. You cannot maintain high agency while ignoring small probability options.  The EA approach is simply more honest. 

I sometimes say, in a provocative/hyperbolic sense, that the concept of "neglectedness" has been a disaster for EA. I do think the concept is significantly over-used (ironically, it's not neglected!), and people should just look directly at the importance and tractability of a cause at current margins. Maybe neglectedness useful as a heuristic for scanning thousands of potential cause areas. But ultimately, it's just a heuristic for tractability: how many resources are going towards something is evidence about whether additional resources are likely to be impactful at the margin, because more resources mean its more likely that the most cost-effective solutions have already been tried or implemented. But these resources are often deployed ineffectively, such that it's often easier to just directly assess the impact of resources at the margin than to do what the formal ITN framework suggests, which is to break this hard question into two hard ones: you have to assess something like the abstract overall solvability of a cause (namely, "percent of the problem solved for each percent increase in resources," as if this is likely to be a constant!) and the neglectedness of the cause. That brings me to another problem: assessing neglectedness might sound easier than abstract tractability, but how do you weigh up the resources in question, especially if many of them are going to inefficient solutions? I think EAs have indeed found lots of surprisingly neglected (and important, and tractable) sub-areas within extremely crowded overall fields when they've gone looking. Open Phil has an entire program area for scientific research, on which the world spends >$2 trillion, and that program has supported Nobel Prize-winning work on computational design of proteins. US politics is a frequently cited example of a non-neglected cause area, and yet EAs have been able to start or fund work in polling and message-testing that has outcompeted incumbent orgs by looking for the highest-v

An informal research agenda on robust animal welfare interventions and adjacent cause prioritization questions Context: As I started filling out this expression of interest form to be a mentor for Sentient Futures' project incubator program, I came up with the following list of topics I might be interested in mentoring. And I thought it was worth sharing here. :) (Feedback welcome!) Last small update to add links: June 18th, 2026.  Animal-welfare-related research/work: 1. What are the safest (i.e., most backfire-proof)[1] consensual EAA interventions? (overlaps with #3.c and may require #6.) 1. How should we compare their cost-effectiveness to that of interventions that require something like spotlighting or bracketing (or more thereof) to be considered positive?[2] (may require A.) 2. Robust ways to reduce wild animal suffering 1. New/underrated arguments regarding whether reducing some wild animal populations is good for wild animals (a brief overview of the academic debate so far here). 2. Consensual ways of affecting the size of some wild animal populations (contingency planning that might become relevant depending on results from the above kind of research). 1. How do these and the safest consensual EAA interventions (see 1) interact? 3. Preventing the off-Earth replication of wild ecosystems. 3. Uncertainty on moral weights (some relevant context in this comment thread and this sequence). 1. Red-teaming of different moral weights that have been explicitly proposed and defended (by Rethink Priorities, Vasco Grilo, ...). 2. How and how much do cluelessness arguments apply to moral weights and inter-species tradeoffs? 3. What actions are robust to severe uncertainty about inter-species tradeoffs? (overlaps with #1.) 4. Considerations regarding the impact of saving human lives (c.f. top-GiveWell charities) on farmed and wild animals. (may require 3 and 5.) 5. The impact of agriculture on soil nematodes and other numerous

In Bruce Friedrich's new book, he writes, "Sometimes when I talk about cultivated meat someone will bring up the handful of states that have banned it. I'm mostly unconcerned. Cultivated meat companies won't be able to supply all 50 US states anytime soon anyway. Once there are multiple companies selling their products in...the majority of cities all across the country, the states that banned will-- I predict-- quietly repeal their laws" (p. 191). It's hard to know how literally to interpret this apparently sanguine attitude, as the book is designed to generate enthusiasm for alternative proteins. But, still it seems raise an important question about the cost-effectiveness of repealing existing bans or preventing further ones. Initial thoughts:   * My guess is that he'd still view preventing additional bans as important, at least in key regions expected to be first-adopters. Maybe Florida and Texas would be laggards in adoption even if cultivated meat were legal. * It'd be interesting to do an outside view analysis to see how quickly bans on other novel products have been undone once they've achieved a certain level of popularity elsewhere. * He's writing as if the industry can definitely succeed in spite of the bans. But, even if the bans spread no further, they already apply to >140M potential consumers across the US and Europe. That, combined with uncertainty about the prospect of additional bans, may chill the sort of public and private investment necessary for industry success.  

Gavi's investment opportunity for 2026-2030 says they expect to save 8 to 9 million lives, for which they would require a budget of at least $11.9 billion[1]. Unfortunately, Gavi only raised $9 billion, so they have to make some cuts to their plans[2]. And you really can't reduce spending by $3 billion without making some life-or-death decisions. Gavi's CEO has said that "for every $1.5 billion less, your ability to save 1.1 million lives is compromised"[3]. This would equal a marginal cost of $1,607 $1,363 per life saved, which seems a bit low to me. But I think there is a good chance Gavi's marginal cost per life saved is still cheap enough to clear GiveWell's cost-effectiveness bar. GiveWell hasn't made grants to Gavi, though. Why? ---------------------------------------- 1. https://www.gavi.org/sites/default/files/investing/funding/resource-mobilisation/Gavi-Investment-Opportunity-2026-2030.pdf, pp. 20 & 43 ↩︎ 2. https://www.devex.com/news/gavi-s-board-tasked-with-strategy-shift-in-light-of-3b-funding-gap-110595 ↩︎ 3. https://www.nature.com/articles/d41586-025-02270-x ↩︎

* Re the new 2024 Rethink Cause Prio survey: "The EA community should defer to mainstream experts on most topics, rather than embrace contrarian views. [“Defer to experts”]" 3% strongly agree, 18% somewhat agree, 35% somewhat disagree, 15% strongly disagree. * This seems pretty bad to me, especially for a group that frames itself as recognizing intellectual humility/we (base rate for an intellectual movement) are so often wrong. * (Charitable interpretation) It's also just the case that EAs tend to have lots of views that they're being contrarian about because they're trying to maximize the the expected value of information (often justified with something like: "usually contrarians are wrong, but if they are right, they are often more valuable for information than average person who just agrees"). * If this is the case, though, I fear that some of us are confusing the norm of being contrarian instrumental reasons and for "being correct" reasons.  Tho lmk if you disagree. 

I'd love to see an 'Animal Welfare vs. AI Safety/Governance Debate Week' happening on the Forum. The risks from AI cause has grown massively in importance in recent years, and has become a priority career choice for many in the community. At the same time, the Animal Welfare vs Global Health Debate Week demonstrated just how important and neglected the cause of animal welfare remains. I know several people (including myself) who are uncertain/torn about whether to pursue careers focused on reducing animal suffering or mitigating existential risks related to AI. It would help to have rich discussions comparing both causes's current priorities and bottlenecks, and a debate week would hopefully expose some useful crucial considerations.