Quick takes

At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing. I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example: * A lot of the most viral pro-AIS people seem to not be reasoning especially carefully * The field’s hiring process is quite insular * Not enough time is spent engaging with a lot of good faith disagreements (imo) * Fieldbuilding strategy doesn’t optimize for careful reasoning amongst those it recruits * People receive lots of social/material benefits for continuing to believe that AI risk is the most important thing and work on it * Even under some of the shortest timelines and most speculative capabilities that superintelligence might have, I really don’t extinction within the next year should be assigned any notable probability mass, due to the need for data centers to be replenished I also see a lot of disagreements that I think are poorly reasoned. For example:  * Poor application of/thinking about EV when conceptualizing risks * Psychoanalysis that doesn’t explain the safety community’s behaviors well * Comments significantly overstating the case for something * Arguments without the slightest amount of research into what the other side actually believes Tbh though, most of these critiques apply to both sides. I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully. My main point, though, is that it’s a little scary to

Thoughts on the American legal system This adversarial model of prosecution vs defense is a bad set up. It encourages the prosecution and indeed police to cut all corners possible on the way to a verdict and we know this happens quite a lot with very little consequence. The prosecution also insists on pursuing cases that have clearly been shown in later years to be false convictions because they want to save face and not admit they got details wrong. A better justice system is built of two sides working together to find the truth and having avenues for admitting mistakes without losing face. There also should never be any convictions at all based on testimonies alone without any non-anecdotal evidence attached to the case. The media is also a problem, in that if a case is sufficiently public the prosecution essentially has to take the case to court to avoid massive criticism. Witness tampering is also severe and has been proven to happen by many governments across the world. Even offering deals to witnesses in exchange for testimony encourages people to lie and then their testimony should be inadmissible. I am not a lawyer I'm just very interested in these failings that seem to be continuing without any change. I don't think people realize how underfunded the Innocence Project is or how many wrongful convictions there are on death row (I believe they claim 10% are wrongful). Innocent until proven guilty is not currently being upheld and the system is quite biased. Mass incarceration is affecting so many people and based on Puritan ideas of 'reform' rather than any evidence based studies. 

I sometimes say, in a provocative/hyperbolic sense, that the concept of "neglectedness" has been a disaster for EA. I do think the concept is significantly over-used (ironically, it's not neglected!), and people should just look directly at the importance and tractability of a cause at current margins. Maybe neglectedness useful as a heuristic for scanning thousands of potential cause areas. But ultimately, it's just a heuristic for tractability: how many resources are going towards something is evidence about whether additional resources are likely to be impactful at the margin, because more resources mean its more likely that the most cost-effective solutions have already been tried or implemented. But these resources are often deployed ineffectively, such that it's often easier to just directly assess the impact of resources at the margin than to do what the formal ITN framework suggests, which is to break this hard question into two hard ones: you have to assess something like the abstract overall solvability of a cause (namely, "percent of the problem solved for each percent increase in resources," as if this is likely to be a constant!) and the neglectedness of the cause. That brings me to another problem: assessing neglectedness might sound easier than abstract tractability, but how do you weigh up the resources in question, especially if many of them are going to inefficient solutions? I think EAs have indeed found lots of surprisingly neglected (and important, and tractable) sub-areas within extremely crowded overall fields when they've gone looking. Open Phil has an entire program area for scientific research, on which the world spends >$2 trillion, and that program has supported Nobel Prize-winning work on computational design of proteins. US politics is a frequently cited example of a non-neglected cause area, and yet EAs have been able to start or fund work in polling and message-testing that has outcompeted incumbent orgs by looking for the highest-v

In Bruce Friedrich's new book, he writes, "Sometimes when I talk about cultivated meat someone will bring up the handful of states that have banned it. I'm mostly unconcerned. Cultivated meat companies won't be able to supply all 50 US states anytime soon anyway. Once there are multiple companies selling their products in...the majority of cities all across the country, the states that banned will-- I predict-- quietly repeal their laws" (p. 191). It's hard to know how literally to interpret this apparently sanguine attitude, as the book is designed to generate enthusiasm for alternative proteins. But, still it seems raise an important question about the cost-effectiveness of repealing existing bans or preventing further ones. Initial thoughts:   * My guess is that he'd still view preventing additional bans as important, at least in key regions expected to be first-adopters. Maybe Florida and Texas would be laggards in adoption even if cultivated meat were legal. * It'd be interesting to do an outside view analysis to see how quickly bans on other novel products have been undone once they've achieved a certain level of popularity elsewhere. * He's writing as if the industry can definitely succeed in spite of the bans. But, even if the bans spread no further, they already apply to >140M potential consumers across the US and Europe. That, combined with uncertainty about the prospect of additional bans, may chill the sort of public and private investment necessary for industry success.  

An informal research agenda on robust animal welfare interventions and adjacent cause prioritization questions Context: As I started filling out this expression of interest form to be a mentor for Sentient Futures' project incubator program, I came up with the following list of topics I might be interested in mentoring. And I thought it was worth sharing here. :) (Feedback welcome!) Last small update to add links: June 18th, 2026.  Animal-welfare-related research/work: 1. What are the safest (i.e., most backfire-proof)[1] consensual EAA interventions? (overlaps with #3.c and may require #6.) 1. How should we compare their cost-effectiveness to that of interventions that require something like spotlighting or bracketing (or more thereof) to be considered positive?[2] (may require A.) 2. Robust ways to reduce wild animal suffering 1. New/underrated arguments regarding whether reducing some wild animal populations is good for wild animals (a brief overview of the academic debate so far here). 2. Consensual ways of affecting the size of some wild animal populations (contingency planning that might become relevant depending on results from the above kind of research). 1. How do these and the safest consensual EAA interventions (see 1) interact? 3. Preventing the off-Earth replication of wild ecosystems. 3. Uncertainty on moral weights (some relevant context in this comment thread and this sequence). 1. Red-teaming of different moral weights that have been explicitly proposed and defended (by Rethink Priorities, Vasco Grilo, ...). 2. How and how much do cluelessness arguments apply to moral weights and inter-species tradeoffs? 3. What actions are robust to severe uncertainty about inter-species tradeoffs? (overlaps with #1.) 4. Considerations regarding the impact of saving human lives (c.f. top-GiveWell charities) on farmed and wild animals. (may require 3 and 5.) 5. The impact of agriculture on soil nematodes and other numerous

Gavi's investment opportunity for 2026-2030 says they expect to save 8 to 9 million lives, for which they would require a budget of at least $11.9 billion[1]. Unfortunately, Gavi only raised $9 billion, so they have to make some cuts to their plans[2]. And you really can't reduce spending by $3 billion without making some life-or-death decisions. Gavi's CEO has said that "for every $1.5 billion less, your ability to save 1.1 million lives is compromised"[3]. This would equal a marginal cost of $1,607 $1,363 per life saved, which seems a bit low to me. But I think there is a good chance Gavi's marginal cost per life saved is still cheap enough to clear GiveWell's cost-effectiveness bar. GiveWell hasn't made grants to Gavi, though. Why? ---------------------------------------- 1. https://www.gavi.org/sites/default/files/investing/funding/resource-mobilisation/Gavi-Investment-Opportunity-2026-2030.pdf, pp. 20 & 43 ↩︎ 2. https://www.devex.com/news/gavi-s-board-tasked-with-strategy-shift-in-light-of-3b-funding-gap-110595 ↩︎ 3. https://www.nature.com/articles/d41586-025-02270-x ↩︎

* Re the new 2024 Rethink Cause Prio survey: "The EA community should defer to mainstream experts on most topics, rather than embrace contrarian views. [“Defer to experts”]" 3% strongly agree, 18% somewhat agree, 35% somewhat disagree, 15% strongly disagree. * This seems pretty bad to me, especially for a group that frames itself as recognizing intellectual humility/we (base rate for an intellectual movement) are so often wrong. * (Charitable interpretation) It's also just the case that EAs tend to have lots of views that they're being contrarian about because they're trying to maximize the the expected value of information (often justified with something like: "usually contrarians are wrong, but if they are right, they are often more valuable for information than average person who just agrees"). * If this is the case, though, I fear that some of us are confusing the norm of being contrarian instrumental reasons and for "being correct" reasons.  Tho lmk if you disagree. 

I'd love to see an 'Animal Welfare vs. AI Safety/Governance Debate Week' happening on the Forum. The risks from AI cause has grown massively in importance in recent years, and has become a priority career choice for many in the community. At the same time, the Animal Welfare vs Global Health Debate Week demonstrated just how important and neglected the cause of animal welfare remains. I know several people (including myself) who are uncertain/torn about whether to pursue careers focused on reducing animal suffering or mitigating existential risks related to AI. It would help to have rich discussions comparing both causes's current priorities and bottlenecks, and a debate week would hopefully expose some useful crucial considerations.