Good question, but I implore you not to take this post too seriously. It's a real phenomenon, but it's a real stretch to claim that this applies in the implied way to cause areas like AI safety. 

The model in the article is just a toy model of a world where existential threats are randomly distributed according to a power law, where genuinely high-probability threats are, by assumption, basically absent from the space of possible threats, and where there's no process of updating based on evidence. 

A more narrow claim like: "single narrowly defined x-risk estimates of genuinely speculative, unresearched causes are likely to be inflated" might be valid, but that doesn't seem to be what titotal is implying. He's making a claim way beyond anything implied by the model, even if the model were a valid representation of the phenomenon. He seems to believe that almost all of today's concern for AI risk is all downstream of a belief cultivated within a narrow subcommunity subject to the optimiser's curse - an extraordinary claim that requires a lot more evidence than that supplied in the article. 

The claim being made is something like:

• Some time in the 2000s, Eliezer Yudkowsky and friends made up some numbers for AI risk
• Community dynamics (rather than the merits of the arguments) spread these numbers from Bostrom to Tegmark to Sam Harris to Elon Musk etc.
• These social dynamics had such an effect that multiple seemingly independent and unrelated people/experts from Geoffrey Hinton to Yoshua Bengio to Chinese academics and tech people from completely different intellectual lineages have absorbed this false belief from the cultural milieu (again, not at all based on the merits of the arguments), leading many of these people, as well as specialists across unrelated fields, to make estimates of AI x-risk that remain orders of magnitude too high

Note that this requires some pretty wild, difficult-to-justify assumptions on how this belief has spread.

The opposing narrative (which I would advocate for) is that:

• AI x-risk is something that has independently been identified as a risk by multiple people - often far before quantifying or ranking risks.
• The spread of these beliefs was obviously affected by community dynamics, but people largely adopted somewhat independent beliefs based on the merits of rational arguments
• Quantitative predictions were inherently imprecise because they're so dependent on messy world models, but they were grounded enough in reason and evidence, and composed of enough independent estimates, that any optimiser's curse is massively weakened
• As certain predictions came to pass, and current LLMs approach AGI in non-ideal geopolitical and competitive circumstances, this is increasingly being seen by a wider range of thinkers as a >1% x-risk
• Even if there were "optimiser curse" risks in initial prioritisation of AI, it's now increasingly recognised that AI will be a massive deal. AI-assisted engineered pandemic uplift work, observed cyber-capabilities, and signs of misalignment/scheming etc. are building on the strong theoretical evidence base that AI-generated catastrophe is possible. 

And on your particular question of how to act, even given the optimiser's curse as stated in the toy model, working on the speculative thing could still be optimal. If a highly uncertain intervention seems exceptionally promising or x-risky, the value of information becomes incredibly high, because accurate or well-reasoned research will lead to this intervention being prioritised or not by far more people. If you have a research focus, it's therefore probably more recommended to focus on a more uncertain, "high-risk-high-reward" area. You could also draw up a toy model for explore/exploit based on the optimiser's curse.

Finally, you don't necessarily have to "pick" from a narrow set of pre-defined cause areas. You can also divide or merge risks, cause areas, skill-sets etc. to be more robust, precise, or coherent with your own world model (e.g. focusing on engineered pandemics because you realise this could interact with AI-related x-risk, GCBRs, and global health).

Your piece went kinda viral on Substack, and most of the comments on the comments thread, and elsewhere on Substack, were constructive and positive. This seems a moderate win for alternative/social media.

I would barely update at all on this paper. Effect sizes are very small and don't show a logical causal mechanism.

Looking at the full paper, in the fully adjusted matched model, smaller tattoos (under one hand palm) give you a higher risk (1.27) than larger ones (1.14). In the unmatched model, large tattoos had no effect. This seems contradictory with their suggested mechanism (ink affecting lymph nodes). 

And their results imply that it's most dangerous getting tattoos before there's a plausible chemical pathway to harm (risk is highest for those who got tattoos under 2 years ago). They say this could be a result of antagonising existing tumours, but seems likely to be a statistical artifact or a non causal correlation.

If you have other evidence, though, I'd be happy to hear it.

I wrote a (draft amnesty) post on a very small subset of this - what global health charities you should donate to given different worldviews.

I agree with the "you don't have to debate on their terms" point here, but I think for 99% of your readers/listeners, it cuts far more strongly in a different way than that you're implying. 

The debate has generally been set in terms of "Anthropic vs. DoW", and, while I know zero people in our community who have taken the government's side on this, I've seen many EAs and adjacent people become increasingly uncritical supporters of Anthropic, just because they're standing against the obviously bad actor in this situation. 

I think it's important to remember:

1. If you thought Anthropic was untrustworthy before this, you shouldn't update too much the other way - especially when they backtracked on their RSP over the same period.
2. If you thought that Anthropic's decision to join the race towards AGI was perilous, you shouldn't really update your view on this based on the Pentagon being absurd and unpredictable.
3. Regardless of the intention and character of the government actors, it's potentially still a worrying sign that the most powerful state in the world has tried to shut down a frontier AI lab and failed spectacularly.
4. The growth and popularity of Anthropic and Claude Code have since caused the AI 2027 team to shorten their AGI timelines. 

Thanks for this post, it's super valuable to get a better sense of this ecosystem.

On the apparent lack of Chinese companies, I think this is a methodological thing; a few possible blind spots:

1. Most obviously, English-language, web-based search is probably going to miss some Chinese AI-aquaculture innovators that would otherwise meet the paper’s inclusion criteria. Using Chinese-language platforms might be necessary.
2. AI for aquaculture in China is often embedded within broader, more integrated "smart aquaculture” systems rather than marketed as standalone AI products. e.g. I'm not sure if Limap 励图高科 came up on your search, but it's a huge aquaculture and fisheries innovation company, deploying integrated aquaculture platforms across China, covering over 50 species etc. Some of their products and systems are explicitly AI-enabled & welfare relevant (e.g. computer-vision-based fish health and disease detection, visible in Chinese-language demos and videos), but they might have been excluded/missed because they're so broad.
3. China's innovation system is more state-led than elsewhere, and a lot of innovation happens through Universities, Agricultural Science and Technology Parks, local government programmes, "demonstration bases" 示范基地 etc. For example, China’s Ministry of Agriculture and Rural Affairs recently released a 主推技术 ("main promoted technology") notice, announcing that AI-enabled “smart aquaculture factory” technologies (including behaviour recognition, automated feeding, inspection robots, and large-model-based decision systems) are being supported through national programmes, implying a state-led deployment process. So you might be missing Chinese AI innovation in aquaculture that's not strictly commercial.

I'd lean towards the World Happiness Report results here. IPSOS uses a fully online sample, which means you end up losing the "bottom half" of the population. World Happiness Report is phone and in-person.

Hi Klara, thanks for the response.

I don't think I am entering the abortion debate by assigning moral value to unborn lives any more than I'm entering any other debate that considers unborn or potential lives (e.g. the ethics of moderate drinking while pregnant, the ethics of having children in space, or the repugnant conclusion). 

I think I'm comfortable with having mostly sidestepped the maternal health issues, given that I was focusing on interventions that are robustly good for the mother. If I were to do a stronger and more robust cost-effectiveness analysis, or tackle more controversial interventions where the interests of the mother and child clearly diverged, I would consider maternal health outcomes separately. I hope my piece makes it clear that we should prioritise uncontroversial and neglected interventions that treat or prevent painful conditions that women suffer from.

Although I do recognise that the ethics of pregnancy, lived experience of the mother, and autonomy trade-offs are important considerations, I'm afraid that attempting to tackle these here would have made this an impossibly long post!

When I say “the economics are looking good,” I mean that the conditions for capital allocation towards AGI-relevant work are strong. Enormous investment inflows, a bunch of well-capitalised competitors, and mass adoption of AI products means that, if someone has a good idea to build AGI within or around these labs, the money is there. It seems this is a trivial point - if there were significantly less capital, then labs couldn’t afford extensive R&D, hardware or large-scale training runs. 

WRT Scaling vs. fundamental research, obviously "fundamental research" is a bit fuzzy, but it's pretty clear that labs are doing a bit of everything. DeepMind is the most transparent about this, they're doing Gemini-related model research, Fundamental science, AI theory and safety etc. and have published thousands of papers. But I'm sure a significant proportion of OpenAI & Anthropic's work can also be classed as fundamental research.