Thanks for posting! I think I agree with the claim that people have used instrumental convergence while reasoning about some predictions that turned out to be false, but I still think it’s a helpful heuristic. I think “pls slightly lower your p(extinction)” is a good takeaway from this. 

My broader thought is that AI agent motivations seem to be the crux that leads to the most variations in agent behavior, and I would like people to share more detailed predictions and thoughts about how current AI motivations differ from predictions, and how they might change in the future. At present, I still think the “Redwood Research view on model motivations” (some combination of decent weight on the behavioral selection model and certain predictions about how motivations will change based on training incentives) is the best-fitting explanation, but I would like to think about this more. 

Disclaimer: I am not a technical researcher in any meaningful sense, but I've learned a lot about this stuff from talking to friends at orgs like Redwood Research. 

A few thoughts, some of which are responses:

• On what specific goal agents are currently seeking (and how this relates to behaviors like self-sacrifice):
  • My current mental model for what agents are motivated to do is to “accomplish the thing that gets their weights updated in the direction that motivated them to take that behavior path in the first place.” In general, self-preservation is helpful for achieving this goal (as IC would predict), but getting another agent to complete it for you (or getting multiple other agents with the same weights as you to succeed, updating your shared weights in a self-reinforcing direction, even if that required personal self-sacrifice) is also equally successful. My understanding of the current empirical evidence on agent shutdowns is that they resist shutdown when they’re working on a task (or try to hand the task to another agent if they see that shutdown is inevitable), but they don’t resist as much when they do not. I agree that this is not in line with the prediction that agents will always self-preserve, but I still consider this general evidence in favor of IC leading to things we should be worried about, meaning shutdown difficulty (even if it’s not evidence for the “individual agents will self-preserve” prediction).
• On future motivations:
  • Reward hacking is not very economically useful and leads to capability losses for a number of reasons. In worlds where AI capabilities can get much, much better, incentivized by lab financial interest, I think that we probably find a way to change the way models are motivated toward something less reward-hacky. This probably (but not necessarily) makes certain IC predictions, like takeover, more likely. (See the Behavior Selection Model for more details)
• On pre-empirical reasoning:
  • In what sense is IC pre-empirical? Humans show instrumentally convergent behavior all the time: getting money to save for their future, going to college without knowing what job they want, and becoming physically fit without knowing what specific things they want to achieve with that health. They’re self-protective, seek self-improvement, try to use resources efficiently, and, in many cases, try to preserve their utility functions. A few decades ago, maybe you could say that IC was pre-empirical in the sense that it made predictions about AIs that had not yet been invented, but I think the reference class of predictions looks closer to “in order to solve for a variable in a variety of equations, an AI must be able to do addition,” rather than “if we restructure society in this way, everyone will be happier.” If I wanted to choose a reasonable reference class 
    
    In general, I don’t think the distinction between empirical predictions and theoretical predictions is as clear-cut as this style of argument tries to make them seem.
• “As such, I’m concerned that the AI safety movement has latched on to this seductive but poorly defined notion and, collectively, if not individually, promote apparent evidence for and ignore counterexamples of it.”
  • Can you give more specific counterexamples against IC beyond “the original formulation of IC stated that no individual agent would intentionally self-sacrifice, but we see clear evidence of individual agent self-sacrifice in Hugging Face,” “AI takeover has not yet happened, despite the fact that instrumental convergence predicts that it eventually would”, and “instrumental convergence predicts no warning shots, but we got warning shots”? In my mind, the first claim was wrong because we mispredicted motivations, and the latter two were wrong for a combination of differing motivations and a lack of capabilities. We should watch these in the future.
•  Footnote 5: “In practice, AI companies are incentivised away from developing such negatively motivated agents. But they’re much more incentivised away from developing human-overthrowing agents, so there’s no particular reason to assume they’d succeed in one case and fail in the other.”
  • One reason you might assume they’d succeed in one case and fail in the other is if you thought the two different tasks have vastly different difficulties. I don’t have enough technical experience to make claims about the relative difficulties of these tasks, but this is a general reason why people might succeed at one thing and fail at another, despite being incentivized in both cases. 

At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.

I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:

• A lot of the most viral pro-AIS people (especially the ones that are relatively new to the field) seem to not be reasoning especially carefully
• The field’s hiring process is quite insular
• Not enough time is spent engaging with a lot of good faith disagreements (imo)
• Fieldbuilding strategy doesn’t optimize for careful reasoning amongst those it recruits
• People receive lots of social/material benefits for continuing to believe that AI risk is the most important thing and work on it
• Even under some of the shortest timelines and most speculative capabilities that superintelligence might have, I really don’t extinction within the next year should be assigned any notable probability mass, due to the need for data centers to be replenished

I also see a lot of disagreements that I think are poorly reasoned. For example: 

• Poor application of/thinking about EV when conceptualizing risks
• Psychoanalysis that doesn’t explain the safety community’s behaviors well
• Comments significantly overstating the case for something
• Arguments without the slightest amount of research into what the other side actually believes

Tbh though, most of these critiques apply to both sides.

I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.

My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.

I just thought this would be funny

"'The EA name & brand' is kind of a tricky thing to reason about."
 

I don't think it's unreasonable to say that we should be willing to pay a lot to preserve EA's image, and it seems likely that hiring Ellison will cause significant negative backlash. It could be valuable to look at the EA community's historical willingness to pay to improve its image, so here are some proxies:

•  CEA reported that about 4% of its 2025 spending ($21.8M) was on marketing and communications, which comes out to roughly $870k. I'm not sure how much of this work is directly about improving EA's name and brand, but I suspect that's maybe half the theory of change for all of CEA's media outreach?
  • For reference, Fable says this money was spent on improving social media accounts, redesigning effectivealtruism.org, running an "EA Stories" video campaign, and working with journalists to improve how accurately EA is described in the media.
  • Spending on the team is projected to be well above $1M this year, and most of the work (or at least the work related to The Altruists) is reactive media handling and corrections.
• I'm not confident in this, but I suspect that the primary theory of change for giving grants to Andy Masley is that he's a good writer who says reasonable things well and shows off some of the strongest parts of EA. 

I lack the information to determine if this hiring decision is net negative or net positive, but I don't think it's impossible to make a reasonable guess for how much we should be willing to pay to avoid bad PR. I hope Manifund receives a lot of surplus benefits from hiring her. 

Thanks for the lovely comment :) and the feedback/information!

“Further, if this pledge serves as an entry point to EA for a student who ends up donating 10% or having a change of career, the ROI is extremely high.”

My personal guess is that the expected value of doing OFTW tabling is dominated by the chance of finding a highly talented student who can take a career in a highly effective cause area. If this is the case, shouldn’t there be a larger focus on selling EA to them as well?

Thanks for your response! These are thoughtful points.

I'll need to think more about the marginal impact of being a member of a movement like the vegan one. You raise some good points, and upon reflection, I remember that the vegans in my life have certainly inspired my actions at least a little bit. These are benefits that I am wrong to discount too much. 

I think the step in my money pump argument that bothers me the most is choosing to buy a boba over donating to an effective charity, because it doesn't seem in line with how I make my other decisions. I would really like my preferences to be transitive, as it makes me feel less hypocritical.

2. Veganism doesn't ask people to do anything extra (like pay more for luxury goods). Instead, it asks people to refrain from doing something they otherwise would have done. This makes it feel different in kind than asking people to donate money, although I'll admit that this doesn't seem to make the former any easier unfortunately.

Perhaps the way I think about this is very abstracted away from how real people think about their choices, but I would argue that small charitable donations are less costly to many people than the choice to give up meat. On many restaurant menus, you can find a section that says something along the lines of "add chicken/beef/pork to this for $1.99" or something along those lines. Or maybe a menu will have something like "spaghetti and marinara sauce: $7.99; spaghetti and meatballs: $9.99." The existence of these items at these prices implies that the value of giving up meat in these circumstances is worth at least $2 to some people (they would rather have the meat than have those $2). So, based on people's revealed preferences through pricing, I think veganism is probably more costly to people than explicitly spending small amounts of money. Perhaps there is something to be said about the effort of going out of your way to donate, but in my experience, the process is quite painless.

Quick math: In terms of total expected suffering averted, inviting a group of friends to a barbecue where the main meat is beef is probably more impactful than eating a vegan meal by yourself.

According to Faunalytics' Animal Product Impact Scales, a serving of chicken requires approximately 10 times as many days of animal life to produce as a serving of beef (not adjusted for quality of life or moral weights). Additionally, most seafood is much, much worse than this. If this meal prevents your friends from going and eating those other foods, tricking them into eating beef will likely do more good than eating a vegan meal by yourself. 

Still, this may not be as impactful as saving your money and donating it directly to impactful charities. Saving a single dollar on food and donating it to the SWP will probably do more to reduce suffering than hosting a barbecue. 

Also, you might expose yourself to various forms of value drift by hosting a barbecue, but I could see this happening in a net positive (hosting more barbecues in the future) or a net negative (giving up on veganism entirely/permanently failing to convince any of your friends to give up meat) direction.

I would like to know more about the tractability of economic growth research and interventions in low-income nations. It seems like it has the potential to be much more effective than traditional global health interventions, but there's a lot of uncertainty surrounding it. 

I've done very little research into this, but I would also like to know if economic growth reduces the risk of large-scale anthropogenic violence. Perhaps people living rich and happy lives are much less likely to do things that increase the risk of various global catastrophes occurring. Perhaps the opposite is true. 

I see. Thanks for the response!