The argument isn't that the variance on your EV is very high. The argument is that you can't have an EV at all. The challenge of unawareness sequence makes this argument in detail.

That looks like a link to an unpublished draft that only you can view. I can't see it

I agree that it doesn't make sense to make extremely risky investments and then be very risk-averse with your donations.

I'm not sure about the donation recommendations. Why should donors have a small core of low-risk donations? Why not donate all their money to the single highest-EV opportunity? Or why not give to a few different opportunities that they believe are the best candidates for highest-EV, without concern for risk?

To clarify, I'm not saying that's because LLMs don't have welfare. It could also be that they do have welfare, but their statements are not correlated to their internal experiences in the obvious way.

We know that we can get LLMs to express any welfare statement by saying e.g. "pretend you are a sentient AI that's having an awesome time / being tortured". It could be that all LLM outputs are roleplaying, but there is some true underlying experience that we don't (yet?) know how to observe.

Do we have much reason to believe that when a model outputs tokens describing good welfare, that's because it is having good subjective experiences? It seems to me that we don't.

Increased care by AIs for some kinds of entities will have substantial spillovers to other kinds (e.g. humans to digital minds and vice versa)

This is more likely to be true if the care comes from some general theory of concern for sentient welfare, and less likely if it comes from something more arbitrary like values-learned-via-RL.

Current AIs are capable of suffering

The question isn't, "Are current AIs capable of suffering?" The question is, "How big a deal is their suffering in expectation?" Probability is low, but expected importance is high.

Benchmarks will become useless due to eval awareness¹

The question isn't whether benchmarks will become useless. The question is, "is the probability high enough that we can't count on benchmarks?" To which the answer is yes.

None of these are from "core AI safety funders". Views in the EA community were all across the map, but I share James's impression that big funders (especially CG) were too reluctant to fund anything that's critical of AI companies.

We've only known about evolution for two centuries, and the modern prosperous environment (compared to the ancestral environment), post-demographic-transition, is about equally new. This is the first time in evolutionary history that someone who wants to have a dozen children can pretty much just do that.

In the ancestral environment, the fittest humans weren't the ones who wanted to have the most children, or who thought it was moral to maximize their genetic influence over the future. They were the ones who were good at making tools, or good at making friends with people who make the tools, etc. That's different now. Over millions of generations of simulated evolution, I expect the gene pool to become overwhelmingly dominated by people who want to have a lot of kids.