I think there's a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.

That makes sense, but I'm feeling skeptical. There are just so many AI safety orgs now, and the technical ones generally aren't even funded by OP. 

For example: https://www.lesswrong.com/posts/9n87is5QsCozxr9fp/the-big-nonprofits-post

While a bunch of these salaries are on the high side, not all of them are.

I guess on one hand, if this were the case, then EAs would be well-represented in America, given that it's population in 1776 was just 2.5M, vs. the UK population of 8M. 

On the other hand, I'd assume that if they were distributed across the US, many would have been farmers / low-income workers / slaves, so wouldn't have been able to contribute much. There is an interesting question on how much labor mobility or inequality there was at the time. 

Also, it seems like EAs got incredibly lucky with Dustin Moskovitz + Good Ventures. It's hard to picture just how lucky we were with that, and what the corresponding scenarios would have been like in 1776. 

Could make for neat historical fiction. 

On AI safety, I think it's fairly likely (40%?) that the risk of x-risk (upon a lot of reflection) in the next 20 years is less than 20%, and that the entirety of the EA scene might be reducing it to say 15%.

This means that the entirety of the EA AI safety scene would help the EV of the world by ~5%.

On one hand, this is a whole lot. But on the other, I'm nervous that it's not ambitious enough, for what could be one of the most [combination of well-resourced, well-meaning, and analytical/empirical] groups of our generation.

One thing I like about epistemic interventions is that the upper-bounds could be higher. 

(There are some AI interventions that are more ambitious, but many do seem to be mainly about reducing x-risk by less than an order of magnitude, not increasing the steady-state potential outcome) 

I'd also note here that an EV gain of 5% might not be particularly ambitious. It could well be the case that many different groups can do this - so it's easier than it might seem if you think goodness is additive instead of multiplicative. 

Yea, I think this setup has been incredibly frustrating downstream. I'd hope that people from OP with knowledge could publicly reflect on this, but my quick impression is that some of the following factors happened:
1. OP has had major difficulties/limitations around hiring in the last 5+ years. Some of this is lack of attention, some is that there aren't great candidates, some is a lack of ability. This effected some cause areas more than others. For whatever reason, they seemed to have more success hiring (and retaining talent) for community than for technical AI safety. 
2. I think there's been some uncertainties / disagreements into how important / valuable current technical AI safety organizations are to fund. For example, I imagine if this were a major priority from those in charge of OP, more could have been done. 
3. OP management seems to be a bit in flux now. Lost Holden recently, hiring a new head of GCR, etc. 
4. I think OP isn't very transparent and public with explaining their limitations/challenges publicly.
5. I would flag that there are spots at Anthropic and Deepmind that we don't need to fund, that are still good fits for talent.
6. I think some of the Paul Christiano - connected orgs were considered a conflict-of-interest, given that Ajeya Cotra was the main grantmaker. 
7. Given all of this, I think it would be really nice if people could at least provide warnings about this. Like, people entering the field are strongly warned that the job market is very limited. But I'm not sure who feels responsible / well placed to do this. 

Around EA Priorities:

Personally, I feel fairly strongly convinced to favor interventions that could help the future past 20 years from now. (A much lighter version of "Longtermism").

If I had a budget of $10B, I'd probably donate a fair bit to some existing AI safety groups. But it's tricky to know what to do with, say, $10k. And the fact that the SFF, OP, and others have funded some of the clearest wins makes it harder to know what's exciting on-the-margin.

I feel incredibly unsatisfied with the public EA dialogue around AI safety strategy now. From what I can tell, there's some intelligent conversation happening by a handful of people at the Constellation coworking space, but a lot of this is barely clear publicly. I think many people outside of Constellation are working on simplified models, like "AI is generally dangerous, so we should slow it all down," as opposed to something like, "Really, there are three scary narrow scenarios we need to worry about."

I recently spent a week in DC and found it interesting. But my impression is that a lot of people there are focused on fairly low-level details, without a great sense of the big-picture strategy. For example, there's a lot of work into shovel-ready government legislation, but little thinking on what the TAI transition should really look like.

This sort of myopic mindset is also common in the technical space, where I meet a bunch of people focused on narrow aspects of LLMs, without much understanding of how their work exactly fits into the big picture of AI alignment. As an example, a lot of work seems like it would help with misuse risk, even when the big-picture EAs seem much more focused on accident risk.

Some (very) positive news is that we do have far more talent in this area than we did 5 years ago, and there's correspondingly more discussion. But it still feels very chaotic.

A bit more evidence - it seems like OP has provided very mixed messages around AI safety. They've provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting, and AI policy. But all of this still represents perhaps ~30% or so of their total budget, and I don't sense that that's about to change. Overall this comes off as measured and cautious. Meanwhile, it's been difficult to convince other large donors to get into this area. (Other than Jaan Tallinn, he might well have been the strongest dedicated donor here).

Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it's just that the AI safety people have migrated to other blogs and organizations.

But again, I'm very hopeful that we can find interventions that will help in the long-term, so few of these excite me. I'd expect and hope that interventions that help the long-term future would ultimately improve animal welfare and more.

So on one hand, AI risk seems like the main intervention area for the long-term, but on the other, the field is a bit of a mess right now.

I feel quite frustrated that EA doesn't have many other strong recommendations for other potential donors interested in the long-term. For example, I'd really hope that there could be good interventions to make the US government or just US epistemics more robust, but I barely see any work in that area.

"Forecasting" is one interesting area - it currently does have some dedicated support from OP. But it honestly seems to be in a pretty mediocre state to me right now. There might be 15-30 full-time people in the space at this point, and there's surprisingly little in terms of any long-term research agendas.

I (with limited information) think the EA Animal Welfare Fund is promising, but wonder how much of that matches the intention of this experiment. It can be a bit underwhelming if an experiment to try to get the crowd's takes on charities winds up determining to, "just let the current few experts figure it out." Though I guess, that does represent a good state of the world. (The public thinks that the current experts are basically right) 

I occasionally hear implications that cyber + AI + rogue human hackers will cause mass devastation, in ways that roughly match "lots of cyberattacks happening all over." I'm skeptical of this causing over $1T/year in damages (for over 5 years, pre-TAI), and definitely of it causing an existential disaster.

There are some much more narrow situations that might be more X-risk-relevant, like [A rogue AI exfiltrates itself] or [China uses cyber weapons to dominate the US and create a singleton], but I think these are so narrow they should really be identified individually and called out. If we're worried about them, I'd expect we'd want to take very different actions then to broadly reduce cyber risks. 

I'm worried that some smart+influential folks are worried about the narrow risks, but then there's various confusion, and soon we have EAs getting scared and vocal about the broader risks. 

Some more discussion in this Facebook Post.

Here's the broader comment against cyber + AI + rogue human hacker risks, or maybe even a lot of cyber + AI + nation state risks. 

Note: This was written quickly, and I'm really not a specialist/expert here. 

1. There's easily $10T of market cap of tech companies that would be dramatically reduced if AI systems could invalidate common security measures. This means a lot of incentive to prevent this.

2. AI agents could oversee phone calls and video calls, and monitor other conversations, and raise flags about potential risks. There's already work here, there could be a lot more.

3. If LLMs could detect security vulnerabilities, this might be a fairly standardized and somewhat repeatable process, and actors with more money could have a big advantage. If person A spend $10M using GPT5 to discover 0-days, they'd generally find a subset compared to person B, who spends $100M. This could mean that governments and corporations would have a large advantage. They could do such investigation during the pre-release of software, and have ongoing security checks as new models are released. Or, companies would find bugs before attackers would. (There is a different question of whether the bug is cost-efficient to fix).

4. The way to do a ton of damage with LLMs and cyber is to develop offensive capabilities in-house, then release a bunch of them at once in a planned massive attack. In comparison, I'd expect that many online attackers using LLMs wouldn't be very coordinated or patient. I think that attackers are already using LLMs somewhat, and would expect this to scale gradually, providing defenders a lot of time and experience.

5. AI code generation is arguably improving quickly. This could allow us to build much more secure software, and to add security-critical features.

6. If the state of cyber-defense is bad enough, groups like the NSA might use it to identify and stop would-be attackers. It could be tricky to have a world where it's both difficult to protect key data, but also, it's easy to remain anonymous when going after other's data. Similarly, if a lot of the online finance world is hackable, then potential hackers might not have a way to store potential hacking earnings, so could be less motivated. It just seems tough to fully imagine a world where many decentralized actors carry out attacks that completely cripple the economy.

7. Cybersecurity has a lot of very smart people and security companies. Perhaps not enough, but I'd expect these people could see threats coming and respond decently.

8. Very arguably, a lot of our infrastructure is fairly insecure, in large part because it's just not attacked that much, and when it is, it doesn't cause all too much damage. Companies historically have skimped on security because the costs weren't prohibitive. If cyberattacks get much worse, there's likely a backlog of easy wins, once companies actually get motivated to make fixes.

9. I think around our social circles, those worried about AI and cybersecurity generally talk about it far more than those not worried about it. I think this is one of a few biases that might make things seem scarier than they actually are.

10. Some companies like Apple of gotten good at rolling out security updates fairly quickly. In theory, an important security update to iPhones could reach 50% penetration in a day or so. These systems can improve further.

11. I think we have yet to see the markets show worry about cyber-risk. Valuations of tech companies are very high, cyber-risk doesn't seem like a major factor when discussing tech valuations. Companies can get cyber-insurance - I think the rates have been going up, but not exponentially.

12. Arguably, there's many trillions of dollars being held to by billionaires and others that they don't know what to do with. If something like this actually causes 50%+ global wealth to drop, it would be an enticing avenue for such money to go. Basically, we do have large reserves to spend, if the EV is positive enough, as a planet.

13. In worlds with much better AI, many AI companies (and others) will be a lot richer, and be motivated to keep the game going.

14. Very obviously, if there's 10T+ at stake, this would be a great opportunity for new security companies and products to enter the market.

15. Again, if there's 10T+ at stake, I'd assume that people could change practices a lot to use more secure devices. In theory all professionals could change to one of a few locked-down phones and computers.

16. The main scary actors potentially behind AI + Cyber would be nation states and rogue AIs. But nation-states have traditionally been hesitant to make these (meaning $1T+ damage) attacks outside of wartime, for similar reasons that they are hesitant to do military attacks outside wartime.

17. I believe that the US leads on cyber now. The US definitely leads on income. More cyber/hacking abilities would likely be used heavily by the US state. So, if they become much more powerful, the NSA/CIA might become far better at using cyber attacks to go after other potential international attackers. US citizens might have a hard time being private and secure, but so would would-be attackers. Cyber-crime becomes far less profitable if the attackers themselves can preserve their own privacy and security. There are only 8 Billion people in the world, so in theory it might be possible to oversee everyone with a risk of doing damage (maybe 1-10 million people)? Another way of putting this is that better cyber offense could directly lead to more surveillance by the US department. (This obviously has some other downsides, like US totalitarian control, but that is a very different risk) 

I wonder if some of the worry on AI + Cyber is akin to the "sleepwalking fallacy". Basically, if AI + Cyber becomes a massive problem, I think we should expect that there will be correspondingly massive resources spent then trying to fix it. I think that many people (but not all!) worried about this topic aren't really imagining what $1-10T of decently-effective resources spent on defense would do.

I think that AI + Cyber could be critical threat vector for malicious and powerful AIs in the case of AI takeover. I also could easily see it doing $10-$100B/year of damage in the next few years. But I'm having trouble picturing it doing $10T/year of damage in the next few years, if controlled by humans.

I think I broadly like the idea of Donation Week. 

One potential weakness is that I'm curious if it promotes the more well-known charities due to the voting system. I'd assume that these are somewhat inversely correlated with the most neglected charities.

Related, I'm curious if future versions could feature specific subprojects/teams within charities. "Rethink Priorities" is a rather large project compared to "PauseAI US", I assume it would be interesting if different parts of it were put here instead. 

(That said, in terms of the donation, I'd hope that we could donate to RP as a whole and trust RP to allocate it accordingly, instead of formally restricting the money, which can be quite a hassle in terms of accounting) 

As a very simple example, I think Amanda Askell stands out to me as someone who used to work on Philosophy, then shifted to ML work, where she now seems to be doing important work on crafting the personality of Claude. I think Claude easily has 100k direct users (more through the API) now and I expect that to expand a lot. 

There's been some investigations into trying to get LLMs to be truthful:
https://arxiv.org/abs/2110.06674

And of course, LLMs have shown promise at forecasting:
https://arxiv.org/abs/2402.18563

In general, I'm both suspicious of human intellectuals (for reasons outlined in the above linked post), and I'm suspicious of our ability to improve human intellectuals. On the latter, it's just very expensive to train humans to adapt new practices or methods. It's obviously insanely expensive to train humans in any complex topic like Bayesian Statistics. 

Meanwhile, LLM setups are rapidly improving, and arguably much more straightforward to improve. There's of course one challenge of actually getting the right LLM companies to incorporate recommended practices, but my guess is that this is often much easier than training humans. You could also just build epistemic tools on top of LLMs, though these would generally target fewer people. 

I have a lot of uncertainty if AI is likely to be an existential risk. But I have a lot more certainty that AI is improving quickly and will become a more critical epistemic tool than it is now. It's also just far easier to study than studying humans. 

Happy to discuss / chat if that could ever be useful! 

Happy to see progress on these.

One worry that I have about them, is that they (at least the forecasting part of the economics one, and the psychology one) seem very focused on various adjustments to human judgement. In contrast, I think a much more urgent and tractable question is how to improve the judgement and epistemics of AI systems. 

I've written a bit more here.

AI epistemics seems like an important area to me both because it helps with AI safety, and because I expect that it's likely to be the main epistemic enhancement we'll get in the next 20 years or so.