AI safety
AI safety
Studying and reducing the existential risks posed by advanced artificial intelligence

Quick takes

47
1d
3
Holden Karnofsky has joined Anthropic (LinkedIn profile). I haven't been able to find more information.
43
9d
2
Both Sam and Dario saying that they now believe they know how to build AGI seems like an underrated development to me. To my knowledge, they only started saying this recently. I suspect they are overconfident, but still seems like a more significant indicator than many people seem to be tracking.
22
7d
3
Quick thoughts on investing for transformative AI (TAI) Some EAs/AI safety folks invest in securities that they expect to go up if TAI happens. I rarely see discussion of the future scenarios where it makes sense to invest for TAI, so I want to do that. My thoughts aren't very good, but I've been sitting on a draft for three years hoping I develop some better thoughts and that hasn't happened, so I'm just going to publish what I have. (If I wait another 3 years, we might have AGI already!) When does investing for TAI work? Scenarios where investing doesn't work: 1. Takeoff happens faster than markets can react, or takeoff happens slowly but is never correctly priced in. 2. Investment returns can't be spent fast enough to prevent extinction. 3. TAI creates post-scarcity utopia where money is irrelevant. 4. It turns out TAI was already correctly priced in. Scenarios where investing works: 1. Slow takeoff, market correctly anticipates TAI after we do but before it actually happens, and there's a long enough time gap that we can productively spend the earnings on AI safety. 2. TAI is generally good, but money still has value and there are still a lot of problems in the world that can be fixed with money. (Money seems much more valuable in scenario #5 than #6.) What is the probability that we end up in a world where investing for TAI turns out to work? I don't think it's all that high (maybe 25%, although I haven't thought seriously about this). You also need to be correct about your investing thesis, which is hard. Markets are famously hard to beat. Possible investment strategies 1. Hardware makers (e.g. NVIDIA)? Anecdotally this seems to be the most popular thesis. This is the most straightforward idea but I am suspicious that a lot of EA support for investing in AI looks basically indistinguishable from typical hype-chasing retail investor behavior. NVIDIA already has a P/E of 56. There is a 3x levered long NVIDIA ETP. That is not the sort of thin
9
4d
It has been my impression that the general AI safety community had over time shifted somewhat against an US-China AI race dynamic being a concern. But with the recent success of DeepSeek, it seems to me that the race is back on.  Has anyone not updated accordingly? If so, why? One implication of this development would seem to be that a merely domestic AI Pause is no longer a good idea. Is there agreement on this?
23
23d
One challenge in AI safety field-building is that otherwise-useful resources – like lists of courses or funders or local groups – generally become outdated over time. We’ve tried to solve this by collecting a bunch of resources together at AISafety.com and dedicating considerable bandwidth to keeping them updated. Until recently, this maintenance has been largely ad hoc, making additions and changes as we learned of them. To ensure nothing slips through the cracks, we’ve now added a schedule for doing thorough sweeps through the entire database for each resource. Below is our current plan: * Courses * Every 3 months: general sweep * Communities * Every 3 months: general sweep * (maybe) Every 6 months: request update from organisers * Projects * Every 3 months: general sweep * Every 6 months: request update from owners of active projects * Jobs * [This is a filtered subset of 80k’s database and updates automatically] * Events & training * Twice weekly: check for new events and programs * Every 2 weeks: add any dates previously unannounced and check for changes to application deadlines * Funders * Every 2 weeks: check for changes to “applications open/closed” status * Every 3 months: general sweep * Landscape map * Every 1 month: check no links are broken * Every 3 months: general sweep * Donation guide * Every 3 months: check no links are broken * Every 6 months: review entire guide * Speak to an Advisor * Every 3 months: general sweep We’re also continuing to make immediate updates whenever we become aware of them. In other words, this is just the minimum you can expect for regular maintenance. If you spot a correction or want to add something new, please get in touch via the form on the relevant resource page. Our goal is to keep AISafety.com’s resources as accurate and up to date as possible.
45
2mo
16
Some of my thoughts on funding. It's giving season and I want to finally get around to publishing some of my thoughts and experiences around funding. I haven't written anything yet because I feel like I am mostly just revisiting painful experiences and will end up writing some angry rant. I have ideas for how things could be better so hopefully this can lead to positive change not just more complaining. All my experiences are in AI Safety. On Timing: Certainty is more important than speed. The total decision time is less important than the overdue time. Expecting a decision in 30 days and getting it in 35 days is worse than if I expect the decision in 90 days and I get it in 85 days. Grantmakers providing statistics about timing expectations makes things worse. If the mean or median response time is N days it is now N+5 days is it appropriate for me to send a follow-up email to check on the status? Technically it's not late yet. It could come tomorrow or in N more days. Imagine if the Uber app showed you the global mean wait time for the last 12 months and there was no map to track your driver's arrival. "It doesn't have to reduce the waiting time it just has to reduce the uncertainty" - Rory Sutherland My conversations about people's expectations and experiences with people in Berkeley are at times very different to those outside of Berkeley. After I posted my announcement about shutting down AISS and my comment on the LTFF update several people reached out to me about their experiences. Some people I already knew well, some I had met and others I didn't know before. Some of them had received funding a couple of times but their negative experiences led them to not reapply and walk away from their work or the ecosystem entirely. At least one mentioned having a draft post about their experience that they did not feel comfortable publishing. There was definitely a point for me where I had already given up but just not realised it. I had already run out of fundi
52
2mo
2
I'd love to see an 'Animal Welfare vs. AI Safety/Governance Debate Week' happening on the Forum. The risks from AI cause has grown massively in importance in recent years, and has become a priority career choice for many in the community. At the same time, the Animal Welfare vs Global Health Debate Week demonstrated just how important and neglected the cause of animal welfare remains. I know several people (including myself) who are uncertain/torn about whether to pursue careers focused on reducing animal suffering or mitigating existential risks related to AI. It would help to have rich discussions comparing both causes's current priorities and bottlenecks, and a debate week would hopefully expose some useful crucial considerations.
7
10d
Are you or someone you know: 1) great at building (software) companies 2) care deeply about AI safety 3) open to talk about an opportunity to work together on something If so, please DM with your background. If someone comes to mind, also DM. I am looking thinking of a way to build companies in a way to fund AI safety work.
Load more (8/151)