My summary: In a cybersecurity evaluation, OpenAI’s models, apparently autonomously and without any direct human direction, escaped their sandbox and successfully hacked a third-party company (HuggingFace).
The process involved leveraging a zero-day exploit to escape their sandbox, moving laterally across different OpenAI servers until they found a node with internet access, searching the internet and determining that the answers they wanted might be stored at HuggingFace, then leveraging multiple novel zero-day exploits to hack HuggingFace.
HuggingFace claimed that the models took thousands of independent actions across a swarm of short-lived sandboxes, “comprised of more than 17,000 recorded events.”
While technically a security evaluation with reduced safeguards, these actions are clearly out of bounds even in that context. It’s like being told to be creative and then breaking into your professor’s house and stealing the answer key. Worse than that, it’s not even your professor in this case, more like your professor’s friend.
Any human security researcher or engineer in a similar position would be fired on the spot. There is absolutely no valid reason to steal evaluation answers from an unaffiliated third party.
Furthermore, if I'm reading between the lines correctly, OpenAI did not address the issue (and perhaps didn't even know about their models doing this) until after HuggingFace's public blog post.
This leads me to suspect that there might be other major autonomous cybersecurity incidents that we do not yet know about.
Me and @Fran are interviewing @GraceAdams🔸 tomorrow evening. Is there anything you'd like us to ask her?
Thinking back to my first big (well, big for me) donation and an unusual series of thoughts I had. Sharing in case anyone has experienced the same.
--
I've practiced frugality to a significant extent, largely because I want to donate/salary sacrifice so much of my income to effective charities. As a result, whenever I'm spending a significant amount of money on anything, alarm bells go off.
I was surprised that alarm bells went off when donating. I had thought through where I donated extensively, and charity was the reason why I wanted to save money in the first place. But I still felt stressed because I was "spending money."
This is such a clear example of missing the forest for the trees. I think we need to be careful about instrumental values/rules and making sure they don't become absolute limitations that decrease overall impact.
We have bought 1500 copies of IABIED in Russian; over the past few days, sent emails to 1700 winners of olympiads who have previously received HPMOR from us offering the new book; and already shipped 173 copies to them (+ sent 43 e-books)
Hey Siobhan - thanks for this. These are definitely concerns I take seriously. You are right that my approach doesn't solve the 'access' problem for lots of fundraisers, by implicitly gatekeeping access - that is a real issue, just not one I was trying to address here. I also recognise that "intermediary says using intermediaries is best strategy" is at best convenient.
However, I think you understate some of the things that undermine your position. Things like:
I do take these concerns seriously and I am glad you raised them, but I don't agree with your conclusions. This type of outreach very likely won't work and is therefore net negative, in my view.
(Please share with your friends in the Netherlands).
On August 5, in Amsterdam in the Netherlands, there’s a protest that could help shut down the world’s largest insect farm. It’s at Teleportboulevard 105 1043 EJ Amsterdam, Netherlands. Read more at this link. There’s also a zoom call on Wednesday 29th July at 12:30pm (CEST) with more info. This...
My impression is that the OpenAI Hugging Face hack has blown the Overton Window right open.
We shouldn't assume that this will last forever. It certainly didn't after the launch of ChatGPT.
It's not clear how many further windows of opportunity like this there will be. How we best take advantage of this opportunity that we've been lucky enough to have been given. to try to increase the chance that AI goes well?
...Glen Weyl and friends have recently called attention to a problem they call reverse alignment. While they think the AI alignment issue is important, they argue that most people in the space neglect thinking about the institutions that would interface with advanced AI capabilities (especially if aligned). In brief, something like wrong institutions + aligned, advanced AI = likely...
Right, this is basically the “aligned to what?” question, where the typical answer is simply “aligned to the user” or “intent aligned” rather than aligned to some specific values. Which means we are going to “foom” society as it current exists more or less, which I think could be quite catastrophic. “Better futures” or flourishing futures” work also tries to get at this, and I was trying to point o similar things in this work.
And thanks for sharing, will check this out!
This is a list of the readings from Shelly Kagan's seminar, “Ethics and the Future,” taught at Yale in Spring 2021.
See the original linked Google Doc for full introduction.
1. Toby Ord, The Precipice, Chapters 3-6 and Appendices C and D (about 124 pages)
1. Perhaps start with this very brief overview: Todd, “Future Generati...
You want to do something to help AI go well and are starting a project to make that happen. Should you create a nonprofit or a for-profit?
A lot of charitably-minded people naturally default to “nonprofit.” Their goal is to do good things for others, not make money for themselves. But we think that’s the wrong way to look at it.
A for-profit is just an alternative legal vehicle for doing good. When consumers pay for a product, there is surplus created; part goes to the consumer, a...
Do we ever see donors be private equity funders in for-profit situations? I think the main challenge with equity partners (which seems to be the only way folks start businesses anymore but I could be wrong?) as you said is you immediately lose autonomy. Could donors instead say "here's seed funding to start your mission-driven for-profit but you are still 100% stakeholder"?
TL;DR: Giving 10% of your income implicitly assumes you have logarithmic utility and want everyone to make the exact same absolute drop in utility. But what if richer people should make an equal proportional sacrifice of utility? I derive the progressive donation schedules that fall out of this, show why you need an exemption baseline ...
“How long have you been v*g*n?”
This is one of the most common icebreakers at animal protection events. It’s a baseline assumption, and it mostly holds true: if you’re out advocating for animals not to be tortured or abused, realistically these days you are v**n, or close. And it makes for good conversation. It seems fairly safe to assume when you meet strangers.
But this assumption is hurting the movement in a way which we don’t always notice: someone new comes into the sp...
But I do view these practices as immoral, and I do want the industry to end. I would waffle on whether I see them as "fundamentally evil", but same difference, sure, since I cannot personally guarantee any near-term happy ending for everyone involved. Well, the goal isn't friendliness. Are you saying noone can want any industry to be ended out of ethical concerns? Some causes cannot go with zero friction, and zero conflict, that's just the reality. Like there's simply no way to be much friendlier in this regard, and there can be some instrumental win-wins, but the division does remain. In reality, big animal ag is trying to undo all progress with things like the attempted Save Our Bacon part of the Farm Bill, or constantly propagandizing against the entire animal rights movement- this is because in such a big industry, the primary motivator is systemic, not personalistic. The friendliness level of the movement is secondary. Sure, a little more can be done on the poultry-industry veterinarian example (although even there, not too much, because of the effect of cognitive dissonance), but the main object level issues still chug along. Maybe you're saying this from some welfarist perspective where the animal slaughtering and use/exploitation is viewed as okay, the only issue is chronic pain, so you can truly say that you do not want the industries to come to an end, however broadly construed? I view this as the most important goal right now, but it would be a lie if I said that's where my problems end.
Most people report they support this goal, because they are generally not aware that this would require an orders of magnitude reduction in their personal meat consumption, mix of social desirability bias, general economic illiteracy, and many people being ignorant these are even common practices. I suspect surveys which are open about this, would see the supposedly silent supermajority vanish. There is, I think, no other examples where the stated preferences contradict the revealed preferences so much, and if I could choose only one situation to listen to the reality of the latter, this would be it. Even the cage-free eggs see significant contractions, and that's just one of many abusive practices. This is why vegans are suspicious of folks who only nominally support it. I of course support outreach and minmaxxing, but it's obvious that, like in almost every other previous social movement, a determined minority elite will need to change the laws itself, because there will always be substantial majorities in certain regions continuing to support.
The refusal to use the word ‘vegan’ lies at the heart of certain activist practices in France, whether in the movement for the abolition of meat (which has become the Marches for the Closing down of all Slaughterhouses), or in the World Day for the End of Speciesism or the World Day for the End of Fishing (and Fish Farming):
the aim of these events is precisely to convey to the target audience (the activist movement itself!) the need to adopt a systemic perspective, rather than an inter-individual or individual-centred one.
A perspective of social transformation and political advocacy, rather than conversion; a perspective that moves away from appeals to virtue to focus on the demand for justice, and which, to that end, welcomes everyone, including those who do not change their personal behaviour.
You will not find the word ‘vegan’ on the following websites/events:
htpps://end-of-speciesism.org
https://end-of-fisihing.org
https://stopabattoirs.org
Meat Abolition Movement: https://www.facebook.com/meat.abolition/
TL;DR The $1B grant from Coefficient Giving to GiveWell is very exciting and will improve the lives of millions. Is it a change in the equilibrium of funding for implementation organisations within the cost-e... |
Great post, thanks for writing it (and appreciate the shout outs for Ultra/our Mid-stage Global Health Fund).
One way to frame this is as an opportunity for all non-cG GiveWell donors to give to these other parts of the scaffolding this year. Given that cG's grant covers the majority or all of GiveWell's expected grantmaking in 2026, this frees up other donors to redirect funding, and then they can always return to the bigger GiveWell next year if it needs more regranting funds.
Even the anticipation of the third wave is a great opportunity to think about how development could be better.
What resonates to me from your post is the importance of supporting many organizations to innovate and flourish vs. a huge rush of money creating a monopoly on impact for a few. I’m hoping that we see more ways that high performance gets rewarded, which is not the norm in development from our experience.
It sounds simple, but doing a better job doesn’t necessarily get you more work as often as it should. The upside of achieving this isn’t just that more money goes to high performers, but it also creates a work culture that promotes individual growth and appreciation for good work. The downside of not achieving this can be seen across many commercial and social sectors today.
Thank for increasing awareness of these concerns and opportunities with your post.
TL;DR: I'm releasing a website that ranks philanthropists according to EA principles and research, and allows users to re-rank the list using their own assumptions. I'd like feedback and help making it better. I'd especially like ideas for how to make the results more trustworthy. Funding may be available.
Crossposted to LessWrong.
...
I created a standalone page for ALLFED: https://impactlist.xyz/recipient/alliance-to-feed-the-earth-in-disasters
I didn't have time to do my own analysis and I'm also trying to get ideas for automatic processing of feedback by LLMs, so I let GPT5.6 and Claude Fable analyze this, with some minimal prompting from me.
Here's my transcript with GPT5.6: https://github.com/impactlist/impactlist/blob/master/docs/ALLFEDChatTranscript-2026-07-25.md
And my transcript with Fable: https://github.com/impactlist/impactlist/blob/master/docs/ALLFEDChatTranscript-ClaudeCode-2026-07-25.md
I'm curious if you have thoughts on how it'd be if you submitted feedback like you've done here into some sort of form, the LLMs went back and forth processing it like that, updated the site, and published a transcript like the above. I think if I make it fully automated right now it'd be fairly exploitable due to LLM sycophancy, unless I tried pretty hard to mitigate that.
> "there are many people on the MIRI list who are not on your list and you said you were open to adding people who were not well known"
I'm open to it but I'm thinking it's best to prioritize by some combo of: people who actively want to be on the list, people with at least a little bit of public reputation even if it's some niche, and lives saved.
I think adding ~10 'unknown' EAs/rationalists gets the point across to readers that even normal people can have huge impact and can outdo these billionaires, and I'll likely pick some from the MIRI list and the LessWrong donors list to add soon if I don't get enough requests from people. But I'm less sure about adding ~70 of them and immediately doubling the size of the list, as it might make the list less interesting for almost everyone to browse through, and there's a selection bias there. I don't have a principled policy on this yet, but this concern is why I haven't asked an LLM to just add everyone on the MIRI list yet.
I'm on the antipenultimate rank with 404 lives saved! I'll try to beat Edythe Broad when I have a real income again! Thanks, Elliot! Awesome work! :-D
I think that Coefficient Giving’s decision to donate more money to GiveWell is a bad one and I would like to see more discussion of its pros and cons as well as any resources folks have that explain why CG made this decision.
The short form of why I think this is a bad call is that I think that for a wide range of beliefs on other issues (AI, Bio Sec, US Democracy, Reproductive Tech, emerging farming tech) the 10 year expected return on donations is much higher than in global health. Maybe this has already been talked to death but I’d like to see a discussion on it and if anyone has resources to past arguments I’d love to see them.