There are lots of really important projects and startups that, when you look at them, you’re like, “Well, obviously someone should be doing that. Ideally five people should be doing that.” But unfortunately nobody is doing it.
So I think for a lot of these things, the bigger risk is not that somebody else takes a better swing at it and your work is obsoleted, but that no one even tries. That’s why we’re trying to bring more founders, more talent into the field to work on some of these problems.
— Max Nadeau
Researcher Owain Evans and his team discovered a ‘dial’ inside AI models that controls how evil they are. Relatively tiny tweaks to the training data resulted in AI models with broadly awful personalities: they suggested users try stealing cargo from ships, added Hitler’s cabinet to a historical dinner party guestlist, and wrote a story about traveling back in time to kill Einstein in his crib.
There are millions available for anyone who can launch a successful nonprofit AI safety startup. The hard part, it turns out, is finding people to take the money.
Coefficient Giving has drawn up a list of dozens of ideas for organisations it would like someone to start — and it’s looking for founders. Today’s guest, Max Nadeau, works on Coefficient Giving’s Technical AI Safety team, where he’s trying to find talented people who can turn neglected AI safety problems into effective organisations.
Project Tailwind is Coefficient Giving’s attempt to get those organisations started.
Preseed grants run $200,000–$2 million, with no preliminary results required.
The gaps Max most wants filled include independent assessment of AI companies’ safety claims, research aimed at aligning far more powerful systems, and shared infrastructure that speeds up the whole field.
But money can’t supply the hardest part: a founder with a convincing account of how their work will actually reduce catastrophic risks. Producing good research is only one step. Someone has to use it, change their decisions, or adopt the safeguards it makes possible.
Max and host Zershaaneh Qureshi discuss what makes a proposal worth backing, why nonprofits can have a bigger impact on safety than frontier companies, and which gaps most urgently need someone to fill them.
Disclosure: Coefficient Giving is 80,000 Hours’s largest donor, though we haven’t received funding directly from Max’s team.
This episode was recorded on August 18, 2026.
Our production team includes:
Video editors: Josh Alward, Dominic Armstrong, Ollie Bignell, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour
Producers: Elizabeth Cox and Nick Stockton
Coordination and support: Katy Moore and Lou Moran
Coordination and support: Katy Moore and Lou Moran
The interview in a nutshell
Max Nadeau, who works on the Technical AI Safety team at Coefficient Giving (formerly Open Philanthropy), is launching Project Tailwind — an effort to get many more philanthropic startups founded in AI safety.
Coefficient Giving is putting startup-scale money behind new AI safety nonprofits
Tailwind is borrowing from the venture capital playbook with its “hits-based giving” approach — where most grants have little impact but a few make the whole portfolio worthwhile. Max says they’re leaning into this harder given rapid AI progress, deliberately avoiding the grantmaker failure mode of scrutinising every cent rather than making sure the best projects are fully funded. They’re offering:
Preseed grants of $200,000–$2 million for founders who may not yet have a team or detailed plan.
Seed grants of $2–20 million for applicants with a founding team covering the important bases and some preliminary results.
Grants up to $200 million — usually a second or third grant once an organisation has a track record, but possible out of the gate for exceptional cases. Max personally predicts Coefficient Giving will make grants larger than $200 million to technical AI safety organisations within the next year or two.
The recent $160 million grant to Resolution shows what earns the biggest cheques: Geoffrey Irving (ex-chief scientist at UK AISI, ex-DeepMind and OpenAI) proposed an ambitious research centre on aligning superintelligence — a problem Max thinks is sorely neglected — with a plan to use AI aggressively to accelerate the safety work itself. Grants at that scale require “pretty hard-to-fake signals” of being a thoughtful founder up front, but Max is enthusiastic about scaling new organisations to that level quickly on the strength of their results rather than the founder’s profile.
Max stresses this is an “all of the above” strategy: Coefficient is also funding existing organisations to scale, and many would-be founders should consider starting a team inside impactful existing organisations instead. Tailwind is for projects that don’t fit existing organisations.
Talent is the main — maybe the only — bottleneck
Every good AI safety organisation has a long list of projects and engagements with AGI companies and governments it can’t staff. What Max wants in founders:
The standard startup traits: perseverance and energy for management, funders, and communication, not just heads-down technical work.
Traits specific to AI safety: being discerning about your theory of impact for risks from systems that don’t exist yet, and being willing to pivot dramatically when a higher-impact avenue appears.
Willingness to engage with speculative questions about where AI is heading. Max points to METR’s early work on time-horizon evaluations, which aged far better than benchmarks built at the same time, as an example of this foresight paying off.
High context on the field’s existing arguments and fault lines. Unlike for-profits, there’s no impossible-to-fake demand signal — “in AI safety, no one will tell you if you’re not having impact.”
Common founder mistakes Max flags:
Not being ambitious enough about the magnitude of the problem — getting sucked into smaller, present-day issues you’ll regret not pivoting away from sooner.
Not thinking about who you actually want to change. Nonprofits don’t need paying customers, so it’s easy to neglect the whole “supply chain” of impact: getting good ideas into practice and into other people’s minds, which requires different strengths from generating them.
On pivots, Max’s case study is Redwood Research: Buck Shlegeris and Ryan Greenblatt concluded mechanistic interpretability wasn’t producing enough progress, shrank the organisation to just the two of them for about a year of rethinking, and emerged with AI control. They got a lot of flak, but Max thinks basically everyone now agrees they’re having far more impact. Founding “totally does require” an appetite for that kind of short-term pain.
Third-party organisations have an edge over AI companies’ safety teams
Max thinks the best opportunities for impact are outside AI companies — and neglected precisely because so many people are inside:
AI companies are in a “brutal, perilous race to the bottom” on safety. Their safety teams are pressured to build cheap techniques, run predeployment evaluations fast, and put out today’s fires.
Companies “just don’t have the top-down organisational prioritisation and resource allocation” needed. Increasing that willingness to pay is best done from outside.
Evidence and assessments from organisations like METR and Redwood have already raised the consequences of getting safety wrong, pushing companies to take alignment more seriously.
What to prioritise if you’re worried about being obsoleted by well-resourced lab teams:
Independent evidence and assessment: companies aren’t credible voices on it, and aren’t doing it well. You’d have understood the state of alignment far better reading METR’s misalignment risk assessment than the companies’ system cards — and been less surprised by the OpenAI Hugging Face incident.
Speculative, ambitious, principled research that companies aren’t doing for contingent reasons. It could be obsoleted — but so could any alignment work. For many projects, “the bigger risk is not that somebody else takes a better swing at it… but that no one even tries.”
In terms of impact, Max sees AI safety nonprofits as much more ambitious than the for-profits. VC funding pushes founders toward well-understood, present-day problems with paying customers — e.g. misbehaving or prompt-injected agents — which are tiny in stakes next to speculative risks like AI-enabled pandemics or loss of human control.
Independent auditing and assessment of AI companies’ safety practices — Max’s “personal favourite,” made salient by the Hugging Face and Mythos UK AISI incidents.
New alignment research centres — e.g. on chain-of-thought monitorability, which already differs a lot between companies.
Better evidence generation on how capable and risky models are. The OpenAI misalignment incidents were detected by Hugging Face, a third party.
Security and verification research, including hardware and training-data supply chain vulnerabilities.
Public goods for the field, on the MATS model — e.g. a compute cluster for safety research.
Fieldbuilding to integrate the growing number of people who want to work on AI risk.
If you’re interested in founding an organisation, develop a vision in depth first — from the Tailwind list or your own idea (Max expects many grants for ideas they didn’t write down) — then apply via cg.org/tailwind.
Zershaaneh Qureshi: So I’m pretty curious here about what you think the bottlenecks actually are to getting all of this kind of work done. What are the main reasons that the field isn’t already in a position, given that it already has several AI safety organisations, to do all of this kind of work?
Max Nadeau: I think the most important bottleneck, maybe the only bottleneck, is talent. There are a number of really exciting AI safety organisations, but at each one of them, there’s a long list of projects that they would love to do, engagements with AGI companies and with governments that they would love to take on, that they just do not have the staff to execute on. Which is a big reason that we’re doing Tailwind, and why we’re trying to scale other parts of the AI safety field: we just need more talented people to come work on these problems.
Zershaaneh Qureshi: So you’re saying that talent is the biggest bottleneck right now for AI safety. Can you tell me more kinds of talent we’re most missing right now, especially in terms of the skills and profiles needed in founders and founding teams?
Max Nadeau: For the purposes of this conversation, I’ll mostly talk about the founders specifically, although I do want to just make clear to the audience that I think there are lots and lots of opportunities for people who are not interested in founding new organisations to have a lot of impact in the AI safety space, and in the third-party AI safety space.
But I think that founders are a really important bottleneck. When it comes to what we’re looking for in founders, and what we think is most important for a founder in the AI safety space, I think there’s a lot of attributes which are shared with normal for-profit startup founding:
It’s really valuable to have perseverance — you know, to be willing to run through walls.
It’s really important to have energy for not just heads-down technical work, but also managing employees, dealing with funders, generally communicating what your organisation does to a wider audience.
And then I think there’s also some distinctive attributes to AI safety philanthropic startups which are not as important for the normal for-profit world. The biggest one is being really discerning and careful about your theory of impact — really thinking deeply about what your organisation is doing, how it’s supposed to make a difference on future risks from AI systems that haven’t been created yet, and being willing to rethink your strategy, to pivot dramatically even, if you see a higher opportunity, higher-impact avenue.
Relatedly, I think it’s very important for people founding organisations in the AI safety space specifically to be willing to think about speculative questions that don’t have clear answers, to think about what the future of AI holds and take best guesses about where these trends are going, and then try and be prescient and try and act in preparation for the AI systems and the risks that are emerging down the line.
That’s something that’s very unique about AI safety, even compared to other philanthropic cause areas: the problems that people are working on mostly haven’t manifested yet. And a lot of people are very averse to that; they want to only work on problems that are more grounded or more empirical or more demonstrable today. So I think that there’s a lot of opportunity for people who are willing to get down in the mud of these speculative questions, and are willing to entertain these sorts of out-there hypotheticals.
Mistakes startups make
Zershaaneh Qureshi: If you are trying to found something new, I think there are some pretty obvious ways that could go wrong — like the founding team could fall out, and that could be pretty bad. But are there any less obvious mistakes that you see founders or early employees making that you think people should be trying to avoid?
Max Nadeau: I think one of the biggest ones is not being ambitious enough about the magnitude of the problems that you’re targeting, and getting sucked into things that are either smaller scale than the sorts of risks that your organisation could be working on; or maybe they’re more present day, and so they’re easier to work on, but they’re just not as large, so in a couple of years you’ll be thinking, “Why was I working on that? Why didn’t I pivot sooner to working on this other thing, which felt speculative and early stage at the time, but like now seems really important?”
So I think just trying to head that off in advance is a mental move that I would want a lot of founders at these new organisations to be making.
Another failure mode that I would call out, especially for nonprofits, is that, because you don’t have to actually get customers in the world to fork over money for your goods and services, there’s a common issue of not thinking hard enough and not spending enough time focusing on the people in the world who you actually want to change their ways on the basis of the work you’re doing. Either you want them to read the reports or the evidence that you’re putting out and change their mind about some important topic, or you want them to implement some practice that you’ve developed that they’re not implementing, or you want them to pursue a line of research that they can pursue but you can’t.
As a philanthropic startup, you need to really be thinking about, like, the entire supply chain, or like every step in the recipe of having impact — which both involves having good ideas, but also involves actually getting those good ideas into practice in the world, and into the minds of other people. And those two parts of the process can rely on different strengths and involve different styles of thinking.
Why AI safety needs outsiders
Max Nadeau: The biggest argument to me for the importance of working outside of AI companies is that we’re in a really brutal, really perilous race to the bottom on safety between many different AI developers. And there’s this really tough pressure that the safety researchers and safety teams at AI companies are under to develop safety techniques that are not very costly to implement and to put in practice; to perform their predeployment evaluations of models very quickly, so that the model can either get deployed internally and start accelerating the company’s research progress, or deployed externally and start making the company money; and to work on problems that are manifesting from the AIs right in front of us, that are happening today — you know, there’s like a lot of fires that are happening at companies that there’s a strong pressure to put out immediately because of this race to the bottom.
And I look at that situation and I think, wow, there’s really not enough willingness to pay at these AI companies. They just don’t have the top-down organisational prioritisation and resource allocation that is necessary to make safety go well. And obviously that fluctuates over time, and there are exceptions to this; sometimes there are some teams or some efforts that have all the resources they need.
But broadly speaking, we do not have good methods for aligning the AI systems of tomorrow, or even the AI systems of today. And we’re going to need more slack and more willingness to pay — for both research, and then also, just in practice, putting in place costly measures while we’re using AIs that are going to slow down things and reduce revenue and otherwise get in the way in order to prevent large negative externalities from being imposed on the world.
And I think that within AI companies there’s a lot of great work that can be done that tries to make the most of the very limited willingness to pay, but that if you want to increase the willingness to pay and release this really harsh pressure of the race to the bottom, I think the best opportunities to do that are outside AI companies.
Impactful work outside AI companies
Max Nadeau: I think there are lots and lots of types of AI safety work and niches that organisations can fill outside of AI companies that really cannot be obsoleted by work happening within AI companies….
One is work that just provides more evidence and more analysis and assessment of what the state of play is on these AI risks. That’s work that AI companies can’t really do — because they’re not credible voices on it, and because it often involves talking about multiple different AIs from multiple different companies and comparing them against each other.
It’s also work that, in practice, I think these AI companies are not doing effectively. I think you would have gotten a very different sense of the state of AI alignment if you just read system cards than if you were reading work like the risk assessment on misalignment that METR put out recently. And I think that you’d be much less surprised by the incidents of misalignment, like the OpenAI Hugging Face incident, if you had been reading the METR work than if you’d just been reading the system cards.
So that sort of work is just not going to be done, it isn’t getting done by AI companies. It has to happen outside.
There’s other work that could happen in AI companies, but isn’t, for just more contingent reasons, I think. There are types of AI safety research that are more speculative and more ambitious and more principled that are just not happening in AI companies. And those are areas of work that totally could be obsoleted by the alignment work that’s happening within AI companies — we can’t rule that out — but I think that’s really true of any work on alignment. You know, maybe whatever technique you’re developing won’t end up mattering, because other alignment techniques were good enough. Or the opposite: maybe your technique just wasn’t good enough, because nothing was good enough anyway.
I think with these sorts of speculative problems, there’s always a risk that the work you’re doing ends up not really fitting in and not really making an impact, and that’s just par for the course.
Is impact possible inside AI companies?
Zershaaneh Qureshi: Something I’m interested in here is whether you think that Anthropic is basically swallowing the field at the moment in terms of absorbing all of the talent and the funding and the influence and so on — whether you think that, and also, if so, how good or bad that is for AI safety overall.
Max Nadeau: I’ll focus primarily on the talent one, because that is, I think, the main bottleneck that a lot of efforts outside of AGI companies face. It’s definitely true that Anthropic has been hiring very rapidly and they’ve been pulling people away from lots of different organisations and institutions across the AI safety field.
One thing I would say is that even for people who are working inside AI companies, and inside Anthropic in particular, there are often lots of opportunities for them to be having more impact than they are in their current role and on their current trajectory. I think a lot of people are just not taking those opportunities, not pursuing those higher-leverage opportunities, even within AI companies. I think it’d be even better for many of those people to leave Anthropic — and maybe other AI companies, although I think it’s a more complicated case — to pursue things in the third-party ecosystem for all the reasons that we’ve been discussing in this podcast.
But I think the fact that a lot of these people, even within Anthropic, are not necessarily pursuing the highest-impact roles or projects or opportunities that they could be just goes to show that in a lot of cases the people working there are not primarily motivated by trying to have as much impact as they can.
And, you know, I don’t think that’s objectively wrong. And I think also many people will say that when you ask them. But I think it’s an important fact about the world, because when people who are new to the field or who are thinking about what they want to do to have the most impact see that there are so many people working on AI safety at Anthropic, it’s easy to assume that’s because they’ve all thought really hard about what the highest-impact thing for them to do is, and decided that their current role is the highest-impact thing. And that is true in some cases, but I think it’s not true in a bunch of other cases too.
So if you’re very morally ambitious, and you’re trying to have as much impact as you can, I think those are not necessarily the role models that you should be looking towards for what it looks like to try and have as much impact as you can.
Epistemic status: Speculation from two decently informed advocates armed with anecdata.
Note on process: After having some version of this conversation several times and saying, “we should probably write about this publicly,” we took the less heroic route: we recorded one of our conversations, fed the transcript into an LLM, and then substantially revised the structure, substance, and framing ourselves. We will not be sharing the transcript, as it is in...
I want to start by noting that I am amazed and somewhat in awe of what's been accomplished by the EA animal welfare movement. I am enthusiastically supportive of all campaigns that make living and dying conditions less bad for farmed animals.
I worked on such campaigns for almost 20 years, and I'm proud of that work.
That said, I do think it's a strategic error that the EA animal movement has so thoroughly moved on from advocacy for animal liberation and diet change.&n...
Author: Grace Ryba (she/her), Executive Director
TL;DR:
* BluePerch is a new animal welfare grantmaker.
* Grant applications aren't open yet - we're aiming to open applications in roughly mid-2027.
* We welcome early expressions of interest from potential grantseekers so we can notify you when applications open - express your interest here or see further details below.
* We're also seeking boa...
