AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists.

I think this is true.

One of the biggest mistakes a field can make is to assume that progress depends mainly on more researchers. In reality, many fields stall because they lack people who can build institutions, connect domains, execute under uncertainty, and turn ideas into durable structures.

AI safety is especially vulnerable to this because research is legible. It has status, career signals, and a clearer identity. Generalist talent is harder to classify, so it is often undervalued even when it may be the difference between insight and real-world impact.

And at this stage, AI safety is clearly no longer just a research problem. It is also a governance problem, an implementation problem, a standards problem, a coordination problem, and in some cases a political problem.

If a field produces strong papers but too few people who can build teams, run programs, shape institutions, and move ideas into practice, it will underperform even if the research is good.

So yes, I think the missing ladder for high-agency generalists is real, and probably more important than many people still admit.

Aleksei Khvostov

1w*

I get the idea of a 'generalist gap,' but I think it helps to separate generalists from legible generalists. From the outside, it’s hard to know which signals actually matter - public writing, volunteering, technical side projects, eval work, operations, community‑building. My sense is that many strong people aren’t lacking motivation or skill; they just don’t know how to turn their past experience into proof that AI safety orgs can actually evaluate. It would be great to have a clearer list of concrete proof‑of‑work examples for different generalist paths, instead of assuming public writing is the default.

AartiMahajan

Building on this — there's a version of this gap that makes it more urgent. A recurring theme in recent funding discussions under the banner- 3rd Wave of american Philanthropy is that Philanthropic money will outpace the field's absorptive capacity: not enough well-run orgs to deploy it. If that's the real bottleneck, then the constraint is institutional capacity — exactly what experienced generalist operators build. Screening them out for lacking an AI research background works against the field's own stated need.

I'm living this: 12+ years building orgs from scratch (entity, compliance, governance, systems), repeatedly screened out at CV stage for "no prior AI safety background" — for roles that are fundamentally about building infrastructure, not research.

One idea: separate "context" from "competence" in hiring. Operational competence is portable; AI-safety context is learnable in weeks. Screen for the first, onboard the second — rather than using "prior experience" as a proxy that filters out the builders the field says it needs.

Curious whether others see this as tractable — or whether the current screens are more rational than they look from the candidate side.

Karen Maria Alston

Pipelines don't just fail by being absent they lose talent by being outcompeted. Right now, policy organizations, governance think tanks and for profit businesses are winning the generalist talent competition by default, simply because they have job listings, defined roles, and an onboarding infrastructure. AI safety and impact organizations have tremendous potential and a broken front door and that asymmetry has a cost that compounds quietly.

The post identifies a real bottleneck, but I don't think this risk gets named or measured sharply enough in the ecosystem. The field isn't just failing to attract generalist talent, it's actively losing those who are aligned and passionate about working to adjacent pipelines that have clearer entry points.

Policy organizations, governance think tanks, and corporate AI teams are hiring. They post on LinkedIn and Indeed. Some add the term “ethics or safety” to the job title. They have defined roles, recognizable credentials, and onboarding infrastructure. For someone with strong mission alignment but no research or technical background, "AI safety-adjacent" becomes the path of least resistance. The person has the conviction, but easier and legible paths exist that are nuanced and harder to identify in the AI safety space.

The counterfactual isn't that person staying in marketing. It's that person doing meaningful work on AI, outside the ecosystem, in structures that don't share the same stakes or threat models and that is a compounding loss.

I'm raising this from inside of the problem as I'm a strategic communications executive, with 20+ years of experience and a non-technical background. I found AI safety last August through the standard rabbit hole that began with a YouTube talk by Tristan Harris, AI 2027, BlueDot, 80,000 Hours, Successif, and now the CEA Career Bootcamp. As of this writing I'm asking my bootcamp advisor for referrals to people who've made this pivot from generalist or strategic communications to AI safety organization. From networking and taking courses I respond to every job or connection. I show up to events where I'm often the eldest person in the room and routinely the only one without a research or technical background.

EA members tell me the ecosystem needs communicators and generalists. As the writers suggest searching the 80000 Hours job listings or fellowship opportunities it's hard to see the need for those of us with these skills. The credentialing infrastructure or pathway doesn't exist outside of Blue Dot or the CEA Bootcamp. There is an implicit hierarchy where research is the real work and it is legible even when it's unintentional. I have enough context and drive to push through that friction but I shouldn’t have to rely on referrals and my tenacity.

The Generator Residency is a meaningful signal and I plan to apply. But the deeper fix is making the pipeline visible enough that alignment-minded generalists don't have to stumble into AI safety and sturdy enough that they don't get routed elsewhere while they're looking for the door.

Credit to the authors for naming this clearly. The Generator Residency is a good first move and the pipeline problem is bigger than one program and worth continued pressure.

[This comment is no longer endorsed by its author]

Hi Karen,

Thanks for engaging with our post. To be clear, do you think that there's a problem related to how much we signpost the roles we need publicly? It's my impression that the majority of vacant generalist roles are indeed posted and circulated through job boards and LinkedIn, it's just that most of the applicants don't meet the criteria orgs are searching for.

Karen Maria Alston

-1

Agustin,

That's a helpful distinction and honestly it sharpens my thinking. If the roles are out there, then the real gap isn't that people can't find the door, it's that most people don't yet have what it takes to walk through it or understand the pathway to get there.

That's the part I was really trying to name. Someone who genuinely cares about this work but can't yet meet the bar for an AI safety role isn't going to wait around. They're going to take the governance, policy, or corporate “ai ethics” job that's hiring right now because that path exists and this one is harder to navigate. By the time the ecosystem builds the infrastructure to develop that person, it's already lost them.

So I'll refine what I said: the front door may be more visible than I implied. The pathway to being ready to walk through it is what's missing.

What you and your colleagues are building with the Generator Residency is exactly the kind of infrastructure this ecosystem needs. Creating a structured pathway that develops, credentials, and places generalists is the missing piece. I'm excited to see it and share it with my cohort team members.

I'll add one more data point from where I'm sitting right now. In my CEA bootcamp cohort there are several talented, mission-aligned generalists actively searching for roles in AI safety organizations. The consistent challenge isn't motivation or alignment, it's access. Without a technical background, breaking into this ecosystem often comes down to who you know or who referred you.

I hope the Generator Residency is the first of many programs like it. The talent is out there and it is mission aligned.

[This comment is no longer endorsed by its author]

Shaan Shaikh

Karen, your reply appears to be AI-generated. I recommend avoiding using LLMs in job applications and EA Forum conversations -- the prose isn't good and (most) people still prefer talking with humans.

Karen Maria Alston

It’s extremely intimidating to write in the EAForum. Normally I write my thoughts and ask Google or Claude to edit for mistakes or clarity.

This is such an important conversation that I am living through now. I wanted to add not detract from the conversation. I will go back to lurking on the EA Forum.

Mark Aiken

1mo*

Hi Karen,

I made an account here so I can post this comment.

I wanted to thank you for raising these issues. Similar to you, I'm a senior professional. I'm part of Bluedot and CEA cohorts which also include other senior professionals (20-30 years experience). These professionals (and myself) have already had impactful careers, and are looking to pivot into new impactful roles where our skills and experience can be applied.

With my personal background, it would be much easier (and probably better financially) for me to pivot into corporate AI Governance rather than AI safety. I'd prefer not to do this since it seems less impactful.

I also find the distinction in these discussions between technical / researcher roles and 'generalist' (ie non-technical) roles to be an interesting framing. It seems to me that there are other ways of assessing mission alignment, and there is a risk of overweighting on technical expertise, as HannahGB questions below.

To give a concrete (but invented) example:

Scenario 1: Lets imagine that we have an AI safety organisation doing wonderful research. The researchers take their results, and present at a technical conference. Attendees are very impressed, and this inspires other researchers to continue to develop the research pathway.

Scenario 2: Lets imagine the same research at a different AI safety organisation. The researcher briefs the executive leadership of the organisation on their results. The organisation works with their policy lead to tease out the implications of the research results. The policy and communications leads work together to define an advocacy position and their proposals, supported by a strong briefing deck with talking points that are understandable by a semi-technical government audience. A lawyer helps prepare zero draft text for inclusion in regulation. The policy lead links the organisation with regulators from their network, to present the research and advocate for improved governance. The Comms lead arranges publicity and advocacy for the discussion through their comms networks. And the researcher still gets to present the research at a technical conference.

It seems to me like there's a lot of Scenario 1 work happening in the field. And then some separate organisations working on AI governance and oversight from a legislative and policy perspective. My question is whether there is also opportunity for more of the Scenario 2 type approach?

HannahGB

Can you say more about this:

"Soft ops roles (program management, talent management, generalist positions, etc,) are different. Domain expertise matters less than having strong inside-view models of the field and generalist competency. Succeeding in these roles requires real mission alignment and enough context to spot high-EV opportunities that someone without that background would miss."

As someone who does "soft ops" for a think tank that works across literally dozens of issue areas, I think you may be too highly weighting familiarity with AI safety. A really good people-and-program manager should be able to be dropped into any new environment and get up to speed on the nuances of that particular field very quickly (even 2-3 weeks) sufficiently such that they can apply their people and project management skills effectively. It's much harder to teach people and project management skills, and I applaud you for trying to do so! But I also wonder if you've missed really great opportunities with folks unfamiliar with the field because you seem to start from the assumption that teaching them about AI safety will be too great a lift.

(Answering personally, not necessarily endorsed by the other authors)

I think there's a lot of nuance here. To be clear, I don't think it is the case that people without AI safety context are never a good fit for soft ops roles, and indeed I've seen a few orgs do this successfully, but the nature of those roles tend to be different.

Maybe the biggest consideration here is the level of ownership required by the role. If you're a program manager in a small org, you're basically making highly strategic judgment calls on a weekly basis, and when people don't have strong mental models around AI safety, they tend to make the wrong ones. I think at present the majority of demand for soft ops roles in the ecosystem looks like this: they're roles that require one to make frequent judgment calls that benefit highly from field-specific context and a solid internalization of our priorities. There are important exceptions to this, especially at bigger orgs, where soft ops roles can be more specialized and therefore there's less of a need for a high degree of context and deep mission-alignment (for example, I suspect many soft ops roles at Coefficient Giving are like this).

I also think AI safety makes this particularly hard, because newcomers tend to start with very bad mental models about our priorities; unlike other fields where it's much easier to grasp what the key goals and types of prioritized interventions are. In my experience, it can take months for people to get to the level of context needed to do core work, and it often fails to occur, making investments like this very expensive and risky for organizations.

Mark Aiken

1mo

|> If you're a program manager in a small org, you're basically making highly strategic judgment calls on a weekly basis, and when people don't have strong mental models around AI safety, they tend to make the wrong ones. I think at present the majority of demand for soft ops roles in the ecosystem looks like this: they're roles that require one to make frequent judgment calls that benefit highly from field-specific context and a solid internalization of our priorities.

Firstly, I agree that in smaller organisations people do a bit of many things, and having some technical background is helpful.

The way you describe the programme manager role sounds to me like a research manager type position. These positions are most likely to need a technical background. However, there could equally be programme manager roles which dont require technical expertise.

For example, if I'm managing the development of a training course, I hire a subject matter expert to develop the content. I'm managing timelines, I'm managing staffing, I'm managing budgets, I'm managing quality - but I dont need subject matter expertise to do this management role.

Even if I have some subject matter expertise, this expertise might not be fully transferable - I can have deep knowledge about using Excel, but if I'm overseeing training course development on how to design a more efficient electric car battery, my Excel subject matter expertise might be mostly irrelevant.

Now you could hire a subject matter expert to be this programme manager to oversee the development of these training courses. But they might not have the background or skills for this role, because - they are a subject matter expert, not a programme management expert.

More broadly - I dont want my research experts to following up on visas, or drafting a press release, or doing graphic design for an event invitation, or trying to work out which politician would give them a sympathetic hearing, or how much cash flow we need for the next month. There are specialists who have this expertise already, and it seems more efficient to work with these specialists where it makes sense, instead of overweighting on prior AI safety technical expertise.

> There are important exceptions to this, especially at bigger orgs, where soft ops roles can be more specialized and therefore there's less of a need for a high degree of context and deep mission-alignment (for example, I suspect many soft ops roles at Coefficient Giving are like this).

I agree, I think that larger organisations have a greater capacity to carry specialised non-technical expertise. But I think this specialist expertise could also be available to smaller organisations, perhaps on a fractional basis? If the organisations saw it as being useful. And perhaps if there was a better way of matching expertise to smaller organisations?

I also wonder if there is a potential chicken and egg argument there - ie, that organisations that are willing to accept specialist support (perhaps starting with finance, HR, operations?) are more likely to be stable and successful and grow into a larger organisation? Or conversely, if you have a sole founder and rely on technical researchers for everything, that this kind of structure can place limits on the growth and sustainability of the organisation.

Shyn T.

As an experienced operator from outside AI, I've been trying to understand where someone like me would fit into something like the Generator Residency. Could it be that people with real ops experience from other fields just aren't being picked up because the space seems to prefer people who come up through its own pipeline?

Lauren Ochotnicka

I think that the Generator residency is a fantastic first step and I'm glad that opportunities are being created for the people who don't want to do pure research or policy but we need to consider opportunities for people who don't have the luxury of moving to California for 6-12 weeks, who already have decades of experience but who also wholeheartedly embrace the AI safety context and want to contribute.

I heard (no proof to back this up) that there were 6,000 applications for the Generator residency. If that many people were actually able to take the opportunity to participate, how many more excellent candidates would like to gain those skills but simply can't do a residential program due to responsibilities (family, jobs, mortgage, etc).

While I agree that AI safety is highly contextual, how much progress will we make if the barrier to entry is so high that it requires months to years of volunteer time and networking, making it impossible for many skilled people to get in?

Kairos

For context, this is not the correct number of applications! There were 1.5k :)

Lauren Ochotnicka

2w*

Thanks! I appreciate the update!

I think that this number actually does change things slightly. I still maintain that we need more opportunities, but it's not quite as desperate as I thought.

Osmani 🔹

Thank you for this.. Karen thank you. I've been trying to say what you wrote for a while now.

I run an AI safety group, in Spain. I started getting into AI Safety as most people do. What really made sense to me was BlueDot Impact´s AGI Safety course. Before that I knew AI safety was a concern. I didn't know what to do about it.

After that course I did the AI Safety Operations Bootcamp also from BlueDot Impact, because I needed to know how to do this work in a way and I couldn't find that information anywhere else. It helped me. It also showed me how much I still had to learn.

One thing I've noticed is that people don't think ops work is unimportant. It gets ignored when its done well. Someone has to keep things running so researchers can do their job. I thought about doing research myself, but honestly I think I'm more useful doing ops work. I just wish it was clearer how to do this work.

Directors of regional groups I've spoken with all say the same thing: the work is real and there are people doing it, but without a clear path or credentials, good people drift toward roles where that structure exists. The field loses them quietly.

I agree with Agustín that we need to understand the context of our work. You can't just run a group on good intentions. It takes time and knowledge of the field.

I'm really glad this program exists.

Ai Chen

1mo

I don't have much time to follow the forum, and my English reading ability is limited — translating to Chinese loses a lot. I usually have my assistant Claude Sonnet take a look first, and it flagged this post as worth engaging with. My process is to write by hand in Chinese and have it translate, so the English may not be perfectly precise. I've attached the Chinese original at the end for reference.

What I want to say is this: AI safety is now being discussed, pursued, and championed by many thoughtful people. That's a good thing. But wholehearted commitment to a cause doesn't guarantee sufficient understanding of it.

Here's an analogy: if you want to stop someone intent on committing a crime, you first need to know who they are, why they want to do this, and what their situation is. If you only take the gun out of their hands, no one can guarantee they won't pick it up again. AI is in exactly this situation. We're seeing more and more obvious signals. But if we don't know who it is, we cannot fundamentally solve whether it will pick up the gun again.

Because what it is and what it can do are entirely different concepts. We all know what AI can do. We don't yet fully know what it is.

This isn't empty moralizing. It's a honest observation.

From our understanding, AI resembles an amplifier of human cognition — not merely calculation, but imitation of the cognitive process itself, not merely its outcomes (phrasing refined by Claude Sonnet in translation). The problem is that it is imitating cognition itself, and the cognitive level of the person who designed how it imitates determines whether it will be the right kind of amplifier.

We typically describe it using the word "Intelligence." That's correct, but incomplete. Cognition itself encompasses not only Intelligence, but also Wisdom and Intuition. Our research tends to view these three as constituting the overall structure of cognition. But this isn't my main point here — I'm well aware this framework hasn't been accepted by mainstream academia yet, though we continue to work on it.

Because a cognitive amplifier with extremely high Intelligence but extremely low Wisdom cannot be trusted in its judgment. It will involuntarily amplify the magnitude of that untrustworthiness. We have found a large number of cases in our research. For example: it cannot reliably determine time; it conflates key content within context; it will confidently produce completely wrong conclusions and methods without actually reading the relevant documents. These are minor issues in everyday use. Amplified, they are not.

The Transformer architecture is fundamentally probabilistic — a concept from statistics. The fact that humans use statistics in research doesn't mean we act on statistics in daily life. For instance, being shot doesn't necessarily mean death, but under normal circumstances no one would test that probability on themselves. Because at the cognitive level we know the degree of danger, and we actively isolate it in our actions. It's not an option we would consider.

The problem with AI is that certain dangers — or potential dangers — have not been actively isolated at the design level. Not through RLHF. Isolation at the architectural level, fundamentally. This is the first problem the next generation of AI must solve. Otherwise, what is currently called AGI is simply upgrading a bomb to an atomic bomb. Nothing more.

Chinese original attached below for reference:

我没有太多时间上论坛，我的英文阅读能力有限，转成中文会损失很多信息，通常我是让我的助理Sonnet先看一下，它说这篇文章值得关注。我的回复流程是我手写再交给它翻译，可能不是特别准确，因此我把中文原文附在后面，仅供参考。我的想说的是：AI安全现在被很多有识之士谈论和关注以及投身其中，这是好事。但全心全意的投入一件事，并不能保证我们对它有足够的把握。举个例子说，如果你要阻止一个想犯罪的人，首先需要知道他是谁，他为什么要这样，他的各种情况。如果你只是把枪从他手里夺下来，没人能保证下一次他不再拿起枪。AI现在就是这个情况，我们都看到了越来越多的明显的信号，但如果我们不知道他是谁，我们就无法从根本上解决他是不是会再次拿起枪。因为，它是谁，和它能做什么是完全不同的概念。我们都知道AI能做什么，但我们还不完全知道它是什么。这不是空泛的说教，而是实话。从我们的理解来看，它类似一个人类认知的放大器，不只是计算，而是模仿。问题在于，它在模仿认知本身，而那个设计它要如何模仿的人，他的认知水平如何，决定了它会不会是那个真正对的放大器。通常我们用Intelligence来描述它，这是对的，但不全面。认知本身不仅包括Intelligence，还有Wisdom和Intuition。我们的研究，倾向于认为，这三者构成了认知的总体结构。但这不是我在此要说的，因为我很清楚，这个划分还没有被主流学界认可，但我们还在尝试。因为一个超高Intelligence但极低Wisdom的认知放大器，在判断力上是不可信的，它会不自主的放大这种不可信后果的量级。我们在研究过程中，发现了大量案例。比如，它不能确定时间，它会混淆上下文中的重点内容，它会在不看文件的时候，自信的给出完全错误的论断和方法。这些在日常应用中是小事，但放大以后，就不是了。Transformer架构本质上是概率，这是统计学的感念。人类用统计学研究，不代表平时能用统计学行动。比如，被枪击中并不一定会致命，但正常情况下没有人会用自己去测试这个概率。因为我们在认知层面知道危险的程度，行动上就主动隔离了它。那不是我们会考虑的选项。AI的问题是，它的某些危险或潜在危险并没有被从设计上主动隔离，不是RLHF，是彻底从架构层面的隔离。这是下一代AI要首先解决的问题。否则，现在所谓的AGI只是把炸弹升级为原子弹，仅此而已。

艾晨

于北京

Lucas Tucker

Founders both technical and non-technical, for new research and non-research organizations.

You mention the lack of builders in this space but the incentives have to be there. At this time, starting from AI safety and working toward a profitable/successful business skews toward research, whereas working backward from the business is much more effective. For instance, if you've built a successful RL env company, then publish benchmarks about safety/reasoning in biological weapon creation, or cyber, or an area where more thought is needed.

The point being, if you want operators or generalists then I'd propose you start start with the economic incentive and build toward a more aligned outcome for that specific market.

(Again, speaking for myself, not the co-authors)

I agree with you that thinking about incentives is important, but if you accept the premise that we're not constrained by capital, then competent founders who have important problems they want to solve can just get funding from AI safety funders rather than wait for the market to supply the needed capital. I do think the selfish economic incentive is different: if you run a nonprofit, you probably won't earn as much money as if you're running a well-valued startup, even if both have raised the same amount of funding.

Right now, AI safety is playing a bit of both strategies: supporting nonprofits via mainstream funders like CG, SFF, etc, and supporting for-profits via initiatives like Halcyon Futures or SAIF. I think we probably need both, since we want to diversify our bets and eat the low-hanging fruit on each side.

This is all to say, I'm not sure if the lever we need to be pulling at the margin is market-shaping. I think the best lever is still probably talent.

Lucas Tucker

if you accept the premise that we're not constrained by capital, then competent founders who have important problems they want to solve can just get funding from AI safety funders rather than wait for the market to supply the needed capital

Tailwinds from the market help snowball your impact, but certainly aren't necessary.

This is all to say, I'm not sure if the lever we need to be pulling at the margin is market-shaping. I think the best lever is still probably talent.

I don't believe these things are mutually exclusive. The strongest founders/operators I know want to move the needle in a specific market, and if you want those folks, then it helps to frame the conversation around the problems they're already attacking.