Top line: If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today.
I work for Anthropic, an industrial AI research lab focussed on safety. We are bottlenecked on aligned engineering talent. Specifically engineering talent. While we'd always like more ops folk and more researchers, our safety work is limited by a shortage of great engineers.
I've spoken to several other AI safety research organisations who feel the same.
Why engineers?
May last year, OpenAI released GPT-3, a system that did surprisingly well at a surprisingly broad range of tasks. While limited in many important ways, a lot of AI safety folk sat up and noticed. Systems like GPT-3 might not themselves be the existential threat that many of us are worried about, but it's plausible that some of the issues that will be found in such future systems might already be present in GPT-3, and it's plausible to think solving those issues in GPT-3 will help us solve equivalent issues in those future systems that we are worried about.
As such, AI safety has suddenly developed an empirical subfield. While before we could only make predictions about what might go wrong and how we might fix those things, now we can actually run experiments! Experiments are not and should never be the entirety of the field, but it's a new and promising direction that leverages a different skill set to more 'classic' AI safety.
In particular, the different skill set it leverages is engineering. Running experiments on a real - if weak - AI system requires a substantial stack of custom software, with projects running from hundreds of thousands to millions of lines of code. Dealing with these projects is not a skillset that many folks in AI safety had invested in prior to the last 18 months, and it shows in our recruitment.
What kind of engineers?
Looking at the engineers at Anthropic right now, every one of them was a great software engineer prior to joining AI safety. Every one of them is also easy to get on with. Beyond that, common traits are
- experience with distributed systems
- experience with numerical systems
- caring about, and thinking a lot about, about AI safety
- comfortable reading contemporary ML research papers
- expertise in security, infrastructure, data, numerics, social science, or one of a dozen other hard-to-find specialities.
This is not a requirements list though. Based on the people working here already, 'great software engineer' and 'easy to get on with' are hard requirements, but the things in the list above are very much nice-to-haves, with several folks having just one or none of them.
Right now our job listings are bucketed into 'security engineer', 'infrastructure engineer', 'research engineer' and the like because these are the noun phrases that a lot of the people we like identify themselves with. But what we're actually most concerned about are generally-great software engineers who - ideally - have some extra bit of deep experience that we lack.
How does engineering compare to research?
At Anthropic there is no hard distinction between researchers and engineers. Some other organisations retain the distinction, but the increasing reliance of research on substantial, custom infrastructure is dissolving the boundary at every industrial lab I'm familiar with.
This might be hard to believe. I think the archetypal research-and-engineering organisation is one where the researchers come up with the fun prototypes, and then toss them over the wall to the engineers to clean up and implement. I think the archetype is common enough that it dissuades a lot of engineers from applying to engineering roles, instead applying to research positions where they - when evaluated on a different set of metrics than the ones they're best at - underperform.
What's changed in modern AI safety is that the prototypes now require serious engineering, and so prototyping and experimenting is now an engineering problem from the get-go. A thousand-line nested for-loop does not carry research as far as it once did.
I think this might be a hard sell to folks who have endured those older kinds of research organisations, so here are some anecdotes:
- The first two authors on GPT-3 are both engineers.
- Some of the most pure engineers at Anthropic spend weeks staring at learning curves and experimenting with architectural variants.
- One of the most pure researchers at Anthropic has spent a week rewriting an RPC protocol.
- The most excited I've ever seen Anthropic folk for a new hire was for an engineer who builds academic clusters as a hobby.
Should I apply?
It's hard to judge sight-unseen whether a specific person would suit AI safety engineering, but a good litmus test is the one given at the top of this post:
With a few weeks' work, could you - hypothetically! - write a new feature or fix a serious bug in a major ML library?
Are you already there? Could you get there with a month or two of effort?
I like this as a litmus test because it's very close to what my colleagues and I do all day. If you're a strong enough engineer to make a successful pull request to PyTorch, you're likely a strong enough engineer to make a successful pull request to our internal repos.
Actually, the litmus test above is only one half of the actual litmus test I give folk that I meet out and about. The other half is
Tell me your thoughts on AI and the future.
with a pass being a nuanced, well-thought-out response.
Should I skill up?
This post is aimed at folks who already can pass the litmus test. I originally intended to pair it with another post on skilling up to the point of being able to pass the test, but that has turned out to be a much more difficult topic than I expected. For now, I'd recommend starting with 80k's software engineering guide.
Take homes
We want more great engineers.
If you could write a pull request for a major ML library, you should apply to one of the groups working on empirical AI safety: Anthropic, DeepMind Safety, OpenAI Safety, Apollo and Redwood Research.
If that's not you but you know one or more great engineers, ask them if they could write a pull request for a major ML library. If yes, tell them to apply to Anthropic.
If that's not you but you'd like it to be, watch this space - we're working on skilling up advice.
Hey, I think I'm a good software developer, but I have roundable-to-zero knowledge in ML.
Can you help me estimate how far I am from the bar of being useful to such a project? [Update: I'm consulting with friends about this]
It would also be useful for me to understand the requirements well since lately I talk to other engineers who are sometimes interested in working on AI Safety if they can
Thanks!
I'm hoping to have some comprehensive advice out on this Soon(TM), but my sense is that if you're familiar with multivariable calculus/linear algebra/probability then it's a few months' full-time-work-equivalent to skill up in ML to the point where it's no longer the constraining factor in being hired as an ML engineer.
If you're not familiar with multivar calc/linear algebra/probability then it's a bit of a longer path and a harder one to predict, since I think ease-with-maths varies more widely than ease-with-ML.
Thank you
P.S
I asked a few friends about this and one of them decided to apply himself, so anyway I think this is turning out well. :)
I was surprised by the mention of Cohere because:
Would you say that Cohere are in fact much more concerned about and/or active on extreme AI catastrophe risks than their site indicates?
(This comment is just quickly written personal views/confusions. Also feel free to reply via DM if that feels more appropriate.)
+1 to this!
If you're a software engineer considering transitioning into AI Safety, we have a guide about how to do it, and attached podcast interview.
There are also many other ways SWE can use their skills for direct impact, including in biosecurity and by transitioning into information security, building systems at EA orgs, or in various parts of govt.
To get more ideas, we have 180+ engineering positions on our job board.
Fantastic article, I ended up sending this to quite a few friends who are great programmers and considering careers in AI Safety.
I think the litmus test is great, we need more like this!
If someone is unsure about whether they are able to make a significant contribution to an AI library, they have to be able to come up with an idea for a meaningful contribution that they can implement. That requires extensive knowledge of the AI library and its shortcomings. Is this part of the litmus test?
If not, it might make sense for AI safety engineers to publicize a difficult toy-problem or two that would allow a prospective candidates to immediately get a sense of their abilities, without needing to think too much about what type of pull-request they would do.
Late to the party here, but I was wondering why these organizations need aligned engineering talent. Anthropic seems like the kind of org that talented, non-aligned people would be interested in...
I'm surprised that this is the case because from what I've seen at EA meetups in both the Bay Area and Boston is that the majority of people seem to be software engineers. Is it because the software engineers who go to the Bay Area / Boston meetups may not have as much ML background? Is it because the software engineers in EA meetups don't realize that this is a problem?
I think it's a combination of
The vague term "great" gets used a lot in this post. If possible, wielding more precise concepts regarding what you're looking for—what counts as "great" in the sense you're using the term—could be helpful moving forward. By honing in on the particular kind of skill you're seeking, you'll help identify those who have those skills. And you may help yourselves confirm which specific skills are truly are essential to the position you're seeking to fill.
(Also, I think there are more ways to be a "great" software engineer than being able to write a substantial pull request for a major machine learning library with minimal ramp-up time. So other wording can help you be more precise, as well as kinder to engineers who are great in other ways.)
I appreciate the feedback, but the spec is intentionally over-broad rather than over-narrow. I and several other engineers in AI safety have made serious efforts to try and pin down exactly what 'great software engineering' is, and - for want of a better phrase - have found ourselves missing the forest for the trees. What we're after is a certain level of tacit, hard-to-specify skills and knowledge that we felt was best characterised by the litmus test given above.
Is Anthropic looking for remote engineers, or only SF-based? (and if the latter, do you feel this limits the available talent)
Current spec is that you should spend at least 25% of the time in SF per month. We've a slight preference for folks who can be here full-time, but that's easily overwhelmed by a promising candidate.
It certainly loses us some talent. So far the feeling is that it's worth it for the cultural benefits, but that might change in future. We've definitely noticed that 'similar timezone' is the majority of the friction in folks working remotely, so that might be the thing to specify rather than explicitly being on-site.