Hide table of contents

Check our our other blogs here. Also, we're hiring for a Software Engineer and a Product Manager! We'd love to see your application.

Introduction

In 2022, we ran the second iteration of the AI Safety Fundamentals: Alignment course. The curriculum was designed by Richard Ngo, and the course was run by Jamie and myself while we worked at EA Cambridge. The course’s objectives were to teach participants about AI alignment, motivate them to work in the field, and help them find opportunities to contribute. We received 450 applications, made 342 offers to join the course, and the cost per student was £440. 

This blog post analyses the career trajectories of the course participants, comparing how many were working on AI safety when they started the course vs how many are working on AI safety today. 

Impact 

In the course’s application form, we asked participants for their CV / LinkedIn profile, their experience in AI safety, and their career plans. By evaluating each individual using this information, we estimate only 18 participants (5% of 342) were working on AI safety when we accepted them onto the course. 

To determine the number of course alumni working on AI safety today, we looked into each individual’s online presence, including their LinkedIn, personal website, blog posts, etc. looking for signs of AI safety-motivated work, and we sent out an alumni survey in January 2024 to learn more about their post-course actions. Our research suggests 123 alumni (37% of 342) work ~full-time on AI safety today, up 105 from 18 initially. 

Of the 105, 20 expressed no interest in pursuing a career in AI safety in their initial application. Either they intended to pursue a different career, or had not done any career planning. We compared their plans from January 2022 with the actions they’ve taken since then and we evaluated their responses to the alumni survey. Based on this, we believe most of the 20 would not be working in AI safety today had they not participated in the course. 

The other 85 expressed a desire to work in AI safety and had tentative plans for how they’d enter the field, but weren’t yet active contributors. Their responses to our alumni survey describe how the course accelerated their impact by helping them find impactful roles sooner, and giving them the tools to have a greater impact in those roles.

Now vs then

Our courses today are a major improvement over the course in 2022. 

In 2022, the curriculum was a static list of resources and discussion plans were just a few open-ended prompts. 

We now provide much higher-quality learning experiences via compelling and coherent course narratives, challenging pre-session exercises, and engaging session activities that drive towards our learning objectives. 

In 2022, we had no idea if people were doing the readings or who attended each session. Some facilitators would let us know who attended via a post-session form, but it was a clunky process and was ignored most of the time.

Now, participants tick off and give feedback on each reading in our course platform, and we track session attendance via an easy two-click login process. We adapt the curriculum on an ongoing basis based on feedback, and we disband low-attendance cohorts so participants are always in cohorts with engaged peers. We can guess at the attendance and engagement during the 2022 course, but we can tell exactly for current courses. 

In 2022, we struggled to recruit enough facilitators to match the demand and quality of our participants. We sent last-minute messages to people in our network asking if they were interested, or had any recommendations. This was a significant barrier to growth.

We now recruit Teaching Fellows who facilitate 5-10 courses each and work 0.5-1 FTE. This has enabled us to scale the courses while improving discussion quality. Teaching Fellows are evaluated via a mock facilitated discussion before they’re selected, and their facilitation skills improve further as a result of specialised training we give them, and consistent practice while in the role. 

In 2022, evaluating applications was 2 weeks of work for two team members, and running the course logistics was a further 3 months full-time for one team member. We had limited time to run events, improve the learning experience, or provide tailored support for the most engaged participants.

Today, most operational tasks are automated, including evaluating applicants against a set criteria, onboarding emails, collecting people’s time availability, creating cohorts, and enabling participants to switch cohorts if their availability changes. Running a course’s logistics takes ~0.2 FTE, and we can now prioritise creating excellent experiences.

Conclusion

The 2022 AI alignment course has 123 graduates who are now working ~full-time on AI safety (37% of the cohort), up from 18 participants at the start of the course. We estimate that 20 of those would not be working on AI safety were it not for them joining the course, based on their 2024 survey responses and their 2022 career plans. We also accelerated the impact of the other 85 participants by helping them find impactful opportunities and performing better in those roles. 

Today, we're delivering better learning experiences at a greater scale, and there are far more opportunities for graduates to have an impact in the AI safety space. Therefore, we expect our current work to be even more impactful, and we’re excited to continue supporting ambitious, driven, altruistic individuals towards impactful careers.

24

0
0
2

Reactions

0
0
2

More posts like this

Comments3


Sorted by Click to highlight new comments since:

Proud to be among that 37%. Keep up the excellent work!

So many incredible achievements!

Executive summary: A 2022 AI alignment course significantly increased the number of participants working on AI safety from 5% to 37%, demonstrating its effectiveness in motivating and preparing individuals for careers in this field.

Key points:

  1. The course received 450 applications, accepted 342 participants, and cost £440 per student.
  2. 105 additional participants began working on AI safety after the course, with 20 of these likely attributable directly to course influence.
  3. Course quality has improved since 2022, with better curriculum, tracking, facilitation, and automation.
  4. Current courses are expected to have even greater impact due to improved delivery and increased opportunities in AI safety.
  5. The course accelerated the impact of 85 participants who were already interested in AI safety careers.
  6. Operational efficiency has increased, allowing more focus on creating excellent learning experiences.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

Curated and popular this week
trammell
 ·  · 25m read
 · 
Introduction When a system is made safer, its users may be willing to offset at least some of the safety improvement by using it more dangerously. A seminal example is that, according to Peltzman (1975), drivers largely compensated for improvements in car safety at the time by driving more dangerously. The phenomenon in general is therefore sometimes known as the “Peltzman Effect”, though it is more often known as “risk compensation”.[1] One domain in which risk compensation has been studied relatively carefully is NASCAR (Sobel and Nesbit, 2007; Pope and Tollison, 2010), where, apparently, the evidence for a large compensation effect is especially strong.[2] In principle, more dangerous usage can partially, fully, or more than fully offset the extent to which the system has been made safer holding usage fixed. Making a system safer thus has an ambiguous effect on the probability of an accident, after its users change their behavior. There’s no reason why risk compensation shouldn’t apply in the existential risk domain, and we arguably have examples in which it has. For example, reinforcement learning from human feedback (RLHF) makes AI more reliable, all else equal; so it may be making some AI labs comfortable releasing more capable, and so maybe more dangerous, models than they would release otherwise.[3] Yet risk compensation per se appears to have gotten relatively little formal, public attention in the existential risk community so far. There has been informal discussion of the issue: e.g. risk compensation in the AI risk domain is discussed by Guest et al. (2023), who call it “the dangerous valley problem”. There is also a cluster of papers and works in progress by Robert Trager, Allan Dafoe, Nick Emery-Xu, Mckay Jensen, and others, including these two and some not yet public but largely summarized here, exploring the issue formally in models with multiple competing firms. In a sense what they do goes well beyond this post, but as far as I’m aware none of t
LewisBollard
 ·  · 6m read
 · 
> Despite the setbacks, I'm hopeful about the technology's future ---------------------------------------- It wasn’t meant to go like this. Alternative protein startups that were once soaring are now struggling. Impact investors who were once everywhere are now absent. Banks that confidently predicted 31% annual growth (UBS) and a 2030 global market worth $88-263B (Credit Suisse) have quietly taken down their predictions. This sucks. For many founders and staff this wasn’t just a job, but a calling — an opportunity to work toward a world free of factory farming. For many investors, it wasn’t just an investment, but a bet on a better future. It’s easy to feel frustrated, disillusioned, and even hopeless. It’s also wrong. There’s still plenty of hope for alternative proteins — just on a longer timeline than the unrealistic ones that were once touted. Here are three trends I’m particularly excited about. Better products People are eating less plant-based meat for many reasons, but the simplest one may just be that they don’t like how they taste. “Taste/texture” was the top reason chosen by Brits for reducing their plant-based meat consumption in a recent survey by Bryant Research. US consumers most disliked the “consistency and texture” of plant-based foods in a survey of shoppers at retailer Kroger.  They’ve got a point. In 2018-21, every food giant, meat company, and two-person startup rushed new products to market with minimal product testing. Indeed, the meat companies’ plant-based offerings were bad enough to inspire conspiracy theories that this was a case of the car companies buying up the streetcars.  Consumers noticed. The Bryant Research survey found that two thirds of Brits agreed with the statement “some plant based meat products or brands taste much worse than others.” In a 2021 taste test, 100 consumers rated all five brands of plant-based nuggets as much worse than chicken-based nuggets on taste, texture, and “overall liking.” One silver lining
 ·  · 1m read
 ·