Effective Altruism Forum
EA Forum

Hide table of contents
Comment Permalink
aog7
5
1

One other potential suggestion: Organizers should consider focusing on their own career development rather than field-building if their timelines are shortening and they think they can have a direct impact sooner than they can have an impact through field-building. Personally I regret much of the time I spent starting an AI safety club in college because it traded off against building skills and experience in direct work. I think my impact through direct work has been significantly greater than my impact through field-building, and I should've spent more time on direct work in college. 

Reply
See in context
by Josh Thorsteinson 🔸
6 min read 8

50

AI safetyAI safety groupsAI governanceBuilding the field of AI safetyEffective altruism groupsUniversity groups
Frontpage

Advice for AI safety university group organizers.

Acknowledgements

I collected most of these ideas in early January while attending OASIS 4.0, a three-day workshop for AI safety university group organizers in Berkeley, CA. Thank you to everyone who gave their input, and a special thanks to Jeremy Kintana, Neav Topaz, Tzu Kit Chan, and Chris Tardy for your feedback on earlier versions of this writeup. The contents of this post don't necessarily reflect the opinion of anyone but myself. All mistakes are my own. 

Summary

Given short timelines, AI safety university groups should:

  • Prioritize grad students and skilled researchers who could have a meaningful impact within 2-3 years
  • Run selective upskilling programs and small, high-context gatherings
  • Protect researcher time for those doing impactful work
  • Have a succession plan with clear documentation
  • Invest in AI governance fieldbuilding and community wellbeing
  • Be cautious and coordinated in advocacy efforts

Context

It seems like everyone’s timelines have been shortening.

One of the most respected voices on AI timelines is Ajeya Cotra. In November 2023, her median estimate for when 99% of currently fully remote jobs will be automatable was 13 years. In January, she wrote that AI progress on tasks that take human experts multiple hours is almost twice as fast as she expected. On timelines, she now defers to engineers at places like METR and Redwood who have hands-on experience with frontier AI.

Daniel Kokotajlo, another highly respected voice on timelines, has had a median timeline of 2027 for years. His probability distribution:[1]

  • 19% - 2025
  • 19% - 2026
  • 12% - 2027
  • 6% - 2028
  • 6% - 2029
  • 4% - 2030
  • 2% - 2031
  • 2% - 2032
  • 2% - 2033
  • 2% - 2034
  • 2% - 2035

When 99% of currently fully remote jobs are automatable, human AI safety researchers will be largely obsolete. If humans still control them, AIs will be the ones doing alignment research, not humans. Therefore, fieldbuilding is on a deadline.

This affects how AI safety university groups should allocate resources. If timelines are indeed likely to be short, here’s what I think AI safety university groups should do.

Resource Allocation

High-Priority Activities

  1. Grad Student Outreach
    • Target grad students with relevant skills
    • Send them relevant papers
    • Invite them to your programs
  2. Prof Outreach
    • Focus on those with relevant research areas
    • Consider visiting their office hours if they don’t reply to your emails
    • Ideas for prof engagement, from least to most involved:
      • Share relevant funding opportunities
      • Invite them to present at or attend your group’s events
      • Host moderated faculty panels and follow up individually
    • If a prof or grad student wants to learn more about AI safety research, you could recommend they request a call with Arkose
  3. Upskilling Programs
    • Run selective programs for highly skilled participants
    • Steal from existing curricula (e.g., BlueDot Impact, ARENA)

Technical vs. Governance

Last year, 80,000 Hours replaced technical AI safety research with AI governance and policy as their top-ranked career path. See their reasoning here. Aside from being more neglected than technical AI safety, governance could also be more tractable given short timelines, especially for students.

Compared to established fields like physics, it’s faster to skill up in technical AI safety. Because it has less foundational research to study, it’s easier to get caught up to the cutting edge. This is even more true of AI governance — there’s still hardly any meaningful AI regulation anywhere in the world.

Don’t neglect governance. Consider:

  • Directing technically-skilled students who can't contribute to technical safety within 3 years toward governance (or technical AI governance)
  • Investing in outreach to those with skills relevant to AI governance, like policy grad students
  • Responding to relevant government Notices of Inquiry or Requests for Comment (RFCs)[2]
    • These likely go by a different name if you’re not in the US
    • Beyond potentially influencing important policy decisions, responding to RFCs is an excellent way to test your fit for policy work; policy think tanks spend lots of time on RFC responses, and their other outputs draw on similar skills
    • When writing a response, treat it as a focused research project targeting 1-3 sub-questions
    • You might find profs willing to supervise this work

Time Management

  1. If you do impactful[3] research, guard your time

    • Historically, the most impactful careers to come out of AI safety university groups have often been those of the organizers themselves
    • Therefore, don't trade full-time research to do more group organizing; prioritize your own development
    • Organize activities that complement your research (e.g., a paper reading group for papers you wanted to read anyway)
  2. If you don’t do impactful research:
    • Seriously evaluate whether you could become an impactful researcher within 2-3 years
    • If yes, consider using your group to support that transition (e.g., organizing study groups for topics you want to learn)
    • If no, make sure your work aligns with your strategic model

Succession Planning

Even with short timelines, it might be important to avoid letting your group die. Why?

  • New capable students arrive each year
  • Timelines could be longer than you think
  • It’s much easier to find a successor than for a future aspiring fieldbuilder to resurrect your group
  • You can focus on activities that don’t heavily trade off against your other priorities
  • Your group could be leveraged for future advocacy efforts (see below)
  • Your group could be important for student wellbeing (see below)

Key elements of succession:

  • Create and maintain documentation that details:
    • Role descriptions and responsibilities
    • Account passwords and assets
    • Past successes and failures
  • Have every leader write transition documents
  • Identify successors well in advance and gradually give them increasing responsibility
    • This allows you to transfer knowledge that isn't easily written down
  • Make information discoverable without relying on current leadership

Warning signs of bad succession planning:

  • Key info is siloed with individual leaders
  • You prioritize current activities at the expense of documentation
  • Interested students can’t find basic information about applying to leadership by themselves

Community Building

Recommendations

  • Prioritize small, high-context gatherings over large events
  • Build tight-knit networks of capable people

This has the benefit of attracting more capable people in the future — your average new group member is likely to be as capable as your average current group member.

Community Support and Wellbeing

Over time, students might become increasingly distressed about AI developments. Juggling academic pressure with the fear of you and everyone you love dying soon is not easy. Feeling a strong sense of urgency and powerlessness is a recipe for burnout. Combined with the sense that the people around you don’t understand your concerns, these ideas can be crushing.

As capabilities accelerate while timelines shrink, your group’s most important function could become emotional support; a healthy community is more likely to maintain the sustained focus and energy needed to be impactful. Some ideas:

  1. As a leader, model healthy behaviors
    • Find the balance between urgency and wellbeing
    • Avoid burnout; this will seriously hurt you and your productivity
    • Normalize talking about emotional impacts
    • Set clear boundaries and take visible breaks
    • Share your coping strategies
  2. Create opportunities for connection
    • Plan social events unrelated to AI
    • Build genuine friendships within the leadership team
    • Include time for personal check-ins during meetings
    • Remember that strong social bonds make hard times easier

Advocacy

A Word of Caution

  1. If you’re funded by Open Philanthropy, your group is not allowed to engage in political or lobbying activities
    • Read the terms of your agreement carefully
    • Reach out to Open Philanthropy if you’re unsure whether something you're planning is allowed
  2. AI safety is in danger of being left-coded
    • This survey found that EAs and alignment researchers identify as left-leaning and nonconservative
    • Given that most universities are also left-leaning, successful university group advocacy could unintentionally strengthen this association, hurting the chances of getting meaningful AI safety regulation from conservative policymakers
    • Yet historically, universities have played an important role in building momentum for social movements (e.g., environmentalism, civil rights)
    • Advocates should therefore frame AI safety as a broadly shared, nonpartisan issue
  3. Contact Kairos leadership before doing anything that risks damaging the AI safety community’s credibility: contact@kairos-project.org 

Potential Activities

  1. Opinion Pieces
    • Focus on well-researched, thoughtful perspectives
    • Start with university publications
    • Build toward high-impact venues
      • This seems difficult as an unknown student; one method would be to ghostwrite for a prominent professor at your school
  2. Open Letters and Statements
    • Leverage academic credibility
    • AI safety university groups have helped get professors to sign the CAIS letter

Should You Stage a Protest?

You might consider it if:

  1. A different (non-crazy) campus group is already planning to stage an AI protest
  2. There was a severe warning shot
    • E.g., a rogue AI is known to have caused a global pandemic but lacked the ability to fully overpower humanity
    • This could dramatically increase public appetite for AI regulation
  3. There is a coordinated (inter)national protest by a well-respected organization
    • This might be communicated by e.g., Kairos or the Center for AI Safety

But you should: 

  • Have a clear (set of) policy objective(s)
  • Be professional and non-violent
  • First communicate with the broader AI safety community
  1. ^

    As of early 2025 (source). I adjusted the probabilities given that 2024 is over, as Daniel says to do.

  2. ^

    Thanks to Aidan Kierans for this idea and the listed details. 

  3. ^

    Given short timelines, many areas of AI safety research are unlikely to be impactful. There are some good considerations for reevaluating your plans here.

50

3
0
1
1

Reactions

3
0
1
1

More posts like this

Comments8


Sorted by
New & upvoted
Click to highlight new comments since:
tzukitchan8
1
0
3

🤘❤️‍🔥 Glad to see you posting this Josh! I imagine it’ll be helpful for future organizers who are also thinking about this :)

Reply
ShafiraNoh2
0
0

yes, agree with thiss, this is super helpful! would want to ask more from you, Josh, via our Discord later if we have timw!

Reply
Buck8
3
0

I think you're maybe overstating how much more promising grad students are than undergrads for short-term technical impact. Historically, people without much experience in AI safety have often produced some of the best work. And it sounds like you're mostly optimizing for people who can be in a position to make big contributions within two years; I think that undergrads will often look more promising than grad students given that time window.

Reply
Josh Thorsteinson 🔸1
2
0

Interesting, thanks for the feedback. That's encouraging for AI safety groups - it's easier to involve undergrads than grad students.

Reply
aog7
5
1

One other potential suggestion: Organizers should consider focusing on their own career development rather than field-building if their timelines are shortening and they think they can have a direct impact sooner than they can have an impact through field-building. Personally I regret much of the time I spent starting an AI safety club in college because it traded off against building skills and experience in direct work. I think my impact through direct work has been significantly greater than my impact through field-building, and I should've spent more time on direct work in college. 

Reply
gergo3
0
0

Great post, thanks for sharing!

Reply
123452
0
0

Great post

Reply
Curated and popular this week
 ·  · 23m read
 · 
Or on the types of prioritization, their strengths, pitfalls, and how EA should balance them   The cause prioritization landscape in EA is changing. Prominent groups have shut down, others have been founded, and everyone is trying to figure out how to prepare for AI. This is the first in a series of posts examining the state of cause prioritization and proposing strategies for moving forward.   Executive Summary * Performing prioritization work has been one of the main tasks, and arguably achievements, of EA. * We highlight three types of prioritization: Cause Prioritization, Within-Cause (Intervention) Prioritization, and Cross-Cause (Intervention) Prioritization. * We ask how much of EA prioritization work falls in each of these categories: * Our estimates suggest that, for the organizations we investigated, the current split is 89% within-cause work, 2% cross-cause, and 9% cause prioritization. * We then explore strengths and potential pitfalls of each level: * Cause prioritization offers a big-picture view for identifying pressing problems but can fail to capture the practical nuances that often determine real-world success. * Within-cause prioritization focuses on a narrower set of interventions with deeper more specialised analysis but risks missing higher-impact alternatives elsewhere. * Cross-cause prioritization broadens the scope to find synergies and the potential for greater impact, yet demands complex assumptions and compromises on measurement. * See the Summary Table below to view the considerations. * We encourage reflection and future work on what the best ways of prioritizing are and how EA should allocate resources between the three types. * With this in mind, we outline eight cruxes that sketch what factors could favor some types over others. * We also suggest some potential next steps aimed at refining our approach to prioritization by exploring variance, value of information, tractability, and the
 ·  · 5m read
 · 
[Cross-posted from my Substack here] If you spend time with people trying to change the world, you’ll come to an interesting conundrum: Various advocacy groups reference previous successful social movements as to why their chosen strategy is the most important one. Yet, these groups often follow wildly different strategies from each other to achieve social change. So, which one of them is right? The answer is all of them and none of them. This is because many people use research and historical movements to justify their pre-existing beliefs about how social change happens. Simply, you can find a case study to fit most plausible theories of how social change happens. For example, the groups might say: * Repeated nonviolent disruption is the key to social change, citing the Freedom Riders from the civil rights Movement or Act Up! from the gay rights movement. * Technological progress is what drives improvements in the human condition if you consider the development of the contraceptive pill funded by Katharine McCormick. * Organising and base-building is how change happens, as inspired by Ella Baker, the NAACP or Cesar Chavez from the United Workers Movement. * Insider advocacy is the real secret of social movements – look no further than how influential the Leadership Conference on Civil Rights was in passing the Civil Rights Acts of 1960 & 1964. * Democratic participation is the backbone of social change – just look at how Ireland lifted a ban on abortion via a Citizen’s Assembly. * And so on… To paint this picture, we can see this in action below: Source: Just Stop Oil which focuses on…civil resistance and disruption Source: The Civic Power Fund which focuses on… local organising What do we take away from all this? In my mind, a few key things: 1. Many different approaches have worked in changing the world so we should be humble and not assume we are doing The Most Important Thing 2. The case studies we focus on are likely confirmation bias, where
 ·  · 2m read
 · 
I speak to many entrepreneurial people trying to do a large amount of good by starting a nonprofit organisation. I think this is often an error for four main reasons. 1. Scalability 2. Capital counterfactuals 3. Standards 4. Learning potential 5. Earning to give potential These arguments are most applicable to starting high-growth organisations, such as startups.[1] Scalability There is a lot of capital available for startups, and established mechanisms exist to continue raising funds if the ROI appears high. It seems extremely difficult to operate a nonprofit with a budget of more than $30M per year (e.g., with approximately 150 people), but this is not particularly unusual for for-profit organisations. Capital Counterfactuals I generally believe that value-aligned funders are spending their money reasonably well, while for-profit investors are spending theirs extremely poorly (on altruistic grounds). If you can redirect that funding towards high-altruism value work, you could potentially create a much larger delta between your use of funding and the counterfactual of someone else receiving those funds. You also won’t be reliant on constantly convincing donors to give you money, once you’re generating revenue. Standards Nonprofits have significantly weaker feedback mechanisms compared to for-profits. They are often difficult to evaluate and lack a natural kill function. Few people are going to complain that you provided bad service when it didn’t cost them anything. Most nonprofits are not very ambitious, despite having large moral ambitions. It’s challenging to find talented people willing to accept a substantial pay cut to work with you. For-profits are considerably more likely to create something that people actually want. Learning Potential Most people should be trying to put themselves in a better position to do useful work later on. People often report learning a great deal from working at high-growth companies, building interesting connection
Recent opportunities in AI safety
3
Applications Open: AI Safety India Phase 1 – Fundamentals of Safe AI (Global Cohort)
adityaraj@eanita
· · 2m read
0
0
7
80,000 Hours is hiring for an Engagement Specialist
2 authors
· · 7m read
4
4
15
AE Studio is hiring!
AE Studio
·
0
0
View more