By: Daniel del Castillo, Chris Leong, and Kat Woods
We made a spreadsheet of resources for learning about AI safety. It was for internal purposes here at Nonlinear, but we thought it might be helpful to those interested in becoming safety researchers.
Please let us know if you notice anything that we’re missing or that we need to update by commenting below. We’ll update the sheet in response to comments.
Highlights
There are a lot of courses and reading lists out there. If you’re new to the field, out of the ones we investigated, we recommend Richard Ngo’s curriculum of the AGI safety fundamentals program. It is a good mix of shorter, more structured, and more broad than most alternatives. You can register interest for their program when the next round starts or simply read through the reading list on your own.
We’d also like to highlight that there is a remote AI safety reading group that might be worth looking into if you’re feeling isolated during the pandemic.
About us: Nonlinear is a new AI alignment organization founded by Kat Woods and Emerson Spartz. We are a means-neutral organization, so are open to a wide variety of interventions that reduce existential and suffering risks. Our current top two research priorities are multipliers for existing talent and prizes for technical problems.
PS - Our autumn Research Analyst Internship is open for applications. Deadline is September 7th, midnight EDT. The application should take around ten minutes if your CV is already written.
Nice initiative, thanks!
Plugging my own list of resources (last updated April 2020, next update before the end of the year).
These aren't entirely about AI, but Brian Tomasik's Essays on Reducing Suffering and Tobias Baumann's articles on S-risks are also worth reading. They contain a lot of articles related to futurism and scenarios that could result in astronomical suffering. On the topic of AI alignment, Tomasik wrote this article on the risks of a "near miss" in AI alignment, and how a slightly misaligned AI may create far more suffering than a completely unaligned AI.
Haven't checked out your spreadsheet, but I do think these sorts of collections are good things to create! And on that note, I'll mention my Collection of AI governance reading lists, syllabi, etc. (so that's for AI governance, not technical AI safety stuff). I suggest people who want to read it read the doc version, but I'll also copy the full contents into this comment for convenience.
What is this doc, and why did I make it?
AI governance is a large, complex, important area that intersects with a vast array of other fields. Unfortunately, it’s only fairly recently that this area started receiving substantial attention, especially from specialists with a focus on existential risks and/or the long-term future. And as far as I’m aware there aren’t yet any canonical, high-quality textbooks or online courses on the topic.[1] It seems to me that this means this is an area where well-curated and well-structured reading lists, syllabi, or similar can be especially useful, helping to fill the role that textbooks otherwise could.[2]
Fortunately, when I started looking for relevant reading lists and syllabi, I was surprised by how many there were. So I decided to try to collect them all in one place. I also tried to put them in very roughly descending order of how useful I’d guess they’d be to a randomly chosen EA-aligned person interested in learning about AI governance.
I think this might help myself, my colleagues, and others who are trying to “get up to speed”, for the reasons given in the following footnote.[3]
I might later turn this doc into a proper post on the EA Forum.
See also EA syllabi and teaching materials and Courses on longtermism.
How can you help
The actual collection
My thanks to everyone who made these lists.
Footnotes
[1] Though there are various presumably high-quality textbooks or courses with some relevance, some high-quality non-textbook books on the topic, some in-person courses that might be high-quality (I haven’t participated in them), and some things that fill somewhat similar roles (like EA seminar series, reading groups, or fellowships).
[2] See also Research Debt and Suggestion: EAs should post more summaries and collections.
[3]
US standards institute NIST also produces recommendations: https://www.nist.gov/itl/ai-risk-management-framework
Here is the curriculum of the ML4Good, an AGI safety camp organized by EffiSciences to tprosaic alignment researchers.
The program contains many programming exercises
I think the Introduction to ML Safety course would be a good addition!
You can add new ones here, I would but you probably have a clearer idea of what a good summary would be.
Oh, thanks, missed that form in the sheet. Might be worth updating this forum post with the form because it currently says: