Still in progress as always, but this talk covers a lot of it! https://forum.effectivealtruism.org/posts/q49obZkQujkYmnFWY/vael-gates-risks-from-advanced-ai-june-2022

(Unfortunately the part about the insights isn't transcribed-- the first 20m is introduction, next ~30m is the description you want, last 10m is questions)

 

Suggestion for a project from Jonathan Yan: 

Given the Future Fund's recent announcement of the AI Worldview Prize, I think it would be a good idea if someone could create an online group of participants. Such a group can help participants share resources, study together, co-work, challenge each other's views, etc. After the AI Worldview Prize competition ends, if results are good, a global AI safety community can be built and grown from there.

("AI can have bad consequences" as a motivation for AI safety--> Yes, but AI can have bad consequences in meaningfully different ways!)

Here's some frame confusion that I see a lot, that I think leads to confused intuitions (especially when trying to reason about existential risk from advanced AI, as opposed to today's systems):

1. There's (weapons) -- tech like nuclear weapons or autonomous weapons that if used correctly involve people dying. (Tech like this exists)

2. There's (misuse) -- tech was intentioned to be anywhere from beneficial <> neutral <> seems high on offense-defense balance,  but it wasn't designed for harm.  Examples here include  social media, identification systems, language models,  surveillance systems. (Tech like this exists)

3. There's (advanced AI pursuing instrumental incentives --> causing existential risk), which is not about misuse, it's about the *system itself* being an optimizer and seeking power (humans are not the problem here, the AI itself is, once the AI is sufficiently advanced). (Tech like this does not exist)

You can say "AI is bad" for all of them, and they're all problems, but they're different problems and should be thought of separately.  (1) is a problem (autonomous weapons is the AI version of it) but is pretty independent  from (3). Technical AI safety discussion is mostly about the power-seeking agent issue (3). (2) is a problem all the time for all tech (though some tech lends itself more to this than others). They're all going to need to get solved, but at least (1) and (2) are problems humanity has any experience with (and so we have at least some structures in place to deal with them, and people are aware these are problems).

^ Yeah, endorsed! This is work in (3)-- if you've got the skills and interests,  going to work with Josh and Lucius seems like an excellent opportunity, and they've got lots of interesting projects lined up. 

I think my data has insights about 3, and not about 1 and 2! You can take a look at https://www.lesswrong.com/posts/LfHWhcfK92qh2nwku/transcripts-of-interviews-with-ai-researchers to see what 11 interviews look like; I think it'd have to be designed differently to get info on 1 or 2. 

Sounds great; thanks Sawyer! "Reaching out to BERI" was definitely listed in my planning docs for this post; if there's anything that seems obvious to communicate about, happy to take a call, otherwise I'll reach out if anything seems overlapping.

Thanks levin! I realized before I published that I hadn't gotten nearly enough governance people to review this,  and indeed was hoping I'd get help in the comment section.

I'd thus be excited to hear more. Do you have specific questions / subareas of governance that are appreciably benefited by having a background in "economics, political science, legal studies, anthropology, sociology, psychology, and history" rather than a more generic "generalist"-type background (which can include any of the previous, but doesn't depend on any of them?)

I view the core of this post as trying to push back a bit on inclusive "social scientists are useful!" framings, and instead diving into more specific instances of what kind of jobs and roles are available today that demand specific skills, or alternatively pointing out where I think background isn't actually  key and excellent generalist skills are what are sought.

"Preventing Human Extinction" at Stanford (first year undergraduate course)

Syllabus (2022)

Additional subject-specific reading lists (AI, bio, nuclear, climate) (2022)

@Pablo Could you also your longtermism list with the syllabus, and with the edit that the class is taught by Steve Luby and Paul Edwards jointly? Thanks and thanks for keeping this list :) .

Great idea, thank you Vaidehi! I'm pulling this from the Forum and will repost once I get that done.

I haven't received much feedback on this video yet, so I'm very curious to know how it's received! I'm interested in critiques and things that it does well, so I can refine future descriptions and know who to send this to.