AI safety is the study of ways to reduce risks posed by artificial intelligence.

AI safety as a career

80,000 Hours' medium-depth investigation rates technical AI safety research a "priority path"—among the most promising career opportunities the organization has identified so far.^[1]^[2]

Arguments against AI safety

AI safety and AI risk is sometimes referred to as a Pascal's Mugging ^[3], implying that the risks are tiny and that for any stated level of ignorable risk the the payoffs could be exaggerated to force it to still be a top priority. A response to this is that in a survey of 700 ML researchers, the median answer to the "the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” was 5% with, with 48% of respondents giving 10% or higher^[4]. These probabilites are too high (by at least 5 orders of magnitude) to be consider Pascalian.

Further reading

Gates, Vael (2022) Resources I send to AI researchers about AI safety, Effective Altruism Forum, June 13.

Krakovna, Victoria (2017) Introductory resources on AI safety research, Victoria Krakovna's Blog, October 19.

Ngo, Richard (2019) Disentangling arguments for the importance of AI safety, Effective Altruism Forum, January 21.

Related entries

AI alignment | AI interpretability | AI risk | cooperative AI | building the field of AI safety

Johan S Daniel

Posts 1

Posts
1