An AI failure mode is a way in which a project to develop machine superintelligence may fail. Malignant AI failure modes are AI failure modes that result in an existential catastrophe.

Nick Bostrom categorizes malignant AI failure modes into three basic types (Bostrom 2014: 119-126): perverse instantiation, which involves the satisfaction of an AI's goals in ways contrary to the intentions of those who programmed it; infrastructure profusion, which involves the transformation of large parts of the accessible universe into infrastructure in the service of some goal that impedes the realization of humanity's longterm potential; and mind crime, which involves the mistreatment of morally relevant computational processes.

Bibliography

Bostrom, Nick (2014) Superintelligence: Paths, Dangers, Strategies, Oxford: Oxford University Press.

Malignant AI failure modes