This post is based on content originally published in the book AGE OF BCI: Existential Risks, Opportunities, Pathways. The full book is available for free to download and share by anyone under a Creative Commons BY-NC-ND license. I encourage to read, explore and discuss broader research and new concepts presented there. In case of any questions, suggestions or ideas, feel free to contact me.
I. Why it's Important to Identify the Basic Risks Factors?
Solving problems at a high-level (instead of at their source) may have some beneficial effects in the short term. However, as long as we do not focus on the root mechanisms of their genesis and escalation, such efforts may have only a limited result. An approach that focuses much of the attention and human potential on high-level factors is particularly risky in the case of existential risks where technological race plays a key role. If technologies are developed by rival actors, for each new powerful technology, a competing actor may develop more powerful technology capable of dethroning the previous one. In the case of technologies carrying an existential risk, the arms spiral may sooner or later lead to the catastrophe due to mistakes arising from constant pressure, fear and haste. The losers here may not only be human beings, but the entire Earth's ecosystem and all living organisms within it.
Therefore it's essential to identify the fundamental sources of existential threat - basic risks factors. In next step, it would be necessary to focus more attention on them in such a way that effective de-escalation takes place at this core level. Such actions can have a broad strictly social dimension. However, at the same time, the technological aspect may prove to be at least equally important. Knowing the basic risks factors of existential threats, we can try, with the help of the potential of technologies (including emerging solutions that appear on the horizon), plan more aware development of them that is focused strictly on the de-escalation of critical threats. Base on the above, I will try in this article indicate the possibly fundamental, basic factors of existential risk.
II. A New Representation of Anthropogenic X-Risks - Main Categories
Some existential risks share many common attributes. Similar may be their physical properties and behaviour as well as their mechanism of emergence and escalation at the social ground. If characteristics of particular risks are similar, grouping them into specific main categories may be possible and valuable. This approach should help in the process of specifying basic escalation factors and further its perception. This is what I will do below as a first step before identifying basic existential risk factors.
First, I’d like to recall all major anthropogenic existential risks:
- Nuclear and chemical weapons
- Environmental degradation
- Misaligned Artificial Intelligence
- Misaligned Intelligence Augmentation
In the section below, I’d like to reorganize these threats into new representation - main categories of anthropogenic existential risks.
Unification of Nuclear/Chemical/Bio/Nano Risks: TFMDR
The first anthropogenic risks listed above, namely nuclear and chemical weapons, biotechnologies, and nanotechnologies, have much in common. All these risks can lead to a global disaster caused by an unfortunate accident, lack of proper safeguards, or hidden defects in the technologies. As a result, they can, unwittingly and unintentionally, become weapons aimed at humanity as a whole. Such a situation may happen, among others as a consequense of the following incidents:
- The release of life-threatening radioactive substances, chemical compounds, pathogens, or nanobes into the external environment.
- Uncontrolled, dangerous modification of improperly designed biological organisms or synthetic nanotechnology-based devices.
- Unintentional, initially unobserved flaws or defects in technologies under development or already in operation.
It’s essential to try to minimize above hazards. As especially biotechnologies and nanotechnology become more widely used and increasingly impact our lives every year it may be a growing challenge to effectively control and keep them safe.
In addition to unintentional, potentially dangerous incidents, above technologies can be strictly used as weapons and deliberately targeted against hostile entities. For thousands of years, people have been creating ever more effective technologies to fight against one another. From that perspective, the use of nuclear technology, chemistry, bioengineering, or nanotechnology as means of destruction is not novel. However, these modern technologies have much evolved from the weapons used in the past. Their scale of acting and unpredictability over time makes the key difference and therefore constitutes a separate, much more dangerous category of weapons that all existed before.
The existing term that describes some types of especially powerful weapons is “weapons of mass destruction” (commonly abbreviated as WMD). It's currently used to refer to military applications of following: nuclear, chemical, radiological and partially biological weapons. Although the concept of “mass destruction” reflects to their scale very well, the concept of a “weapon” itself, in the reality of the increasingly ambiguous, multidimensional nature of war, no longer reflects accurately the full spectrum of dangers from technology. Moreover, the term WMD, as commonly understood, doesn’t refer to the new directions of threats from, for example, modern techniques of DNA synthesis and recombination, CRISPR/Cas9 genome editing, and the use of inorganic nanotechnologies. With the increasing number of entities developing new branches of bio and nano technologies, and the intensification of non-military applications, it’s important that the public properly understands any risks from these new directions. Because of the relatively strong and consistent perception of the technologies so far understood under the “WMD” term, its use in the context of new, not only strictly military, applications implies a risk of perceiving threats selectively, according to the traditional understanding of this term. In the face of new dangers and the growing number of potentially risky uses, it’s important to perceive all of them as consistently and comprehensively as possible. In view of the above, I’d like to propose a new term in this place to allow broader perception of the threats from all the above-mentioned technologies:
Technologies Fraught with Mass Destruction Risk - TFMDR
The new term and the word “technologies” covers both the military and civilian applications. The phrase “fraught with mass destruction risk” indicates that these technologies may increase the danger of mass destruction, even if the work is by design intended only for civilian and peaceful applications. The dynamically developed technologies, not only on the military but also on the civilian ground, will determine our security in the 21st century.
Unification of AI and IA Risks
Both AI and IA are types of technologies that could be interpreted as a subset of TFMDR. However, because of their exponential, potentially unlimited abilities of information analysis, rapid adaptation, and transformation of themselves and the external environment, AI and IA represent a particularly powerful, distinct class of threats. In the publications concerning existential risks to date, one of the most alarming is the threat of AI. Since there’s also an existential risk posed by IA, I’d like to propose a new concept that includes this second important factor. Part of the public may understand the already existing term “Misaligned AI” incompletely to the spectrum of risks that may bring and perceive the misalignment problem as only related to dangerous actions of AI itself that escaped from human control and acts independently or at least misinterpreted our expectations. It’s because the term “Misaligned AI” may suggest (especially for the people unfamiliar with existential risk in detail) that it’s strictly concerns AI. However, such a view ignores another critical threat associated with using powerful synthetic intelligence by narrow groups of representatives of the human species, e.g., an influential terrorist organization or the elite of a totalitarian government. From a terminological point of view, such a threat should be perceived as a part of the IA risk because it more precisely indicates the direct entity that triggers an existential catastrophe – humans who use technology to achieve their own potentially narrow goals and values. Given the above, the full term that covers the risks from any kind of intelligent entities (AI itself or humans) that are misaligned with the values and goals of humanity should be extended to the following form:
The new definition with the “IA” abbreviation introduced here covers the full spectrum of actors. Powerful entities can be based entirely on synthetic structures (AI) or a combination of biological and synthetic structures (IA). In both cases it can be seen as new species on the Earth that is much more intelligent than homo sapiens. In the case of IA, actors can use indirect interfaces of communication between biological and synthetic intelligence (based on current perceptual-motoric channels of human body). In the perspective of current decade, indirect communication seems much more likely. However, in the perspective of the upcoming decades, we should expect a much more powerful and effective integration of biological and synthetic intelligence based on high-bandwidth BCI implants.
Final representation of anthropogenic existential risks main categories
As a result, the main categories of anthropogenic existential risks are as follows:
- Technologies Fraught with Mass Destruction Risk (TFMDR)
- Misaligned AI/IA
- Environmental Degradation
This new form is the one that will be use below for indicating basic risks factors.
III. TFMDR – Basic Risk Factors
Basic factor 1: Immanent set of physical attributes and behavioral tendencies
TFMDR are characterized by low safety even if we assume that multiple countermeasures are adapted. This is because of the correlation of the following, highly unfavorable attributes and trends:
1. Scale and irreversibility of damages even in case of a single incident:
The potential damage caused by TFMDR can cover a large area of the planet or its entire surface. What’s essential, even a single incident can lead to such situation. An example of a single global event might be a nuclear explosion of a powerful warhead, contamination of water with harmful chemical substances, escape of designed viruses into the natural environment, or nanotechnologies that can damage biological organisms.
2. Unpredictability of behavior – latent period in the environment, possible mutations and self-replication:
TFMDR can be extremely hard or even impossible to control outside the laboratory environment because of the period of temporary secrecy during development, risk of mutations, and self-replication. The latent period in the environment can lead to their large-scale propagation in ways that are difficult to detect. They may manifest a tendency to mutate themselves (bio / nano) or the organisms with which they interact (bio / nano / nuclear / chemical). Biotechnologies and nanotechnologies may also carry the potential to replicate uncontrollably, covering more and more area.
3. Highly undetectable source of origin:
TFMDR have potential of being untraceable once they enter the natural environment. In most cases, they don’t bear a signature which could clearly identify their origin and as a result may be a highly desirable type of weapon, e.g., during a war or conflict in which the parties wish to be perceived as positively as possible by the general public. Due to above, TFMDR can be a particularly tempting field of secret research and development. The potential of weapons based on advances in biotechnology and nanotechnology can carry a particularly high risk of conducting research under strict secrecy.
The characteristics above constitute the first basic risk factor. The nature of TFMDR and their critical level of security may, unfortunately, bring about the situation in which these technologies will pose an increasing threat year after year. Thoughtful use of TFMDR will become more challenging assuming increased progression in the sophistication of these technologies in the coming years. If we want to increase the public awareness of the risk from this direction, it must be related to broad countermeasures focused on improving security in both the military and civilian spheres.
To summarize: The first basic risk factor stems from the immanent, highly unfavorable nature of the technologies in this area. The lack of widespread awareness and countermeasures to develop the necessary security measures will be crucial factors in the escalation of threats from this direction in the coming years.
Basic factor 2: Social tensions and conflicts
Even the most adverse characteristics of TFMDR don’t fully determine whether the threats from this direction will increase. In addition to the above-mentioned first basic factor, there is also a second factor that originates from the social processes. The question that must be asked at this point is: Why would anyone want to use the potential of TFMDR against other people? The most common reasons are:
1. The desire to broadly improve current status is a valid reason for the use of any weapon. In this case, using the technology in question ensures the achievement of an objective aimed at improving status at the expense of others. In this situation, the motivation based on both pragmatic and emotional factors can play an important role.
2. A specific type of weapon may be used for strictly defensive purposes. Situations where one entity fears to maintain its current status may lead to actions in which broad types of weapons are used to prevent negative changes. Both pragmatic and emotional motivations can play a role in this case.
3. The intention of causing suffering or at least making the attacker feel a kind of relief or satisfaction. The suffering inflicted in this case affects the emotions of the other party, and the expected gratification is to be a changed internal psychological state: a feeling of relief from the retaliation carried out, revenge for wrongs, etc. Emotional motivation plays a key role here.
In all of the cases above, the motive for using a weapon will be the subjective perception of the surrounding world and how other entities, events, and processes are perceived and analyzed consciously and subconsciously in the mind. The internal state of mind related to the external situation at a given moment and the desire to keep the current status or to make it better lead to emotional discomfort and internal tension. This state pushes people to make specific actions that may be perceived by others as more or less desirable. The complex web of interactions between individual members of society can induce social tensions. Ultimately, it can lead to actions such as using TFMDR. It’s worth noting that social tensions can be both conscious and unconscious. Both types can lead to conflicts at a later stage. Internal discomfort of a person in the reality of complex interaction with other members of society may lead to social tensions. This situation may result in conflicts in which a particular weapon will be used. The internal state of tension of a particular person and, at a higher level, social tensions can be the effect of differences of a tangible and intangible nature:
Tangible differences – common examples:
- natural resources,
- created goods (incl. technology),
- physical body features.
Intangible differences – common examples:
- knowledge and skills,
- believed values,
It should be strongly emphasized that neither tangible nor intangible differences lead to any tensions and conflicts by definition. In a very large number of cases, differences are accepted and perceived as desirable by people. The precondition for tensions is how these differences will be interpreted by specific individuals and groups. If they cause discomfort for some people, the way is open for social tensions and potential conflicts (in which specific weapons can be used against the opposing party).
It’s also important to note that some of the tangible and intangible differences can be referred to as inequalities. However, the term “inequality” is narrower and doesn’t include all phenomena that can lead to social tensions. For example, the differences in the access to natural resources, such as drinking water in one region, or the difference in the possessions between a dictator and the average citizen of a country can be labeled “inequality”. On the other hand, with regard to differences in skin complexion or cultural affiliation, it’s hard to determine inequality in a measurable way. In this case, these can be referred to as “differences”. The term “differences” incorporates inequalities, while it also captures a broader spectrum of phenomena taking place within a society.
Lastly, it is worth to mention particularly troubling social issues taking place nowadays. They affect the lives of large groups of people – numbering in millions. All of them have a significant potential to induce social tensions on the grounds of tangible and intangible differences. Low awareness of their existence and a lack of countermeasures to de-escalate, among others, the issues below will determine the level of threats stemming from TFMDR in the coming years.
- Geopolitical rivalry between the US and China. The technological “arms race” of the superpowers for dominance.
- Shrinking or impoverishing (depending on criteria) of the middle class in some developed countries. The shift of capital, labor, or new technologies to emerging countries as well as accumulation of wealth by a relatively narrow group.
- Revision of the international order by some states seeking to profit from the absence of an unequivocal global leader, which increases the risk of regional tensions.
- The accelerating pace of change and the growing complexity of the modern world; May make it harder to understand and adapt to constantly changing environment.
- Increasing social polarization, post-truths, filter bubbles, and echo chambers.
To summarize: The second basic risk factor stems from social tensions based on both tangible and intangible differences. Such tensions can lead to conflicts in which one or more types of technology (weapons) are used. The lack of widespread awareness and countermeasures to de-escalate current social tensions will determine the growth of the threat from the direction of TFMDR.
IV. Misaligned AI/IA – Basic Risk Factors
Basic factor 1: Immanent set of physical attributes and behavioral tendencies
Both the AI and IA technologies are hard to maintain safety, even assuming a broad countermeasures. This follows from the correlation of below set of risky attributes and tendencies:
1. Scale and irreversibility of impact beyond the critical point:
The level at which AI/IA becomes sufficiently sophisticated and can act on its own may be the point beyond which there will be no turning back from the direction of its supremacy. After achieving the critical level of development, a sufficiently advanced technology may be uncontrollable and its further expansion unstoppable. The potential scale of damage caused by AI/IA actions can span the globe. It’s worth bearing in mind that it can occur as a result of both intentional or unintentional actions of their developers.
2. Unpredictability of values and goals evolved over time:
It’s difficult to predict the evolution of AI/IA values over a long period. With powerful enough analytical skills and a bit of time for “reflection”, AI/IA worldviews and values can change dramatically. Even the most altruistic and utilitarian attitude may quickly evolve to a far cry from the original one, as a result setting completely different goals and strategies for further action.
3. Exponential power that may encourages consolidation of overall potential:
AI/IA power can enable significant, growing advantages over time for those who develop a sufficiently advanced technology first. Such prospect of “benefits” can be extremely tempting, in consequence favoring their narrow, selective consolidation. This attribute may promotes the secrecy of the development and selective distribution. When the public is aware of advanced works on AI/IA, it may try to pressure the entities involved in the development to stop them. From the perspective AI/IA funders, secrecy may be important aspects, providing as comfortable as possible conditions for further development.
The above characteristics of AI/IA nature represent the first basic risk factor that can increase existential risk. The above-mentioned issues will be more threatening if we are poorly aware of their existence. We should also realize that even if we increase public awareness and expenditure for technological safeguards of AI/IA, we may still have problems with transparency and the direction of development. Nonetheless, we should, wherever possible, increase efforts to raise critical levels of security and transparency.
To summarize: The first basic risk factor of misaligned AI/IA stems from the correlation of unfavorable attributes and tendencies of these technologies. The lack of widespread awareness and countermeasures to implement the safeguards will be the key determinant of the increasing risk from this direction.
Basic factor 2: Social tensions and conflicts
To a large extent, the level of risks from the AI/IA direction is determined by the global social situation. The escalation/de-escalation mechanism in this case is same as described for TFMDR (basic factor 2). As with TFMDR, the more tensions and conflicts within society, the more often they may be resolved through force and aggression. More conflicts imply greater risk that some of them will be resolved intentionally with technologies developed as part of the arms race or unintentionally as a result of their misuse. In conflict situations, AI or IA may be used deliberately to gain an advantage over hostile entities. As with TFMDR, the threat level from the direction of AI/IA depends on the widespread awareness of this dependency and the implementation of countermeasures to de-escalate tensions.
To summarize: The second basic factor is social tensions and conflicts within society. The lack of widespread awareness and countermeasures to de-escalate social tensions will determine the level of this risk.
V. Environmental degradation – Basic Risk Factors
Basic factor 1: The limitation of space and resources which, at the present level of development of life on the Earth, must be responsibly used to maintain the balance of the environment
The Earth’s ecosystem is a limited resource. Soil, water, oxygen, minerals, and the space in which organisms live are limited. Any species’ life depends on a fragile balance of interdependent organisms that use common natural assets present on the Earth. Continuous adaptation to the surrounding environment is inherent to every biological organism. It’s especially efficient in the case of the most advanced of them. A sufficiently mature and organized species can learn to use the Earth’s resources in such a way as not to adversely affect the individual components of the complex system. Moreover, it can begin to tap into the vast resources available beyond the mother planet, such as the Solar System and more distant regions of space. For this to be possible, it must organize its immediate environment in which it claims dominant status and survives the critical transition period to a more mature stage of development. The first of the basic factors is the limited space and resources at the current level of life’s advancement on the Earth. They must be used especially carefully and responsibly now to maintain the fragile balance of the ecosystem.
To summarize: The Earth has a limited amount of wealth. Individual organisms are dependent on each other. The first basic factor stems from the limitation of space and resources, which, at the present stage of development life must be responsibly used to keep the balance of the environment.
Basic factor 2: A Model of our civilization development that is to some extent in conflict with the limited nature of the ecosystem (at the current stage of our civilization’s advancement).
The global changes over the recent decades in the Earth’s environment, including climate change, devastation of natural areas, and the mass extinction of species are commonly perceived as the most notable manifestations of the Anthropocene. This term is used to describe a new epoch in the history of our planet, when human activities have a key impact on the natural environment. Every species transforms at least its immediate space in one way or another. Also, every human being wants to improve, enhance, and change at least its closest environment in a desirable way to create the right living conditions for themselves and their descendants. However, no other species in the history of our globe has had as significant an impact on the environment as humans have today. The problem is that the current development of homo sapiens is taking place largely in a reckless manner, ultimately harming both other species and ourselves.
On the surface, it may seem that this situation is mainly the result of the fact that there is still no broad consensus on the degree of human destructive impact on the ecosystem. It’s true that some people disagree on this matter. Not everyone realizes the limited nature of the space and resources on the Earth and the need to use them wisely and share them with other species. Some people consciously or subconsciously deny the processes taking place (denial syndrome). This is largely caused by the fear of not being easily able to answer the questions that arise when our negative impact is acknowledged. However, even if our environmental impact is widely accepted, it still can be difficult to stop current trends. This is because the problem of climate change is grounded on deeper issues. Some contemporary thinkers believe that people should focus more on the pursuit of being than on the possession of goods (the “to have or to be” dilemma). A way of living based on “being” rather than “having” involves being mindfully present in the here and now, appreciating those around us and the surrounding nature, rather than pursuing material objects that can distract us from our relationship with people and nature. It might seem that this second approach is a simple answer to our questions about why we currently have such big problems with negative impacts on the Earth’s ecosystem and how we should live if we want to change this situation. However, the root of the problem lies deeper.
As humans, we want not so much an abundance of tangible possessions as a wealth of challenges, experiences, and goals. Of course, this kind of wealth can have, among other things, a material dimension, e.g., the goods we buy for our purposes or creative processes of transforming physical resources. Nonetheless, it’s also largely nonmaterial, e.g., satisfaction from self-development, tasks we’ve done, things we’ve built and the meaning we give to them. Every morning, we want a new day to be full of challenges and goals that we can strive for and that fit our aspirations and our understanding of what should be done. No sane person wants to spend their life staring at the ceiling – without goals, challenges, and experiences. This triad can be oriented toward others and the relation with nature as well as toward the world of matter. We don’t want to only “be” or to only “have”. We want to experience both in a balance that satisfies us. We want to orient ourselves toward other people and nature, but also toward creative development, transforming the surrounding matter, our world and ultimately changing our lives for the better.
Developed countries can provide a much broader spectrum of such wealth. Their large economies are the result and a further base of the human need of fulfillment. The public is afraid of ideas such as central control of economy and lockdowns. The first instinctive fear is that this can lead to a loss of control over the life existential situation of a hitherto relatively free human being. However, what scares us at least as much is the fear of the collapse of the present diversity of challenges, experiences, and goals that make our lives more meaningful. Are we capable, in the limited conditions that our civilization currently can use, of providing ourselves with a continuously satisfactory wealth of challenges, experiences, and goals, without having a negative impact on the Earth’s ecosystem?
To summarize: The current ecological crisis stems largely from the conflict between the rising aspirations of 20th and 21st century humans and the limitations of the Earth’s ecosystem. The current model of development is to some extent in conflict with the limitation of the environment we inhabit at the current stage of our civilization’s advancement. We want to continually provide ourselves with rewarding challenges, experiences, and goals within the limited space and resources that our civilization currently uses.
VI. Basic Risk Factors – Summary
In the previous sections, I outlined basic factors that have a fundamental impact on the existential risks. The table below presents them all in summary form.
|Threat type||Basic risk factors|
1. Immanent set of physical attributes and behavioral tendencies:
2. Social tensions and conflicts arise from the following differences:
1. Immanent set of physical attributes and behavioral tendencies:
2. Social tensions and conflicts arise from the following differences:
1. Limitation of space and resources which, at the present level of development of life on the Earth, must be responsibly used to maintain the balance of the natural environment.
2. Model of our civilization development that is to some extent in conflict with the limited nature of the ecosystem (at the current stage of our civilization’s advancement). We want to provide ourselves with continually rewarding challenges, experiences, and goals within the limited space and resources that we currently utilizes.
VIII. Solving problems at root level
In order to reduce existential risk permanently instead of only temporarily, it is necessary to focus more on the fundamental factors. What's important is that our actions can have a comprehensive social and technological dimension. Knowing the basic factors, we can try to evaluate many existing or upcoming social concepts and technologies in terms of their potential on permanent de-escalation of existential risks.
If we are engaged in or plan to devote our time to developing a specific sociological/economic concept or technology, we can attempt to assess its impact on basic risks factors and thus impact on general de-escalation/escalation of existential risks.
Based on the assessments, if our current or planned actions have a positive, de-escalating impact on existential risks, this will be important motivation to keep going despite the obstacles we face. On the other hand, if our activities seem to have only a neutral or, even worse, a negative, escalating impact, this can be a sign to refocus towards a more desired area. With such a strategy of acting, we can increase our chance for a fundamental, long-term de-escalation of existential risks, which will have a crucial impact on our and the next generations future.