Hide table of contents

We must begin our serious discussion of fatal risks from the basic premise that we may need to change our strategies.

The Misstep Named Balance

Some people cannot escape the concept of balance when thinking about risks.

Balance refers to a rational strategy when there's a loss on one side and a benefit on the other.

It aims to balance both sides by giving a slight advantage to the benefits. If that balance can be maintained over time, even if losses occur occasionally, it indicates that overall gain is being achieved.

This model cannot handle fatal risks. Balance should only be considered a rational strategy if even a partial failure allows for an overall benefit. Walking a dangerous path with a cliff on the right side, in the name of maintaining balance, is not a rational strategy. It's reckless and by no means rational.

The term balance becomes a misstep when it comes to fatal risks. The wise do not approach fatal risks.

The Misstep Named Risk Management

Some people cannot escape the concept of risk management when thinking about risks.

Risk management refers to a rational strategy that involves managing the likelihood and impact of risks to minimize them while maximizing benefits.

If you manage to minimize risks and maximize benefits, you can eventually gain significant benefits. This way of thinking assumes that even if the minimized risk materializes, it's acceptable as long as the expected value favors benefits.

This model also cannot handle fatal risks. When risks materialize, no matter how much profit is made, it will always end up as a loss. Even if the risk probability is low, it's challenging to quantify the impact of fatal risks. Therefore, we cannot calculate the expected value. Thus, dealing with fatal risks through risk management is not a rational strategy. It's arrogant and by no means rational.

The term risk management becomes a misstep when it comes to fatal risks. The wise do not approach fatal risks.

The Misstep Named Risk Minimization

Some people cannot escape the concept of risk minimization when thinking about risks.

The idea is that it's good if risks can be minimized. However, this way of thinking lacks the concept of reducing risks to zero.

Fatal risks demand that risk zero is the baseline. If you cannot reduce fatal risks to zero, it should not be because you desire benefits. That would lead to the misstep named risk management. The only cases in which fatal risks cannot be reduced to zero are when other fatal risks increase. Only when suppressing other fatal risks does taking on fatal risks become rational.

By excluding benefits not involving fatal risks and considering other fatal risks, minimizing overall fatal risks becomes a rational strategy.

The term risk minimization becomes a misstep when faced with a single fatal risk. The wise only consider risk minimization when facing multiple fatal risks.

The Gap Between Impossible and Difficult

Sometimes people say that achieving risk zero is difficult. Indeed, it is difficult. But if we look at that fatal risk alone, it's not impossible. If we don't take the risk, the risk of the adventure is zero.

On the other hand, some people suggest that we should strive to keep fatal risks from materializing by controlling them. However, it's almost never guaranteed that we can control everything completely. Overlooking risk modes, human errors, and the emergence of unexpected new risks make it nearly impossible to handle everything flawlessly in most cases.

We need to focus on the gap between the words difficult and impossible. They are entirely different. Difficulties can be overcome. But what is impossible cannot be made possible. This difference is vast. Exceedingly so.

Therefore, while the difficulty of avoiding risk and the impossibility of full risk control may seem similar in wording, it's crucial to understand that there's a decisive break between them in reality.

The Dichotomy between Logical Impossibility and Empirical Impossibility

There are two types of 'impossible': logically impossible and empirically impossible.

In the case of logical impossibility, it is absolutely never possible.

However, empirical impossibility might be logically possible. The probability is not zero. In other words, it can be rephrased as 'difficult'.

The dichotomy between these two types of impossibility is larger than the impression of the word itself. Therefore, when faced with the word 'impossible', it's essential to clearly distinguish between these two.

The Dual Difficulty in Dealing with Existential Risks

When it's difficult to both cease the adventure to make the existential risk zero and also take measures while adventuring to make the existential risk zero, the best strategy becomes to exert effort on both fronts.

In particular, the effort to stop the adventure is important. Stopping the adventure is essentially a one-time difficulty. On the other hand, taking measures while continuing the adventure is an endlessly continuing effort, where the risk is always required to be kept at zero. Clearly, the former is a more rational risk measure.

Judgment by Values, Decision by Resolve, or a Strategy of Ignoring

On the other hand, we are frequently exposed to fatal risks with very low probabilities in our daily lives. We don't always act based solely on risk rationality.

When we cross the street, get in a car, train, airplane, or boat, enjoy sports or outdoor activities, or enjoy alcohol and delicious food. When we go to unknown places or intentionally ignore the doctor's advice. We expose ourselves to fatal risks with minute probabilities.

This is different from risk balance or risk management. It's likely one of the following three:

Judgment by Values

Even if our lifespan might be shortened a bit, enjoying alcohol and delicious food is part of life. Sports, outdoor activities, and visiting unknown places enrich our lives. What's the point of living long by giving up enjoyment? We take risks with such a value perspective on life and enjoyment.

Decision by Resolve
Getting on an airplane can be a little nerve-wracking and scary. However, if something happens, so be it. Although it might be small, we take risks based on such resolve.

A Strategy of Ignoring
Living while considering all risks is difficult. Humans are not made to live while paying attention to all worries. Risks with minute probabilities associated with things everyone does as a matter of course, or things repeated many times in the past, have become invisible. We expose ourselves to fatal risks with such a strategy.

Ethics in Fatal Risks

The above is the basic literacy we should understand beforehand when thinking about fatal risks.

With this literacy as a prerequisite, we can finally have a sincere discussion about risks. Especially in the case of fatal risks, if we don't base our discussions on this literacy, we will not be able to communicate effectively.

Bringing up balance, risk management, and risk minimization in discussions as if they were rational strategies, while having this literacy, is neither wise nor conscientious. Also, discussing difficulties and impossibilities as if they were on the same level is neither rational nor responsible.

If an adventure is absolutely necessary, we need to ask those affected by the fatal risks for judgments based on their values and decisions based on their resolve. Of course, based on the concept of informed consent, it's a prerequisite to carefully and clearly explain what we know and what we don't know about the risks.

Neglecting this, venturing on our own judgment, or pretending to be responsibly handling risks with expressions like balance, risk management, and risk minimization, is undoubtedly an act against ethics.

The former violates respect for human rights, while the latter violates the duty of care of a good administrator.
 

1

0
0

Reactions

0
0

More posts like this

Comments


No comments on this post yet.
Be the first to respond.
More from Aino
Curated and popular this week
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Dr Kassim
 ·  · 4m read
 · 
Hey everyone, I’ve been going through the EA Introductory Program, and I have to admit some of these ideas make sense, but others leave me with more questions than answers. I’m trying to wrap my head around certain core EA principles, and the more I think about them, the more I wonder: Am I misunderstanding, or are there blind spots in EA’s approach? I’d really love to hear what others think. Maybe you can help me clarify some of my doubts. Or maybe you share the same reservations? Let’s talk. Cause Prioritization. Does It Ignore Political and Social Reality? EA focuses on doing the most good per dollar, which makes sense in theory. But does it hold up when you apply it to real world contexts especially in countries like Uganda? Take malaria prevention. It’s a top EA cause because it’s highly cost effective $5,000 can save a life through bed nets (GiveWell, 2023). But what happens when government corruption or instability disrupts these programs? The Global Fund scandal in Uganda saw $1.6 million in malaria aid mismanaged (Global Fund Audit Report, 2016). If money isn’t reaching the people it’s meant to help, is it really the best use of resources? And what about leadership changes? Policies shift unpredictably here. A national animal welfare initiative I supported lost momentum when political priorities changed. How does EA factor in these uncertainties when prioritizing causes? It feels like EA assumes a stable world where money always achieves the intended impact. But what if that’s not the world we live in? Long termism. A Luxury When the Present Is in Crisis? I get why long termists argue that future people matter. But should we really prioritize them over people suffering today? Long termism tells us that existential risks like AI could wipe out trillions of future lives. But in Uganda, we’re losing lives now—1,500+ die from rabies annually (WHO, 2021), and 41% of children suffer from stunting due to malnutrition (UNICEF, 2022). These are preventable d
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f