Hide table of contents

This post was written for Convergence Analysis. Cross-posted to LessWrong.

The concept of information hazards is highly relevant to many efforts to do good in the world (particularly, but not only, from the perspective of reducing existential risks. I’m thus glad that many effective altruists and rationalists seem to know of, and refer to, this concept. However, it also seems that:

  • People referring to the concept often don’t clearly define/explain it
  • Many people (quite understandably) haven’t read Nick Bostrom’s original (34 page) paper on the subject
  • Some people misunderstand or misuse the term “information hazards” in certain ways

Thus, this post seeks to summarise, clarify, and dispel misconceptions about the concept of information hazards. It doesn’t present any new ideas of my own.


In his paper, Bostrom defines an information hazard as:

A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.

He emphasises that this concept is just about how true information can cause harm, not how false information can cause harm (which is typically a more obvious possibility).

Bostrom’s paper outlines many different types of information hazards, and gives examples of each. The first two types listed are the following:

Data hazard: Specific data, such as the genetic sequence of a lethal pathogen or a blueprint for making a thermonuclear weapon, if disseminated, create risk.

[...] Idea hazard: A general idea, if disseminated, creates a risk, even without a data-rich detailed specification.

For example, the idea of using a fission reaction to create a bomb, or the idea of culturing bacteria in a growth medium with an antibiotic gradient to evolve antibiotic resistance, may be all the guidance a suitably prepared developer requires; the details can be figured out. Sometimes the mere demonstration that something (such as a nuclear bomb) is possible provides valuable information which can increase the likelihood that some agent will successfully set out to replicate the achievement.

He further writes:

Even if the relevant ideas and data are already “known”, and published in the open literature, an increased risk may nonetheless be created by drawing attention to a particularly potent possibility.

This leads him to his third type:

Attention hazard: mere drawing of attention to some particularly potent or relevant ideas or data increases risk, even when these ideas or data are already “known”.

Because there are countless avenues for doing harm, an adversary faces a vast search task in finding out which avenue is most likely to achieve his goals. Drawing the adversary’s attention to a subset of especially potent avenues can greatly facilitate the search. For example, if we focus our concern and our discourse on the challenge of defending against viral attacks, this may signal to an adversary that viral weapons—as distinct from, say, conventional explosives or chemical weapons—constitute an especially promising domain in which to search for destructive applications.

The significance of these and other types of potential information hazards is that it will sometimes be morally best to avoid creating or spreading even true information. (Exactly when and how to attend to and reduce potential information hazards is beyond the scope of this post; I begin to explore that topic here.)

Context and scale

Those quoted examples all relate to fairly large-scale risks (perhaps even existential risks). Some also relate to risks from advancing the development of potentially dangerous technologies (i.e., from going against the principle of differential progress). It seems to me that the concept of information hazards is most often used in relation to such large-scale, existential, and/or technological risks, and indeed that that’s where the concept is most useful.

However, it’s also worth noting that information hazards don’t have to relate to these sorts of risks. Information hazards can occur in a wide variety of contexts, and can sometimes be very mundane or minor. Some of Bostrom’s types and examples highlight that. For example:

Spoilers constitute a special kind of disappointment. Many forms of entertainment depend on the marshalling of ignorance. Hide-and-seek would be less fun if there were no way to hide and no need to seek. For some, knowing the day and the hour of their death long in advance might cast shadow over their existence.

[Thus, it is also possible to have a] Spoiler hazard: Fun that depends on ignorance and suspense is at risk of being destroyed by premature disclosure of truth.

Who’s at risk?

Spoiler hazards are a type of information hazards that risk harm only to the knower of the true information themselves, and as a direct result of them knowing the information. (In contrast, in Bostrom’s earlier examples, the knower might eventually be harmed, but this would be (1) along with many other people, and (2) a result of catastrophic or existential risks that were made more likely by the knowledge the knower spread, rather than as a direct result of them having that knowledge.)

Bostrom also lists other such types of information hazards where the risk of harm is to the knower themselves, and directly results from their knowledge. But there appears to be no established term for the entire subset of information hazards that fit that description. Proposed terms I’m partial to include “knowledge hazards” and “direct information hazards”. (Further discussion can be found in that comments section. If you have thoughts on that, please comment here or there.)

But I should emphasise that that is indeed only a subset of all information hazards; information hazards can harm people other than the knower themselves, and, as mentioned above, this will be the case in the contexts where information hazards are perhaps most worrisome and worth attending to. (I emphasise this because some people misunderstand or misuse the term “information hazards” as referring only to what we might call “knowledge hazards”; this misuse is apparent here, and is discussed here.)

Information hazards are risks

As noted, an information hazard is “A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm” (emphasis added). Thus, as best I can tell:

  • Something can be an information hazard even if no harm has yet occurred, and there’s no guarantee it will ever occur.

  • E.g., writing a paper on a plausibly dangerous technology can be an information hazard even if it turns out to be safe after all.

  • But if we don’t have any specific reason to believe that there’s even a “risk” of harm from some true information, and we just think that it’d be worth bearing in mind that there might be a risk, it may be best to say there’s a “potential information hazard”.

  • So it’s probably not helpful to, e.g., slap the label of “information hazard” on all of biological research, for example.

Closing remarks

I hope you’ve found this post clear and useful. To recap:

  • The concept of information hazards relates to risks of harm from creating or spreading true information (not from creating or spreading false information).
  • The concept is definitely very useful in relation to existential risks and risks from technological development, but can also apply in a wide range of other contexts, and at much smaller scales.
  • Some information hazards risk harm only to the knower of the true information themselves, and as a direct result of them knowing the information. But many information hazards harm other people, or harm in other ways.
  • Information hazards are risks of harm, not necessarily guaranteed harms.

In my next post, I’ll attempt to similarly summarise and clarify the concept of differential progress (a generalisation of Bostrom’s differential technological development). And in a later post, Convergence will suggest some rules-of-thumb regarding why, when, and how one should deal with potential information hazards (aiming for more nuance than just “Always pursue truth!” or “Never risk it!”).

My thanks to David Kristoffersson and Justin Shovelain for feedback on this post.

Other sources relevant to this topic are listed here.

Comments3


Sorted by Click to highlight new comments since:

I did find this post clear and useful; it will be my main recommendation if I want to explain this concept to someone else.

I also really like your proposition of "potential information hazards", as at that point in the post, I was wondering if all basic research should be considered information hazards, which would make the whole concept rather vacuous. Maybe one way to address the potential information hazards is to try to quantify how removed are they from potential concrete risks?

Anyway, I'm looking forward to the next posts on dealing with these potential information hazards.

Thanks! That's great to hear.

And yes, I think that section you point at was important, and I David Kristoffersson for pushing me to attend to that distinction between actual harms, "information hazards" (where it's just a risk), and "potential information hazards" (where we don't have a specific way it would be harmful in mind, or something like that). (He didn't formulate things in that way, but highlighted the general issue as one worth thinking about more.)

Just so you know, we've now (finally!) published the post on how to deal with potential information hazards over on LessWrong.

We'll be putting most of our posts on the topic on that forum, as part of a "sequence".

Curated and popular this week
Sam Anschell
 ·  · 6m read
 · 
*Disclaimer* I am writing this post in a personal capacity; the opinions I express are my own and do not represent my employer. I think that more people and orgs (especially nonprofits) should consider negotiating the cost of sizable expenses. In my experience, there is usually nothing to lose by respectfully asking to pay less, and doing so can sometimes save thousands or tens of thousands of dollars per hour. This is because negotiating doesn’t take very much time[1], savings can persist across multiple years, and counterparties can be surprisingly generous with discounts. Here are a few examples of expenses that may be negotiable: For organizations * Software or news subscriptions * Of 35 corporate software and news providers I’ve negotiated with, 30 have been willing to provide discounts. These discounts range from 10% to 80%, with an average of around 40%. * Leases * A friend was able to negotiate a 22% reduction in the price per square foot on a corporate lease and secured a couple months of free rent. This led to >$480,000 in savings for their nonprofit. Other negotiable parameters include: * Square footage counted towards rent costs * Lease length * A tenant improvement allowance * Certain physical goods (e.g., smart TVs) * Buying in bulk can be a great lever for negotiating smaller items like covid tests, and can reduce costs by 50% or more. * Event/retreat venues (both venue price and smaller items like food and AV) * Hotel blocks * A quick email with the rates of comparable but more affordable hotel blocks can often save ~10%. * Professional service contracts with large for-profit firms (e.g., IT contracts, office internet coverage) * Insurance premiums (though I am less confident that this is negotiable) For many products and services, a nonprofit can qualify for a discount simply by providing their IRS determination letter or getting verified on platforms like TechSoup. In my experience, most vendors and companies
 ·  · 4m read
 · 
Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar. We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window. More details on our website. Why we exist We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared. Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future. Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization. This area of work feels to us like the early days of EA: we’re exploring unusual, neglected ideas, and finding research progress surprisingly tractable. And while we start out with (literally) galaxy-brained schemes, they often ground out into fairly specific and concrete ideas about what should happen next. Of course, we’re bringing principles like scope sensitivity, impartiality, etc to our thinking, and we think that these issues urgently need more morally dedicated and thoughtful people working on them. Research Research agendas We are currently pursuing the following perspectives: * Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destr
Dr Kassim
 ·  · 4m read
 · 
Hey everyone, I’ve been going through the EA Introductory Program, and I have to admit some of these ideas make sense, but others leave me with more questions than answers. I’m trying to wrap my head around certain core EA principles, and the more I think about them, the more I wonder: Am I misunderstanding, or are there blind spots in EA’s approach? I’d really love to hear what others think. Maybe you can help me clarify some of my doubts. Or maybe you share the same reservations? Let’s talk. Cause Prioritization. Does It Ignore Political and Social Reality? EA focuses on doing the most good per dollar, which makes sense in theory. But does it hold up when you apply it to real world contexts especially in countries like Uganda? Take malaria prevention. It’s a top EA cause because it’s highly cost effective $5,000 can save a life through bed nets (GiveWell, 2023). But what happens when government corruption or instability disrupts these programs? The Global Fund scandal in Uganda saw $1.6 million in malaria aid mismanaged (Global Fund Audit Report, 2016). If money isn’t reaching the people it’s meant to help, is it really the best use of resources? And what about leadership changes? Policies shift unpredictably here. A national animal welfare initiative I supported lost momentum when political priorities changed. How does EA factor in these uncertainties when prioritizing causes? It feels like EA assumes a stable world where money always achieves the intended impact. But what if that’s not the world we live in? Long termism. A Luxury When the Present Is in Crisis? I get why long termists argue that future people matter. But should we really prioritize them over people suffering today? Long termism tells us that existential risks like AI could wipe out trillions of future lives. But in Uganda, we’re losing lives now—1,500+ die from rabies annually (WHO, 2021), and 41% of children suffer from stunting due to malnutrition (UNICEF, 2022). These are preventable d
Relevant opportunities
16
Eva
· · 1m read