Epistemics: (Part 2: Examples)

Certain sub-areas within biosecurity … are highly neglected. For example, the Biological Weapons Convention (BWC) of 1972, which is the main disarmament treaty of its kind and is considered to have established a near-universal norm against the use of biological weapons, has been run since 2006 by an Implementation Support Unit with only four employees and a budget smaller than the average McDonald’s.

Giving What We Can, “Improving biosecurity and pandemic preparedness” [Edit: An earlier version incorrectly attributed this passage to GiveWell. Sorry, GiveWell!].

1. Recap

This is Part 2 in my series on epistemics: practices that shape knowledge, belief and opinion within a community. In this series, I focus on areas where community epistemics could be productively improved.

Part 1 introduced the series and briefly discussed the role of funding, publication practices, expertise and deference within the effective altruist ecosystem.

Today, I want to look at the role of examples in discussions by effective altruists. Effective altruists offer many striking examples of small changes that could be made to yield large benefits for current or future people. We are invited to imagine that these diamonds in the rough can be easily mined, if only a courageous reader has the will (and funds) to help.

The problem is that diamonds in the rough are rare. Many of the examples presented by effective altruists are substantially more complex than they appear. Once these examples are unpacked, it is no longer obvious that they support the original point being made. Let’s look at two ways that examples are misused, and passed down across texts, in discussions of biological weapons and bioterrorism.

2. Enforcing the Biological Weapons Convention

One of the existential risks that effective altruists are most concerned about is the risk of biological catastrophe (biorisk). For example, Toby Ord assigns 3% probability to the claim that an intentionally engineered pandemic will lead to existential catastrophe by 2100, and a 2008 conference at the Future of Humanity Institute returned a median probability of 2% to the same claim.

Suppose we are concerned about biorisk. What might we do to mitigate it? Many texts written by effective altruists give the impression that mitigating biorisk would be deceptively simple: just give more money to the body in charge of enforcing the Biological Weapons Convention, an international treaty prohibiting the stockpiling of many types of biological weapons.

For example, when reviewing policy proposals that could offer large amounts of existential risk reduction at a minimal cost, Toby Ord says in The Precipice:

What about smaller changes—improvements to international coordination that offer a large amount of security for their cost? … there may be more obvious [changes], such as simply strengthening existing institutions related to existential risks. For example, the Biological Weapons Convention could be brought into line with the Chemical Weapons Convention: taking its budget from $1.4 million up to $80 million, granting it the power to investigate suspected breaches, and increasing its staff from a mere four people to a level more appropriate for its role.

This claim is echoed elsewhere. The 80,000 Hours problem profile on Preventing catastrophic pandemics offers strengthening the Biological Weapons Convention as one of three “clear actions we can take to reduce these [biological] risks”:

In a similar vein, when Giving What We Can argues that biosecurity is neglected, despite its relatively high level of current funding, they point to the underfunding of the Biological Weapons Convention:

Effective altruists often claim to have spotted diamonds in the rough: simple examples where a small push could promote dramatic gains. Ord lists funding the Biological Weapons Convention as one of several efforts that “offer a large amount of security for their cost”; 80,000 Hours cites it as one of several “clear actions we can take to reduce [biological] risks”; and 80,000 Hours treats work on the Biological Weapons Convention as a “highly neglected” subarea of biosecurity.

The thing about diamonds in the rough is that they tend to be rare and hard to find. When someone tells you they are the first one to find a diamond, it is worth digging around and asking whether others have found it first but judged it to be a doozy.

In fact, the weakness of the Biological Weapons Convention is extremely well-known and has survived persistent efforts at correction. The problem is not that no willing donor has found a few million dollars to pony up for a good cause. The problem is that there are strong political obstacles to enforcing the Biological Weapons Convention, and these political obstacles reveal clear downsides to increased enforcement that never seem to show up on effective altruists’ ledgers.

Here is an intentionally exhausting description of the fraught political history of the Biological Weapons Convention, quoted from an article by the scholar of international affairs Jez Littlewood. I hope that this litany may help to illustrate why strengthening the Biological Weapons Convention may be more difficult than building another McDonald’s.

The life of the convention has never been easy or simple. Five years after entry into force, it weathered a diplomatic spat between the United States (US) and the then Union of Soviet Socialist Republics (USSR) over the allegations related to the USSR’s offensive programme. Suspicions about the USSR’s activities in this realm were confirmed in the early 1990s, but remain subject to contestation in the contemporary period. In addition, concerns about the assurance of compliance in general resulted in innovative procedures that contributed to a very gradual strengthening of the procedural aspects of its implementation, albeit far from complete. However, 15 years after entry into force, the end of the Cold War (1989-1991) and the emergence of the Chemical Weapons Convention inspired States Parties and others to reimagine what a more robust BTWC would entail. This led to a decade of efforts to `strengthen the Convention’ between 1991 and 2001, which came to nothing in the summer of 2001 with the US rejection of a draft Protocol to the BTWC. A slow recovery from its `doldrums’ ensued, but a period of revitalization after 2006 proved short-lived, and in 2011, the ambitions of some actors to reinvigorate the operation of the Convention were stymied by differences related to the balance between compliance and development agendas. This produced a “modest outcome” at the Seventh Review Conference in 2011, but also signaled the reemergence of a group of recalcitrant States Parties opposed to treaty evolution that involved efforts outside a multilaterally negotiated agreement. By mid-2016, on the eve of its Eighth Review Conference, there was a perception of change, but even the limited ambitions of providing the Convention with a scientific advisory panel and a slightly larger implementation support unit of five individuals were not realized in 2016: an outcome categorized as “one of disappointment and frustration”.

Why exactly has the Biological Weapons Convention been so politically fraught? At least two types of constituencies have voiced important fears.

Many large, industrialized nations are afraid of two things. First, they are afraid of sharing information: large pharmaceutical companies worry that inspectors will reveal trade secrets, and militaries worry that bioweapons inspections will compromise sensitive information about military plans and capabilities. Second, many developed nations (prominently, the United States) are remarkably hesitant to accept that international conventions might apply to them, and fear that the treaty may be used to restrict their activities. As we have seen, this was a major cause of tension during the Cold War, because there were credible allegations that the USSR was in violation of the Biological Weapons Convention. For a good example of how these fears are voiced and used to stifle efforts at strengthening the Biological Weapons Convention, readers may want to look through US Ambassador Donald Mahley’s remarks to an Ad Hoc Group convened in 2001 to discuss why the United States could not accept strengthened verification mechanisms for treaty compliance, an incident described in the passage above.

By contrast, smaller nations voice opposite fears. They are afraid of being branded as “rogue states”, whose developing chemical or biological capabilities will be used to justify punitive inspections followed by sanctions or even outright war. Indeed, the (second) US-led invasion of Iraq was justified largely on the grounds that Iraq was stockpiling chemical and biological “weapons of mass destruction”. For example, then Secretary of State Colin Powell told the United Nations:

We know that Saddam Hussein is determined to keep his weapons of mass destruction; he’s determined to make more. Given Saddam Hussein’s history of aggression, given what we know of his grandiose plans, given what we know of his terrorist associations and given his determination to exact revenge on those who oppose him, should we take the risk that he will not some day use these weapons at a time and the place and in the manner of his choosing at a time when the world is in a much weaker position to respond? The United States will not and cannot run that risk to the American people. Leaving Saddam Hussein in possession of weapons of mass destruction for a few more months or years is not an option, not in a post-September 11th world.

The Iraqis told a different story: before the invasion, they welcomed the return of inspectors, all the while claiming that these inspections were being weaponized to create a pretext for invasion. As it turns out, they may well have been right: the United States led a bloody and largely fruitless invasion of Iraq that recovered no serious evidence of a chemical or biological weapons program.

So far, we have seen that strengthening the Biological Weapons Convention is nothing like building a McDonald’s. It is a tremendous political task that many serious political actors have tried, and failed, to accomplish since the inception of the Convention. They have failed because there are genuine costs to strengthening the Convention, and these costs are far greater than the cost of building a McDonald’s.

This does not mean that it is not worth strengthening the Biological Weapons Convention. But it does mean that the Biological Weapons Convention should not be presented as a `diamond in the rough’ that can be quickly fixed. Discussions of the Biological Weapons Convention should honestly acknowledge the difficult political situation surrounding the Convention, as well as the real harms that states are likely to suffer if they agree, somehow, to strengthen enforcement.

By contrast, existing discussions by effective altruists rarely mention these facts. Neither the 80,000 Hours problem profile nor the Giving What We Can analysis quoted above says anything about the political difficulties surrounding the Biological Weapons Convention. Likewise, although Ord mentions two obstacles to strengthening the Biological Weapons Convention, this discussion says very little about politics and generally reinforces the impression of a diamond in the rough. Ord writes in The Precipice:

The most famous international protection comes from the Biological Weapons Convention (BWC) of 1972. This is an important symbol of the international taboo against these weapons and it provides an ongoing international forum for discussion of the threat. But it would be a mistake to think it has successfully outlawed bioweapons. There are two key challenges that limit its ability to fulfill this mission. First, it is profoundly underfunded. This global convention to protect humanity has just four employees, and a smaller budget than an average McDonald’s. Second, unlike other arms control treaties (such as those for nuclear or chemical weapons) there is no effective means of verification of compliance with the BWC. This is not just a theoretical issue. The vast Soviet bioweapons programme, with its deadly anthrax and smallpox accidents, continued for almost twenty years after the Soviets had signed the BWC, proving that the convention did not end bioweapons research. And the Soviets were not the only party in breach. After the end of apartheid, South Africa confessed to having run a bioweapons programme in violation of the BWC. After the first Gulf War, Iraq was caught in breach of the convention. At the time of writing, the United States has said it believes several nations are currently developing bioweapons in breach of the BWC. Israel has refused to even sign. And the BWC offers little protection from non-state actors.

Here there is barely a hint of the elephant in the room: the vast political hurdles to be cleared before the Biological Weapons Convention can be strengthened. Nor will we be told about these political difficulties when Ord offers strengthening the Biological Weapons Convention as one of his key policy recommendations. Strikingly, in the passage above Ord does run across one of the key hurdles to enforcement: that “Iraq was caught in breach of the convention”, a fact which Ord neglects to mention was instrumental in one of the most destructive wars in recent memory.

Again, none of this is meant to suggest that it is unimportant to work to strengthen the Biological Weapons Convention. But readers need to be told what they are getting involved in. When the full political story is told, the estimated cost of success should go up, the estimated probability of success should go down, and regrettable geopolitical consequences of success should be fully accounted for.

3. Omnicidal bioterrorism: The case of Aum Shinrikyo

In the previous section, we looked at an example where the tractability of preventing biorisks was overstated by inflating the ease and deflating the consequences of increasing funding allocated to support the Biological Weapons Convention. In this section, I want to look at an example where biorisks appear to have been inflated.

I think there are many reasons to believe that estimates of biorisk are inflated. I hope to address some of the strongest objections in my series Exaggerating the risks. In the meantime, let’s ask a deceptively simple question: how likely is it that a serious group of people with the technology, funding, and organization to pull off an extinction-level pandemic would actually have the desire to kill us all?

It might seem easy to wheel out examples of omnicidal bioterrorists. But oddly enough, effective altruists always seem to wheel out the same example. Aum Shinrikyo is a Japanese doomsday cult that carried out a series of sarin gas attacks in the 1990s, based on a belief in the need to bring about a cleansing Armageddon in which non-believers would be killed.

Many effective altruists, including several who should (and probably do) know better, suggest that Aum Shinrikyo was omnicidal, aiming to destroy all living humans. Here are Piers Millett and Andrew Snyder-Beattie, writing in one of the leading effective altruist arguments for taking biorisk seriously, which we discussed in Part 7 of the series Exaggerating the Risks:

The Aum Shinrikyo cult in Japan sought biological weapons for the express purpose of causing extinction.

Likewise, here is Rob Reid on the Sam Harris podcast (discussed here on the EA forum):

Some groups do have bizarre urges to sweep the Earth of humanity. There are plenty of doomsday cults out there, and at some point, one of them will get bored and decide to speed things along. Japan’s Aum Shinrikyo cult did this. It gathered over a thousand members, including several biologists, and it meant to bring about the end of the world, but the tools to do that just weren’t around in 1995. So it made its big move with the sarin gas attack in the Tokyo subway. When the next Aum Shinrikyo comes along, I doubt they’ll limit their arsenals to deadly gases.

Readers might complain that Reid, unlike Millett and Snyder-Beattie, is not a central member of the effective altruist community. Fair enough. But here is Toby Ord in The Precipice:

People with the motivation to wreak global destruction are mercifully rare. But they exist. Perhaps the best example is the Aum Shinrikyo cult in Japan, active between 1984 and 1995, which sought to bring about the destruction of humanity. They attracted several thousand members, including people with advanced skills in chemistry and biology. And they demonstrated that it was not mere misanthropic ideation. They launched multiple lethal attacks using VX gas and sarin gas, killing 22 people and injuring thousands. They attempted to weaponise anthrax, but did not succeed. What happens when the circle of people able to create a global pandemic becomes wide enough to include members of such a group? Or members of a terrorist organisation or rogue state that could try to build an omnicidal weapon for the purposes of extortion or deterrence?

Here, as in Millett and Snyder-Beattie’s paper, we have the direct suggestion that Aum Shinrikyo aimed to destroy humanity, together with an admission that it would be quite difficult to replace the example of Aum Shinrikyo (hence the repeated reliance of different authors on this example).

The problem is that it is simply not true to say that Aum Shinrikyo wanted to bring about human extinction. They wanted to bring about the extermination of nonbelievers: the followers of Aum were very much meant to survive. In a book-length study of the Aum Shinrikyo cult, the psychiatrist Robert Jay Lifton writes of the group’s founder and guru (Shoko Asahara)

As he saw it, Armageddon, the final global conflagration, would be survived only by himself and a small band of followers, who would then create the world’s “religious future.”

Aum Shinrikyo did not merely intend to survive the apocalypse. They intended to rebuild their own communities, then all of Japan, and then the entire world. Here again is Lifton:

Asahara did offer his Shambhala plan, in which a proliferation of Aum communes, or “Lotus Villages,” would ensure the peaceful salvation of Japan (and eventually humankind)

This was not mere empty rhetoric. For example, Aum Shinrikyo attracted a large number of doctors, who took themselves to be responsible for ensuring that the movement would have the medical capability to survive a biological attack. Lifton writes:

A leading Aum physician declared during a radio dialogue with Asahara that, although World War III would destroy the global industrial and medical infrastructure, “Aum Shinrikyo would not have a problem” because it would establish its own industrial infrastructure and preserve its medical knowledge and supplies, rendering it “capable of rebuilding the medical infrastructure itself from the ground up.

In short, far from aiming to bring about human extinction, Aum Shinrikyo meant to survive and rebuild human civilization. This is important for at least two reasons.

First, effective altruists have correctly stressed that there is a vast normative gulf between events which kill nearly all living humans and events which kill all living humans. Events of the first variety may well lead to survival and rebuilding, and indeed we have seen that Aum Shinrikyo planned to do just that. By contrast, extinction-level events preclude the future regrowth of human civilization. And this is no accident. While the world is replete with groups that want to kill most of humanity, it is extremely rare to find sophisticated groups with the desire to kill all of humanity.

Second, it is vastly more difficult to engineer a pandemic that kills everyone outside of a certain group than to engineer a pandemic that kills everyone. While I hope to argue elsewhere that it is already quite difficult to engineer an extinction-level pandemic, it is worth pausing to think about what a dedicated group could do to kill everyone else, while surviving themselves. Could they retreat to safety within a bunker or on a small island? Innoculate themselves with a vaccine? The problem with these suggestions is that most of these survival strategies would be presumably available to others. It is difficult to describe a credible route to protecting a single group from an extinction-level pandemic that would not, by its ready availability, prevent the pandemic from killing everyone else.

So far, we have seen that many leading discussions of biorisk rely on a single historical example to illustrate the plausibility of a sophisticated group of omnicidal bioterrorists: Aum Shinrikyo. We have seen that, contrary to the direct claims of leading authors, Aum Shinrikyo was not omnicidal. And we have seen two reasons why this matters: near-extinction is far better than extinction, and also more difficult to bring about in a targeted way.

I want to close by expressing my disappointment at the claims quoted earlier from some leading authors. Millett and Snyder-Beattie said directly that Aum Shinrikyo “sought biological weapons for the express purpose of causing extinction.” And Toby Ord said that Aum Shinrikyo “sought to bring about the destruction of humanity.” We have seen that these claims are demonstrably untrue in ways that matter a great deal to the authors’ arguments.

Crucially, Ord, Millett and Snyder-Beattie are in a position to know that these claims are untrue. When experts in a position to know better echo the same false claims across many papers, one gets the impression that examples are being passed along and repeated by rote rather than checked for accuracy, even when they drive key claims in leading texts. Responsible research and communication (or as effective altruists would have it, good epistemics) demands better.

4. Taking stock

In this post, we looked at how examples are wielded by effective altruists to suggest that threats are imminent, or cost-effective interventions are just around the corner. We focused on two cases: calls to strengthen the Biological Weapons Convention, and the example of Aum Shinrikyo as a group of omnicidal bioterrorists.

In each case, we saw that the same example is misused by a variety of authors. The examples are misused by leaving out relevant context which, when properly incorporated, substantially complicates or challenges the original claim.

This gives us reason to take examples used by effective altruists with a grain of salt. It is all too easy to present readers with diamonds in the rough, or frightening historical actors, without providing readers with sufficient context to judge the quality of the diamonds or the intentions of historical actors. Readers should take pains to do their own research to verify that examples are being used responsibly. Likewise, authors should take care not to misuse examples by leaving out relevant information or exaggerating the intentions of historical actors.

I suspect that these are not the only cases in which examples have been systematically misused. Are there other key examples in the effective altruist tradition which may be more complicated than they appear? Alternatively, is there more to the story than my own discussion of these two examples suggests? Let me know in the comments below.