OR The Tragedy of the Einstein Letter and the Gaither Report; Cautionary Lessons from the Manhattan Project and the ‘Missile Gap’; Beware Assuming You’re in an AI Race; The illusory Atomic Gap, the illusory Missile Gap and the AGI Gap

Summary

In both the 1940s and 1950s, well-meaning and good people – the brightest of their generation – were convinced they were in an existential race with an expansionary, totalitarian regime. Because of this belief, they advocated for and participated in a ‘sprint’ race: the Manhattan Project to develop a US atomic bomb (1939-1945); and the ‘missile gap’ project to build up a US ICBM capability (1957-1962). These were both based on a mistake, however - the Nazis decided against a Manhattan Project in 1942, and the Soviets decided against an ICBM build-up in 1958. The main consequence of both was to unilaterally speed up dangerous developments and increase existential risk. Key participants, such as Albert Einstein and Daniel Ellsberg, described their involvement as the greatest mistake of their life.

Our current situation with AGI shares certain striking similarities and certain lessons suggest themselves: make sure you’re actually in a race (information on whether you are is very valuable), be careful when secrecy is emphasised, and don’t give up your power as an expert too easily.

I briefly cover the two case studies, discuss the atmosphere at RAND, then draw the comparison with AGI and explain my three takeaways. This short piece is mainly based on Richard Rhodes’ The Making of the Atomic Bomb and Daniel Ellsberg’s The Doomsday Machine. It was inspired by a Slack discussion with Di Cooke.

The ‘atomic gap’, the Einstein-Szilárd Letter, and the Manhattan Project

Here is a rough timeline of some key events around the Manhattan Project:

• 12 September 1933: Szilárd conceives the idea of a nuclear chain reaction, and keeps it a secret for the next six years.
• 2 August 1939: Einstein-Szilárd letter to Roosevelt advocates for setting up a Manhattan Project.
• 1 September 1939: Nazi invasion of Poland.
• 9 October 1941: Roosevelt approves the atomic program, subsequently Manhattan Project receives serious funding (eventually, 0.4% of GDP).
• June 1942: Hitler decides against an atomic program for practical reasons.
• December 1942: First chain reaction in Chicago. Szilard notes “I shook hands with Fermi and I said I thought this day would go down as a black day in the history of mankind.”
• 30 April 1945: Hitler kills himself.
• 7 May 1945: Nazi surrender.
• July 1945: Szilárd petition (signed by 70 scientists) calls for the bomb to be used only after Japan had refused to surrender, and for the decision to be made by Truman personally – and reiterated the original intention was to defend against the Nazis.
• 6 and 9 August 1945: The USA bombs Hiroshima and Nagasaki.
• 29 August 1949: First successful Soviet nuclear test.

The Manhattan Project (and other US projects like the Apollo program) was a ‘sprint’ project, in which the peak year funding reached 0.4% of GDP (Stine, 2009, see also Grace, 2015).

Why did nuclear scientists like Szilárd, who kept the chain reaction secret and opposed nuclear weapons for decades after the war, advocate for and participate in the Manhattan Project? In Ellsberg’s words:

“How could he? The answer is he believed, even before others, that they were racing Hitler to the attainment of this power. It was German scientists, after all, who had first accomplished the fission of a heavy element. There seemed no reason to suppose that Germany could not stay ahead of any competitors in harnessing this unearthly energy to Hitler’s unlimited ambitions for conquest. The specter of a possible German monopoly, even a temporary one, on an atomic bomb drove the Manhattan Project scientists – above all the Jewish emigres from Europe […] until the day of Germany’s surrender.” (p.28)

For comparison with the ‘missile gap', we could describe this as a perception of, or fear of, an ‘atomic gap’.

However, this was based on a mistake. During World War 2, the USA and UK were not in a desperate race with a powerful, totalitarian opponent. Neither the Nazis, the Japanese Empire nor the USSR had a serious nuclear program during the war. The Nazis considered a nuclear sprint, but they decided against it for three reasons. First, Speer and the nuclear physicists thought it would take three to four years to deliver - too late to make a difference to the war. Second, the Nazis were severely constrained in terms of raw materials and manpower, which were needed elsewhere in armaments production (Tooze, 2006). Third, Werner Heisenberg, principal scientist on the ‘Uranverein’ nuclear weapons program, was not able to guarantee that fission would not ignite the atmosphere. The Manhattan Project did not need to, and did not in practice, deter Hitler from using a nuclear weapon.

So the crucial effect of the advocacy and participation of the nuclear scientists in the Manhattan Project was to bring forward in time the advent of nuclear weapons. This is because it is reasonable to assume that the USA would not have ‘sprinted’ if many scientists did not advocate and volunteer for it. Indeed, it is plausible that the advent was brought forward by perhaps a decade. It is unclear whether the USA would have ‘sprinted’ to the same extent (or at all) outside of the context of WW2. As a reminder, the USA spent 0.4% of GDP on the Project. This would have been harder to justify after WW2. The Soviets were only able to catch up in four years after the war due to immense espionage from the Manhattan Project.

In addition to timing, one can also speculate about the manner in which nuclear weapons were introduced to the world. The signers of the Szilárd petition were concerned that if used on in ‘anger’, it would launch an arms race:

“If after this war a situation is allowed to develop in the world which permits rival powers to be  in uncontrolled possession of these new means of destruction, the cities of the United States as well as the cities of other nations will be in continuous danger of sudden annihilation”

If there was not a perception that the use of nuclear weapons had ‘ended the war’ in Asia, then the intensity of the nuclear arms race may have been lessened. The really wrenching “what-if” conjecture is if the advent of nuclear weapons had been delayed until the mid 1950s. Early Cold War proposals for international control of nuclear weapons (or limits on their development, stockpiling and use) failed due to US lack of interest (Zaidi & Dafoe, 2021), but also Stalin’s paranoia and distrust (Gaddis, 1997). What if the nuclear bomb was not developed until after Stalin’s death on the 5 March 1953? The prospects for international controls on the development, stockpiling and use of nuclear weapons may have been much improved.

The ‘missile gap’, the Gaither Report, and RAND

Here is a rough timeline of some key events around the missile gap:

• 26 August 1957: USSR’s first successful ICBM test.
• 4 October 1957: Sputnik launch.
• 3 November 1957: Laika launch.
• 7 November 1957: Gaither Report claims a ‘missile gap’.
• 11 June 1957, 6 December 1957: Failed US tests.
• 28 November 1958: USA’s first successful ICBM test.
• Summer 1958: DARPA and NASA established; National Defense Education Act passed.
• 7 June 1961: New National Intelligence Estimate (NIE) released internally (above Top Secret classification) – only 4 ICBMs had been observed.
• 21 October 1961: Gilpatric speech (influenced by Ellsberg) signals to Soviets that the USA knew that there was no ‘missile gap’.
• 30 October 1961: Tsar Bomba, most powerful nuclear test ever.
• 16 October 1962: Cuban Missile Crisis. Arguably the closest the world has ever come to nuclear war. Later, Kennedy says the odds of war were between “1/3 and even”.

The perception of a missile gap prompted a US ‘sprint’ project to develop and stockpile ICBMs and develop nuclear war plans.

Why did scientists like Ellsberg, who as a schoolboy in 1944 wrote an essay against nuclear weapons and who would spend the rest of his career as a famed whistleblower opposing nuclear weapons, advocate for and participate in this sprint? In his words:

“In the late fifties, I was given what seemed good reason to believe – on the basis of highly classified official information – that we were again in a desperate race with a powerful, totalitarian opponent […] This apprehension was based on illusion.” (p. 29)

Ellsberg notes that summer 1958 was the “high point of secret intelligence predications of an imminent vast Soviet superiority in deployed ICBMs, the ‘missile gap.’” (p. 34). The Air Force & CIA estimated that the USSR would have an ICBM fleet of “several hundred, perhaps as early as 1959 (with a crash effort), almost certainly by 1960-61, with thousands in the sixties” (p.35). Crucially, if there were a missile gap, the US would have been vulnerable to a first strike - Soviet ICBMs could have destroyed most of the US bomber fleet, preventing a US retaliatory strike. Just like Hitler would have had he had a nuclear monopoly, this advantage would either be a strong incentive for the USSR to launch a nuclear war, or be used as nuclear blackmail to force the USA to accept Soviet expansion.

However, this was based on a mistake, as would be established in National Intelligence Estimate NIE 11-8-61 THE SOVIET ICBM PROGRAM—EVIDENCE AND ANALYSIS. The estimate was that “the Soviets had exactly four ICBMS, soft, liquid-fuelled missiles at one site, Plesetsk. Currently we had about forty operational Atlas and Titan ICBMs […] the numbers were ten to one in our favour” (p.164).

The new estimate “totally contradicted the fundamental basis for [Ellsberg’s] concerns and work for the past several years.

It wasn’t just a matter of numbers, though that alone invalidated virtually all the classified analyses and studies I’d read and participated in for years. Since it seemed clear that the Soviets could have produced and deployed many, many more missiles in the three years since their first ICBM test, it put in question – it virtually demolished – the fundamental premise that the Soviets were pursuing a program of world conquest like Hitler’s.” (p.162)

“The 1959-62 period was their only opportunity to have such a disarming capability with missiles, either for blackmail purposes or an actual attack. […] Four missiles in 1960-61 was strategically equivalent to zero, in terms of such an aim. […]

Khrushchev had been totally bluffing about his missile production rates. He had said he was turning them out “like sausages”. […] about ICBMs it was a flagrant lie. Moreover, it meant that he had consciously forsworn the crash effort needed to give him a credible first-strike capability in the only period when that might have been feasible.

Our assumptions about his aims […] were now entirely in question.” (p. 162-3)

So the crucial effect of the advocacy and participation of the experts at RAND and elsewhere was to bring forward in time the advent of ICBMs, and to heighten the destabilising arms race. This is because it is reasonable to assume that the USA would not have sprinted for ICBMs if many intelligence, military and scientific experts were not convinced of a missile gap and advocated and participated in a US sprint. It is plausible that this brought forward the development and stockpiling of ICBMs by around five years (which is when the Soviets began building up their forces). It would have been harder to justify funding a sprint to the same extent without the missile gap fear. While missile technology progress would have occured, more may have gone into space research, as the Soviets did. 

In addition to timing, one can speculate about the manner in which ICBMs were introduced. That is to say in a quick and fearful manner, at an intense moment of the Cold War, and in a way which played straight into the security dilemma. This intensified the nuclear arms race. The manner in which the USA signalled to the USSR may also have contributed to the intensity of the Berlin Crisis of 1961 and the Cuban Missile Crisis of 1962. The Gilpatric speech came during a Soviet Party Congress, four days after Khrushchev had offered an opening to the USA by withdrawing his ultimatum that the USA negotiate a peace treaty with East Germany by the end of 1961. The Gilpatric speech was interpreted as Kennedy’s response - to deliberately humiliate Khrushchev. The immediate response was two nuclear tests, including Tsar Bomba, the largest test ever (50-58 megatons). This may have contributed to Soviet elite perceptions of Kennedy as a risky militarist uninterested in agreements, and therefore contributed to the Cuban Missile Crisis (p.176-177).

However, it is important to note that this period was also incredibly creative on the arms control side. 1961 saw the publication of Bull’s The Control of the Arms Race, Schelling & Halperin’s Strategy and Arms Control; and Brennan’s Arms Control, Disarmament, and National Security. Together with the 1960 Daedalus special issue on Arms Control, these are seen as the “four bibles” of arms control. They are all linked to the Harvard-MIT joint seminar, many participants of which went into government, and later contributed to the first bilateral nuclear agreement on nuclear arms control fifty years ago in 1972 (Schelling 1985; Adler 1992).

 Other possible examples

Two other examples of mistaken races come to mind. The first, in between our two cases in the early 1950s, is the ‘bomber gap’. From 1954-1957 the US National Intelligence Estimates estimated large numbers of Soviet long-range bombers, in the hundreds. In response, over that same period the US built up its bomber fleet to over 2,500 bombers. However, this was also based on a mistake. There were only 30 Soviet M-4 bombers in 1956, only 93 were ever produced, and indeed design flaws meant that they could not reach the continental United States. So the main consequence was to unilaterally speed up dangerous developments and increase existential risk.

The second is the Soviet bioweapons program of the 1970s, what we might call the ‘bioweapons gap’. The USSR believed the US were ahead (e.g. in genetics and genomics) and assumed that the US would cheat on the Biological Weapons Convention (BWC), signed in 1972. In response (at least in part), it cheated on the BWC, and carried out the largest bioweapons program in history. However, this was also based on a mistake. The Nixon Administration had in fact destroyed the US’ biological weapons and disbanded the program, as Nixon had announced on November 25, 1969. So the main consequence was to unilaterally speed up dangerous developments and increase existential risk.

Another possible example is long-range heavy strategic bombers in the 1930s - the US and UK invested heavily, the fascists did not. A smaller scale one could be Western mistaken intelligence about Iraq’s WMD programmes. I’d be interested to what extent these dynamics were also present in the US and Russian (and others’) cyber weapons development programs of the 2000s (e.g. the fear of a 'Cyber Pearl Harbor')– and interested in any other cases people are familiar with. More generally, these mistaken races could be seen as a subset of the security dilemma: when actions taken by state A (to increase its security) cause reactions by state B, decreasing the security of A and B (Herz 1950; Jervis 1978).

There are, of course, examples of mistaken intelligence in the other direction, that is overestimates of how long an adversary would take to achieve a capability - for example, the US estimate that the Soviets would take a decade to build a nuclear bomb. And there are examples of correct intelligence about other’s capabilities and intentions, for example perhaps the Dreadnought programme in the early 1900s. 

The atmosphere at RAND

Ellsberg writes vividly and evocatively about the charged, almost messianic atmosphere of RAND during the ‘missile gap’ – the “obsessive ideation” that surrounded it. (Rhodes describes the Manhattan Project in similar vivid detail, but I have not quoted him.) In reading these, I was repeatedly struck by a strange, gnawing sense of familiarity – see if you have the same reaction.

On the importance, neglectedness and tractability of the problem – and the sense of mission:

“I found myself immersed in what seemed the most urgent concrete problem of uncertainty and decision-making that humanity had ever faced […] the challenge looked both more difficult and more urgent than almost anyone outside RAND seemed able to imagine.” (p. 35)

“nearly all the departments and individual analysts at RAND were obsessed with solving the single problem […] in the next few years […] The concentration of focus, the sense of a team effort of the highest urgency, was very much like that of the scientists in the Manhattan Project.” (p.36)

“there was our sense of mission, the burden of believing we knew more about the dangers ahead, and what might be done about them, than did the generals […] or Congress or the public, or even the President. It was an enlivening burden.” (p.37)

“From the analyses by men who became my mentors and closest colleagues, I had come to believe – like Szilard and Rotblat a generation earlier – that this was the best, indeed the only way, of increasing the chance of [survival]. (p.39)

On the intellectual culture:

“From my academic life, I was used to being in the company of very smart people, but it was apparent from the beginning that this was as smart a bunch of [people] as I had ever encountered. […] And it was even better than that. In the middle of the first session, I ventured – though I was the youngest, assigned to be taking notes, and obviously a total novice on the issues – to express an opinion. Rather than showing irritation or ignoring my comment, Herman Kahn […] looked at me soberly and said “You’re absolutely wrong.”

A warm glow spread throughout my body. This was the way my undergraduate fellows […] had routinely spoken to each other [… At Cambridge or Harvard] arguments didn’t remotely take this gloves-off, take-no-prisoners form. I thought, “I’ve found a home.”

And I had. […] I shared with my colleagues a sense of brotherhood, living and working with others for a transcendent cause.” (p.36)

In the late 1950s, it was overwhelmingly male (the previous quote was actually “as smart a bunch of men”, Ray Acheson highlights this point in her review of the book):

“During the cocktail interval at the frequent dinners that our wices took turns hosting, two or three men at a time would cluster in a corner to share secret reflections, sotto voce; the women didn’t have clearances. After the meal the wives would go together into the living room - for security reasons - leaving the men to talk secrets at the table.

There were almost no cleared women professionals at RAND then. The only exceptions I remember were [...] the daughter of Fleet Admiral Chester Nimitz; Alice Hsieh, a China analyst; and Albert Wohlstetter’s wife”

On the privileges and intensity of their life:

“Materially we led a privileged life. I started at RAND, just out of graduate study, at the highest salary my father had ever attained […] Working conditions were ideal […]

But my colleagues were driven men. They shared a feeling – soon transmitted to me – that we were in the most literal sense working to save the world. […]

The work was intense and unrelenting. The RAND building’s lights were kept on all night because researchers came in and out at all hours, on self-chosen schedules. At lunch […] we talked shop – nothing else.” (p.37)

“The first summer there, I worked seventy-hour weeks, devouring secret studies and analyses until late every night, to get up to speed on the problems and possible solutions.” (p. 38)

On the sense of urgency, consider this:

“Enthoven and I were the youngest members of the department. Neither of us joined the extremely generous retirement plan RAND offered. Neither of us believed, in our late twenties, we had a chance of collecting on it.” (p.38)

Ellsberg ends the chapter:

“When my former Harvard faculty advisor heard in 1959 that I was going back to RAND as a permanent employee, he told me bitterly that I was “selling out (as an economist) for a high salary. I told him that after what I had learned the previous summer at RAND, I would gladly work there without pay. It was true. I couldn’t imagine a more important way to serve humanity.” (p.40)

To those who have been involved with effective altruism, rationality, existential risk, AGI research and development, and AI risk: does any of this sound at all familiar?

To make this explicit, I am stating that several of these aspects of the atmosphere, working life and intellectual culture at RAND are strikingly similar. This is especially true of the sense of urgency, secret insight, community and mission. The non-stop “shop” talk, self-chosen work schedules, high starting salaries (for some), the gloves-off intellectual debate will be familiar to many - as will the lack of diversity. I have heard the exact same points made about not taking pensions and being willing to work for free. 

Note that many of these parallels also hold between RAND and the Manhattan Project - such as the isolation, urgency and secrecy. Two quotes from Rhodes:

"[Szilard’s] deepest ambition, more profound even than his commitment to science, was somehow to save the world." (p. 20)

"This informal collegiality partly explains the feeling among scientists of Szilard’s generation of membership in an exclusive group, almost a guild, of international scope and values" (p. 25)

Three takeaways

By this point, I hope the similarities with our current situation are jumping out at you. To make things more explicit:

In general, the AI risk community is very concerned about AI races (Cave & Ó hÉigeartaigh, 2018; Armstrong et al, 2013). Why would AI risk experts - who argued for AI risk when it was an early, fringe belief; have dedicated large amounts of money and talent to AI governance and alignment research; and are particularly concerned about the dangers of racing - ever advocate for and participate in an AGI sprint?

I am concerned that at some point in the next few decades, well-meaning and smart people who work on AGI research and development, alignment and governance will become convinced they are in an existential race with an unsafe and misuse-prone opponent. They might perceive that there is an ‘AGI gap’: that the opponent has some non-negligible (>10%) chance of being ‘first’ to AGI. They will therefore advocate for and participate in a ‘sprint’ to AGI (e.g. with a yearly budget of 0.4% of GDP, or ~$84bn). This advocacy could be the equivalent of the Einstein-Szilard Letter or the Gaither Report.

However, this could be based on a mistake - the opponent may not be racing and there may be no AGI gap. If so, the main consequence of such a sprint would be to unilaterally speed up the dangerous development of AGI (with safety and structural problems unsolved), increasing existential risk. If mistaken like this, we are likely to view our advocacy and participation as the greatest mistake of our careers.

The most likely way this could manifest is a US (or USA and close allies) project motivated by the fear of a Chinese project. This is implicit in current claims from some in the AI risk community (that I do not deny) on the lack of alignment researchers in China (safety risk) and the undesirability of authoritarian AI development (misuse risk). To a lesser and near-term extent, it can be seen in some current DC rhetoric (from e.g. Senator Tom Cotton) about the risk of China and the existence of an AI arms race. 

I draw three general lessons: make sure you’re actually in a race (information on whether you are is very valuable), be careful when secrecy is emphasised, and don’t give up your power as an expert too easily.

Make sure you’re actually in a race

Before any sprint is advocated for or participated in, we should be highly confident that there is a rival sprint occurring. The importance of this point is generally accepted in the AI risk community, but is worth underlining. Information about whether one is actually in a race is very important – accurate information on the Nazis in 1942 or the Soviets in 1957 could have avoided dangerous escalation. 

In our case, evidence and data might take several forms - for example linked to key AI inputs of talent or compute. We may be able to track top talent, for example if researchers ‘go dark’ and stop publishing publicly. We may also be able to track, monitor and verify compute location and usage. We may be able to track government spending on various projects. Cyber espionage and human intelligence on rival’s officials and researchers could also provide evidence. We may also be able to track progress through open-source intelligence. More research and development on ways to obtain accurate information on whether a sprint has been launched by a rival is sorely needed.

Some important work so far on this has been done by the Center for Security and Emerging Technology, a national security think tank in DC, introducing empirical reality into the discussion and therefore deflating some of the DC AI race rhetoric. For example, their work on the USA and its allies’ dominance of the semiconductor supply chain and high-impact AI research; and their work on AI talent (demonstrating that over 85% of Chinese PhD students in the USA intend to and generally do stay in the USA).

You should also be careful when people tell you you’re in a race, as we see in my next point.

Be careful about secrecy 

In both our cases, secrecy was used to sustain the gap myths and sideline those concerned about racing.

This is exemplified by the Rotblat case:

“Joseph Rotblat, after learning from a British associate in the fall of 1944 that there was no German program to deter, promptly resigned from the Manhattan Project. The only scientist to do so, Rotblat was induced, by threat of deportation, not to reveal his reasons for leaving, lest he inspire others to emulate him.” (p.29)

Leslie Groves, Director of the Manhattan Project was infamously secretive, and successfully excluded scientists from many of the targeting and use decisions. Secretary of State, James F. Byrnes, prevented the Szilard Petition from reaching Truman - it is unclear to what extent Truman had preapproved (with full knowledge of the effects) the Hiroshima bombing. After the war, many researchers such as Szilard and Oppenheimer had their security clearances revoked, were blacklisted from government projects, and were cut out of policy-making.

Secrecy also preserved the missile gap myth. In the 1950s, dissenting opinions on the missile gap from the Army and Navy were sidelined. It is unclear to this day to what extent Kennedy actually believed there was in fact a missile gap, as opposed to using it as a useful political attack on the Republicans. In 1961, the NIE revealing that the missile gap was a mistake was classified above ‘Top Secret’. One of the key effects of this was to sustain the myth of the missile gap, and the motivation for the ICBM sprint.

Some levels of secrecy are justified, for example to prevent proliferation of dangerous knowledge to opponents. But one should be careful about demands for secrecy. Secrecy can also be used to obscure the truth, sustain gap myths and sideline those with concerns about racing. Fear of losing clearances, and therefore career progression and policy influence, can be a powerful means to restrict important information and induce conformity. Demands for secrecy may sometimes be a way to keep you from knowing the full truth. It is also important to reflect on a point that Ellsberg raises. Secrecy has a certain glamour to it. It indicates belonging to a select group with special insights into what is really going on. This can be very tempting, but can mislead and distract:

“My clearances had been my undoing. And not only mine. Precisely because we were exposed to secret intelligence estimates [...] I and my colleagues at the RAND Corporation were preoccupied in the late fifties with the urgency of averting nuclear war by deterring a Soviet surprise attack that would exploit an alleged “missile gap.” That supposed dangerous US inferiority was exactly as unfounded in reality as the earlier Manhattan Project fear of a Nazi crash bomb program had been [...]

Working conscientiously, obsessively, on a wrong problem, countering an illusory threat, I and my colleagues at RAND had distracted ourselves and helped distract others from dealing with real dangers posed by the mutual superpower pursuit of nuclear weapons - dangers which we were helping make worse - and from real opportunities to make the world more secure. Unintentionally, yet inexcusably, we made our country and the world less safe.” (p. 296)

Scientists have a lot of power! Don’t give it up easily

As noted above, a US sprint to the Bomb may not have occured when it did without the advocacy of top nuclear scientists, and would not have been successful without the participation of those experts. The US sprint to ICBMs would also have not occurred without the advocacy of many intelligence, military and scientific experts (for example in the Gaither Report), and their subsequent participation.

Similarly, an AGI sprint may not occur without the advocacy of top AI scientists, and will not succeed without the participation of those experts. At a smaller level, researchers being “overwhelmingly opposed” to working on Lethal Autonomous Weapons (LAWS) (Zhang et al. 2021) has meaningfully slowed the USA’s drive towards LAWS.

Final thoughts

Finally, I want to return to the character of the Manhattan Project scientists. These were very good people, heroes even. Several of them kept the idea of a nuclear chain reaction secret throughout the 1930s; they worked incredibly hard (for what they thought was necessary) during WW2; and many of them drew attention to the dangers of nuclear weapons after the war, at heavy cost to their careers. They were arguably right, ex ante, to advocate for and participate in a project to deter the Nazi use of nuclear weapons. They were also amongst the smartest of their generation. Nevertheless, they were convinced by a mistake.

Our current generation is vulnerable to also being convinced by a mistake. Not to put too fine a point on it, you the reader are not smarter than Einstein, Fermi and Oppenheimer. You’re not smarter than Kahn, Wohlstetter, von Neumann, the developers of game theory - the “best and the brightest, the “Whiz Kids”. They were mistaken, and you could be too. 

More generally, we are not in a completely unique, unprecedented situation. We don’t need to figure everything out from first principles. We can and must learn from previous generations, and their mistakes.

(Thanks to colleagues at CSER, CFI, GovAI and Rethink Priorities – especially Di Cooke, Matthijs Maas, Helen Toner, Alex Lintz and Markus Anderljung – for feedback.)
As of 2022-08-04, the certificate of this article is owned by Haydn Belfield (100%).