Epistemic status: Spent a few hours reading and thinking about the report. I buy a weak variant of its thesis (if you want to have influence, you better have a good model of the institutions you’re working within / with), and am sympathetic to the strong variant (‘in order to reliably affect decision-making, you must yourself be the decision-maker’).
TLDR
This is a review of a report by Samo Burja and Zachary Lerangis on how Leo Szilard and other scientists involved in the Manhattan project failed to influence decision-making around the nuclear bomb, including the choice to use it in Japan and nuclear proliferation after the war. There are strong parallels to X-risk governance, in particular around AI. To quote the report,
The implication of these historical outcomes is that in order to reliably affect decision-making, you must yourself be the decision-maker. Prestige, access to decision-makers, relevant expertise, and cogent reasoning are not sufficient; even with all these you are liable to be ignored.
The Report
Some of the scientists involved in the building of the atomic bomb (primarily Leo Szilard, author of the famous Einstein-Szilard letter, and Niels Bohr) tried to influence decision-making around nuclear governance, including
- Increasing the influence of scientists on how the Manhattan project was led
- Convincing Truman not to use the bomb in Japan
- Convincing Truman that the Soviets would soon develop their own nuclear weapons (which US gov leadership at the time did not believe)
- Establishing international governance of nuclear weapons / arms control regime.
Both Szilard and Bohr had access to powerful people within the US and UK governments, including FDR and Churchill. They nevertheless failed at achieving their goals.
I’ll briefly excerpt two of the four examples from the report:
Bohr talks to Churchill and FDR to convince them to coordinate with the Soviets
In 1944, Niels Bohr, one of the great, Nobel Prize-winning physicists of the 20th century and whose reputation was very well established at the time, was very concerned about the threat of nuclear war and a nuclear arms race after the inevitable end of the war. He wanted the Allies to cooperate with the Soviets on the control of nuclear technology so as to avoid a nuclear arms race or all out war. Towards this end, he wanted the Allies to tell the Soviets about the existence of the atomic bomb project, in the hopes that this would build goodwill and foster future coordination.
[...] Off the bat, Churchill was aggressively opposed to any cooperation with the Russians, or even to telling them of the existence of the bomb project. The meeting went nowhere.
[...] FDR told Bohr that he read Bohr’s memo, that he agrees that the Soviets ought to be approached, and that he is optimistic about the prospects of cooperation based on his assessment of Soviet leader Joseph Stalin. He also said that he was optimistic that Churchill would come around [...]
The next month, FDR and Churchill convened in Hyde Park. They had a short, secret agreement written up on the nuclear question that included the following resolutions: the nuclear program will continue to be kept absolutely secret; the US and Britain will continue to develop nuclear weapons after the end of the war; Professor Bohr should be put under surveillance.
Leo Szilard attempts to prevent the use of the bomb against Japan
[Szilard] went to Washington in May 1945, while the Interim Committee—the committee commissioned to make recommendations about the use of the atomic bomb—was meeting. There he met with Oppenheimer, one of four scientists on the scientific advisory panel to the Interim Committee, and tried to convince him that the bomb shouldn’t be used against Japanese cities. Oppenheimer disagreed. He may have contacted others involved in the committee, but we have not found a record of this. The scientific panel eventually recommended immediate use of the bomb against Japanese cities.
Soon afterwards, he drafted a petition to the President [...] There is a good chance that the petition never reached Truman.
Despite Szilard’s prominence within the scientific hierarchy of the Manhattan Project from day one, he was never able to gain any meaningful standing in the state hierarchy
My Takeaway
The opinions of leading researchers in a field do not (by themselves) determine how the technology they develop will be deployed.
There are many AI researchers who take AI X-risk seriously, including researchers in leading university and industry labs. By itself, this is not enough, if X-risk conscious people are not in position to actually make the crucial decisions (e.g. whether or not to pour another 100x of compute into scaling up a seemingly safe model, or to decide what publication norms should be like).
More generally, in order to have influence it is important to have a good understanding of the institutions one is working within and a concrete model of how exactly one will affect decisionmaking.
Thanks to Ben Landau-Taylor for a good discussion and for pointing me to this report, and to Sam Clarke for feedback and encouraging me to write this.
Thanks for the post. I'm afraid the conclusion (that you need inside info and influence to affect policy and decision making) is right - but I already believed it even before the examples. However, I don't think Bohr and Szilard would be regarded as anywhere close to being naïve or powerless... I mean, Szilard is the guy who gets to convince people that a controlled chain reaction could be feasible in the first place; they had way more influence than the average scientist (though Szilard was not in Los Alamos)... I think they failed because it was a really hard task.
Moreover: they were regarded as experts concerning the development of the bomb, but not on its deployment. As Feynman remarks in his memoires, the military (and other hawks) were not stupid - actually, he praises their decision-making abilities. They had no reason to see the opinion of Szilard and others as authoritative on this matter.
Maybe sometimes there are too many pressures and interests for one to succeded in such tasks - but I'm afraid this hypothesis might be unwelcome to those who embrace Great Founder Theory. And contingencies: Szilard says that
If Roosevelt had died some days later, perhaps Szilard would have succeeded.
And yet, Asilomar Conference succeeded, and the Montreal protocol... I'm not sure if I'd call it a success, but the Warnock Report on embryology also did it. I don't think these people were more influencial than Bohr and Szilard - it's just that the interests of government decision-makers coincided a bit more with their conclusions.
Good points!
Yeah I buy this. There's a report from FHI on nuclear arms control [pdf, section 4.8] that concludes that the effort for international control in 1945/46 was doomed from the start, because of the political atmosphere at the time:
On the other hand, I have to disclose that I sometimes (e.g., when I think about Schelling Nobel Lecture) consider a "dismal hypothesis": given human nature, if the world hadn't seen what happened to Hiroshima, it's quite possible that people wouldn't have developed the same level of aversion to nukes, and we might have had something like a nuclear WW III.
I guess people often need a concrete "proof of concept" to take risks seriously - so they can regard them as imminent . Possibly that's an additional factor in the explanation of why we succeeded with smallpox and CFCs, and why biosecurity gained more track after covid-19.
The authors' takeaway is:
I'm sympathetic to this view, and think they're right about this case study (eg see my Are you really in a race? The Cautionary Tales of Szilárd and Ellsberg).
Nevertheless, I think this claim is overconfident and unfounded. We can't just generalise from one case to the entire universe of cases! A more accurate assessment needs to reckon with the success of the nuclear and biological weapons arms control epistemic community in the early 1970s (such as Kissinger and Meselson) - as well as the many other examples of scientific advisers being influential.
Thanks the good points and the links! I agree the arms control epistemic community is an important story here, and re-reading Adler's article I notice he even talks about how Szilard's ideas were influential after all:
Despite this, in my reading Adler's article doesn't contradict the conclusions of the report: my takeaway is that "Prestige, access to decision-makers, relevant expertise, and cogent reasoning" (while not sufficient on its own) is a good foundation that can be leveraged to gain influence, if used by a community of people working strategically over a long time period, whose members gain key positions in the relevant institutions.