The Weapon of Openness is an essay published by Arthur Kantrowitz and the Foresight Institute in 1989. In it, Kantrowitz argues that the long-term costs of secrecy in adversarial technology development outweigh the benefits, and that openness (defined as "public access to the information needed for the making of public decisions") will therefore lead to better technology relative to adversaries and hence greater national security. As a result, more open societies will tend to outperform more secretive societies, and policymakers should tend strongly towards openness even in cases where secrecy is tempting in the short-term.
The Weapon of Openness presents itself as a narrow attack on secrecy in technological development. In the process, however, it makes many arguments which seem to generalise to other domains of societal decision-making, and can hence be viewed as a more general attack on certain kinds of secretiveness. As such, it seems worth reviewing and reflecting on the arguments in the essay and how they might be integrated with a broader concern for information hazards and the long-term future.
The essay itself is fairly short and worth reading in its entirety, so I've tried to keep this fairly brief. Any unattributed blockquotes in the footnotes are from the original text.
Secrecy in technological development
The benefits of secrecy in adversarial technological development are obvious, at least in theory. Barring leaks, infiltration, or outright capture in war, the details of your technology remain opaque to outsiders. With these details obscured, it is much more difficult for adversaries to either copy your technology or design countermeasures against it. If you do really well at secrecy, even the relative power level of your technology remains obscured, which can be useful for game-theoretic reasons.
The costs of secrecy are more subtle, and easier to miss, but potentially even greater than the benefits. This should sound alarm bells for anyone familiar with the failure modes of naïve consequentialist reasoning.
One major cost is cutting yourself off from the broader scientific and technological discourse, greatly restricting the ability of experts outside the project to either propose new suggestions or point out flaws in your current approach. This is bad enough by itself, but it also makes it much more difficult for project insiders to enlist outside expertise during internal disputes over the direction of the project. The result, says Kantrowitz, is that disputes within secret projects have a much greater tendency to be resolved politically, rather than on the technical merits. That means making decisions that flatter the decision-makers, those they favour and those they want to impress, and avoiding changes of approach that might embarrass those people. This might suffice for relatively simple projects that involve making only incremental improvements on existing technology, but when the project aims for an ambitious leap in capabilities (and hence is likely to involve several false starts and course corrections) it can be crippling.
This claimed tendency of secret projects to make technical decisions on political grounds hints at Kantrowitz's second major argument: that secrecy greatly facilitates corruption. By screening not only the decisions but the decision-making progress from outside scrutiny, secrecy greatly reduces the incentive for decision-makers to make decisions that could be justified to outside scrutinisers. Given the well-known general tendency of humans to respond to selfish incentives, the result is unsurprising: greatly increased toleration of waste, delay and other inefficiencies, up to and including outright corruption in the narrow sense, when these inefficiencies make the lives of decision-makers or those they favour easier, or increase their status (e.g. by increasing their budget).
This incentive to corruption is progressive and corrosive, gradually but severely impairing general organisational effectiveness in ways that will obviously impair the effectiveness of the secret project. If the same organisation performs other secret projects in the future, the corrosion will be passed to these successor projects in the form of normalised deviance and generalised institutional decay. Since the corrupted institutions are the very ones responsible for identifying this corruption, and are screened from most or all external accountability, this problem can be very difficult to reverse.
Hence, says Kantrowitz, states that succumb to the temptations of secret technological development may reap some initial gains, but will gradually see these gains eaten away by impaired scientific/technological exchange and accumulating corruption until they are on net far less effective than if they'd stayed open the whole time. The implication of this seems to be that the US and its allies should be tend much more towards openness and less towards secrecy, at least in the technological domain in peacetime.
Secrecy as a short-term weapon
Finally, Kantrowitz makes the interesting argument that secrecy can be a highly effective short-term weapon, even if it isn't a viable long-term strategy.
When a normally-open society rapidly increases secrecy as a result of some emergency pressure (typically war) they initially retain the strong epistemic institutions and norms fostered by a culture of openness, and can thus continue to function effectively while reaping the adversarial advantages provided by secrecy. In addition, the pressures of the emergency can provide an initial incentive for good behaviour: "the behavior norms of the group recruited may not tolerate the abuse of secrecy for personal advancement or interagency rivalry."
As such, groups that previously functioned well in the open can continue to function well (or even better) in secret, at least for some short time. If the emergency persists for a long time, however, or if the secret institutions persist past the emergency that created them, the corroding effects of secrecy – on efficacy and corruption – will begin to take root and grow, eventually and increasingly compromising the functionality of the organisation.
Secrecy may therefore be good tactics, but bad strategy. If true, this would explain how some organisations (most notably the Manhatten Project) produce such impressive achievements while remaining highly secretive, while also explaining why these are exceptions to the general rule.
Speculating about this myself, this seems like an ominous possibility: the gains from secrecy are clearly legible and acquired rapidly, while the costs accrue gradually and in a way difficult for an internal actor to spot. The initial successes justify the continuation of secrecy past the period where it provided the biggest gains, after which the accruing costs of declining institutional health make it increasingly difficult to undo. Those initial successes, if later made public, also serve to provide the organisation with a good reputation and public support, while the organisations declining performance in current events are kept secret. As a result, the organisation's secrecy could retain both public and private support well past the time at which it begins to be a net impediment to efficacy.
If this argument is true, it suggests that secrecy should be kept as a rare, short-term weapon in the policy toolbox. Rather than an indispensible tool of state policy, secrecy might then be regarded analogously to a powerful but addictive stimulant: to be used sparingly in emergencies and otherwise avoided as much as possible.
Final thoughts
The Weapon of Openness presents an important-seeming point in a convincing-seeming way. Its arguments jibe with my general understanding of human nature, incentives, and economics. If true, they seem to present an important counterpoint to concerns about info hazards and information security. At the same time, the piece is an essay, not a paper, and goes to relatively little effort to make itself convincing beyond laying out its central vision: Kantrowitz provides few concrete examples and cites even fewer sources. I am, in general, highly suspicious of compelling-seeming arguments presented without evidentiary accompaniment, and I think I should be even more so when those arguments are in support of my own (pro-academic, pro-openness) leanings. So I remain somewhat uncertain as to whether the key thesis of the article is true.
(One point against that thesis that immediately comes to mind is that a great deal of successful technological development in an open society is in fact conducted in secret. Monetised open-source software aside, private companies don't seem to be in the habit of publicly sharing their product before or during product development. A fuller account of the weapon of openness would need to account for why private companies don't fail in the way secret government projects are alleged to.)
If the arguments given in the Weapon of Openness are true, how should those of us primarily concerned with value of the long-term future respond? Long-termists are often sceptical of the value of generalised scientific and technological progress, and in favour of slower, more judicious, differential technological development. The Weapon of Openness suggests this may be a much more difficult needle to thread than it initially seems. We may be sanguine about the slower pace of technological development, but the corrosive effects of secrecy on norms and institutions would seem to bode poorly for the long-term preservation of good values required for the future to go well.
Insofar as this corrosion is inevitable, we may simply need to accept serious information hazards as part of our narrow path towards a flourishing future, mitigating them as best we can without resorting to secrecy. Insofar as it is not, exploring new ways to be secretive about certain things while preserving good institutions and norms might be a very important part of getting us to a good future.
Thanks for this, both the original work and your commentary was an edifying read.
I'm not persuaded, although this is mainly owed to the common challenge that noting considerations 'for' or 'against' in principle does not give a lot of evidence of what balance to strike in practice. Consider something like psychiatric detention: folks are generally in favour of (e.g.) personal freedom, and we do not need to think very hard to see how overruling this norm 'for their own good' could go terribly wrong (nor look very far to see examples of just this). Yet these considerations do not tell us what the optimal policy should be relative to the status quo, still less how it should be applied to a particular case.
Although the relevant evidence can neither be fully observed or fairly sampled, there's a fairly good prima facie case for some degree of secrecy not leading to disaster, and sometimes being beneficial. There's some wisdom of the crowd account that secrecy is the default for some 'adversarial' research; it would surprise if technological facts proved exceptions to the utility of strategic deception. Bodies that conduct 'secret by default' work have often been around decades (and the states that house them centuries), and although there's much to suggest this secrecy can be costly and counterproductive, the case for their inexorable decay attributable to their secrecy is much less clear cut.
Moreover technological secrecy has had some eye-catching successes: the NSA likely discovered differential cryptanalysis years before it was on the open literature; discretion by early nuclear scientists (championed particularly by Szilard) on what to publish credibly gave the Manhattan project a decisive lead over rival programs. Openness can also have some downsides - the one that springs to mind from my 'field' is Al-Qaeda started exploring bioterrorism after learning of the United States expressing concern about the same.
Given what I said above, citing some favourable examples doesn't say much (although the nuclear weapon one may have proved hugely consequential). One account I am sympathetic to would be talking about differential (or optimal) disclosure: provide information in the manner which maximally advantages good actors over bad ones. This will recommend open broadcast in many cases: e.g. where there aren't really any bad actors, where the bad actors cannot take advantage of the information (or they know it already, so letting the good actors 'catch up'), where there aren't more selective channels, and so forth. But not always: there seem instances where, if possible, it would be better to preferentially disclose to good actors versus bad ones - and this requires some degree of something like secrecy.
Judging the overall first-order calculus, leave along weighing this against second order concerns (such as noted above) is fraught: although, for what it's worth, I think 'security service' norms tend closer to the mark than 'academic' ones. I understand cybersecurity faces similar challenges around vulnerability disclosure, as 'don't publish the bug until the vendor can push a fix' may not perform as well as one might naively hope: for example, 'white hats' postponing their discoveries hinders collective technological progress, and risks falling behind a 'black hat' community avidly trading tips and tricks. This consideration can also point the other way: if the 'white hats' are much more able than their typically fragmented and incompetent adversaries, the greater the danger of their work 'giving bad people good ideas'. The FBI or whoever may prove much more adept at finding vulnerabilities terrorists could exploit than terrorists themselves. They would be unwise to blog their red-teaming exercises.
Thanks Greg! I think a lot of what you say here is true, and well-put. I don't yet consider myself very well-informed in this area, so I wouldn't expect to be able to convince someone with a considered view that differs from mine, but I would like to get a better handle on our disagreements.
I basically agree with this, with the proviso that I'm currently trying to work out what the considerations to be weighed even are in the first place. I currently feel like I have a worse explicit handle on the considerations mitigating in favour of openness than those mitigating in favour of secrecy. I do think these higher-level issues (around incentives, institutional quality, etc) are likely to be important, but I don't yet know enough to put a number on that.
Given that, and given how little actual evidence Kantrowitz marshals, I don't think someone with a considered pro-secrecy view should be persuaded by this account. I do suspect that, if such a view were to turn out to be wrong, something like this account could be an important part of why.
Do you think there is any evidence for institutional decay due to secrecy? I'm interested in whether you think this narrative is wrong, or just unimportant relative to other considerations.
My (as yet fairly uninformed) impression is that there is also evidence of plenty of hidden inefficiency and waste in secret organisations (and indeed, given that those in those orgs would be highly motivated to use their secrecy to conceal this, I'd expect there to be more than we can see). All else equal, I would expect a secret organisation to have worse epistemics and be more prone to corruption than an open one, both of which would impair its ability to pursue its goals. Do you disagree?
I don't know anything about the NSA, but I think Kantrowitz would claim the Manhattan project to be an example of short-term benefits of secrecy, combined with the pressures of war, producing good performance that couldn't be replicated by institutions that had been secret for decades (see footnote 7). So what is needed to counter his narrative is evidence of big wins produced by institutions with a long history of secret research.
By "second order concerns", do you mean the proposed negative effect of secrecy on institutions/incentives/etc? Because if so that does seem to me to weigh more clearly in one direction (i.e. against secrecy) than the first-order considerations do. Though this probably depends a lot on what you count as first vs second order...
No I agree with these pro tanto costs of secrecy (and the others you mentioned before). But key to the argument is whether these problems inexorably get worse as time goes on. If so, then the benefits of secrecy inevitably have a sell-by date, and once the corrosive effects spread far enough one is better off 'cutting ones losses' - or never going down this path in the first place. If not, however, then secrecy could be a strategy worth persisting with if the (~static) costs of this are outweighed by the benefits on an ongoing basis.
The proposed trend of 'getting steadily worse' isn't apparent to me. One can find many organisations which typically do secret technical work have been around for decades (the NSA is one, most defence contractors another, (D)ARPA, etc.). A skim of what they were doing in (say) the 80s versus the 50s doesn't give an impression they got dramatically worse despite the 30 years of secrecy's supposed corrosive impact. Naturally, the attribution is very murky (e.g. even if their performance remained okay, maybe secrecy had gotten much more corrosive but this was outweighed by countervailing factors like much larger investment; maybe they would have fared better under a 'more open' counterfactual) but the challenge of dissecting out the 'being secret * time' interaction term and showing it is negative is a challenge that should be borne by the affirmative case.
Yeah, I was thinking about this yesterday. I agree that this ("inexorable decay" vs a static cost of secrecy) is probably the key uncertainty here.
I think the points each of you make there are true and important.
As a further indication of the value of Will's point, I think a big part of the reason we're having this discussion at all is probably Bostrom's paper on information hazards, which is itself much more a list of considerations than an attempt to weigh them up. Bostrom makes this explicit:
(We could describe efforts such as Bostrom's as "mapping the space" of consequences worth thinking about further, without yet engaging in that further thought.)
It seems possible to me that we've had more cataloguing of the considerations against openness than those for it, and thus that posts like this one can contribute usefully to the necessary step that comes before weighing up all the considerations in order to arrive at a well-informed decision. (For the same reason, it could also help slightly-inform all the other decisions we unfortunately have to make in the meantime.)
One caveat to that is that a post that mostly covers just the considerations that point in one direction could be counterproductive for those readers who haven't seen the other posts that provide the counterbalance, or who saw them a long time ago. But that issue is hard to avoid, as you can't cover everything in full detail in one place, and it also applies to Bostrom's paper and to a post I'll be making on this topic soon.
Another caveat in this particular case is that there are two related reasons why decisions on whether to develop/share (potentially hazardous) information may demand somewhat more caution than the average decision: the unilateralist's curse, and the fact that hard-to-reverse decisions destroy option value.
I personally think that it's still a good idea to openly discuss the reasons for openness, even if a post has to be somewhat lopsided in that direction for brevity and given that other posts were lopsided in the other direction. But I also personally think it might be good to explicitly note those extra reasons for caution somewhere within the "mostly-pro" post, for readers who may come to conclusions on the basis of that one post by itself.
(Just to be clear, I don't see this as disagreeing with Greg or Will's comments.)