Over the long run, technology has improved the human condition. Nevertheless, the economic progress from technological innovation has not arrived equitably or smoothly. While innovation often produces great wealth, it has also often been disruptive to labor, society, and world order. In light of ongoing advances in artificial intelligence (“AI”), we should prepare for the possibility of extreme disruption, and act to mitigate its negative impacts. This report introduces a new policy lever to this discussion: the Windfall Clause.
What is the Windfall Clause?
The Windfall Clause is an ex ante commitment by AI firms to donate a significant amount of any eventual extremely large profits. By “extremely large profits,” or “windfall,” we mean profits that a firm could not earn without achieving fundamental, economically transformative breakthroughs in AI capabilities. It is unlikely, but not implausible, that such a windfall could occur; as such, the Windfall Clause is designed to address a set of low-probability future scenarios which, if they come to pass, would be unprecedentedly disruptive. By “ex ante,” we mean that we seek to have the Clause in effect before any individual AI firm has a serious prospect of earning such extremely large profits. “Donate” means, roughly, that the donated portion of the windfall will be used to benefit humanity broadly.
Properly enacted, the Windfall Clause could address several potential problems with AI-driven economic growth. The distribution of profits could compensate those rendered faultlessly unemployed due to advances in technology, mitigate potential increases in inequality, and smooth the economic transition for the most vulnerable. It provides AI labs with a credible, tangible mechanism to demonstrate their commitment to pursuing advanced AI for the common global good. Finally, it provides a concrete suggestion that may stimulate other proposals and discussion about how best to mitigate AI-driven disruption.
Motivations Specific to Effective Altruism
Most EA AI resources to-date have been focused on extinction risks from AI. One might wonder whether the problems addressed by the Windfall Clause are really as pressing as these.
However, a long-term future in which advanced forms of AI like AGI or TAI arrive but primarily benefit a small portion of humanity is still highly suboptimal. Failure to ensure advanced AI benefits all could "drastically curtail" the potential of Earth-originating intelligent life. Intentional or accidental value lock-in could result if, for example, a TAI does not cause extinction but is programmed to primarily benefit shareholders of the corporation that develops it. The Windfall Clause thus represents a legal response to this sort of scenario.
There remain significant unresolved issues regarding the exact content of an eventual Windfall Clause, and the way in which it would be implemented. We intend this report to spark a productive discussion, and recommend that these uncertainties be explored through public and expert deliberation. Critically, the Windfall Clause is only one of many possible solutions to the problem of concentrated windfall profits in an era defined by AI-driven growth and disruption. In publishing this report, our hope is not only to encourage constructive criticism of this particular solution, but more importantly to inspire open-minded discussion about the full set of solutions in this vein. In particular, while a potential strength of the Windfall Clause is that it initially does not require governmental intervention, we acknowledge and are thoroughly supportive of public solutions.
We hope to contribute an ambitious and novel policy proposal to an already rich discussion on this subject. More important than this policy itself, though, we look forward to continuously contributing to a broader conversation on the economic promises and challenges of AI, and how to ensure AI benefits humanity as a whole. Over the coming months, we will be working with the Partnership on AI and OpenAI to push such conversations forward. If you work in economics, political science, or AI policy and strategy, please contact me to get involved.
Thanks very much for sharing this. It is nice to see some innovative thinking around AI governance.
I have a bunch of different thoughts, so I'll break them over multiple comments. This one mainly concerns the incentive effects.
I think this is a bit of a strawman. While it is true that many people don't understand tax incidence and falsely assume the burden falls entirely on shareholders rather than workers and consumers, the main argument for the optimality of a 0% corporate tax rate is Chamley-Judd (see for example here) and related results. (There are some informal descriptions of the result here and here.) The argument is about disincentives to invest reducing long-run growth and thereby making everyone poorer, not a short-term distributional effect. (The standard counter-argument to Chamley Judd, as far as I know, is to effectively apply lots of temporal discounting, but this is not available to longtermist EAs).
This is sort of covered in B.1., but I do not think the responses are very persuasive. The main response is rather glib:
There are a lot desirable investments which would be rendered uneconomic. The fact that some investment will continue at a reduced level does not mean that missing out on the other forgone projects is not a great cost! For example, a 20% pre-tax return on investment for a moderately risky project is highly attractive - but after ~25% corporate taxes and ~50% windfall clause, this is a mere 5% return* - almost definitely below their cost of capital, and hence society will probably miss out on the benefits. Citation 231, which seems like it should be doing most of the work here, instead references a passing comment in a pop-sci book about individual taxes:
But corporations are much less motivated by fame and love of their work than individuals, so this does not seem very relevant, and furthermore it does not address the inter-temporal issue which is the main objection to corporation taxes.
I also think the sub-responses are unsatisfying. You mention that the clause will be voluntary:
But this does not mean it won't reduce incentives to innovate. Firms can rationally take actions that reduce their future innovation (e.g. selling off an innovative but risky division for a good price). A firm might voluntarily sign up now, when the expected cost is low, but then see their incentives dramatically curtailed later, when the cost is large. Furthermore, firms can voluntarily but irrationally reduce their incentives to innovate - for example a CEO might sign up for the clause because he personally got a lot of positive press for doing so, even at the cost of the firm.
Additionally, by publicising this idea you are changing the landscape - a firm which might have seen no reason to sign up might now feel pressured to do so after a public campaign, even though their submission is 'voluntary'.
The report then goes on to discuss externalities:
Here you approvingly cite Seb's paper, but I do not think it supports your point at all. Firms have both positive and negative externalities, and causing them to internalise them requires tailored solutions - e.g. a carbon tax. 'Being very profitable' is not a negative externality, so a tax on profits is not an effective way of minimising negative externalities. Similarly, the Malicious Use paper is mainly about specific bad use cases, rather than size qua size being undesirable. Moreover, size has little to do with Seb's argument, which is about estimating the costs of specific research proposals when applying for grants.
I strongly disagree with this non-sequitur. The fact that we have achieved some level of material success now doesn't mean that the future opportunity isn't very large. Again, Chamley-Judd is the classic result in the space, suggesting that it is never appropriate to tax investment for distributional purposes - if the latter must be done, it should be done with individual-level consumption/income taxation. This should be especially clear to EAs who are aware of the astronomical waste of potentially forgoing or delaying growth.
Elsewhere in the document you do hint at another response - namely that by adopting the clause, companies will help avoid future taxation (though I am sceptical):
However, it seems that the document equivocates on whether or not the clause is to reduce taxes, as elsewhere in the document you deny this:
\* for clarity of exposition I am assuming the donation is not tax deductible, but the point is not dramatically altered if it is.
This same reasoning also shows why firms might seek positional goods. E.g., executives and AI engineers might really care about being the first to develop AGI. Thus, the positional arguments for taxing windfall come back into play to the same extent that this is true.
This is certainly true. I think we as a community should discuss (as here) what the tradeoffs are. Reduced innovation in AI is a real cost. So too are the harms identified in the WC report and more traditional X-risk harms. We should set the demands of firms such that the costs to innovation are outweighed by benefits from long-run wellbeing.
As a blanket note about your next few points, I agree that the WC would disincentivize innovation to some extent. It was not my intention to claim—nor do I think I actually claimed (IIRC)—that it would have no socially undesirable incentive effects on innovation. Rather, the points I was making were more aimed at illuminating possible reasons why this might not be so bad. In general, my position is that the other upsides probably outweigh the (real!) downsides of disincentivizing innovation. Perhaps I should have been more clear about that.
Yep, that seems right.
However, it's very hard to get individuals to sign a WC for a huge number of reasons. See
I think both outcomes are possible. The second point is simply to point out that the WC does not and cannot (as a legal matter) prevent a state from levying taxes on firms. The first two points, by contrast, are a prediction that the WC will make such taxation less likely.
The report then goes on to discuss externalities:
I agree that the WC does not target the externalities of AI development maximally efficiently. However, I think that the externalities of such development are probably significantly correlated with windfall-generation. Windfall-generation seems to me to be very likely to accompany a risk of a huge number of negative externalities—such as those cited in the Malicious Use report and classic X-risks.
A good analogy might therefore be to a gas tax for funding road construction/maintenance, which imperfectly targets the thing we actually care about (wear and tear on roads), but is correlated with it so it's a decent policy.
To be clear, I agree that it's not the best way of addressing those externalities, and that the best possible option is to institute a Pigouvian tax (via insurance on them like Farquhar et al. suggest or otherwise).
Thanks a ton for your substantial engagements with this, Larks. Like you, I might spread my replies out across a few posts to atomize my replies.
Thanks for this. TBQH, I was primarily familiar with the concerns cited as the reasons for opposition to corporate income taxation. I do wish in retrospect I had been able to get more acquainted with the anti-corporate-tax literature like you cited. Since I'm not an economist, I was not aware of and wasn't able to find some of the sources you cited. I agree that they make good points not adequately addressed by the Report.
For some more recent discussion in favor of capital taxation, see Korinek (2019). Admittedly, it's not clear how much this supports the WC because it does not necessarily target rents or fixed factors.
I agree that the problem (that investors will prefer to invest in non-signatories, and hence it will reduce the likelihood of pro-social firms winning, if pro-social firms are more likely to sign) does seem like a credible issue. I found the description of the proposed solution rather confusing however. Given that I worked as an equity analyst for five years, I would be surprised if many other readers could understand it!
Here are my thoughts on a couple of possible versions of what you might be getting at- apologies if you actually intended something else altogether.
1) The clause will allow the company to make the required payments in stock rather than cash.
Unfortunately this doesn't really make much difference, because it is very easy for companies to alter this balance themselves. Consider that a company which had to make a $1 billion cash payment could fund this by issuing $1 billion worth of stock; conversely a company which had to issue stock to the fund could neutralise the effect on their share count by paying cash to buy back $1 billion worth of ordinary shares. This is the same reason why dividends are essentially identical to share buybacks.
2) The clause will allow subsequent financing to be raised that is senior to the windfall clause claim, and thus still attractive to investors.
'Senior' does not mean 'better' - it simply means that you have priority in the event of bankruptcy. However, the clause is already junior to all other obligations (because a bankrupt firm will be making ~0% of GDP in profit and hence have no clause obligations), so this doesn't really seem like it makes much difference. The issue is dilution in scenarios when the company does well, which is when the most junior claims (typically common equity, but in this case actually the clause) perform best.
The fundamental reason these two approaches will not work is that the value of an investment is determined by the net present value of future cashflows (and their probability distribution). Given that the clause is intended to have a fixed impact on these flows (as laid out in II.A.2), the impact on firm valuation is also rather fixed, and there is relatively little that clever financial engineering can do about it.
3) The clause will have claim only to profits attributable to the existing shares at the time of the signing on. Any subsequent equity will have a claim on profits unencumbered by the clause. For example, if a company with 80 shares signs on to the clause, then issues 10 more shares to the market, the maximum % of profits that would be owed is 50%*80/(80+10) = 44.4%
This would indeed avoid most of the problems in attracting new capital (save only the fear that a management team willing to screw over their previous investors will do so to you in the future, which is something investors think about a lot).
However, it would also largely undermine the clause by being easy to evade due to the fungibility of capital. Consider a new startup, founded by three guys in a basement, that signs the clause. Over the next few years they will raise many rounds of VC, eventually giving up the majority of the company, all excluded from the clause. Additionally, they pay themselves and employees in stock or stock options, which are also exempt from the clause. Eventually they IPO, having successfully diluted the clause-affected shares to ~1%. In order to finish the job, they then issue some additional new equity and use the proceeds to buy back the original shares.
One interesting point on the other side, however, is the curious tendency for tech investors to ignore dilution. Many companies will exclude stock-based-comp from their adjusted earnings, and analysts/investors are often willing to go along with this, saying "oh but it's a non-cash expense". Furthermore, SBC is excluded from Free Cash Flow, which is the preferred metric for many tech investors. So it is possible that (for a while) investors would simply ignore it.
Apologies that this was confusing, and thanks for trying to deconfuse it :-)
Subsequent feedback on this (not reflected in the report) is that issuing low-value super-junior equity at the time of signing (and then holding it in trust) is probably the best option for this.
I wonder how such a commitment would actually impact a company's balance sheet.
In the example of Windfall Clause worth $649.34 million (in 2010 dollars), I guess that, according to IAS 37, it would be considered a contingent liability of remote possibility - and so wouldn't even need to be disclosed by the company.
Moreover, due to hyperbolic discount, it would probably be perceived as much less costly than $650m. (and I thought time preferences were evil...)
Yep, thinking through the accounting of this would be very important. Unfortunately I'm not an accountant but I would very much like to see an accountant discuss how to structure this in a way that does not prematurely burden a signatory's books.
(Epistemic status: there must be some flaw, but I can't find it.)
Sure. But, let me be clearer: what drew my attention is that, apparently, there seems to be no down-side for a company to do this ASAP. My whole point:
First, consider the “simple” example where a signatory company promises to donate 10% of its profits from a revolutionary AI system in 2060, a situation with an estimated probability of about 1%; the present value of this obligation would currently amount to U$650 million (in 2010 dollars). This seems a lot; however, I contend that, given investors’ hyperbolic discount, they probably wouldn’t be very concerned about it – it’s an unlikely event, to happen in 40 years; moreover, I’ve checked with some accountants, and this obligation would (today) be probably classified as a contingent liability of remote possibility (which, under IAS 37, means it wouldn’t impact the company’s balance sheet – it doesn’t even have to be disclosed in its annual report). So, I doubt such an obligation would negatively impact a company’s market value and profits (in the short-term); actually, as there’s no “bad marketing”, it could very well increase them.
Second (all this previous argument was meant to get here), would it violate some sort of fiduciary duty? Even if it doesn’t affect present investors, it could affect future ones: i.e., supposing the Clause is enforced, can these investors complain? That’s where things get messy to me. If the fiduciary duty assumes a person-affecting conception of duties (as law usually does), I believe it can’t. First, if the Clause were public, any investor that bought company shares after the promise would have done it in full knowledge – and so wouldn’t be allowed to complain; and, if it didn’t affect its market value in 2019, even older investors would have to face the objection “but you could have sold your shares without loss.” Also, given the precise event “this company made this discovery in such-and-such way”, it’s quite likely that the event of the promise figures in the causal chain that made this precise company get this result – it certainly didn’t prevent it! Thus, even future investors wouldn’t be allowed to complain.
There must be some flaw in this reasoning, but I can’t find it.
(Could we convince start-ups to sign this until it becomes trendy?)
Interesting. I don't think it's relevant, from a legal standpoint, that investors might discount hyperbolically rather than exponentially. I assume that a court would apply standard exponential discounting at market rates. But this is a promising psychological and pragmatic fact!
If this is right, this is very helpful indeed :-)
See § III of the report :-)
I wonder if, in addition to the section B.2, the clause could framed as a compensation scheme in favor of a firm's shareholders - at least if it was adopted conditionally to other firms adopting it (a kind of "good cartel"). Since the ex ante probability of one specific firm A obtaining future windfall profits from an AGI is lower than the probability of any of its present or future competitors doing it (so driving A out of business), it might be in the interest of these firms' shareholders to hedge each other by committing to a windfall clause. (Of course, the problem with this argument is that it'd only justify an agreement covering the shareholders of each agreeing firm)
You are not the only person to have expressed interest in such an arrangement :-) Unfortunately I think there might be some antitrust problems with that.
I imagined so; but the idea just kept coming to my head, and since I hadn't seen it explicitly stated, I thought it could be worth mentioning.
I agree that, with current legislation, this is likely so.
But let me share a thought: even though we don't have hedge for when one company succeeds so well it ends up dominating the whole market (and ruining all competitors in the process), we do have some compensation schemes (based on specific legislation) for when a company fails, like deposit insurance. The economic literature usually presents it as a public good (they'd decrease the odds of a bank run and so increase macroeconomic stability), but it was only accepted by the industry because it solved a lemons problem. Even today, the "green swan" (s. section 2) talk in finances often appeals to the risk of losses in a future global crisis (the Tragedy of the Horizon argument). My impression is that an innovation in financial regulation often starts with convincing banks and institutions that it's in their general self-interest, and then it will become compulsory only to avoid free-riders.
(So, yeah, if tech companies get together with the excuse of protecting their investors (& everyone else in the process) in case of someone dominating the market, that's collusion; if banks do so, it's CSR)
(epistemic status about the claims on deposit insurance: I shoud have made a better investigation in economic history, but I lack the time, the argument is consistent, and I did have first hand experience with the creation of a depositor insurance fund for credit unions - i.e., it didn't mitigate systemic risk, it just solved depositors risk-aversion)
Do you think this could be more effective as legislation rather than corporate policy? That is, could political advocacy be better than corporate campaigning for achieving this?
I am fairly confident that corporate policy is better. Corporate policy has a number of advantages:
I just want to document that this idea was mentioned in the book Superintelligence by Nick Bostrom.
Does anybody know who first proposed the idea of a windfall clause?
Superintelligence is the earliest reference that I know.