Jan wrote this thoughtful critique of the Windfall Clause back in 2020, and I thought it should be posted publicly. Posted with Jan's permission. Key excerpt:
A core challenge when trying to design a windfall clause is that there is an incredibly strong incentive to find a loophole once the clause takes effect. If you run an organization who signed a windfall clause and in the future the unlikely comes to pass and you actually end up making $11 trillion in annual profits, it would be rational for you to spend up to $10 trillion on legal fees to try to get out of that clause just for that year; preferably in a way that doesn't cost you too much credibility. Companies are doing this already–this is why the big internet companies pay hardly any taxes.
So how can we implement a windfall clause so that it withstands a multi-trillion dollar search for loopholes? This is incredibly hard, because we need to write and implement the clause today, before the windfall happens (which might never happen at all). This means it has to be done on a budget that is many orders of magnitude smaller than the actual windfall. Let's call this asymmetry the legal offense/defense ratio: We have to defend the "spirit" of the windfall clause against a future version of the organization (possibly with different people in charge) who is experiencing a windfall and wants to get out of the clause. So if you tried to figure out the implementation of a legally tight windfall clause on an insanely high budget like 10 million dollars to safeguard a $10 trillion windfall, you need to withstand a legal offense/defense ratio of at least 1:1,000,000.[1]
These numbers are certainly made up, but let's roll with them and look at some historical precedent: What are legal battles where one party has spent that much more money than the other? I'm certainly not a lawyer, but I want to highlight two examples.
The Tobacco Master Settlement Agreement was one of the highest stakes legal cases in history: a bunch of US states sued the big tobacco companies for the increased burden of long-time smokers on the healthcare system. The tobacco companies ultimately lost and were sentenced to pay $206 billion over 25 years.[2] Because of incentive fees, the states' legal fees exceeded $8 billion. I don't know how much the tobacco companies spent on legal fees, but in order to get above the 1:1,000,000 ratio, they would have needed to keep their expenses below $8,000. We can safely assume that they spent much more than that; $8,000 probably wouldn't even enable you to attend most of the court sessions even if you are defending yourself, live right next to the court, and pay yourself minimum wage.
This may sound ridiculous, but this was almost the case in the famous "McLibel" case: Two people were sued by MacDonald's in 1990 for libel because they distributed some flyers accusing the corporation of various malpractices: destroying rainforests, mistreating animals–the usual stuff. One of the defendants was a part-time bar-worker earning a maximum of £65 a week, and the other one was an unemployed postal worker. The two defendants were denied free legal aid because apparently the UK libel system is not great at this. Therefore they represented themselves, plus spending about £30,000 over ten (!) years, not counting their personal time and various pro bono legal advice. In contrast, MacDonald's allegedly spent several million dollars on the case. Still, this is a legal offense/defense ratio of below 1:1,000. The two defendants ultimately lost the case and had to pay £40,000 to MacDonald's, but the Judge also upheld some of their claims. The European Court for Human Rights later ruled that they weren't given a fair trial because of this asymmetry and lack of legal aid. Ultimately we don't know whether the trial would have turned out differently if the two defendants had spent comparable resources to MacDonald's, but it's not hard to imagine that it would since they weren't trained lawyers, and there were even witnesses the defendants couldn't present because they couldn't pay for the plane tickets.
Overall, it's unclear whether legal cases with an offense/defense ratio of significantly upwards of 1:1,000 have ever taken place; in these cases it's probably easy enough for the better resourced party to keep the conflict out of the courts altogether. Thus it is probably safe to assume that our legal system has never before been tested on a legal offense/defense ratio of 1:1,000,000. So we really don't have evidence that the legal system would hold up.
This post is part of what inspired my current thinking on law-following AI.
Important caveat pointed out by Richard Ngo: the benefactors of the windfall clause are also incentivized to spend trillions in legal fees to stop the organization from getting out of the windfall clause; at least assuming they can raise the necessary funds. ↩︎
This is so much money that some states ended up creating financial derivatives for these payments in order to hedge the debt. ↩︎
I think you perhaps should consider the possibility that legal defense and offense might have declining marginal returns... After X number of dollars exploring all available evidence, hiring excellent legal minds to develop, refine, and deliver the most robust and persuasive arguments, etc., there is going to be an asymptote regarding the value obtained from additional funds. Then it will depend more on the judge, fact finders, and underlying evidence than spending a trillion dollars on lawyers, despite what legal dramas would have you believe...
Thanks for sharing this critique, Cullen.
I was curious about who would be the firm's opponent in this scenario, i.e. the actor trying to legally implement the Windfall Clause.
In a world where a Windfall of this order of magnitude is possible, I would anticipate a number of additional actors of somewhat comparable magnitude. I'd also expect states to have more wealth too (even if the AI company didn't pay tax, an AI advanced enough to generate Windfall profits is likely to grow the economy dramatically). If this were true, I might expect there to be incentives (or the possibility of providing incentives) for sufficiently wealthy states or other actors to use their resources to keep the legal offense-defence ratio more manageable.
That being said, I'm very uncertain about the above. There is certainly precedence for companies to become dramatically richer than some states. Moreover, states benefiting considerably from transformative AI may not necessarily see defending a Windfall Clause as a priority. Nevertheless, I do think there's merit in thinking carefully about what kind of actors might exist in a world where the Windfall Clause looks like it will soon trigger.
Agreed, and in part why I'm not very sold on this critique.
This is underdetermined by the concept of the WC itself, but is a very important design consideration.
The worst-case scenario for this failure mode is that some very large number of people are plaintiffs in their individual capacity. Coordinating to enforce would be hard for them, but class action mechanisms (of which I'm not an expert!) could probably help.
A better approach would be to have some identifiable small number (including one) of recipients. This is in fact what we suggest in the report (Appendix II). This helps that actor internalize the costs of seeking to enforce the WC.
I think we can improve on that too, as suggested in the report to some degree. For example, I strongly believe the WC should include fee-shifting provisions for exactly this reason, so that the AI developer would be on the hook for legal fees from those trying to enforce the WC. And a variety of standard covenants in debt arrangements—such as accounting, indemnification, and domicile requirements—could further reduce risk. I also think the gold standard is securitizing the WC payment instrument, which would probably make enforcement easier for a variety of reasons.