Jan Leike: On the windfall clause

Jan wrote this thoughtful critique of the Windfall Clause back in 2020, and I thought it should be posted publicly. Posted with Jan's permission. Key excerpt:

A core challenge when trying to design a windfall clause is that there is an incredibly strong incentive to find a loophole once the clause takes effect. If you run an organization who signed a windfall clause and in the future the unlikely comes to pass and you actually end up making $11 trillion in annual profits, it would be rational for you to spend up to $10 trillion on legal fees to try to get out of that clause just for that year; preferably in a way that doesn't cost you too much credibility. Companies are doing this already–this is why the big internet companies pay hardly any taxes.

So how can we implement a windfall clause so that it withstands a multi-trillion dollar search for loopholes? This is incredibly hard, because we need to write and implement the clause today, before the windfall happens (which might never happen at all). This means it has to be done on a budget that is many orders of magnitude smaller than the actual windfall. Let's call this asymmetry the legal offense/defense ratio: We have to defend the "spirit" of the windfall clause against a future version of the organization (possibly with different people in charge) who is experiencing a windfall and wants to get out of the clause. So if you tried to figure out the implementation of a legally tight windfall clause on an insanely high budget like 10 million dollars to safeguard a $10 trillion windfall, you need to withstand a legal offense/defense ratio of at least 1:1,000,000.^[1]

These numbers are certainly made up, but let's roll with them and look at some historical precedent: What are legal battles where one party has spent that much more money than the other? I'm certainly not a lawyer, but I want to highlight two examples.

The Tobacco Master Settlement Agreement was one of the highest stakes legal cases in history: a bunch of US states sued the big tobacco companies for the increased burden of long-time smokers on the healthcare system. The tobacco companies ultimately lost and were sentenced to pay $206 billion over 25 years.^[2] Because of incentive fees, the states' legal fees exceeded $8 billion. I don't know how much the tobacco companies spent on legal fees, but in order to get above the 1:1,000,000 ratio, they would have needed to keep their expenses below $8,000. We can safely assume that they spent much more than that; $8,000 probably wouldn't even enable you to attend most of the court sessions even if you are defending yourself, live right next to the court, and pay yourself minimum wage.

This may sound ridiculous, but this was almost the case in the famous "McLibel" case: Two people were sued by MacDonald's in 1990 for libel because they distributed some flyers accusing the corporation of various malpractices: destroying rainforests, mistreating animals–the usual stuff. One of the defendants was a part-time bar-worker earning a maximum of £65 a week, and the other one was an unemployed postal worker. The two defendants were denied free legal aid because apparently the UK libel system is not great at this. Therefore they represented themselves, plus spending about £30,000 over ten (!) years, not counting their personal time and various pro bono legal advice. In contrast, MacDonald's allegedly spent several million dollars on the case. Still, this is a legal offense/defense ratio of below 1:1,000. The two defendants ultimately lost the case and had to pay £40,000 to MacDonald's, but the Judge also upheld some of their claims. The European Court for Human Rights later ruled that they weren't given a fair trial because of this asymmetry and lack of legal aid. Ultimately we don't know whether the trial would have turned out differently if the two defendants had spent comparable resources to MacDonald's, but it's not hard to imagine that it would since they weren't trained lawyers, and there were even witnesses the defendants couldn't present because they couldn't pay for the plane tickets.

Overall, it's unclear whether legal cases with an offense/defense ratio of significantly upwards of 1:1,000 have ever taken place; in these cases it's probably easy enough for the better resourced party to keep the conflict out of the courts altogether. Thus it is probably safe to assume that our legal system has never before been tested on a legal offense/defense ratio of 1:1,000,000. So we really don't have evidence that the legal system would hold up.

This post is part of what inspired my current thinking on law-following AI.