Recently, I have been learning about industry norms, legal discovery proceedings, and incentive structures related to companies building risky systems. I wanted to share some findings in this post because they may be important for the frontier AI community to understand well. 

TL;DR

Documented communications of risks (especially by employees) make companies much more likely to be held liable in court when bad things happen. The resulting Duty to Due Diligence from Discoverable Documentation of Dangers (the 6D effect) can make companies much more cautious if even a single email is sent to them communicating a risk. 

Companies tend to avoid talking about risk through documented media.

Companies often intentionally avoid discussing the risks of what they are doing through permanent media such as email. For example, this article gives some very shady advice on how companies can avoid liability by using “safe communication” practices to avoid the creation of incriminating “bad documents”. 

Often the drafters of these documents tend to believe that they are providing the company with some value to the business. For example, an engineer notices a potential liability in a design so he informs his supervisor through an email. However, the engineer’s lack of legal knowledge and misuse of legal vocabulary in the communication may later implicate the company with notice of the problem when a lawsuit arises.

I personally enjoyed the use of “when” and not “if” in the excerpt. 

This is a perverse consequence of how it is relatively hard for companies to be held liable for risks when it cannot be proven they knew about them, even if they did. When an incident happens and a company is sued, evidence about its role in the problem is gathered during what is known as the “discovery” phase of a lawsuit (emails are usually discoverable). When records showing that a company had knowledge of the problem are found in discovery, they are much more likely to be found liable.

One email can have a lot of power.

The unfortunate consequence of how discovery works is that companies strategically avoid communicating risks via documented media. But there is a silver lining. The threat of liability due to documented communications of risks can have a lot of influence over how cautious a company is. One discoverable record of a risk can be very impactful.

I like to call this the 6D effect – the Duty to Due Diligence from Discoverable Documentation of Dangers. 

 A few examples

Here are some notable examples of companies being held liable for damages because they ignored documented communication of risks (but there are many throughout legal history). 

• In Grimshaw v. Ford Motor Company, 1981, Ford was held liable for damages involving a fatal crash with a Ford Pinto because it was shown that leadership within the company ignored warnings about problems with the vehicle’s fuel system. 
• In April of this year, a large settlement was reached after the 2017 Grenfell Tower fire in London, which killed 72 people. A big factor in the lawsuit was that the company managing the tower had ignored numerous fire safety warnings which were found in discovery.
• Last year, the Hardwick v. 3M case ended. It was a class action lawsuit from 2018 about the presence of harmful “forever chemicals” (PFAS) in consumer products. The company behind these chemicals was found to have known about risks since the 1970s but was knowingly negligent, which led to a ruling against them. 

Miscellaneous notes

1. The 6D effect can result from any discoverable communication, but it is especially powerful when the warning comes from an employee of the company itself. 
2. If you communicate a risk, it is important to speak up and bring documentation of it to the attention of a court during the discovery phase of a lawsuit. 
3. If you are aware that something a company has done is hazardous, it is your ethical obligation to inform the company, but it is NOT your ethical obligation to help them fix it without compensation. Make sure not to let a company take advantage of you. 

Three takeaways

1. If you work at a company doing potentially risky things, insist on discussing dangers through documented media. If you are retaliated against for documenting communication of risks, you may have grounds for legal recourse. #notlegaladvice
2. If you notice something risky, say something. If the thing you predicted happens, point out the fact that you communicated it. 
3. Safety-focused companies (such as those working on frontier AI systems) should have explicit policies about documenting all discussions of risk.