This is super awesome! Thanks for sharing the specifics of what you did—it will definitely be useful info for us in the future. We’ve considered having people fill out fellowship apps during our intro talk but have worried that this might lower the quality of applicant responses. I’d be interested in knowing what your experience with it was.

Can you tell us a little bit about how this project and partnership came together? What was OpenPhil’s role? What is it like working with such a large number of organizations, including governments? Do you see potential for more collaborations like this? 

Question for either James or Julia: Is this specifically for lead policy or just policy advocacy in general? And can you elaborate why?

This is awesome! Any details you can share on how this whole thing came together? It could be really impactful to try to aim for more coalitions like this for other cost-effective opportunities.

To clarify, I agree that that the ways you can be liable mostly fall into the two categories you delineate but think that your characterization of the categories might be incorrect.

You say that a developer would be liable

1. if you developed a covered model that caused more than $500M harm
2. if you violated any of the prescribed transparency/accountability mechanisms in the bill

But I think a better characterization would be that you can be liable 

1. if you developed a covered model that caused more than $500M harm -> if you fail to take reasonable care to prevent critical harms
2. if you violated any of the prescribed transparency/accountability mechanisms in the bill

It's possible "to fail to take reasonable care to prevent critical harms" even if you do not cause critical harms. The bill doesn't specify any new category of liability specifically for developers who have developed models that cause critical harm. 

To use Casado's example, if a self-driving car was involved in an accident that resulted in a person's death, and if that self-driving car company did not "take reasonable care to prevent critical harms" by having a safety and security protocol much worse than that of other companies, it seems plausible that the company could be fined 10% of their compute/have to pay other damages. (I don't know if self-driving cars actually would be affected by this bill.) 

I think the best reason this might be wrong is that courts might not be willing to entertain this argument or that in tort law "failing to take reasonable care to avoid something" requires that you "fail to avoid that thing"—but I don't have enough legal background/knowledge to know.

Thanks for your reply! I'm a bit confused—I think my understanding of the bill matches yours. The Vox article states "Otherwise, they would be liable if their AI system leads to a 'mass casualty event' or more than $500 million in damages in a single incident or set of closely linked incidents." (See also eg here and here). But my reading of the bill is that there is no mass casualty/$500 million threshold for liability like Vox seems to be claiming here.

Kelsey Piper’s article on SB 1047 says

This is one of the questions animating the current raging discourse in tech over California’s SB 1047, newly passed legislation that mandates safety training for that companies that spend more than $100 million on training a “frontier model” in AI — like the in-progress GPT-5. Otherwise, they would be liable if their AI system leads to a “mass casualty event” or more than $500 million in damages in a single incident or set of closely linked incidents.

I’ve seen similar statements elsewhere too. But after I spent some time today reading through the bill, this seems to be wrong? Liability for developers doesn’t seem to be dependent on whether “critical harm” is actually done. Instead, if the developer fails to take reasonable care to prevent critical harm (or some other violation), even if there is no critical harm done, violations that cause death/bodily harm/etc can lead to fines of 10% or 30% of compute. Here’s the relevant section from the bill:

(a) The Attorney General may bring a civil action for a violation of this chapter and to recover all of the following:

(1) For a violation that causes death or bodily harm to another human, harm to property, theft or misappropriation of property, or that constitutes an imminent risk or threat to public safety that occurs on or after January 1, 2026, a civil penalty in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.

Has there been discussion about this somewhere else already? Is the Vox article wrong or am I misunderstanding the bill?

SummaryBot hallucinated an acronym! UGAP is the University Group Accelerator Program, not the “Undergraduate Priorities Project.”

What are those non-AI safety reasons to pause or slow down?

Checked to see if it had been released, but it looks like the release date has been pushed back to August 23!