Today I'm launching SB53.info, a resource to help people understand California's Senate Bill 53. The site shows the text of SB 53 enriched with annotations to explain its political and scientific context.
In a nutshell, SB 53 says seven things. These are the four I consider most important:
And these are the three major provisions that I expect will be less important:
For my most thorough analysis of these key provisions, I encourage you to read SB53.info. Here, I'll give a high-level overview of what each provision does and how it departs from the status quo.
Large AI developers are not currently mandated to adopt safety policies, but under SB 53, they would be. This wouldn't require most frontier developers to do much, since they already have published safety policies, and they've already made non-enforceable commitments to follow those policies. Anthropic, OpenAI, Google DeepMind, and Meta have all written safety policies that satisfy many of the requirements in § 22757.12.a, and xAI has a draft safety policy that satisfies a few of the requirements. So if SB 53 were to pass, one industry laggard would have to write a safety policy for the first time, other frontier developers would have to make their existing safety policies more robust, and every frontier developer would be legally mandated to follow their own safety policy.
Moreover, they must get an independent auditor to certify that they are following their safety policy, and this explicitly includes certifying that the safety policy is clear enough for it to be determinate whether the developer is complying with it. As far as is publicly known, no major AI developer has ever undergone a safety audit, but such audits are completely routine in other risky industries like aviation. Every airline in the US is required to write a plan explaining what measures they will follow to ensure safety, and they must regularly commission independent audits to confirm that they're following the plan. SB 53 would require companies developing frontier AI systems to do the same.
It is already a widely accepted best practice in the AI industry that when a company releases a new frontier model, they should publish a report called a model card describing the model's capabilities to consumers and to the scientific community. Anthropic, OpenAI, and Google DeepMind have consistently released model cards alongside all of their recent frontier models, and all three companies' model cards likely comply with most of the requirements in SB 53. These cards generally explain how the developer assessed the risks posed by their model, how they intend to mitigate those risks, and whether their model reached any prespecified risk or capability thresholds. If the bill were to pass, the big three AI developers would have to disclose more detailed information about third party assessments run on their models, and developers like xAI that generally don't publish model cards would have to start publishing them.
SB 53 would make large developers civilly liable for breaches of the above rules. No AI company executives will go to jail for failing to publish a safety policy or model card, but their companies can be faced with heavy fines—up to millions of dollars for a knowing violation of the law that causes material catastrophic risk. This is a major change from the status quo. Today, frontier AI developers have no legal obligation to disclose anything about their safety and security protocols to government, let alone to the public. When a company releases a new AI system more powerful than any system before, it is entirely optional under present law for them to tell consumers what dangerous things that system can do. And if a company does choose to adopt a safety policy or publish a model card, there is no force of law to guarantee the safety policy is being implemented or that the model card is accurate. This would all change under SB 53. We'd no longer have to rely on AI developers' good will to share critical safety information with the public.
There is currently no official channel for the California state government to collect reports of safety incidents involving AI. If a frontier AI developer discovered tomorrow that the weights of their leading model had been stolen, the best they could do to alert state authorities would probably be to email the Attorney General's office. If a member of the public witnessed an AI autonomously causing harm in the wild, the fastest way for them to tell the authorities would probably be to tweet about it. SB 53 would replace these slow, informal information channels with an official incident reporting mechanism run by the AG. Just like California has an official website to collect reports of data breaches, there would be another site for reports of critical AI safety incidents.
Existing California law already offers whistleblower protection to AI company employees who report a violation of federal, state, or local law to public officials or to their superiors. Companies may not make rules or enforce contracts that would prevent their employees from blowing the whistle, nor can they retaliate against an employee who becomes a whistleblower. SB 53 expands the scope of these protections in two ways. First, it would grant whistleblower protection to actors who are currently not protected. Independent contractors, freelancers, unpaid advisors, and external groups that help developers to assess and manage catastrophic risk are not protected by existing law if they become whistleblowers, but they would be under SB 53. Second, the bill would protect disclosures of evidence that an AI developer's activities pose a catastrophic risk, whereas existing law only protects disclosures of evidence that a developer is breaking the law. Of course, many ways that a developer could cause a catastrophic risk would also involve breaking the law, but it's conceivable that a developer could do something catastrophically dangerous yet legal. It might also be easier for many would-be whistleblowers to tell whether their employer is causing a catastrophic risk than to tell whether their employer is breaking a specific law.
Finally, SB 53 calls for California to build a publicly owned AI compute cluster called CalCompute. The cluster's purpose would be to support AI research and innovation for the public benefit. Nothing like CalCompute currently exists in California, but similar projects have been announced or are already underway in several other jurisdictions. New York has already built a compute cluster under their Empire AI initiative, the UK has given academics compute access through its AI Research Resource, and the US National Science Foundation's National AI Research Resource aims to provide the same for American researchers. SB 53 does not specify how much funding California will put behind CalCompute, nor how many AI chips it aims to acquire, so it's hard to tell how much this section of the bill will accomplish. If CalCompute is funded generously in the next state budget, it could be a big deal, but if the project only gets a meager budget, it may not achieve much.
I have intentionally kept SB53.info neutral. The purpose of the site is to help readers with low context on AI safety and governance get up to speed and understand SB 53, not to editorialize in favor of the bill or against it. But of course I do have an opinion on SB 53. I think it would probably be good for California and for the world if the bill passes.
Requiring every frontier AI company to follow a safety policy and to publish model cards alongside their strongest models is just about the least we can ask them to do for safety. In the first place, we know it's not too hard for the companies to comply since Anthropic, OpenAI, and GDM already have safety policies and model cards that satisfy most of the requirements in SB 53. It's not asking much for them to be somewhat more transparent and for less responsible companies near the frontier to get serious about safety and transparency.
And if we look beyond the AI industry, it becomes even clearer how light SB 53's safety and disclosure requirements are in absolute terms. We make airlines write detailed safety plans, and we hold them legally accountable if they fail to follow those plans. Why shouldn't we make the largest AI companies do the same? Would it really make sense for companies that are trying to build godlike superintelligence to do less safety planning than companies that fly planes? Again, if a company wants to sell children's crayons, we make them test those crayons for safety hazards and publish a report explaining why they believe it is safe to sell their crayons to consumers. Would it really be sensible for a company to deploy AGI to the public with less safety information than Crayola provides when it sells crayons?[1] My point is that the safety and transparency requirements SB 53 would impose on large AI developers are not out of the ordinary. We already impose them on just about any business that creates even a small risk to the public.
On the other hand, SB 53 is just about the strongest AI safety regulation that we can know right now will be robustly good. In most other risky industries, we pretty much understand what steps businesses need to take to avoid accidents and keep the public safe, so we pass detailed regulations requiring them to take those steps. If you're cooking food, you need to store it at such and such a temperature, protect it from contaminants, and so on. If you're driving a truck, you have to rest for so many hours in each twenty-four hour window, and you have to check regularly that your truck is in good repair. But we are not yet in a position to pass such detailed safety regulations for AI developers. Our best threat modeling is still so janky and the sciences of alignment and control are still so immature that nobody knows precisely what you have to do to guarantee that an advanced AI will be safe. There is no checklist yet.
The best we can do in this situation is to ask large AI developers for more transparency. As the California Report repeatedly stresses, the key benefit of transparency is that it aligns the AI developers' incentives with public welfare. We may not know now exactly what safety measures we'll want developers to be taking in three years,[2] but we'll definitely want to know then what measures they actually are taking. That way, we can assess whether their safety practices are adequate in light of all the evidence we don't have yet but will have gained in the future. We can determine what practices are standard across the AI industry and create a race to the top on safety by applauding leaders and pressuring laggards to improve. And if needed, we can write more detailed, prescriptive regulation from a position of greater knowledge than we have now. None of this is possible without basic transparency.
To their credit, many of our leading AI developers are already quite transparent. They've chosen to share far more information about their safety protocols and their cutting edge models than they have to for purely commercial reasons. This is great, but we shouldn't keep relying on AI companies' good will to keep the public informed of critical safety information. It's time to make transparency non-optional.
One might question whether Crayola should have to test its products and publish safety reports. Maybe the toy industry is over-regulated, and it would be better on balance if we scrapped even the most basic disclosure requirements. For my part, I do not think this is the case, but my higher level response to someone arguing along these lines is that they're objecting to all consumer safety regulation, not to AI regulation in particular. If we want to have that debate, we certainly can, but we should notice that the opponent of AI regulation is now making a much more sweeping claim than they were before.
Why should we be uncertain about what safety measures will be appropriate in three years? What might have happened by then to invalidate our current best guesses? (1) The way we train and deploy frontier AI might have changed dramatically by then. Eg, we might switch from discrete deployments to continuous learning. (2) The background risk environment may have changed. Eg, we may have imposed universal nucleic acid synthesis screening, reducing the need for stringent bio-misuse mitigations. (3) Alignment and control techniques that aren't feasible now might have become feasible, or the inverse. Eg, chain-of-thought monitoring looks like an effective strategy for controlling AIs now, but it could soon become useless if companies train their models to reason less legibly.
My impression is that SB 53 is almost good, but there is one bit that makes me worry it could end up being very harmful. Specifically this part:
By 2030, we may already be dead, or we may have turned over control to AI to an extent that puts us on an inevitable track toward death, or perhaps death is not yet inevitable but AI has nonetheless changed the political landscape to such an extent that this rule is meaningless.
My concern is that a law requiring audits by 2030 may prevent us from getting a law that requires audits sooner than that. I would much rather require audits by 2026, or 2027 at the latest.
How likely is it that this law would prevent us from getting safety regulations that come into effect sooner? That seems like an important question to answer to determine whether SB 53 is net positive.
Thanks for your comments, Michael.
The section of SB 53 that talks about external auditing was added to the bill by the Assembly Committee on Privacy and Consumer Protection. They wrote that the purpose of the four year grace period before auditing becomes mandatory is "to give time for the nascent industry of AI auditors to grow." Now, I don't think the auditing industry needs that much time. Deloitte has already helped Anthropic to audit Claude 4, and I suspect the other Big Four firms will get involved soon. They can pull in AI experts from RAND, METR, or AISI if they need to.
It's worth noting that even if the relevant parts of SB 53 pass unamended, some other state or the federal government could still pass an external auditing requirement that kicks in before 2030. I don't see an obvious reason why passing SB 53 makes it less likely that such a law passes in a jurisdiction other than CA.
The solution to the problem of AI developers choosing lax auditors is § 22757.16.b. The bill says that if an auditor is negligent or "knowingly include[s] a material misrepresentation or omit[s] a material fact" in their report to the AG, they're civilly liable for up to $10k in fines. Now, I think that penalty figure is probably too low, but if you raise it enough, it will solve the incentive problem. Auditors won't go easy on AI developers because they know they can be fined if they do.
CalCompute's effect might indeed be somewhat accelerationist. FWIW, all that SB 53 does is appoint a board to explore setting up CalCompute. The bill does not appropriate funds for a new cluster. Given how many hurdles CalCompute would still have to clear even if SB 53 passed, I don't think it should drive our net assessment of whether SB 53 is good or bad.
I was thinking policy-makers might see that there's already an auditing requirement and decide not to impose another auditing requirement because it doesn't seem important anymore. (Even though on my view it would still be important to get a requirement that comes into effect sooner.) I don't know whether policy-makers are likely to think that way, it just seems like a possibility that's worthy of concern.
Two more minor concerns:
I expect this part is weakly net harmful because it shortens timelines a bit by increasing demand for AI. But it's not a big enough deal for me to care too much about.
My second concern is that auditors will end up being toothless because companies will look for auditors who will give them a passing grade even though they don't deserve to pass. I don't know how to fix this*, and I still think mandating audits is better than not mandating audits.
*Well I have some unrealistic ideas about how to fix it, like "any group conducting audits must be approved by the Machine Intelligence Research Institute."