Effective Altruism Forum
EA Forum

Hide table of contents
by Oliver Z, Dan H
5 min read 1

35

AI safetyExistential riskPolicyAI governancePublic communication on AI safety
Frontpage
This is a linkpost for https://newsletter.safe.ai/p/ai-safety-newsletter-3

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

---

Policy Proposals for AI Safety

Critical industries rely on the government to protect consumer safety. The FAA approves new airplane designs, the FDA tests new drugs, and the SEC and CFPB regulate risky financial instruments. Currently, there is no analogous set of regulations for AI safety.  

This could soon change. President Biden and other members of Congress have recently been vocal about the risks of artificial intelligence and the need for policy solutions.

From guiding principles to enforceable laws. Previous work on AI policy such as the White House Blueprint for an AI Bill of Rights and the NIST AI Risk Management Framework has articulated guiding principles like interpretability, robustness, and privacy. But these recommendations are not enforceable – AI developers can simply choose to ignore them.

A solution with more teeth could be on its way. Axios reports that Senator Chuck Schumer has been circulating a draft framework for AI governance among experts over the last several weeks. To help inform policy making efforts, the Department of Commerce has issued a request for comments on how to effectively regulate AI.

The European Union debates narrow vs. general AI regulation. In Europe, policy conversations are centering around the EU AI Act. The Act focuses on eight “high-risk” applications of AI, including hiring, biometrics, and criminal justice. But the rise of general purpose AI systems like ChatGPT calls into question the wisdom of regulating only a handful of specific applications.

An open letter signed by over 50 AI experts, including CAIS’s director, argues that the Act should also govern general purpose AI systems, holding AI developers liable for harm caused by their systems. Several members from all political blocs of the EU parliament have publicly agreed that rules are necessary for “powerful General Purpose AI systems that can be easily adapted to a multitude of purposes.”

Specific policy proposals for AI safety. With politicians promising that AI regulation is coming, the key question is which proposals they will choose to carry forward into law. Here is a brief compilation of several recent sets of policy proposals:

  1. Create an AI regulatory body. A national agency focused on AI could set and enforce standards, monitor the development of powerful new models, investigate AI failures, and publish information about how to develop AI safely.
  2. Clarify legal liability for AI harm. When ChatGPT falsely accused a law professor of sexual harassment, legal scholars argued that OpenAI should face legal liability for libel and defamatory statements made by its models. Others propose AI developers should be strictly liable for harm caused by AI, but questions remain about where to draw the line between an unsafe product versus deliberate misuse.
  3. Compute governance. AI regulations could be automatically enforced by software built into the cutting edge computer chips used to train AI systems.
  4. Nuclear command and control. Despite persistent problems with the security and reliability of AI systems, some military analysts advocate using AI in the process of launching nuclear weapons. A simple proposal: Don’t give AI influence over nuclear command and control.
  5. Fund safety research. Organizations promoting work on AI safety such as NIST and NSF could use more funding from federal sources.

China proposes many AI regulations. Last week, China released its own set of AI regulations that go much further than current Western efforts. Under these regulations, AI developers would be required to conduct security assessments, protect user data privacy, prevent impersonation via generative AI, and take legal responsibility for harm caused by their models. While some have opposed safety measures on the grounds that they would slow progress and allow countries like China to catch up, these regulations provide an opportunity for cooperation and taking precautions without fears of competitive loss.

---

Competitive Pressures in AI Development

The AI developer landscape is changing quickly—one organization is shifting its strategy, two organizations are merging, and one organization is emerging, all in order to adapt to competitive pressures.

Anthropic shifts its focus to products. Anthropic was originally founded by former OpenAI employees who were concerned about OpenAI’s product-focused direction coming at the expense of safety. More recently, however, Anthropic has been influenced by competitive pressures and has shifted their focus toward products. TechCrunch obtained a pitch deck for Anthropic’s Series C fundraising round, which includes plans to build a model that is “10 times more capable than today’s most powerful AI” that will require “a billion dollars in spending over the next 18 months.” 

Elon Musk will likely launch a new AI company. Elon Musk is apparently launching a new artificial intelligence start-up to compete with OpenAI. Musk has already “secured thousands of high-powered GPU processors from Nvidia” and begun recruiting engineers from top AI labs.” 

While Musk now seeks to compete with OpenAI, he was originally one of OpenAI’s co-founders. He was allegedly inspired to start the company due to concerns that Google co-founder Larry Page was not taking AI safety seriously enough. Many years ago, when Musk brought up AI safety concerns to Page, the Google co-founder allegedly responded by calling Musk a “speciesist.” The emergence of a new major AI developer will likely increase competitive pressures.

Google Brain and DeepMind merge into Google DeepMind. Google announced a merger between Google Brain and DeepMind, two major AI developers. This restructuring was likely spurred by products from Google’s competitors, OpenAI and Microsoft. Google’s announcement stated this was to make them move “faster” and to “accelerate” AI development. The new organization, Google DeepMind, will be run by Demis Hassabis (former CEO of DeepMind).

From an AI safety perspective, the effect of this decision will largely be determined by whether the new organization will have a safety culture more similar to DeepMind or Google Brain. DeepMind leadership has a much stronger history and track record of being concerned about AI safety, whereas Google Brain has never had a safety team. If DeepMind leadership has more influence over the new organization, this might represent a win for AI safety. If not, then society has essentially lost one of the (relatively) responsible leading AI developers. 

Competitive pressures shape the AI landscape. These news updates relate to a larger trend in the AI development landscape: the role of competitive pressures. Building safer AI systems can incur a substantial cost. Competitive pressures can make it difficult for AI developers—even ones that care about safety—to act in ways that put safety first. The pressures to race ahead can cause actors to make sacrifices, especially when there are tradeoffs between safety and competitiveness.

---

See also: CAIS website, CAIS twitter, A technical safety research newsletter

35

0
0

Reactions

0
0

More posts like this

Comments1


Sorted by
New & upvoted
Click to highlight new comments since:
Vasco Grilo🔸4
0
0

Thanks for sharing, I have really liked the 1st 3 editions!

Reply
More from Oliver Z
157
Introducing the ML Safety Scholars Program
6 authors
· · 4m read
42
42
136
Announcing the Introduction to ML Safety Course
3 authors
· · 9m read
4
4
64
Announcing an Empirical AI Safety Program
3 authors
· · 2m read
7
7
View more
Curated and popular this week
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Dr Kassim
 ·  · 4m read
 · 
Hey everyone, I’ve been going through the EA Introductory Program, and I have to admit some of these ideas make sense, but others leave me with more questions than answers. I’m trying to wrap my head around certain core EA principles, and the more I think about them, the more I wonder: Am I misunderstanding, or are there blind spots in EA’s approach? I’d really love to hear what others think. Maybe you can help me clarify some of my doubts. Or maybe you share the same reservations? Let’s talk. Cause Prioritization. Does It Ignore Political and Social Reality? EA focuses on doing the most good per dollar, which makes sense in theory. But does it hold up when you apply it to real world contexts especially in countries like Uganda? Take malaria prevention. It’s a top EA cause because it’s highly cost effective $5,000 can save a life through bed nets (GiveWell, 2023). But what happens when government corruption or instability disrupts these programs? The Global Fund scandal in Uganda saw $1.6 million in malaria aid mismanaged (Global Fund Audit Report, 2016). If money isn’t reaching the people it’s meant to help, is it really the best use of resources? And what about leadership changes? Policies shift unpredictably here. A national animal welfare initiative I supported lost momentum when political priorities changed. How does EA factor in these uncertainties when prioritizing causes? It feels like EA assumes a stable world where money always achieves the intended impact. But what if that’s not the world we live in? Long termism. A Luxury When the Present Is in Crisis? I get why long termists argue that future people matter. But should we really prioritize them over people suffering today? Long termism tells us that existential risks like AI could wipe out trillions of future lives. But in Uganda, we’re losing lives now—1,500+ die from rabies annually (WHO, 2021), and 41% of children suffer from stunting due to malnutrition (UNICEF, 2022). These are preventable d
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
Recent opportunities in AI safety
57
Quick nudge to apply to the LTFF grant round (closing on Saturday)
calebp
· · 1m read
7
7
35
AIS Netherlands is looking for a Founding Executive Director (EOI form)
7 authors
· · 5m read
4
4
5
Nationwide Action Workshop: Contact Congress about AI Safety!
Felix De Simone
· · 1m read
0
0
View more