Effective Altruism Forum
EA Forum

Hide table of contents
by Garrison
2 min read 3

29

AI safetyExistential riskPolicyPodcasts
Frontpage
This is a linkpost for https://garrisonlovely.substack.com/p/35-yoshua-bengio-on-why-ai-labs-are

This is exactly what I'm afraid of. That some human will build machines that are going to be - not just superior to us - but not attached to what we want, but what they want. And I think it's playing dice with humanity's future. I personally think this should be criminalized, like we criminalize cloning of humans. 

- Yoshua Bengio

My next guest is about as responsible as anybody for the state of AI capabilities today. But he's recently begun to wonder whether the field he spent his life helping build might lead to the end of the world. 

Following in the tradition of the Manhattan Project physicists who later opposed the hydrogen bomb, Dr. Yoshua Bengio started warning last year that advanced AI systems could drive humanity extinct. 

Dr. Bengio is the second-most cited living scientist and one of the so-called “Godfathers of deep learning.” He and the other “Godfathers,” Geoffrey Hinton and Yann LeCun, shared the 2018 Turing Award, computing’s Nobel prize.

In November, Dr. Bengio was commissioned to lead production of the first “State of the Science” report on the “capabilities and risks of frontier AI” — the first significant attempt to create something like the Intergovernmental Panel on Climate Change (IPCC) for AI.

I spoke with him last fall while reporting my cover story for Jacobin’s winter issue, “Can Humanity Survive AI?” Dr. Bengio made waves last May when he and Geoffrey Hinton began warning that advanced AI systems could drive humanity extinct.  

You can find The Most Interesting People I Know wherever you find podcasts and a full transcript here. If you'd like to support the show, sharing it with friends and reviewing it on Apple Podcasts is the most helpful! You can also subscribe to my Substack for updates on all my work. 

We discuss:

  • His background and what motivated him to work on AI
  • Whether there's evidence for existential risk (x-risk) from AI
  • How he initially thought about x-risk
  • Why he started worrying
  • How the machine learning community's thoughts on x-risk have changed over time
  • Why reading more on the topic made him more concerned
  • Why he thinks Google co-founder Larry Page’s AI aspirations should be criminalized
  • Why labs are trying to build artificial general intelligence (AGI)
  • The technical and social components of aligning AI systems
  • The why and how of universal, international regulations on AI
  • Why good regulations will help with all kinds of risks
  • Why loss of control doesn't need to be existential to be worth worrying about
  • How AI enables power concentration
  • Why he thinks the choice between AI ethics and safety is a false one
  • Capitalism and AI risk
  • The "dangerous race" between companies
  • Leading indicators of AGI
  • Why the way we train AI models creates risks

29

0
0

Reactions

0
0

More posts like this

Comments3


Sorted by
New & upvoted
Click to highlight new comments since:
Mayowa Osibodu3
0
0

Interesting podcast - I read the transcript.

My main takeaway was that building AI systems to have self-interest is dangerous because that has the potential to explicitly conflict with humanity's own interest, leading to a major existential risk with super-intelligent AIs.

I wonder if there's any advantage of self-interest in AI though. Is there any way self-interest could possibly make an AI more effective at accomplishing its goals? In biological entities, self-interest obviously helps with e.g. avoiding threats, seeking more favourable living conditions, etc. I wonder if this applies in a similar manner to AIs, or if self-interest in an AI is inconsequential at best.

 

I'm curious, what exactly is the worry with AGI development in e.g. Russia and China? Is the concern that they are somehow less invested in building safe AGI (which seems to strongly conflict with their own self-interest)?

Or is the concern that they could somehow build AGI which selectively harms people/countries of their choosing? In this latter case it seems to me that the problem is exclusively a human one, and isn't ethically different from concerns about super-lethal computer viruses or bio/nuclear weapons. It's not clear how this precise risk is specific to AI/AGI.

Reply
Garrison2
0
0

I think building AI systems with some level of autonomy/agency would make them much more useful, provided they are still aligned with the interests of their users/creators. There's already evidence that companies are moving in this direction based on the business case: https://jacobin.com/2024/01/can-humanity-survive-ai#:~:text=Further%2C%20academics%20and,is%20pretty%20good.%E2%80%9D

This isn't exactly the same as self-interest, though. I think a better analogy for this might be human domestication of animals for agriculture. It's not in the self-interest of a factory farmed chicken to be on a factory farm, but humans have power over which animals exist so we'll make sure there are lots of animals who serve our interests. AI systems will be selected for to the extent they serve the interests of people making and buying them.

RE international development: competition between states undercut arguments for domestic safety regulations/practices. These are exacerbated by beliefs that international rivals will behave less safely/responsibly, but you don't actually need to believe that to justify cutting corners domestically. If China or Russia built an AGI that was totally safe in the sense that it is aligned with its creators interests, that would be seen as a big threat by the US govt. 

If you think that building AGI is extremely dangerous no matter who does it, then having more well-resourced players in the space increases the overall risk. 

Reply
Vasco Grilo🔸2
0
0

Thanks for sharing, Garrison. I have read Yoshua's How Rogue AIs may Arise and FAQ on Catastrophic AI Risks, but I am still thinking annual extinction risk over the next 10 years is less than 10^-6. Do you know Yoshua's thoughts on the possibility of AI risk being quite low due to the continuity of potential harms? If deaths in an AI catastrophe follow a Pareto distribution (power law), which is a common assumption for tail risk, there is less than 10 % chance of such a catastrophe becoming 10 times as deadly, and this severely limits the probability of extreme outcomes. I also believe the tail distribution would decay faster than that of a Pareto distribution for very severe catastrophes, which makes my point stronger.

Reply
More from Garrison
88
What the Headlines Miss About the Latest Decision in the Musk vs. OpenAI Lawsuit
Garrison
· · 7m read
10
10
152
Why Did Elon Musk Just Offer to Buy Control of OpenAI for $100 Billion?
Garrison
· · 7m read
2
2
114
Is Deep Learning Actually Hitting a Wall? Evaluating Ilya Sutskever's Recent Claims
Garrison
· · 10m read
7
7
View more
Curated and popular this week
Paul Present
 ·  · 28m read
 · 
Note: I am not a malaria expert. This is my best-faith attempt at answering a question that was bothering me, but this field is a large and complex field, and I’ve almost certainly misunderstood something somewhere along the way. Summary While the world made incredible progress in reducing malaria cases from 2000 to 2015, the past 10 years have seen malaria cases stop declining and start rising. I investigated potential reasons behind this increase through reading the existing literature and looking at publicly available data, and I identified three key factors explaining the rise: 1. Population Growth: Africa's population has increased by approximately 75% since 2000. This alone explains most of the increase in absolute case numbers, while cases per capita have remained relatively flat since 2015. 2. Stagnant Funding: After rapid growth starting in 2000, funding for malaria prevention plateaued around 2010. 3. Insecticide Resistance: Mosquitoes have become increasingly resistant to the insecticides used in bednets over the past 20 years. This has made older models of bednets less effective, although they still have some effect. Newer models of bednets developed in response to insecticide resistance are more effective but still not widely deployed.  I very crudely estimate that without any of these factors, there would be 55% fewer malaria cases in the world than what we see today. I think all three of these factors are roughly equally important in explaining the difference.  Alternative explanations like removal of PFAS, climate change, or invasive mosquito species don't appear to be major contributors.  Overall this investigation made me more convinced that bednets are an effective global health intervention.  Introduction In 2015, malaria rates were down, and EAs were celebrating. Giving What We Can posted this incredible gif showing the decrease in malaria cases across Africa since 2000: Giving What We Can said that > The reduction in malaria has be
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
LewisBollard
 ·  · 8m read
 · 
> How the dismal science can help us end the dismal treatment of farm animals By Martin Gould ---------------------------------------- Note: This post was crossposted from the Open Philanthropy Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post. ---------------------------------------- This year we’ll be sharing a few notes from my colleagues on their areas of expertise. The first is from Martin. I’ll be back next month. - Lewis In 2024, Denmark announced plans to introduce the world’s first carbon tax on cow, sheep, and pig farming. Climate advocates celebrated, but animal advocates should be much more cautious. When Denmark’s Aarhus municipality tested a similar tax in 2022, beef purchases dropped by 40% while demand for chicken and pork increased. Beef is the most emissions-intensive meat, so carbon taxes hit it hardest — and Denmark’s policies don’t even cover chicken or fish. When the price of beef rises, consumers mostly shift to other meats like chicken. And replacing beef with chicken means more animals suffer in worse conditions — about 190 chickens are needed to match the meat from one cow, and chickens are raised in much worse conditions. It may be possible to design carbon taxes which avoid this outcome; a recent paper argues that a broad carbon tax would reduce all meat production (although it omits impacts on egg or dairy production). But with cows ten times more emissions-intensive than chicken per kilogram of meat, other governments may follow Denmark’s lead — focusing taxes on the highest emitters while ignoring the welfare implications. Beef is easily the most emissions-intensive meat, but also requires the fewest animals for a given amount. The graph shows climate emissions per tonne of meat on the right-hand side, and the number of animals needed to produce a kilogram of meat on the left. The fish “lives lost” number varies significantly by
Recent opportunities in AI safety
15
Apply to the Cambridge ERA:AI Fellowship 2025
Harrison 🔸
· · 3m read
0
0
57
Quick nudge to apply to the LTFF grant round (closing on Saturday)
calebp
· · 1m read
7
7
23
Last week to apply for the Futurekind AI Fellowship! (deadline: April 1)
Jay Luong
· · 1m read
0
0
View more