Effective Altruism Forum
EA Forum

Hide table of contents
Comment Permalink
Peter Berggren2
2
1

As an aside, the idea that we should prioritize optics over intellectually honest exploration of the epistemic landscape is deeply harmful to effective altruism as a whole.

Reply
See in context
by Peter Berggren
3 min read 19

0

AI safetyExistential riskPhilosophyBounty (open)
Frontpage

UPDATE: The phrasing of this post has caused some confusion. I would like to say that I am not at all confident as to whether the people in question have written up their object-level arguments before, and do not mean to imply either that they have or that they haven't. All I'm saying is, in my examination of their work, I have yet to find them. By offering this bounty, I do not mean to devalue the time of anyone by insisting on a new, extensive response; linking to a pre-existing object-level argument, as Torres did, would be more than sufficient. I'm mostly doing this for my own sake, so that I can familiarize myself with their object-level arguments.

I am prepared to pay out anywhere between $20 and $100 to AI ethicists of the DAIR/"Stochastic Parrots" school of thought if they provide their object-level arguments against the idea that preventing AI from killing everyone is a real and important issue. This pay will depend on their notability within AI ethics, as well as the clarity and persuasiveness of their arguments.

Conditions for the bounty

  1. The bounty must be claimed by an AI ethicist of the DAIR/"Stochastic Parrots" school of thought. Ethicists from other schools of thought (such as the "what if self-driving cars face trolley problems" school of thought) may be given bounties on a case-by-case basis, but probably not. Any member of DAIR or coauthor of the "Stochastic Parrots" paper counts for this, but people outside of these specific circles may qualify at my discretion, if I believe that their intellectual output is similar to or connected with DAIR or the "Stochastic Parrots" coauthors.
  2. The arguments provided by the claimant must be posted publicly, ideally in the comment section of this thread (or in the comment section of the corresponding LessWrong thread: https://www.lesswrong.com/posts/uTRafHCcjNfbAByyo/bounty-available-ai-ethicists-what-are-your-object-level).
  3. The arguments provided by the claimant must be object-level. This means that they must discuss concrete subjects specific to the issues at hand. This is in contrast to meta-level arguments, which focus on facts about the question (rather than about the issues it addresses), such as difficulties involved in future prediction, the cultural milieu of contemporary AI notkilleveryoneism, the framing of my questions, etc. Note that I have nothing against meta-level arguments; it's just that I've already seen plenty of meta-level arguments by AI ethicists against AI notkilleveryoneism, and I want to see some object-level arguments.
  4. The arguments provided by the claimant must be a good-faith summary of the claimant's actual object-level arguments against AI notkilleveryoneism. For example, "AI notkilleveryoneism is unimportant because paperclips are shiny" will not count, even if made by a qualifying claimant, even though it is object-level. I do not expect that I will need to invoke this condition, but I may do so at my discretion.
  5. The following AI ethicists will be presumptively considered valid claimants, and will fall into the most notable category (meaning that I will pay each of them the maximum $100 bounty assuming they follow all the terms of the bounty, unless I notice loophole abuse):
    Emily Bender
    Timnit Gebru
    Margaret Mitchell
    Melanie Mitchell

Note that there is no requirement for the arguments to change my mind, or even to be persuasive in the slightest. The only requirements are the above ones. If someone manages to abuse a loophole to get there, I will pay them the minimum bounty of $20, and then modify the rules for all future claimants to preempt this loophole.

So far, Emile Torres has already responded to the bounty (to my understanding, they believe that AI extinction risk is real, but that the field of AI notkilleveryoneism is broken beyond repair) by recommending their book as the place where their object-level arguments have been written. I will judge this as soon as I am able to check this book out from a library near me.

Note that I may need to close this bounty if I get too many claims from it, because I have a limited budget. All the more reason to get your arguments in here soon!

0

0
0

Reactions

0
0

More posts like this

Comments19


Sorted by
New & upvoted
Click to highlight new comments since:
Devin Kalish14
3
1

This is a quick PSA, Emile Torres does think “Preventing AI from killing everyone is a real and important issue”. The last time this was pointed out to you (that I’m aware of) you clarified that Torres’ disagreement was basically with longtermism. Please, pleeease clarify this in the post, it isn’t remotely how this challenge comes off and is borderline spreading misinformation, which is especially bad for important coalition building.

Reply
Peter Berggren3
0
0

I didn't mean to imply that Emile Torres didn't think that this was an extinction risk. I'm sorry that I misspoke on that part.

Reply
Devin Kalish4
1
0

Thanks for changing it.

Reply
cdkg10
5
1

Just FYI, many people in the AI ethics community find this kind of thing offensive. They have published their arguments in numerous scholarly venues and also in major newspapers and magazines and on places like Medium and Twitter. This kind of post is interpreted as "I'm too lazy to look at your work to find your arguments but I bet I can make you dance with small sums of money." Bad optics.

Reply
Geoffrey Miller10
5
1

Many people in the AI ethics community seem to find almost everything offensive. 

I've seen no evidence that they're worth engaging with on the topic of AI X-risk. They routinely caricature and demonize AI Safety researchers and EAs on social media, they seem not to have read any of the key works on AI X-risk, and their epistemic standards seem very weak.

While I admire Peter Breggren's attempt to entice them to engage in object-level objections to AI Safety research, I very much doubt that they will engage in any serious discussion of this issue.

Reply
cdkg4
0
0

Well, you can dismiss them and their argument if you want to — I personally don't find their arguments terribly convincing, and their social media presence is, as you point out, strident.

But one must be aware that to a surprising extent, they control the narrative about AI safety in academia and the mainstream media. So if one cares about making AI safety seem credible, it's worth engaging with them.

Reply
Peter Berggren1
0
0

Do they really control the narrative in the "mainstream media," though, or just a few far-left content mills that tend to get clicks by being really outrageous?

Reply
Peter Berggren3
0
0

I never denied that they have published their arguments in many places. I just can't find any such arguments that are object-level.

Reply
cdkg2
0
0

The object-level argument, as I understand it, is that worries about human-level AI capabilities of the sort that could pose an existential threat are based on a misunderstanding of what is going on under the hood in neural networks. This is what Bender means when she talks about "AI Hype". See for example her paper with Koller "Climbing towards NLU" for criticisms of attributing some kinds of mental states to neural networks.

Reply
Larks8
2
0

The paper you mentioned doesn't seem to discuss existential risk or AGI at all, so I don't see how it could represent the sort of object-level argument against existential risk that Peter is asking for.

Reply
cdkg4
2
1

Have a little imagination.

Suppose I am very worried that ghosts will steal things out of my closet. It seems like a perfectly object-level argument against my position to provide reasons for thinking that beliefs in paranormal activity are not scientifically respectable. This can be true even if the reasons provided do not mention ghosts.

People like Bender take themselves to be offering reasons for thinking that worries about AGI are not scientifically respectable. This can be true even if the reasons they provide do not mention AGI.

Note that I think Bender's arguments are bad. But I don't see what is so mysterious about them.

Reply
Peter Berggren3
1
1

It seems to me that, while the form/meaning distinction in this paper is certainly a fascinating one if your interests tend towards philosophy of language, this has very little to say about supposed inherent limitations of language models, and does not affect forecasts of existential risk.

Reply
cdkg4
0
0

Ok, let me spell it out explicitly. In a section called "Large LMs: Hype and analysis," the linked paper says that claims that LLM can "understand," "comprehend," and "know" are "gross overclaims." The paper supports this contention by pointing to evidence that "in fact, far from doing the “reasoning” ostensibly required to complete the tasks, [LLMs] were instead simply more effective at leveraging artifacts in the data than previous approaches."

Here is where the imagination comes in. Imagine that you think that all mental state attributions to artificial systems are confused in exactly this way. Imagine that you think that artificial neural nets can't reason at all. Now imagine that someone tells you that we should all be very concerned that misaligned superintelligent AI systems will destroy us.

Your response to that would be something like: it is deeply confused to think that superintelligent AI systems are something we need to worry about, and the people who are worried about them simply do not understand what is going on under the hood in machine learning models. Worries about existential risk from superintelligent AI stem from the same kind of confusion as attributing understanding to existing systems: the tendency of people who are not technically literate to anthropomorphize the systems they interact with.

Reply
Linch2
0
0

Imagine that you think that artificial neural nets can't reason at all.

Is this a real position that real living intelligent people actually hold, or is it just one of the funny contrarian philosopher beliefs that some philosophers like to around with for fun?

Reply
zchuang2
1
0

I think this is an actual position. It's the stochastic parrots argument no? Just a recent post by a cognitive scientist holds this belief.

Reply
Linch2
0
1

I don't think there were any factual claims in that article from a skim; entirely just normative claims and a few rhetorical question.

Reply
cdkg2
0
0

I think this is really the position of the stochastic parrots people, yes.

I don't think it's plausible, but I think it partly explains their relentless opposition to work on AI safety.

Reply
Peter Berggren2
2
1

As an aside, the idea that we should prioritize optics over intellectually honest exploration of the epistemic landscape is deeply harmful to effective altruism as a whole.

Reply
cdkg5
4
1

I didn't endorse that idea and, as an academic, obviously wouldn't. Also as an academic, I think paying people to explain themselves to you when you haven't first shown that you have read their work by e.g. explaining why you don't find the arguments they have already made in print convincing is not a shining exemplar of intellectually honest exploration.

Reply
Curated and popular this week
Ronen Bar
 ·  · 10m read
 · 
"Part one of our challenge is to solve the technical alignment problem, and that’s what everybody focuses on, but part two is: to whose values do you align the system once you’re capable of doing that, and that may turn out to be an even harder problem", Sam Altman, OpenAI CEO (Link).  In this post, I argue that: 1. "To whose values do you align the system" is a critically neglected space I termed “Moral Alignment.” Only a few organizations work for non-humans in this field, with a total budget of 4-5 million USD (not accounting for academic work). The scale of this space couldn’t be any bigger - the intersection between the most revolutionary technology ever and all sentient beings. While tractability remains uncertain, there is some promising positive evidence (See “The Tractability Open Question” section). 2. Given the first point, our movement must attract more resources, talent, and funding to address it. The goal is to value align AI with caring about all sentient beings: humans, animals, and potential future digital minds. In other words, I argue we should invest much more in promoting a sentient-centric AI. The problem What is Moral Alignment? AI alignment focuses on ensuring AI systems act according to human intentions, emphasizing controllability and corrigibility (adaptability to changing human preferences). However, traditional alignment often ignores the ethical implications for all sentient beings. Moral Alignment, as part of the broader AI alignment and AI safety spaces, is a field focused on the values we aim to instill in AI. I argue that our goal should be to ensure AI is a positive force for all sentient beings. Currently, as far as I know, no overarching organization, terms, or community unifies Moral Alignment (MA) as a field with a clear umbrella identity. While specific groups focus individually on animals, humans, or digital minds, such as AI for Animals, which does excellent community-building work around AI and animal welfare while
Max Taylor
 ·  · 9m read
 · 
Many thanks to Constance Li, Rachel Mason, Ronen Bar, Sam Tucker-Davis, and Yip Fai Tse for providing valuable feedback. This post does not necessarily reflect the views of my employer. Artificial General Intelligence (basically, ‘AI that is as good as, or better than, humans at most intellectual tasks’) seems increasingly likely to be developed in the next 5-10 years. As others have written, this has major implications for EA priorities, including animal advocacy, but it’s hard to know how this should shape our strategy. This post sets out a few starting points and I’m really interested in hearing others’ ideas, even if they’re very uncertain and half-baked. Is AGI coming in the next 5-10 years? This is very well covered elsewhere but basically it looks increasingly likely, e.g.: * The Metaculus and Manifold forecasting platforms predict we’ll see AGI in 2030 and 2031, respectively. * The heads of Anthropic and OpenAI think we’ll see it by 2027 and 2035, respectively. * A 2024 survey of AI researchers put a 50% chance of AGI by 2047, but this is 13 years earlier than predicted in the 2023 version of the survey. * These predictions seem feasible given the explosive rate of change we’ve been seeing in computing power available to models, algorithmic efficiencies, and actual model performance (e.g., look at how far Large Language Models and AI image generators have come just in the last three years). * Based on this, organisations (both new ones, like Forethought, and existing ones, like 80,000 Hours) are taking the prospect of near-term AGI increasingly seriously. What could AGI mean for animals? AGI’s implications for animals depend heavily on who controls the AGI models. For example: * AGI might be controlled by a handful of AI companies and/or governments, either in alliance or in competition. * For example, maybe two government-owned companies separately develop AGI then restrict others from developing it. * These actors’ use of AGI might be dr
Joris 🔸
 ·  · 5m read
 · 
Last week, I participated in Animal Advocacy Careers’ Impactful Policy Careers programme. Below I’m sharing some reflections on what was a really interesting week in Brussels! Please note I spent just one week there, so take it all with a grain of (CAP-subsidized) salt. Posts like this and this one are probably much more informative (and assume less context). I mainly wrote this to reflect on my time in Brussels (and I capped it at 2 hours, so it’s not a super polished draft). I’ll focus mostly on EU careers generally, less on (EU) animal welfare-related careers. Before I jump in, just a quick note about how I think AAC did something really cool here: they identified a relatively underexplored area where it’s relatively easy for animal advocates to find impactful roles, and then designed a programme to help these people better understand that area, meet stakeholders, and learn how to find roles. I also think the participants developed meaningful bonds, which could prove valuable over time. Thank you to the AAC team for hosting this! On EU careers generally * The EU has a surprisingly big influence over its citizens and the wider world for how neglected it came across to me. There’s many areas where countries have basically given a bunch (if not all) of their decision making power to the EU. And despite that, the EU policy making / politics bubble comes across as relatively neglected, with relatively little media coverage and a relatively small bureaucracy. * There’s quite a lot of pathways into the Brussels bubble, but all have different ToCs, demand different skill sets, and prefer different backgrounds. Dissecting these is hard, and time-intensive * For context, I have always been interested in “a career in policy/politics” – I now realize that’s kind of ridiculously broad. I’m happy to have gained some clarity on the differences between roles in Parliament, work at the Commission, the Council, lobbying, consultancy work, and think tanks. * The absorbe
Recent opportunities in AI safety
102
AI Moral Alignment: The Most Important Goal of Our Generation
Ronen Bar
· · 10m read
21
21
35
Center on Long-Term Risk: Summer Research Fellowship 2025
Center on Long-Term Risk
· · 1m read
0
0
28
Apply to the Cambridge ERA:AI Fellowship 2025
Harrison 🔸
· · 3m read
0
0
View more