Comments
Nice quality user research!
Consider adding a TL;DR including your calls to action - looking for collaborators and ideas for future projects, which I think will interest people
Nice quality user research!
Consider adding a TL;DR including your calls to action - looking for collaborators and ideas for future projects, which I think will interest people
Thanks for doing this!
The strength of the arguments is very mixed as you say. If you wanted to find good arguments, I think it might have been better to focus on people with more exposure to the arguments. But knowing more about where a diverse set of EAs is at in terms of persuasion is good too, especially for AI safety community builders.
This solidifies a conclusion for me: when talking about AI risk, the best/most rigorous resources aren't the ones which are most widely shared/recommended (rigorous resources are e.g. Ajeya Cotra's report on AI timelines, Carlsmith's report on power-seeking AI, Superintelligence by Bostrom or (to a lesser extent) Human Compatible by Russell).
Those might still not be satisfying to skeptics, but are probably more satisfying than " short stories by Eliezer Yudkowsky" (though one can take an alternative angle: skeptics wouldn't bother reading a >100 page report, and I think the complaint that it's all short stories by Yudkowsky comes from the fact that that's what people actually read).
Additionally, there appears to be a perception that AI safety research is limited to MIRI & related organisations, which definitely doesn't reflect the state of the field—but from the outside this multipolarity might be hard to discover (outgroup-ish homogeneity bias strikes again).
Personally I find Human Compatible the best resource of the ones you mentioned. If it were just the others I'd be less bought into taking AI risk seriously.
I agree that it occupies a spot on the layperson-understandability/rigor Pareto-frontier, but that frontier is large and the other things I mentioned are at other points.
Indeed. It just felt more grounded in reality to me than the other resources which may appeal more to us laypeople and the non laypeople prefer more speculative and abstract material.
Seconded/thirded on Human Compatible being near that frontier. I did find its ending 'overly optimistic' in the sense of framing it like 'but lo, there is a solution!' while other similar resources like Superintelligence and especially The Alignment Problem seem more nuanced in presenting uncertain proposals for paths forward not as oven-ready but preliminary and speculative.
I'm not quite sure I read the first two paragraphs correctly. Are you saying that Cotra, Carlsmith and Bostrom are the best resources but they are not widely recommended? And people mostly read short posts, like those by Eliezer, and those are accessible but might not have the right angle for skeptics?
Yes, I think that's a fair assessment of what I was saying.
Maybe I should have said that they're not widely recommended enough on the margin, and that there are surely many other good & rigorous-ish explanations of the problem out there.
I'm also always disappointed when I meet EAs who aren't deep into AI safety but curious, and the only things they have read is the List of Lethalities & the Death with Dignity post :-/ (which are maybe true but definitely not good introductions to the state of the field!)
As a friendly suggestion, I think the first paragraph of your original comment would be less confusing if the parenthetical clause immediately followed "the best/most rigorous resources". This would make it clear to the reader that Cotra, Carlsmith, et al are offered as examples of best/most rigorous resources, rather than as examples of resources that are widely shared/recommended.
There are short stories by Yudkowsky? All I ever encountered were thousands-of-pages-long sequences of blog posts (which I hence did not read, as you suggest).
Lots of it is here
If you're unconvinced about AI danger and you tell me what specifically are your cruxes, I might be able to connect you with Yudkowskian short stories that address your concerns.
The ones which come immediately to mind are:
I think I would have found Ajeya's cold takes guest post on "Why AI alignment could be hard with modern deep learning" persuasive back when I was skeptical. It is pretty short. I think the reason why I didn't find what you call "short stories by Eliezer Yudkowsky" persuasive was because they tended to not use concepts / terms from ML. I guess even stuff like orthogonality thesis and instrumental convergence thesis was not that convincing to me on a gut level even though I didn't disagree with the actual argument for them because I had the intuition that whether misaligned AI was a big deal depended on details of how ML actually worked, which I didn't know. To me back then it looked like most people I knew with much more knowledge of ML were not concerned about AI x-risk so probably it wasn't a big deal.
Thanks! I thought this was great. I really like the goals of fostering a more in-depth discussion and understanding skeptics' viewpoints.
I'm not sure about modeling a follow-up project on Skeptical Science, which is intended (in large part) to rebut misinformation about climate change. There's essentially consensus in the scientific community that human beings are causing climate change, so such a project seems appropriate.
If the answer to either of these questions is "no," then maybe more foundational work (in the vein of this interview project) should be done first. I like your idea of using double crux interviews to determine which arguments are the most important.
One other idea would be to invite some prominent skeptics and proponents to synthesize the best of their arguments and debate them, live or in writing, with an emphasis on clear, jargon-free language (maybe such a project already exists?).
Is there an equally high level of expert consensus on the existential risks posed by AI?
There isn't. I think a strange but true and important fact about the problem is that it just isn't a field of study in the same way e.g. climate science is — as argued in this Cold Takes post. So it's unclear who the relevant "experts" should be. Technical AI researchers are maybe the best choice, but they're still not a good one; they're in the business of making progress locally, not forecasting what progress will be globally and what effects that will have.
Thanks! I agree - AI risk is at a much earlier stage of development as a field. Even as the field develops and experts can be identified, I would not expect a very high degree of consensus. Expert consensus is more achievable for existential risks such as climate science and asteroid impacts that can be mathematically modeled with high historical accuracy - there's less to dispute on empirical / logical grounds.
A campaign to educate skeptics seems appropriate for a mature field with high consensus, whereas constructively engaging skeptics supports the advancement of a nascent field with low consensus.
One other idea would be to invite some prominent skeptics and proponents to synthesize the best of their arguments and debate them, live or in writing, with an emphasis on clear, jargon-free language (maybe such a project already exists?).
This is a pretty good idea!
We could use kialo, a web app, to map those points and their counterarguments
Do you have a sense of which argument(s) were most prevalent and which were most frequently the interviewees crux?
It would also be useful to get a sense of which arguments are only common among those with minimal ML/safety engagement. If basic AI safety engagement reduces the appeal of a certain argument, then there's little need for further work on messaging in that area.
Do you think the wording "Have you heard about the concept of existential risk from Advanced AI? Do you think the risk is small or negligible, and that advanced AI safety concerns are overblown? " might have biased your sample in some way?
E.g. I can imagine people who are very worried about alignment but don't think current approaches are tractable.
In case "I can imagine" was literal, then let me serve as proof-of-concept, as a person who thinks the risk is high but there's nothing we can do about it short of a major upheaval of the culture of the entire developed world.
The sample is biased in many ways: Because of the places where I recruited, interviews that didn't work out because of timezone difference, people who responded too late, etc. I also started recruiting on Reddit and then dropped that in favour of Facebook.
So this should not be used as a representative sample, rather it's an attempt to get a wide variety of arguments.
I did interview some people who are worried about alignment but don't think current approaches are tractable. And quite a few people who are worried about alignment but don't think it should get more resources.
Referring to my two basic questions listed at the top of the post, I had a lot of people say "yes" to (1). So they are worried about alignment. I originally planned to provide statistics on agreement / disagreement on questions 1/2 but it turned out that it's not possible to make a clear distinction between the two questions - most people, when discussing (2) in detail, kept referring back to (1) in complex ways.
Once again, I’ll say that a study which analyzed the persuasion psychology/sociology of “x-risk from AI” (e.g., what lines of argument are most persuasive to what audiences, what’s the “minimal distance / max speed” people are willing to go from “what is AI risk” to “AI risk is persuasive,” how important is expert statements vs. theoretical arguments, what is the role of fiction in magnifying or undermining AI x-risk fears) seems like it would be quite valuable.
Although I’ve never held important roles or tried to persuade important people, in my conversations with peers I have found it difficult to walk the line between “sounding obsessed with AI x-risk” and “under emphasizing the risk,” because I just don’t have a good sense of how fast I can go from someone being unsure of whether AGI/superintelligence is even possible to “AI x-risk is >10% this century.”
Just added a link to the "A-Team is already working on this" section of this post to my "(Even) More EAs Should Try AI Safety Technical Research," where I observe that people who disagree with basically every other claim in this post still don't work on AI safety because of this (flawed) perception.
I'd be very interested to see how many of them have changed their minds now.
TL;DR I interviewed 22 EAs who are skeptical about AI safety. My belief is that there is demand for better communication of AI safety arguments. I have several project ideas aimed in that direction, and am open to meeting potential collaborators (more info at the end of the post).
I interviewed 22 EAs who are skeptical about existential risk from Artificial General Intelligence (AGI), or believe that it is overrated within EA. This post provides a comprehensive overview of their arguments. It can be used as a reference to design AI safety communication within EA, as a conversation starter, or as the starting point for further research.
In casual conversation with EAs over the past months, I found that many are skeptical of the importance of AI safety. Some have arguments that are quite well reasoned. Others are bringing arguments that have been convincingly refuted somewhere - but they simply did not encounter that resource, and stopped thinking about it.
It seems to me that the community would benefit from more in-depth discussion between proponents and skeptics of AI Safety focus. To facilitate more of that, I conducted interviews with 22 EAs who are skeptical about AGI risk. The interviews circled around two basic questions:
Only people who said no to (1) or yes to (2) were interviewed. The goal was to get a very broad overview of their arguments.
My goal was to better understand the viewpoints of my interview partners - not to engage in debate or convince anyone. That being said, I did bring some counterarguments to their position if that was helpful to gain better understanding.
The results are summarized in a qualitative fashion, making sure every argument is covered well enough. No attempt was made to quantify which arguments occured more or less often.
Most statements are direct quotes, slightly cleaned up. In some cases, when the interviewee spoke verbosely, I suggested a summarized version of their argument, and asked for their approval.
Importantly, the number of bullet points for each argument below does not indicate the prevalence of an argument. Sometimes, all bullet points correspond to a single interviewee and sometimes each bullet point is from a different person. Sub-points indicate direct follow-ups or clarifications from the same person.
Some interviewees brought arguments against their own position. These counterarguments are only mentioned if they are useful to illuminate the main point.
General longtermism arguments without relation to AI Safety were omitted.
Some of these arguments hint towards specific ways in which AI safety resources could be improved. Others might seem obviously wrong or contradictory in themselves, and some might even have factual errors.
However, I believe all of these arguments are useful data. I would suggest looking behind the argument and figuring out how each point hints at specific ways in which AI safety communication can be improved.
Also, I take responsibility for some of the arguments perhaps making more sense in the original interview than they do here in this post (taken out of context and re-arranged into bullet points).
Interview partners were recruited from the /r/EffectiveAltruism subreddit, from the EA Groups slack channel and the EA Germany slack channel, as well as the Effective Altruism Facebook group. The invitation text was roughly like this:
Have you heard about the concept of existential risk from Advanced AI? Do you think the risk is small or negligible, and that advanced AI safety concerns are overblown? I'm doing research into people's beliefs on AI risk. Looking to interview EAs who believe that AI safety gets too much attention and is overblown.
How much time each week do you spend on EA activities, including your high-impact career, reading, thinking and meeting EAs?
How much time did you spend, in total, reading / thinking / talking / listening about AI safety?
I greatly enjoyed having these conversations. My background is having studied AI Safety for about 150 hours throughout my EA career. I started from a position of believing in substantial existential AGI risk this century. Only a small number of arguments seemed convincing to me, and I have not meaningfully changed my own position through these conversations. I have, however, gained a much deeper appreciation of the variety of counterarguments that EAs tend to have.
Well, the fundamental piece behind justification of [AI safety] research is: a meaningful probability that something superintelligent is going to exist in the next decades... And I feel that this underlying thesis is motivated by some pieces of evidence, like success of modern ML algorithms (GPT-3)... But I feel this is not good evidence. The causality between "GPT-3 being good at mimicking speech" and "therefore superintelligence is closer than we think". I think that link is faulty because current algorithms all capture statistical relationships. This to me is a necessary condition to AGI but not sufficient. And the sufficient is not known yet. Evidence that statistical learning is very successful is not evidence that AGI is close.
[Interviewer: Wouldn't it be possible that we discover this missing piece quite soon?] You'd have to think about what your definition of general intelligence is. If you think about human intelligence, humans have many features that statistical learning doesn't have. For example, embodiment. We exist in a world that is outside of our control. By evolution we have found ways to continue to exist and the requirement of continual existence is to develop the ability to do statistical learning ourselves... but also that state learning is something that has sprouted off something else, and THAT something else is the fact that we were placed in an environment and left to run for a long time. So maybe the intelligence is a consequence of that long process...
[Interviewer: I don't quite understand. Would you say there is a specific factor in evolution that is missing in machine learning?] I think we need, in addition to statistical learning, to think "what else did evolution provide that together comprises intelligence?" And I do not think this is going to be as simple as designing a slightly better statistical learning algorithm and then all of the sudden it works. You can't just take today's AI and put it in a robot and say "now it's general."
Therefore... My main argument against the notion that superintelligent AI is a serious immediate risk is that there is something missing (and I think everyone would agree), and for me the progress in the things that we do have does not count as evidence that we are getting closer to the thing that's missing. If anything we are no closer to the thing that's missing than we were 5 years ago. I think the missing pieces are mandatory, and we don't even know what they are.
One could also argue that I'm moving the goalposts, but also I suppose... I feel like somebody who is worried about AGI risk now would have a lot in common with someone who is worried about AGI risk 30 years ago when Deep Blue beat Kasparov. I feel the actuality of both situations is not so different. Not much has changed in AI except there are better statistical learning algorithms. Not enough to cause concern.
Maybe researching towards "what is the missing link towards AGI? How would we get there, is it even possible" might be more impactful than saying "Suppose we have a hypothetical superintelligence, how do we solve the control problem?"
I truly believe that there is no impetus for growth without motivation. AI does not have an existential reason, like evolution had. This constrains its growth. With computers, there is no overarching goal to their evolution. Without an existential goal for something to progress towards, there is no reason for it to progress.
[Interviewer: Could AI training algorithms provide such a goal?] I just don't know. Because we have been running training algorithms for decades (rudimentary at first, now more advanced) but they are still idiots. They don't understand what they are doing. They may have real world applications, but it doesn't know that it's identifying cancer... it is identifying a pattern in a bunch of pixels.
[Interviewer: But what is it that you think an AI will never be able to do?] Verbal and nonverbal communication with humans. [Explanation about how their job is in medicine and that requires very good communication with the patient]
[Interviewer: Anything else?] You can ask the computer any number of things, but if you ask it a complex or nuanced question its not going to give you an answer. *[Interviewer: What would such a question be?] e.g. "What timeframe do you think AGI is going to come about?" - they might regurgitate some answer from an expert on an internet, its never going to come up with a reasonable prediction on its own. Because it doesn't understand that this is a complex question involving a bunch of fields.
The world is just so complicated - if you look at a weather model that predicts just the weather, which is much more simple than the world in total, to somewhat predict it a day ahead or two that's possible, but further ahead it often fails. Its a first-order chaotic system. But the world around us is a second-order chaotic system. I dont see how you could suddenly have an instance to destroy humanity accidentially or otherwise. The world is just way too complicated - an AI couldn't just influence it easily.
I don’t think superintellience will be all that powerful. Outcomes in the real world are dominated by fundamentally unpredictable factors, which places limits on the utility of intelligence.
The speed of AI development so far does not warrant even human level AGI any time soon. What really underscores that is there is a rate-limiting effect of neuroscience and cognitive science. We need to first understand human intelligence before we build AGI. AGI will not come before we have a good understanding of what makes the brain work.
In terms of how we understand intelligence, it is all biological and we don't have any idea how it really works. We have a semblance of an idea, but when we are working with AI, we are working in silicon and it is all digital, whereas biological intelligence is more like stochastic and random and probabilistic. There are millions of impulses feeding into billions of neurons and it somehow makes up a person or a thought. And we are so far away from understanding what is actually going on - the amount of processing in silicon needed to process a single second of brain activity is insane. And how do we translate this into the transistors? It is a very big problem that I don't think we have even begun to grasp.
I think future AI systems will be more general than current ones, but I don't think they will be able to do "almost anything." This strikes me as inconsistent with how AI systems, at least at the moment, actually work (implement fancy nonlinear functions of training data).
I'm not totally bought into the idea that as soon as we have an AI a bit smarter than us, it's automatically going to become a billion times smarter than us from recursive self-improvement. We have not seen any evidence of that. We train AI to get better at one thing and it gets better at this thing, but AI doesn't then take the reins and make itself better at the other things.
Self-improvement is doubtful. I can't think of an approach to machine learning in which this kind of behavior would be realistic. How would you possibly train an AI to do something like this? Wouldn't the runtime be much too slow to expect any sort of fast takeoff? Why would we give an AI access to its own code, and if we didn't intend to, I don't understand what kind of training data would possibly let it see positive consequences to this kind of behavior.
[Interviewer: Could a brain be simulated and then improve itself inside the computer?] I suspect this would run significantly more slowly than a real human brain. More fundamentally, I am not convinced a [digital] human even with 1000 years of life could easily improve themselves. We don't really know enough about the brain for that to be remotely feasible, and I don't see any reason an AI should either (unless it was already superintelligent).
I don’t believe fast takeoff is plausible. Creation of advanced AI will be bottlenecked by real-word, physical resources that will require human cooperation and make it impossible for AGI to simply invent itself overnight.
I don't believe exponential takeoff is possible. It would require the AGI to find a way to gather resources that completely supercedes the way resources are allocated in our society. I don't see a risk where the AGI, in an exponential takeoff scenario, could get the kind of money it would need to really expand its resources. So even from an infrastructure perspective, it will be impossible. For example: Consider the current safety measures for credit card fraud online. We assume in an exponential takeoff scenario that the AI would be able to hack bank systems and override these automated controls. Perhaps it could hack a lot, but a lot of real-world hacking involves social engineering. It would need to call someone and convince that person - or hire an actor to do that. This sounds extremely far fetched to me.
I don't think that silicon will ever get to the point where it will have the efficiency to run something on the level of complexity of a human mind without incredible amounts of power. Once that happens, I really doubt it could get out of control just because of the power constraints.
Well, the silicon scaling laws with silicon are running out. They're running into physical boundaries of how small and power-efficient transistors can get. So the next generation of GPUs is just trying to dump twice as much wattage on them. The transistors are getting smaller but barely. Eventually we get to the point where quantum tunneling is going to make silicon advancement super difficult.
Yes we might be able to get AGI with megawatts of power on a supercomputer.... but
An AI would also be limited by the physical manifacturing speed of new microchips.
Yes we might be able to get AGI with megawatts of power on a supercomputer.... but could you afford to use that when you could be using the computer power to do literally anything else?
There are material science constraints to building faster computers that are difficult to circumvent.
Surely you can affect the world without having a physical body - but many things you cannot affect that way. Surely you can influence everything on the internet, but I fail to see how an AGI could, for example, destroy my desk. Yes I know there are robots, but I would think there will be possibilities to stop it as soon as we realize it's going awry.
The idea that all you need is internet access is often used to support the idea of an AGI expanding their resources. But in reality you need a physical body or at least agents who have that.
It's nice to talk about the paperclip problem, but we don't have a mechanism to describe scientifically how a machine would directly convert matter into processing power. Entirely within the realm of sci-fi. Right now, such a machine would only steal people's money, crash Netflix... But it could not affect the real world, so it's in a way less dangerous than a person with a gun.
The AI will absorb some human morality from studying the world - they [human morals] will not be peeled out! There will be some alignment by default because we train it in material obtained from humans.
If it becomes superintelligent, it will probably get some amount of wisdom, and it will think about morality and will probably understand the value of life and of its own existence and why it should not exterminate us.
I don't see why it would exterminate humanity. I agree it can prevent itself from being switching off, but I don't think it will try to exterminate humanity if it has at least a little wisdom.
All the amazing people working on AI Safety now, they will be able to prevent this existential threat. I have a lot of faith in them. These research institutes are very strong and they are working so hard on this. Even if other fields have more resources and more people, I don't think those resources & people are as competent as those in the AI safety field in many cases.
Yes, I think AI safety is overrated within EA. You have so many amazing people working on AI safety. I think they already have enough people and resources. I have met some of these people and they are truly amazing. Of course these people wouldn't do any more good if they did another topic because this is the topic they are expert on, so their resources are in the right place. And if other people want to go into AI Safety too, then I say go for it [because then they will be working from their strengths].
We do not have factual evidence that AI kills people, and certainly not a large number of people. AI has not eradicated humanity yet. And I doubt it will ever happen, simply because previous predictions were wrong so often.
I would say this [meaning AGI existential risk scenarios] is an instance of Pascal's mugging, as we don't have enough evidence.
The difference between longtermists and non-longtermists is their degree of willingness to put trust in expert opinion.
A lot of AI writers like Yudkowsky and maybe Scott Alexander, they throw a lot of numbers around of the likelihood of AI. But that's just the reflection of someone's belief, it is not the result of actual research.
What I'm really missing is more concrete data, especially on examples of AI that have gone awry. I haven't seen enough examples from researchers being like "this is the kind of thing it has done in the past, and these are some examples of how it could really go wrong in the future". At the same time, I do believe some of the catastrophe scenarios, but I also have a big emotional thing telling me that maybe people will do the right thing and we don't need to invest that much resources into safety. This comes perhaps from a lack of communication or data from people who are doing this research. I heard lots of big arguments, but I want more specifics: Which AI things already went wrong? What unexpected results did people get with GPT-3 and DALL-E? I would also like to hear more about DeepBrain.
I wonder if I have a basic misunderstanding of EA because as it sounds from an outsider point of view, if it is EA and trying to find good opportunities for philantropy to have the most effect then it does seem like focus on AI safety is overrated - there is no tangible evidence I have seen that AGI is possible to create or even that in our present-day world there is evidence for its possibility. I haven't seen any examples that lead me to think that it could happen. If somebody showed me a nice peer-reviewed research paper and trials that said "oh yeah we have this preliminary model for an AGI" then I would be a little bit more spooked but its something i have never seen but I do see people starving.
I would like to suggest Skeptical Science, which is a library of climate change arguments, as a role model for how AI safety material can be made accessible to the general public. In particular, I like that the website offers arguments in several levels of difficulty. I would like to start something like this and am looking for collaborators (see below).
There could be an AI safety "support hotline", allowing EAs who are curious and not involved with the field to ask questions and get referred to the right resources for them. Something similar, but for people who are already considering moving into the field, is AI Safety Support.
It would be interesting to conduct these interviews in more detail using the Double Crux technique, bringing in lots of counterarguments and then really uncovering the key cruxes of interview partners.
I am looking for ways to improve the AI safety outreach and discourse. Either by getting involved in an existing project or launching something new. Send me a message if you're interested to collaborate, would like help with your project, or would just like to bounce ideas around.
I think it would be interesting to have various groups (e.g. EAs who are skeptical vs worried about AI risk) rank these arguments and see how their lists of the top ones compare.