Note: This is a submission for the 2023 Open Philanthropy AI Worldviews contest, due May 31, 2023. It addresses Question 1: “What is the probability that AGI is developed by January 1, 2043?”
Overview
People tend to view harmful things as evil, and treat them as evil, to minimize their spread and impact. If enough people are hurt, betrayed, or outraged by AI applications, or lose their jobs, professional identity, and sense of purpose to AI, and/or become concerned about the existential risks of AI, then an intense public anti-AI backlash is likely to develop. That backlash could become a global, sustained, coordinated movement that morally stigmatizes AI researchers, AI companies, and AI funding sources. If that happens, then AGI is much less likely to develop by the year 2043. Negative public sentiment could be much more powerful in slowing AI than even the most draconian global regulations or formal moratorium, yet it is a neglected factor in most current AI timelines.
Introduction
The likelihood of AGI being developed by 2043 depends on two main factors: (1) how technically difficult it will be for AI researchers to make progress on AGI, and (2) how many resources – in terms of talent, funding, hardware, software, training data, etc. – are available for making that progress. Many experts’ ‘AI timelines’ for predicting AI development assume that AGI likelihood will be dominated by the first factor (technical difficulty), and assume that the second factor (available resources) will continue increasing.
In this essay I disagree with that assumption. The resources allocated into AI research, development, and deployment may be much more vulnerable to public outrage and anti-AI hatred than the current AI hype cycle suggests. Specifically, I argue that ongoing AI developments are likely to provoke a moral backlash against AI that will choke off many of the key resources for making further AI progress. This public backlash could deploy the ancient psychology of moral stigmatization against our most advanced information technologies. The backlash is likely to be global, sustained, passionate, and well-organized. It may start with grass-roots concerns among a few expert ‘AI doomers’, and among journalists concerned about narrow AI risks, but it is likely to become better-organized over time as anti-AI activists join together to fight an emerging existential threat to our species. (Note that this question of anti-AI backlash likelihood is largely orthogonal to the issues of whether AGI is possible, and whether AI alignment is possible.)
I’m not talking about a violent Butlerian Jihad. In the social media era, violence in the service of a social cause is almost always counter-productive, because it undermines the moral superiority and virtue-signaling strategies of righteous activists. (Indeed, a lot of ‘violence by activists’ turns out to be false flag operations funded by vested interests to discredit the activists that are fighting those vested interests.)
Rather, I’m talking about a non-violent anti-AI movement at the social, cultural, political, and economic levels. For such a movement to slow down the development of AGI by 2043 (relative to the current expectations of Open Philanthropy panelists judging this essay competition), it only has to arise sometime in the next 20 years, and to gather enough public, media, political, and/or investor support that it can handicap the AI industry’s progress towards AGI, in ways that have not yet been incorporated into most experts’ AI timelines.
An anti-AI backlash could include political, religious, ideological, and ethical objections to AI, sparked by vivid, outrageous, newsworthy failures of narrow AI systems. An anti-AI backlash could weakly delay AI research through government regulation. But it could strongly delay AI research through socio-cultural dynamics such as AI research becoming morally taboo, socially stigmatized, religiously condemned, and/or politically polarized. For example, if being an AI researcher became as publicly stigmatized as being a white nationalist, a eugenicist, a sexist, or a transphobe, then AI research would be largely abandoned by any researchers sensitive to social pressure, and AGI would not be developed for a long time.
Thus, we can invert the question of AGI timelines, and consider the possible timelines for an anti-AI backlash. Rather than asking ‘What is the likelihood that we’ll have AGI by 2043?’, we could ask ‘What is the likelihood that we will see an anti-AI backlash by 2043 – a backlash that is strong enough to slow down AGI development?’ I’d argue that the answer to this second question is fairly high. Even just in the last few weeks (as of May 31, 2023), we’ve seen a dramatic increase in public attention on AI risk, public and government concern about AI, and the beginnings of an anti-AI backlash on social media, such as Twitter.
(Note that in this essay I’m not taking a position on whether an anti-AI backlash would be a good thing or a bad thing; I’m just doing a preliminary analysis of how such a backlash could slow down AGI timelines.)
Triggers for an anti-AI backlash
The general public is already culturally primed for an anti-AI backlash. Ever since the novel Frankenstein (1818), we’ve had generations of science fiction novels, movies, TV shows, computer games, and other media portraying the dangers of creating artificial intelligence. Most living people in developed countries have been exposed to these cautionary tales. They’ve mostly seen 2001: A Space Odyssey, The Terminator, Ex Machina, Black Mirror, and Westworld. They’re often the first things that ordinary people think about when they think about AI. And most adults have first-hand experience of playing computer games against powerful (but narrow) AI, e.g. trying to win ‘Civilization’ on ‘god mode’ difficult level.
The triggers for an anti-AI backlash don’t need to create moral stigma from scratch. They just need to connect these latent cultural fears of AI to current real-world AI issues. I’ll call these issues ‘triggers’, and there are several kinds that seem quite likely to provoke moral stigmatization of AI within the next 20 years.
Trigger 1: Unemployment
People get pretty upset when they lose their jobs. The closer we get to AGI, the more job losses we’ll see. And, for any ‘new jobs’ that open up due to increased economic activity, AI systems will probably be able to learn the new job faster than humans will be able to re-train to do them.
Insofar as Large Language Models are making faster progress in human-style information processing than autonomous robotics are making in doing physical tasks, AI job losses may start hitting white-collar professional who do ‘brain work’ before they hit blue-collar workers doing physical work. These white-collar professionals may include millions of suddenly unemployed lawyers, accountants, journalists, teachers, academics, medical staff, pharmacists, software engineers, graphic designers, architects, and civil engineers.
Such people are typically highly educated, politically engaged, and prone to adopting new moral stigmas through social media. If they’re unemployed, they would have all the time in the world to organize an anti-AI backlash movement. If they have some real estate equity, investment assets, and credit, they may have the money to keep fighting for a while, even without an income. If they have kids, who face poor career prospects in turn due to ongoing AI developments, they may feel the righteous fury of parents who are motivated to do anything necessary to secure a viable future for their next generation. Thus, AI-imposed unemployment is likely to provoke an anti-AI backlash, probably in the time scale of 5-20 years from now.
Trigger 2: Sex
Moral stigmatization often focuses on human sexuality. Sexual practices outside the mainstream have often provoked furious moral condemnation, across cultures and across history – whether it’s incest, polygamy, prostitution, cheating, BDSM, polyamory, or porn. As narrow AI gets applied to goods and services related to human sexuality, there are likely to be all kinds of moral backlashes from diverse groups, ranging from Christian conservatives to woke feminists.
New information technologies are often applied first to create new sexual content. Internet Rule 34 says ‘If it exists, there is porn of it; no exceptions’. A variant will be ‘If AI can make porn of it, there will be porn of it’. Possible applications of AI in the sexual domain have focused on AI-generated porn and erotica (whether photos, audio, video, or stories), deepfake porn, interactive girlfriends and boyfriends, and sexbots.
A key trigger for an anti-AI backlash could be the moral outrage and sexual disgust provoked by sexual applications of narrow AI, such as highly habit-forming interactive VR porn, or customized erotic chatbots with the voices, mannerisms, ad personality traits of someone’s neighbors, co-workers, or ex-lovers.
The most salient, intimate, and controversial application of AI in the next couple of decades will be, essentially, the production of interactive sex slaves – whether in real physical bodies, VR avatars, 2-D deepfake porn, or auditory chatbots. The moral condemnation of slavery remains very strong – it just hasn’t been applied yet to digital slaves. When AI researchers start to be seen as breeders and traders of digital sex slaves, they’re likely to be strongly stigmatized.
Many of these sexual AI applications will take highly controversial forms. Pedophiles will buy AI sexbots with children’s bodies. Sadistic psychopaths will use disposable AI sexbots that can be flogged, cut, and branded, and that scream in realistic pain. Guys who like futanari porn will use sexbots that combine the primary and secondary sexual traits of males and females. AI-generated deepfake porn of politicians, tech billionaires, media celebrities, journalists, and activists is especially likely to provoke the wrath of the rich, powerful, and influential.
The marketing and use of these sexual AI applications may be private at first, but there will inevitably by news coverage, and it will be written to provoke maximum moralistic outrage, because moralistic outrage sells, and gets clicks, and gets shares on social media.
Trigger 3: Violence
Many AI researchers have signed pledges not to develop lethal autonomous weapons (LAWs), such as ‘slaughterbots’. However, there are many other applications of narrow AI that could lead to widespread dangers, injuries, and deaths. Such violence often provokes moral outrage and intense stigmatization of the technologies involved.
The big danger here is not so much that AI safety engineers will stupidly overlook some obviously dangerous failure mode in their systems. Rather, the danger is that rogue nation-states, terrorists, bad actors, resentful former employees, aggrieved nihilists, creepy stalkers, or mischievous youth will manipulate or hack the AI systems to cause targeted deaths of mass carnage. Bad actors could hack self-driving cars to cause huge pile-ups on highways that lead to dozens of deaths. AI drones could be modified by terrorists, criminal gangs, or violent activists to cause mass shootings or explosions at public events. Autonomous assassination drones with face-recognition abilities and long-term loitering abilities could kill major heads of state. Obsessive stalkers could use AI systems to track, harass, and harm their sexual victims. Anarchists, anti-capitalists, and eco-activists who hate resource-intensive industries could hack AI factory control systems to cause horrific industrial accidents. Religious extremists could use AI propaganda systems to promote religious radicalization, terrorism, and warfare.
All of these violent AI applications will, of course, be dismissed and disavowed by the AI industry. But the public may notice the common denominator: AI allows highly effective, targeted violence that is displaced in time and space from the humans directing the violence. This increases the effectiveness and decreases the risks of doing all kinds of mayhem. This will strike many ordinary people as horrifying and outrageous, and will reinforce anti-AI sentiment.
Other triggers
Apart from unemployment, sex, and violence, there are many other applications of narrow AI that could exacerbate an anti-AI backlash. These include harmful effects on AI on women, children, elders, racial minorities, and sexual minorities. These include harmful effects of AI propaganda in political polarization and religious intolerance. Biomedical AI systems for drug discovery could lead to new, highly addictive, psychosis-inducing recreational drugs rather than cures for cancer. AI applied to consumer advertising, gambling, and investments could lead people into over-spending, debt, bankruptcy, divorce, and ruin. The number of harmful things that could go wrong with narrow AI systems is almost limitless – but each new type of harm will be an occasion for sensationalist news coverage, public outrage, virtue signaling, political condemnation, and moral stigmatization of AI.
AI chokepoints that could delay AGI
So what if there’s an anti-AI backlash? What could ordinary people actually do to slow down AI research, given the arms race dynamics between AI companies (such as Microsoft vs. Google) and nation-states (such as the US vs. China)? This section addresses some key resources required for AGI development that could be choked off by an anti-AI backlash. It’s not an exhaustive list of the ways that moral stigmatization of AI could handicap AI research. It’s just intended to give a sense of how strongly and comprehensively an anti-AI backlash could lead to another ‘AI winter’, or even to a decades-long ‘AI ice age’.
Chokepoint 1: AI Talent
If AI research becomes strongly morally stigmatized, all the prestige and coolness of being an AI researcher would evaporate. Moral stigmatization of a career does not just mean the career suffers a slight decline in status relative to other careers. No. The psychology of moral stigmatization means the general public views the career as evil, and views the people working in the career as morally tainted by that evil. Intense moral stigma against AI would mean that being an AI researcher is seen as being about as reputable as being a convicted sex offender, a Nazi racist, an arms dealer, or a mass murderer. The public would view AI researchers as hubris-driven mad scientists with psychopathic traits and genocidal aspirations.
Moral stigmatization of AI research would render AI researchers undateable as mates, repulsive as friends, and shameful to family members. Parents would disown adult kids involved in AI. Siblings wouldn’t return their calls. Spouses would divorce them. Landlords wouldn’t rent to them.
Once the anti-AI backlash renders AI researchers socially, sexually, and professionally toxic, this would radically reduce the quantity and quality of talent working in AI. People with the technical skills to do AI research would exit the field, and would work instead on cybersecurity, or crypto, or non-AI software, or robotics, or whatever. They would have many career options that aren’t viewed as evil by lots of people they meet.
People in some fields have already developed pretty thick skins for resisting stigma. Researchers in controversial areas such as behavior genetics, evolutionary psychology, intelligence research, sex differences, and race differences have been subject for decades to moral stigmatization, hostile stereotyping, career handicaps, lack of funding, and attacks by journalists. They’ve become self-selected for orneriness, disagreeableness, intellectual courage, emotional stability, and self-sufficiency, and they’re learned many coping strategies. By contrast, AI researchers have little experience of being morally stigmatized. They’re used to high status, prestige, income, and coolness. They may be shocked when the public suddenly turns against them and paints them as evil mad scientists consumed by hubris and misanthropy. In other words, the pool of AI talent is highly vulnerable to stigmatization, and has few defenses against it. Faced with a choice between staying in a highly stigmatized field (AI) versus switching to another highly-paid, intellectually engaging computer science field that is not highly stigmatized (e.g. gaming, cybersecurity, crypto), most AI researchers may jump ship and leave AI.
Chokepoint 2: AI Funding
A strong enough anti-AI backlash would lead to AI funding drying up. Investors have become quite sensitive to ‘ESG criteria’ concerning environmental, social, and governance issues. If AI becomes morally stigmatized, ESG criteria could quickly and easily include AI as a disqualifying taboo. Any company involved in AI would receive low ESG scores, and would attract less ‘ethical investment’.
Apart from formal ESG criteria, individual and institutional investors tend to avoid companies widely perceived as reckless, evil, and inhumane. Many investors already avoid companies involved in weapons, alcohol, tobacco, porn, or gambling. If AI becomes seen as a horrifying new weapon, an addictive entertainment, and/or an insanely risky species-level gamble, it would combine all the worst evils of these already-stigmatized industries.
Investors may aspire to be rational maximizers of risk-adjusted returns. But investors are also social primates, subject to the same social and moral pressures that shape human behavior in every other domain of life. High Net Worth Individuals (‘rich people’) often set up family offices to handle their investments, assets, and trusts for their kids and grand-kids. These family offices are specifically designed to take a long-termist, multi-generational perspective on the preservation and enhancement of dynastic wealth and power. That long-termist perspective naturally leads to a concern about multi-decade technological changes, geopolitical risks, global catastrophic risks, and existential risks. If AI becomes morally stigmatized as a major existential risk, family offices and their investment professionals will not want to deploy their capital in AI companies that could lead the rich people’s kids and grand-kids not to die out before the end of the 21st century.
The investment world, like every human world, is prone to moral fads and fashions. Some companies and industry sectors become viewed as morally righteous, saintly, and inspiring; others become viewed as morally disgusting, sinful, and degrading. The psychology of moral disgust runs on the logic of contagion: anything in, around, or near a morally stigmatized activity becomes morally stigmatized by proxy. This means that if a large publicly traded corporation such as Microsoft or Google happens to include a much smaller organization (such as OpenAI or DeepMind) that becomes stigmatized, the large corporation also becomes morally stigmatized. Fewer people want to invest in it. They don’t want their portfolio contaminated by the second-hand evil. As fewer investors are buying and more are selling, the share price falls. As the share price falls, other investors see the writing on the wall, and panic-sell. Hedge funds start aggressively shorting the stock. Soon the corporations face a dilemma: either they shut down or sell off the tainted AI organization poisoning their shareholder value from within, or they continue seeing their share price fall off a cliff – until they get acquired in a hostile takeover by new investors who are willing to cut the AI cancer out of the corporation, to save the rest of the company.
A few anti-ethical investors might see AI as a clever contrarian play, and might think AI company stocks are temptingly under-valued, and will become great investments after the moral stigma fades. But the stigma might not fade, and they may be left facing huge capital losses.
Chokepoint 3: Suppliers
Moral contagion flows out in all directions. If AI starts to be seen as evil, any other organization that does business with AI researchers or AI companies will be seen as evil, or at least evil-adjacent. They will be stigmatized by association, as often happens in ‘cancel culture’. AI research depends on all kinds of suppliers of goods and services, utilities, computational infrastructure, and business infrastructure.
A sort of ‘ethical back-propagation’ would happen, where the moral stigma of AI would propagate backwards along the supply chain, tainting every person and company that provides essential goods and services to AI research.
In response, every supplier who is sensitive to the anti-AI backlash may withdraw their support from AI research groups. This may include everyone supplying GPU hardware, software, cloud computing resources, office space, legal services, accounting services, banking services, and corporate recruiting services. AI businesses may find that no reputable lawyers, bookkeepers, banks, or headhunters are willing to work with them. At a more mundane level, AI groups may find that they cannot find reputable businesses willing to supply them with tech support, back-office staff, office temps, caterers, drivers, janitors, or security staff. If some companies are not willing to do business with cannabis shops, porn producers, drug gangs, arms dealers, racketeers, human traffickers, or other stigmatized forms of economic activity, and if AI becomes stigmatized to a similar level, AI research will be handicapped, and will slow down.
The supplier issue could also affect AI researchers in their personal lives. If AI is widely seen as a work of reckless, hubristic evil, AI researchers may find that landlords are not willing to rent to them, coop boards are not willing to let them buy condos, and daycare centers and private schools are not willing to care for their kids. Bodyguards and police may think they’re too disgusting to protect. Therapists may advise them to ‘seek help elsewhere’. They may even find spiritual services getting choked off, as their priest, pastor, or rabbi shun them for the sinful way they make a living.
Chokepoint 4: Laws and regulations
Informal moral stigmatization often leads to formal government regulations and laws governing new activities and technologies. Indeed, it’s often difficult to coordinate bipartisan support for new regulations and laws constraining something unless there is already a foundation of public stigmatization against that thing. Once the horrors of chemical weapons were witnessed in World War 1, and the public viewed mustard gas and other agents as morally outrageous, it was fairly easy to develop international bans on chemical weapons. Once human cloning became morally stigmatized in the 1990s, it was fairly easy to implement government bans and scientific norms against human cloning. Conversely, it’s quite difficult to sustain regulations and laws against something if the moral stigma against the thing erodes – as in the case of cannabis use gradually becoming destigmatized in the US since the 1960s, and legalization of recreational cannabis following in many states. Thus, moral stigma and government regulation often have mutually reinforcing functions.
In the case of AI, if an anti-AI backlash was sufficiently global in scale, and became a major focus of public concern in both the US and China, it may be much easier to develop international agreements to pause, constrain, or ban further AGI research. With global moral stigmatization of AI, global regulation of AI becomes feasible. Without global moral stigmatization of AI, global regulation of AI is probably impossible. Yet much of the work on AI governance seems to have ignored the role of informal moral stigmatization in creating, energizing, and sustaining formal international agreements.
If an anti-AI backlash gets formalized into strong laws and regulations against AGI development, leading governments could make it prohibitively difficult, costly, and risky to develop AGI. This doesn’t necessarily require a global totalitarian government panopticon monitoring all computer research. Instead, the moral stigmatization automatically imposes the panopticon. If most people in the world agree that AGI development is evil, they will be motivated to monitor their friends, family, colleagues, neighbors, and everybody else who might be involved in AI. They become the eyes and ears ensuring compliance. They can report evil-doers (AGI developers) to the relevant authorities – just as they would be motivated to report human traffickers or terrorists. And, unlike traffickers and terrorists, AI researchers are unlikely to have the capacity or willingness to use violence to deter whistle-blowers from whistle-blowing.
Laws and regulations by themselves would not be enough to significantly slow down AGI development. Bad actors would always be motivated to evade detection and accountability. However, it’s a lot harder to evade detection if there is a global moral stigma against AGI development, with strong public buy-in. From the public’s point of view, laws and regulations are simply ways to articulate, formalize, and implement moral stigmas that are already widely accepted in public discourse. In short, the public has already figured out what’s evil, and they just want government to use its monopoly on the legitimate use of force to deter and punish what’s evil. Thus, moral stigmatization super-charges the effectiveness of any formal laws and regulations around AI.
Often, if some activity becomes sufficiently stigmatized, regulators and law enforcement can apply existing laws in highly targeted ways to deter the activity. For example, laws against reckless endangerment and public endangerment could be applied to prosecute AGI research – if there was sufficient public and institutional belief that AGI imposes existential risks on citizens without their consent. The FBI could switch its focus from ‘white supremacy as the leading domestic terrorist threat’ to ‘AGI research as the leading domestic terrorist threat’ – and investigate and prosecute AI researchers accordingly. Note that government regulators and law enforcement agencies are often motivated to find and capitalize on any new threats that the public perceives. This provides pretexts for increasing their budgets, staff, and powers. If an anti-AI backlash becomes popular, many government workers will see this as a great opportunity to increase their status and power. Fighting against something widely considered an existential threat to humanity would sound like a pretty cool mission to a lot of FBI agents (in the US) or Ministry of State Security agents (in China). Thus, moral stigmatization of AI could lead quite quickly and directly to government investigations, audits, litigation, and prosecution of AI researchers and companies. Result: AGI development is slowed or stopped.
Conclusion
The social-psychological processes of moral stigmatization have evolved genetically and culturally over thousands of generations. Moral stigma plays crucial roles in solving group coordination problems, enforcing social norms, punishing anti-social behavior, and minimizing existential threats to groups. Stigmatization is both a deep human instinct and a powerful cultural tradition. It can solve problems that can’t be solved in any other way. This may include solving the problem of delaying AGI development until we have a better idea whether AI alignment is possible at all, and if it is possible, how to achieve it.
I think a concerted effort to make the public aware of some of the underlying motivations, consequences, and goals of AGI research would likely trigger public backlash:
the singularity-flavored motivations of many AGI researchers: creation of superior successor beings to realize quasi-religious promises of a heavenly future, etc
the economic-flavored motivations of AGI labs: "highly autonomous systems that outperform humans at most economically valuable work". This is literally on OpenAI's website
increasing the likelihood of total human extinction is regarded as acceptable collateral damage in pursuit of the above goals. Some are even fine with human extinction if it's replaced by sufficiently-advanced computers that are deemed to be able to experience more happiness than humans can.
human economic/social disempowerment is sought out in pursuit of these goals, is realistically achievable well within our lifetime, and is likely to occur even with "aligned" AGI
AI "alignment" is theater: even the rosiest visions of futures with AGI are ones in which humans are rendered obsolete and powerless, and at the mercy of AGI systems: at best an abyssal unescapable perpetual pseudo-childhood with no real work to do, no agency, no meaningful pursuits nor purpose -- with economic and social decision-making and bargaining power stripped away
post-scarcity of human-level intelligence directly translates to human work being worthless, and "human work output is valuable" underpins whatever stability and beneficence has existed in every social and economic system we have and have ever had. AGI shreds these assumptions and worsens the situation: at the mercy of powerful systems at whose hands we are entirely powerless.
specifically, the promises of AGI-granted universal basic income used to legitimate AGI-granted economic cataclysm are unlikely to be upheld: with human labor output being worthless, there's nothing humans can do if the promise is reneged on. What are we going to do, go on strike? AGI's doing all the work. Wage an armed insurrection? AGI's a powerful military tool. Vote the AGI overlords out of office? AGI shreds the assumptions that make democracy a workable system (and not solely a box-ticking theater like it is in totalitarian states).
the accelerationist and fundamentalist-utilitarian ideologies driving and legitimating this work place vanishingly little value on human power/agency -- or even continued human existence. Some regard human desires for power/agency/meaningful work/existence as pathological, outdated, and to be eliminated by psychotechnological means, in order for humans to better cope with being relegated to the status of pets or zoo animals.
many of those working towards AGI openly revel in the thought of humanity's obsolescence and powerlessness in the face of the systems they're building, cheer that AI progress is so fast that political systems cannot react fast enough before AGI happens, and are stricken by a lack of faith in humanity's ability to cope with the world and its problems -- to be solved by domination and/or replacement by AGI.
the outcomes of enduring fruitful cooperation between AGI and humans (because of alleged comparative advantage) are laughably implausible: at a certain level of AGI capabilities, the costs of human involvement in AGI systems will be greater than the value of human involvement, and it will be more efficient to simply remove humans from the loop: by analogy, there's no horse-attachment point even on the first automobile because there's no gains to be had from having a horse in the loop. The same will be true of humans and AGI systems. To put it crudely, "centaurs" get sent to the glue factory too.
strategic deceit is used to obscure both the "total technological unemployment" cluster of motivations and the "singularity" cluster of motivations: for instance, the arguments that accelerating AGI is necessary for surviving problems which don't actually need AI: climate change, pandemics, asteroid impact, etc (and thus the risks of AGI are justified). "we need to build AGI so I can become immortal in a really powerful computer because I don't want to die" doesn't quite have the same ring to it
RLHF'ing language models is not even "alignment": it's mostly meant to evade society cracking down on the AGI industry by making it unlikely that their language models will say politically sensitive things or obscenities. This matters because it is remarkably easier to crystallize political crackdowns on entities whose products wantonly say racial slurs than it is to crystallize political crackdowns on entities that are working towards the creation of systems that will render humans obsolete and powerless. The Lovecraftian absurdity is withering.
"AI took my job" is low-status ("can't adapt? skill issue") to admit seriously thinking about, but even in the dream AGI/ASI-is-aligned scenarios, the catastrophic consequences of AGI/ASI will likely look like "AI took my job" extrapolated to the entire human species: full-spectrum human obsolescence, total technological unemployment, loss of human socioeconomic bargaining power, loss of purpose, loss of human role in keeping civilization running, and a degradation of humanity to perma-NEETs tranquilized via AI-generated AR-media/games/pornography, etc.
To put it very bluntly, the overwhelming majority of humanity doesn't want to be aboard a metaphorical Flight 93 piloted by nihilists with visions of a paradisiacal post-scarcity post-human future dancing in their heads as they make the final turns towards the singularity.
Thank you so much for this extremely insightful comment! I strongly agree with all of your points.
“‘AI took my job’ is low-status (‘can’t adapt? skill issue’) to admit seriously thinking about, but even in the dream AGI/ASI-is-aligned scenarios, the catastrophic consequences of AGI/ASI will likely look like ‘AI took my job’ extrapolated to the entire human species…”
My guess: The point at which “AI took my job” changes from low-status to an influential rallying cry is the point when a critical mass of people “wake up” to the fact that AGI is going to take their jobs (in fact, everyone’s) and that this will happen in the near future.
My fear is that there won't be enough time in the window between "critical mass of people “wake up” to the fact that AGI is going to take their jobs (in fact, everyone’s)" and AGI/ASI actually being capable of doing so (which would nullify human social/economic power). To be slightly cynical about it, I feel like the focus on doom/foom outcomes ends up preventing the start of a societal immune response.
In the public eye, AI work that attempts to reach human-level and beyond-human-level capabilities currently seems to live in the same category as Elon's Starship/Super Heavy adventures: an ambitious eccentric project that could cause some very serious damage if it goes wrong -- except with more at stake than a launchpad. All the current discourse is downstream of this: opposition towards AGI work thus gets described as pro-stagnation / anti-progress / pro-[euro]sclerosis / pro-stagnation / anti-tech / anti-freedom and put in the same slot as anti-nuclear-power environmentalists, as anti-cryptocurrency/anti-encryption efforts, etc.
There's growing public realization that there's ambitious eccentric billionaires/corporations working on a project which might be Really Dangerous If It Things Go Wrong — "AI researchers believe that super-powerful AI might kill us all, we should make sure it doesn't" is entering the Overton window — but this ignores the cataclysmic human consequences even if things go right, even if the mythical (which human values? which humans? how is this supposed to be durable against AI systems creating AIs, how is this supposed to be durable against economic selection pressures to extract more profit and resources?) "alignment with human values" is reached.
Even today, "work towards AGI is explicitly and openly seeks to make the overwhelming majority of human work economically unviable" is still not in the Overton window of what it's acceptable/high-status to express, fear, and coordinate around, even though "nobody finds your work to be valuable, it's not worth it to train you to be better, and there's better replacements for what you used to do" is something which:
most people can easily understand the implications of (people in SF can literally go outside and see what happens to humans that are rendered economically unviable by society)
is openly desired by the AGI labs: they're not just trying to create better protein-folding AIs, they're not trying to create better warfighting or missile-guidance AIs. They're trying to make "highly autonomous systems that outperform humans at most economically valuable work". Says it right on OpenAI's website.
is not something that the supposed "alignment" work is even pretending to be able to prevent.
@havequick
Your comment is valuable, because it's a very pointed criticism of how (a lot of) EAs think about this topic, but, unlike most things in that genre, expressed in a way that will make intuitive sense to most EAs (I think). You should turn it into a post of your own if you have time.
havequick -- thanks for an excellent reply.
You add some crucial additional context about some factors that will probably amplify the fury of an anti-AI backlash. Most centrally, the posthumanist ambitions of the Singularity enthusiasts who look forward to humans being replaced by machine intelligences represent an ideology that's profoundly repulsive, outrageous, & treasonous to most people. Once those people are widely seen as traitors to our species, and their profound and pervasive influence on the AI industry is understood, an anti-AI backlash seems highly likely.
The main challenge at that point will be to keep the anti-AI backlash peaceful and effective, rather than counter-productively violent. This will basically require 'just the right amount' of moral stigmatization of AI: enough to pause it for a few decades, but not so much that AI researchers or AI business leaders get physically hurt.
I agree with keeping it peaceful and effective; but I don't think that trying to calibrate "just the right amount" is really feasible or desirable. At least not the way that EA/LW-style approaches to AGI risk has approached this; which feels like it very easily leads to totalizing fear/avoidance/endless rumination/scrupulosity around doing anything which might be too unilateral or rash. The exact same sentiment is expressed in this post and the post it's in reply to; so I am confident this is very much a real problem: https://twitter.com/AISafetyMemes/status/1661853454074105858
"Too unilateral or rash" is not a euphemism for "non-peaceful": I really do specifically mean that in these EA/LW/etc circles there's a tendency to have a pathological fear (that can only be discharged by fully assuaging the scrupulosity of oneself and one's peers) of taking decisive impactful action.
To get a bit more object-level, I believe it is cowardice and pathological scrupulosity to not take a strong assertive line against very dangerous work (GoF virology on pandemic pathogens, AGI/ASI work, etc) because of fears that some unrelated psychotic wacko might use it as an excuse to do something violent. Frankly, if unstable wackos do violent things then the guilt falls quite squarely on them, not on whatever internet posts they might have been reading.
"Don't upset the AGI labs" and "don't do PR to normies about how the efforts towards AGI/ASI explicitly seek out human obsolescence/replacement/disempowerment" and the like feel to me like outgrowths of the same dynamic that led to the pathological (and continuing!) aversion to being open about the probable lab origin of Covid because of fears that people (normies, wackos, political leaders, funding agencies alike) might react in bad ways. I don't think this is good at all. I think that contorting truth, misrepresenting, and dissembling in order to subtly steer/manipulate society and public/elite sentiment leads to far worse outcomes than the consequences of just telling the truth.
To get very object-level, the AGI-accelerationist side of things does not hold themselves to any sort of scrupulosity or rumination about consequences. If anything, that side of things is defined by reckless and aggressive unilateral action; both in terms of discourse and capabilities development. Unilateral to the point that there is a nontrivial contingent of those working towards AGI who openly cheer that AI progress is so fast that political systems cannot react fast enough before AGI happens.
pmarca's piece about AI (published when I was writing this) is called "Why AI Will Save the World", the thesis is that fears about AI are the outgrowth of "irrational" "hysterical" "moral panic" from Ludditic economic innumerates, millenarianist doom-cultists, and ever-present hall-monitors, that accelerating AI "as fast and aggressively" as possible is a "moral obligation", and strongly insinuates that those not on-board act as enablers to the PRC's dreams of AI-fueled world domination.
There is, of course, no counterargument for the contention that at a certain level of AGI capabilities, the costs of human involvement will be greater than the value of human involvement, and it will be more efficient (gains from trade / comparative advantage / productivity growth didn't save draught horses from the glue factory when tractors/trucks/cars rolled around) to simply remove humans from the loop: which leads to near-total human technological unemployment/disempowerment. There is no mention that the work towards AGI explicitly and openly seeks to make the overwhelming majority of human work economically unviable. There is no mention of any of the openly-admitted motivations, consequences, and goals of AGI research which are grievously opposed to most of humanity (those which I had highlighted in my previous comment).
But that sort of piece doesn't need any of that, since the point isn't to steelman a case against AI doomers, but to degrade the credibility of AI doomers and shatter their ability to coordinate by depicting them as innumerates, Luddites, cultists, disingenuous, or otherwise useful idiots of the PRC.
I cannot help but see the AGI-accelerationist side of things winning decisively, soon, and irreversibly if those who are opposed continue to be so self-limitingly scrupulous about taking action because of incredibly nebulous fears. At some point those who aren't on-board with acceleration towards AGI/ASI have to start assertively taking the initiative if they don't want to lose by default.
Again, to be totally explicit and clear, "assertively taking the initiative" does not mean "violence". I agree with keeping it peaceful and effective. But it does mean things like "start being unflinchingly open about the true motivations, goals, and likely/intended/acceptable consequences of AGI/ASI research" and "stop torturing yourselves with scrupulosity and fears of what might possibly conceivably go wrong".
Respect for this comment.
In the original conception of the unilateralist’s curse, the problem arose from epistemically diverse actors/groups having different assessments of how risky an action was.
The mistake was in the people with the rosiest assessment of the risk of an action taking the action by themselves – in disregard of others’ assessments.
What I want more people in AI Safety to be aware of is that there are many other communities out there who think that what “AGI” labs are doing is super harmful and destabilising.
We’re not the one community concerned. Many epistemically diverse communities are looking at the actions by “AGI” labs and are saying that this gotta stop.
Unfortunately, in the past core people in our community have inadvertently supported the start-up of these labs. These were actions they chose to make by themselves.
If anything, unilateralist actions were taken by the accelerationists, as tacitly supported by core AI Safety folks who gave labs like DeepMind, OpenAI and Anthropic leeway to take these actions.
The rest of the world did not consent to this.
Remmelt - I agree. I think EA funders have been way too naive in thinking that, if they just support the right sort of AI development, with due concern for 'alignment' issues, they could steer the AI industry away from catastrophe.
In hindsight, this seems to have been a huge strategic blunder -- and the big mistake was under-estimating the corporate incentives and individual hubris that drives unsafe AI development despite any good intentions of funders and founders.
This is an incisive description, Geoff. I couldn't put it better.
I'm confused what the two crosses are doing on your comment.
Maybe the people who disagreed can clarify.
I second this. I further think there are a lot of image and tribe concerns that go into these sentiments. Many people in EA and especially AI Safety sort of see themselves in the same tribe with AGI companies, whether they are working toward the singularity or just generally being a tech person who understands that tech progress improves humanity and guides history. Another aspect of this is being drawn to technocracy and disdaining traditional advocacy (very not grey tribe). Some EAs actually work for AGI companies and others feel pressure to cooperate and not “defect” on others around them have made alliances with AGI companies.
Your comments are very blunt and challenging for EAs, but they are, I think, very accurate in many cases.
The AGI-accelerationists are at the very center of AGI X-risk -- not least because many of them see human extinction as a positively good thing. In a very real sense, they are the X risk -- and the ASI they crave is just the tool they want to be able to use to make humanity obsolete (and then gone entirely).
And, as you point out, the e/acc enthusiasts often have no epistemic standards at all, and are willing to use any rhetoric they think will be useful (e.g. 'If you don't support American AI hegemony, you want Chinese AI hegemony'; 'If you don't support AGI, you want everyone to die without the longevity drugs AGI could discover'; 'If you oppose AGI, you're a historically & economically ignorant luddite', etc.)
I think such a broad and intense backlash against AI research broadly is extremely unlikely to happen, even if we put all our resources on it.
Thank you so much, Geoffrey, for this compelling argument! I completely agree that a moral backlash against AI is very plausible (and in my estimation, imminent), especially from people whose career paths are or will be indefinitely automated away by AI, and who will not thrive in the new niche of 'startup founder who only hires AGI employees.'
OpenAI's mission is to create "highly autonomous systems that outperform humans at most economically valuable work" (their definition of AGI).
I cannot overstate how broadly unpopular this mission is. In fact, in my experience, whenever I told someone occupationally far-removed from AI about OpenAI's mission, they immediately diagnosed this mission as dystopian. They were also very open-minded about the plausibility of catastrophic and existential dangers in such an AGI-led future, with very few exceptions.
The only bottleneck is that most people currently don't believe that AGI is imminent. This is, of course, starting to change: for example, with the recent CAIS statement signed by leading AI researchers and notable figures.
We should tirelessly prepare for a future in which AGI leaves large numbers of people jobless, purposeless, and angry. I don't know all the answers to how we should prepare for this dystopia, but I'm confident that we should prioritize the following two things:
(1) Truthful, frequent, and high-quality communication to angry people. e.g., communication about the true cause of their hardships—AGI companies—so that they don't blame a scapegoat.
(2) Make preparations for the prevention and resolution of the first (near-catastrophic or catastrophic) AI disaster, as well as for proposing and implementing effective AI-disaster-prevention policies after the first AI disaster.
Peter -- thanks for a very helpful reply. I think your diagnosis is correct.
Your point about the wrong scapegoats getting blamed is very important. If we start seeing mass AI-induced unemployment, the AI companies will have every incentive to launch disinformation campaigns leading people to blame any other scapegoats for the unemployment -- e.g. immigrants, outsourcing to China/SE Asia, 'systemic racism', whatever distracts attention from the problem of cognitive automation. The AI companies don't currently have the stomach to manipulate public opinion in these ways -- they don't have too. But the history of corporate propaganda suggests that every business facing a public backlash will do anything necessary to scapegoat others outside their industry. Big Tech will be no different in this regard. And they'll have the full power of mass-customized AI propaganda systems to shape public opinion.
Something to add is that this sort of outcome can be augmented/bootstrapped into reality with economic incentives that make it risky to work to develop AGI-like systems while simultaneously providing economic incentives to report those doing so -- and again, without any sort of nightmare global world government totalitarian thought police panopticon (the spectre of which is commonly invoked by certain AI accelerationists as a reason not to regulate/stop work towards AGI).
These two posts (by the same person, I think) give an example of a scheme like this (ironically inspired by Hanson's writings on fine-insured-bounties): https://andrew-quinn.me/ai-bounties/ and https://www.lesswrong.com/posts/AAueKp9TcBBhRYe3K/fine-insured-bounties-as-ai-deterrent
Things to note not in either of those posts (though possibly in other writings by the author[s]) is:
the technical capabilities to allow for decentralized robust coordination that creates/responds to real-world money incentives have drastically improved in the past decade. It is an incredibly hackneyed phrase but...cryptocurrency does provide a scaffold onto which such systems can be built.
even putting aside the extinction/x-risk stuff there are financial incentives for the median person to support systems which can peaceably yet robustly deter the creation of AI systems which would take any of the jobs they could get ("AGI") and thereby leave them in an abyssal state of dependence without income and without a stake or meaningful role in society for the rest of their life
Are there other technologies besides AGI whose development has been slowed by social stigma or backlash?
Nuclear power and certain kinds of genetic engineering (e.g. GoF research) seem like plausible candidates off the top of my head. OTOH, we still have nuclear bombs and nuclear power plants, and being a nuclear scientist or a geneticist is not widely stigmatized. Polygenic screening is apparently available to the general public, though there are some who would call the use and development of such technology immoral.
I think this is an interesting point overall, but I suspect the short-term benefits of AI will be too great to create a backlash which results in actual collective / coordinated action to slow frontier capabilities progress, even if the backlash is large. One reason is that AI capabilities research is currently a lot easier to do in private without running afoul of any existing regulations, compared to nuclear power or genetic engineering, which require experimentation on controlled materials, human test subjects, or a high biosaftey level lab.
Max - I'm working on a follow-up post about this issue.
Preview: we do have many examples of research fields (if not technologies themselves) being stigmatized so hard that they get radically slowed down in their progress. For example, the Blank Slate Left's stigmatization of behavior genetics, evolutionary psychology, intelligence research, and sex differences research has been extremely successful in slowing down the growth and influence of these fields, deterring students and researchers from joining them, choking off government funding for them, exiling them from leading science journals and publishing houses, limiting their policy influence, etc. There are common, well-established, battle-tested activist strategies for achieving this 'moral stigmatization of allegedly dangerous scientific fields'. These same strategies could be deployed by an anti-AI backlash (but for higher net social good than the stigmatization of these other scientific fields, IMHO).
Conditional on this happening, does this lead to a non-negligible likelihood that humanity would never create AGI since the stigma increases to greater proportions than developing AGI becomes easier?
Leon - yes, I think an anti-AI backlash could delay AGI by many decades (or centuries), and may even reduce the likelihood that we ever develop AGI. (Although 'never' is a very long time in the cosmic history of life and technology).
Despite that, the US military has been investing billions in automating network-centric warfare over the last years.
Check out #KillCloud on Twitter: https://twitter.com/search?q=%23KillCloud&src=typed_query
Remmelt - thanks for the link the Vice article (here).
I agree that military applications of (narrow) AI will continue, especially in the US and China, and I think they're likely to result in some appalling carnage, including 'collateral damage' to innocent civilians, in the coming years -- feeding into my point about 'violence' being a major trigger for an anti-AI backlash.