Epistemic Status: Quickly written, uncertain. I'm fairly sure there's very little in terms of the public or government concerned about AGI claims, but I'm sure there's a lot I'm missing. I'm not at all an expert on government or policy and AI.
This was originally posted to Facebook here, where it had some discussion. Many thanks to Rob Bensinger, Lady Jade Beacham, and others who engaged in the discussion there.
Multiple tech companies now are openly claiming to be working on developing AGI (Artificial General Intelligence).
As written in a lot of work on AGI (See Superintelligence, as an example), if any firm does establish sufficient dominance in AGI, they might have some really powerful capabilities.
- Write bots that could convince (some) people to do almost anything
- Hack into government weapons systems
- Dominate vital parts of the economy
- Find ways to interrupt other efforts to make AGI
And yet, from what I can tell, almost no one seems to really mind? Governments, in particular, seem really chill with it. Companies working on AGI get treated similarly to other exciting AI companies.
If some company were to make a claim like,
"We're building advanced capabilities that can hack and modify any computer on the planet"
or,
"We're building a private nuclear arsenal",
I'd expect that to draw attention.
But with AGI, crickets.
I assume that governments dismiss corporate claims of AGI development as overconfident marketing-speak or something.
You might think,
"But concerns about AGI are really remote and niche. State actors wouldn't have come across them."
That argument probably applied 10 years ago. But at this point, the conversation has spread a whole lot. Superintelligence was released in 2014 and was an NYT bestseller. There are hundreds of books out now about concerns about increasing AI capabilities. Elon Musk and Bill Gates both talked about it publicly. This should be one of the easiest social issues at this point for someone technically savvy to find.
The risks and dangers (of a large power-grab, not of alignment failures, though those too) are really straightforward and have been public for a long time.
Responses
In the comments to my post, a few points were made, some of which I was roughly expecting. Points include:
- Companies saying they are making AGI are ridiculously overconfident
- Governments are dramatically incompetent
- AGI will roll out gradually and not give one company a dominant advantage
My quick responses would be:
- I think many longtermist effective altruists believe these companies might have a legitimate chance in the next 10 to 50 years, in large part because of a lot of significant research (see everything on AI and forecasting on LessWrong and the EA Forum). At the same time, my impression is that most of the rest of the world is indeed incredibly skeptical of serious AGI transformation.
- I think this is true to an extent. My impression is that government nonattention can change dramatically and quickly, particularly in the United States, so if this is the crux, it might be a temporary situation.
- I think there's substantial uncertainty here. But I would be very hesitant to put over a 70% chance that: (a) one, or a few, of these companies will gain a serious advantage, and (b) the general-purpose capabilities of these companies will come with significant global power capabilities. AGI is general-purpose, it seems difficult to be sure that your company can make it without it being an international security issue of some sort or other.
Updates
This post was posted to Reddit and Hacker News, where it had a total of around 100 more comments. The Hacker News crowd mostly suggested Response #1 ("AGI is a pipe dream that we don't need to worry about")
This isn't intended to be a complete response to your post, but for comparison, here are some other things that ambitious tech companies have serious plans to accomplish:
SpaceX has a pretty credible plan to create a revolutionary space-launch system (Starship) and use it to literally colonize another planet. The USA doesn't seem as excited about this as it rationally ought to be, nor do China or other countries seem as concerned as one might expect.
Various crypto companies and projects have big plans to literally replace fiat currency, undermining one of the fundamental ways that governments wield power. Crypto does seem to be on the radar of most governments, but still less than one might expect given the long-term stakes.
Nuclear fusion companies like Helion, General Fusion, and others have collectively raised billions of dollars to make "energy too cheap to meter" a reality. You'd think governments would be more excited about subsidizing potentially incredible energy sources like this, but maybe they've been burned too many times before or maybe they're just slow.
There is a lot we could straightforwardly do to be better prepared for future pandemics, but we're mostly not doing that stuff (hugely scaled-up metagenomic sequencing, improved ventilation and PPE, making vacaccines in advance for all potential pandemic viruses, etc) even though it's totally legible, it doesn't require any risky breakthroughs, and it's extremely salient due to covid.
Obviously the consequences of even Mars colonization, full cryptoization of the economy, abundant power from fusion, and significantly mitigated biorisk pale in comparison to the transformative power of AGI. But from a government's perspective they might all seem to be in the same reference class. Yet it's surprisingly close to "crickets" on all counts. (I admit that AGI might be especially neglected even here, though -- SpaceX at least gets normal NASA contracts, etc.)
I think SpaceX's regular non-Mars-colonization activities are in fact taken seriously by relevant governments, and the Mars colonization stuff seems like it probably won't happen and also wouldn't be that big a deal if it did (in terms of, like, national security; it would definitely affect who gets into the history books). So it doesn't seem to me like governments are necessarily acting irrationally there.
Same with cryptocurrency; its implications for investor protection, tax evasion, capital controls evasion, and facilitating illicit transactions are indeed taken seriously, and while governments would obviously care quite a lot if it displaced fiat currency, I just don't think there's any way that's happening. If it does, then this is probably because fiat currency itself somehow stopped working and something was needed to fill the void; if governments think this scenario is at all plausible, then presumably their attention would be on the first part where fiat currency fails, since that's much more within their control and cryptocurrency isn't really a relevant input.
The scientific and regulatory culture around fusion power seems to be shaped, as you suggest, by the long history of failures in that domain; judging by similar situations in other fields, I wouldn't be surprised if no one wanted to admit to putting any credence in it, so that they wouldn't look stupid in case it fails again.
The state of pandemic preparedness does indeed seem like just straight-up government incompetence.
That's a good point, and I like the examples, thanks!