Why am I not bullish on civilizational refuges for reducing bio xrisk? I believe bio xrisk is concentrated in E2H scenarios or AI takeover, and neither of these things benefit much from refuges. For both of these threat models, there is value in refuges but only if they are a safe staging area to take back control of the environment or against an AI, which means you probably need a full industrial base worth of people/infrastructure to be protected. Society-wide biohardening and PPE is therefore the better option even on purely longtermist grounds (and has the additional property of also being much more valuable on non-longtermist grounds as well!)
Tentative thesis: China is unlikely to accept a subordinate position in AI capabilities to the US, just as the US is unlikely to accept a subordinate position in AI capabilities to China or anyone else.
This thesis suggests that there are two likely paths for the future international AI (non)regulation:
1) a continuing race between the US and China, perhaps joined by some late entrants, possibly with some rules established (e.g. mutual ban on autonomous weapons or something like that), or
2) a “pause or stop deal” that will establish some sort of ceiling on capabilities, which will be equal for China and the US.
What I think is far less likely is a deal in which China would accept lower AI capabilities than the US.
I simply don’t see a good reason why Chinese leaders would accept subordinate position. They know that China has the ability to catch-up to the US in various technological domains, as evidenced by the fact that in, like, 1990, China was behind in more or less everything, and now they are pushing the technological frontier in many areas.
Some estimates floating around the web suggest that, if the US would stop AI development now, China could reach current US capabilities within months. That is maybe overly optimistic/pessimistic depending on where you stand, but I very much doubt that catch-up would take more than a decade. And Chinese government, being patriotic about the abilities of the Chinese nation, probably will not have an absurdly pessimistic estimate.
Moreover, the Chinese regime is in many ways oriented around this idea of catching up (this is a big difference between China and EU). It is a central plank of the official historical narrative of the People’s Republic of China (see for example preamble to their constitution: https://english.www.gov.cn/archive/lawsregulations/201911/20/content_WS5ed8856ec6d0b3f0e9499913.html) that the century between First Opium War and the Communist victory in the Chinese civil war in 1949 was the century of humiliation, during which various imperialist powers exploited China due to the fact that it was technologically backward country unable to stand up to them. I happen to think this narrative is largely correct, but even if it isn’t, what matters is that Chinese government believes it and is determined to never again allow China to be put in the position where a technologically superior power could walk over it.
This has some implications:
A) Since China is currently behind, any sort of “pause or stop deal” between the US and China would likely have to involve either the US stopping further development and allowing China to catch up, or the US somehow crippling its own capabilities to a current Chinese level. Of course, for either option, the US would have to place a lot of trust in the Chinese government.
B) As long as China is behind, export controls on hardware and other attempts to limit Chinese capabilities are at best of doubtful utility as a way to get to a "pause or stop" agreement. In fact, imho there are making a deal less likely. They antagonize Chinese leaders, make them even more paranoid and less inclined to a constructive agreement. But also, to the extent they are successful in limiting Chinese capabilities, they make it less likely to that the US government would agree to an equal deal, since that would mean sacrificing larger lead, which is harder to politically accept for the US. Perhaps this is a reason why nuclear arms treaties in the Cold war were agreed only after the USSR more or less caught up to US nuclear arsenal?
Now, attempts to limit Chinese capabilities can still be seen as useful if you think it is important for the US to “win” the competition, for example because there are reasons to believe US AI will be safer. Or because you think the world would be a better place if the US “wins”, even though both countries would develop similarly dangerous systems. Or of course if you think that the US victory would be preferable even if American AI would be more dangerous than Chinese AI.
Btw. for a well written snapshot of Chinese opinion/vibes, I recommend this: https://www.chinatalk.media/p/how-anthropic-became-chinas-goliath
"We cannot choose not to be vegan, even for a moment, even if it would be pragmatic to do so."
I genuinely don't mean to be glib, but have you tried? One can just choose to become non-vegan, so if one truly believed that non-vegans were better advocates and we need more of them, surely it should be something to consider.
YC never had any sort of monopoly on the best startups though, its own terms are much better than they used to be, and its still definitely the most prestigious. The question isn't "why didn't Anthropic join YC", it's "why, if AI truly makes most developers 10x more productive and transforms the unit economics of customers, are the valuations of a cohort of prestigious "AI-enabled" startups round about their non-AI enabled 2019 peers?". There really aren't enough people founding research labs for that to be the issue
Of course, any early stage valuations graph is as much a graph of investor sentiment as anything else, but if AI was really making these companies significantly more productive, investors would have to be very bearish on AI or YC selection effect to have gone down the toilet for that not to show in the data. I presume median data looks similar?
The best counterargument is that AI also makes it easier for competitors leading to less defensible business models even if AI actually enables them to grow faster, but if that was the case investor sentiment should be much more bearish on incumbents...
Isn't another advantage of using multiple agents (on top of speed) that you get around the context window limits that you would hit if you tried to put the same number of tokens through a single agent?
Or is that effect already accounted for in the analysis? Or a contributing factor to the "plateau" effect already mentioned?
Thanks for the post. I think this is an important strategic debate to have, so I'll reply in more length than usual.
I agree with you that we should be honest about what we believe, pursue a variety of strategies, and try to expand the moral circle. I also think you're right that animal liberation played an important role in recruiting many of the movement's current leaders. And I have immense respect for liberation-focused activists like Wayne Hsiung who do courageous work to help animals. None of the following is intended as a criticism of their work. That said, I disagree with you on a number of other points:
Thanks again for this thoughtful post, which I think makes an important point.