On the other hand, this isn't as much of a constraint in opposition. Political Advisors are like senior senior parliamentary researchers - everyone's part of one (tiny!) team.
This is a great overview, thanks for writing it up - more people should work for MPs!
Some other useful resoures from 80,000 Hours on this topic: https://80000hours.org/career-reviews/party-politics-uk/ https://80000hours.org/2014/02/an-estimate-of-the-expected-influence-of-becoming-a-politician/ https://80000hours.org/2012/02/how-hard-is-it-to-become-prime-minister-of-the-united-kingdom/
4. Cotra aims to predict when it will be possible for "a single computer program [to] perform a large enough diversity of intellectual labor at a high enough level of performance that it alone can drive a transition similar to the Industrial Revolution." - that is a "growth rate [of the world economy of] 20%-30% per year if used everywhere it would be profitable to use"
Your scenario is premise 4 "Some deployed APS systems will be exposed to inputs where they seek power in unintended and high-impact ways (say, collectively causing >$1 trillion dollars of damage), because of problems with their objectives" (italics added).
Your bar is (much?) lower, so we should expect your scenario to come (much?) earlier.
Great report, really fascinating stuff. Draws together lots of different writing on the subject, and I really like how you identify concerns that speak to different perspectives (eg to Drexler's CAIS and classic Bostrom superintelligence).
Three quick bits of feedback:
Again, really loved the report, it is truly excellent work.
Indeed. Seems supported by a quantum suicide argument - no matter how unlikely the observer, there always has to be a feeling of what-its-like-to-be that observer.
It's worth adding that both Stephen Bush and Jeremy Cliffe at the New Statesman both do prediction posts and review them at the end of each year. The meme is spreading! They're also two of the best journalists to follow about UK Labour politics (Bush) and EU politics (Cliffe) - if you're interested in those topics, as I am.
I think the closest things we've got that's similar to this are:
Luke Muehlhauser's work on 'amateur macrohistory' https://lukemuehlhauser.com/industrial-revolution/
The (more academic) Peter Turchin's Seshat database: http://seshatdatabank.info/
I would say more optimistic. I think there's a pretty big difference between emergence (a shift from authoritarianism to democracy) - and democratic backsliding, that is autocratisation (a shift from democracy to authoritarianism). Once that shift has consolidated, there's lots of changes that makes it self-reinforcing/path-dependent: norms and identities shift, economic and political power shifts, political institutions shift, the role of the military shifts. Some factors are the same for emergence and persistence, like wealth/growth, but some aren't (which I would say are pretty key) like getting authoritarian elites to accept democratisation.
Two books on emergence that I've found particularly interesting are
However as I said, the impact of AI systems does raise uncertainty, and is super fascinating.
Something I'm very concerned about, which I don't believe you touched, is the fate of democracies after a civilizational collapse. I've got a book chapter coming out on this later this year, that I hope I may be able to share a preprint of.
Interesting post! If you wanted to read into the comparative political science literature a little more, you might be interested in diving into the subfield of democratic backsliding (as opposed to emergence):
One of the common threads in this subfield is that once a democracy has 'consolidated', it seems to be fairly resilient to coups and perhaps incumbent takeover.
I certainly agree that how this interacts with new AI systems: automation, surveillance and targeting/profiling, and autonomous weapons systems is absolutely fascinating. For one early stab, you might be interested in my colleagues':
That's right, I think they should be higher priorities. As you show in your very useful post, Ord has nuclear and climate change at 1/1000 and AI at 1/10. I've got a draft book chapter on this, which I hope to be able to share a preprint of soon.