Great post, and glad to see someone working through this concretely. I wrote my first EA Forum post earlier this year on much the same intuition you're operating on: the technologies that enable lock-in or front-running coups require strictly weaker AI than ASI does. Yours is far more grounded than mine, but the more people draw attention to this the better.
Concentration of power doesn't need superintelligence. It doesn't even need AGI. That's precisely what makes it scary.
I wrote about this in The Front-Running Catastrophe. The core argument: the technologies required for stable totalitarianism — omni-surveillance, autonomous lethal systems, AI-assisted internal security — require strictly weaker AI than the technologies required for extinction via misalignment
Thanks! I think there's a lot of thinking that needs to be done wrt all the different contingencies conditioned on the "best case scenario of pausing / not reaching ASI"
I have always thought that there is a lot of unpicked low-hanging fruit for animation among the most popular blogposts. Examples might be "I only believe in the paranormal" and "Great minds might not think alike"from lesswrong.
Great post, and glad to see someone working through this concretely. I wrote my first EA Forum post earlier this year on much the same intuition you're operating on: the technologies that enable lock-in or front-running coups require strictly weaker AI than ASI does. Yours is far more grounded than mine, but the more people draw attention to this the better.
Concentration of power doesn't need superintelligence. It doesn't even need AGI. That's precisely what makes it scary.
I wrote about this in The Front-Running Catastrophe. The core argument: the technologies required for stable totalitarianism — omni-surveillance, autonomous lethal systems, AI-assisted internal security — require strictly weaker AI than the technologies required for extinction via misalignment
Thanks! I think there's a lot of thinking that needs to be done wrt all the different contingencies conditioned on the "best case scenario of pausing / not reaching ASI"
I have always thought that there is a lot of unpicked low-hanging fruit for animation among the most popular blogposts. Examples might be "I only believe in the paranormal" and "Great minds might not think alike"from lesswrong.