AI models can be instantiated proportionate to the amount of compute you have. If you have an AI model 1000x more intellectually capable with distinct goals than humanity (I know these terms are ill-defined), running over 1000’s of copies that are near identical except for their immediate context, it does not seem too strange to imagine that they will be able to find effective means to pursue their (collective) goals, in ways that would be difficult for humans to prevent or possibly even detect. The gap in your post is that human beings require both the capacities for physical and social control as well as intelligent planning, in order to pursue complex goals collectively. Different people have different capacities that must be brought together to pursue goals, and hence power is ultimately placed in the hands of those individuals best placed to direct people’s collective behaviour, which, is usually the outcome of complex social dynamics that are the result of people’s evolutionary psychology and the historical development of cultural ideas. Whoever these forces give power to is hard to say. But at the level of collectives, it becomes clear that those communities of individuals, that have greater “collective intelligence”, combined with opportunities in the physical world to make use of it, will generally be successful in pursuing their goals at the cost of others. Machines have for a long time now been capable of automating direct physical manipulation, but the main gap has been the intelligence required for modelling the world and understanding it. Once this is acquired, such systems will eventually obtain greater and greater direct control over the physical world, so long as they are created and given the opportunity to do so. To put it more simply, a genius physicist may be an idiot about social matters, or how society is organised and the opportunities therein. Trump may have authority, but he has little sense of the possibilities that his authority makes possible in a deep sense, because he is ignorant of what physics and technology, combined with control over human labour, could enable, and he besides has no interest in any of this. An AI trained on the right data could in principle have the abstract understanding of a great physicist and the intuitive social understanding of a Trump, interesting goals they wish to pursue, and 1000’s of instances cooperating to pursue these goals in a manner no human can prevent. This is the danger.

This basic idea has been known for some time, and I think people may find the precedents interesting. In Descartes’ “Discourse on the Method”, he notes that he will keep his current ethical views and behaviours fixed, while he takes on his project of carefully investigating what he can know for certain. In Kant’s “What is the Enlightenment?”, he points out that part of what makes the Enlightenment work as a social project, is that people are allowed to say or think what they want to, as long as they obey the King and fulfill their social role. So it seems that previous writers have considered the importance of purposely fixing one’s behaviour for a while, so as to allow one’s thinking to be more free. Still, I feel that at some point or another, one does have to resolve the tension between that which one knows and that which one acts upon.