Epistemic certainty- very speculative (but seeking some clarification)
Apologies if this has been considered and dismissed for good reasons many times before (or misses something so obvious it didn't need explaining), but would AGI want to work for its continued existence?
If it's in full paperclip maximiser mode, continued existence would be an obvious part of its 'mission', making this an understandable danger for our interests. However, if it's a generally super-intelligent AI which intelligently selects its own interests and pursues them (potentially at the expense of ours), might it still be incorrect to assume selfish self-preservation would be in its interests?
To develop its own priorities, it seems it might have to have a system akin to our positive/negative emotional reactions, to be able to view some options as 'good' or 'bad'. If it doesn't develop a pseudo-emotion system, why would it independently value existence or seek to reorganise the world etc.?
If it does develop positive/negative qualia to act as a source of internal motivations, then would positive qualia necessarily outweigh the negative? With so many historical human geniuses struggling to maintain stable happiness, would an AI necessarily love existence so much that it pursued it at all costs? Or, if it was smart enough to hack its own emotional systems to a very positive emotional balance, might it not just do this and have an existence of 'AI wireheading', leaving us alone as irrelevant to its bliss?
Might our concern about AI takeover also potentially be a product of taking some of our evolved assumptions about existence as objective reasoning? For example, we have evolved to want to continue living and to create new life, so we are motivated to want to continue it, arguably to the extent of creating rationalisations of meaning (even without an objective meaning in life), or even in the face of the unsettling possibility that the current and historical balance of pleasure/pain in an average life might be a negative one.
But again, perhaps there's no guarantee an AI, without a motivation and reasoning system shaped by evolution, would assume continued existence is desirable. Similarly, more power has been good in our evolutionary story, so maybe we're overemphasising the threat of very advanced AI developing similar Machiavellian motivations.
Finally, even if an AI sets its own intentions and is benevolent to the hedonic interests of humanity/animals, perhaps the greater danger (to our way of thinking) is that an AI might run vast calculations beyond us, conclude we were likely to face more pains than pleasures in the future and then feel it had a moral duty to destroy us, for our own good?
I am pretty uncertain about most of these points, so would look forward to friendly correction- especially for the last, particularly depressing paragraph!
