When I first read about the orthogonality thesis in the book "Superintelligence", my first impression was "what do you mean by goals?". (Others had the same reaction, although this is only an anecdote, and so was my first statement.) Looking back, the Orthogonality Thesis does seem to prove too much. The space of possible minds is huge; there does not exist a second example of a sapient lifeform. Also, it might be going too far to discount the closest examples: good old fashioned AI and computer programs. It seems more likely now that AGI will share characteristics with their inanimate relatives; some of the important ones relevant to safety being a lack of personality, autonomy, and intrinsic rewards. It’s hard to give precise words to these characteristics since only recently did we have examples of systems that were smart but lacked these things.
It was cool to read about the issue with IC in precise language from someone trained in philosophy. It does seem important to understand philosophically what goals are, and probably we will understand this better with the use of AI. I guess goals are identified with states of mind distinct from the “default network” state. I tend to be passive and reactive personality-wise, so maybe this contributes to my intuitions about AI-risk. Anyway, this was just me thinking out loud, both here and above.
The question in the title is good food for thought, and I'm not sure if one should aim for survival or flourishing from a reasoned point of view. If making the decision based on intuition, I'd choose flourishing, just because it's what excites me about the future.
Currently I'm not convinced that a near-best future is hard to achieve, after reading (some of) part 2 of 6 from the Forethought website. One view is that the value of the future should be judged by those in the future, and that as long as the world is changing we don't know the value of it yet [1]. So this future we're talking about could take a long time to arrive, when things become stable in the cosmic scale. Then, in this world of technological maturity and a stable reference class of moral patients, we ask if they find that the world is near-best (a eutopia).
Under this rubric a eutopia seems more likely because the close ancestors of the people of the future will have navigated to it. Although oppression and other ways of non-flourishing aren't ruled out, the assumption of a stable equilibrium rules out war and active conflict.
Another view is that the value of the future is not judged only by those living in that future, and that the people of the future may be oblivious to some value, or that in some cases our present judgment is better than those that of the inhabitants of the future. This is possible, but I don't think is likely, given that they will be so much smarter than we are, and understand themselves better. Also, it's logically possible that this future value should be judged by present people, or by people living between now and then, or there's some objective moral truths by which the future fails to flourish. But, these don't make much sense to me, and for the latter I don't think that moral truths exist. They may exist for the current human condition, and appear to be unchanging as parts of us stay immutable, but the future is for our ways of being to be highly liquid.
The _Common-sense utopia_ scenario cuts against my assumption that only the long term future matters, so maybe I should say more. At any time there could be ongoing moral catastrophes from the perspectives of the people living at that time, but what matters is the state of the long term, stable future.
When I first read about the orthogonality thesis in the book "Superintelligence", my first impression was "what do you mean by goals?". (Others had the same reaction, although this is only an anecdote, and so was my first statement.) Looking back, the Orthogonality Thesis does seem to prove too much. The space of possible minds is huge; there does not exist a second example of a sapient lifeform. Also, it might be going too far to discount the closest examples: good old fashioned AI and computer programs. It seems more likely now that AGI will share characteristics with their inanimate relatives; some of the important ones relevant to safety being a lack of personality, autonomy, and intrinsic rewards. It’s hard to give precise words to these characteristics since only recently did we have examples of systems that were smart but lacked these things.
It was cool to read about the issue with IC in precise language from someone trained in philosophy. It does seem important to understand philosophically what goals are, and probably we will understand this better with the use of AI. I guess goals are identified with states of mind distinct from the “default network” state. I tend to be passive and reactive personality-wise, so maybe this contributes to my intuitions about AI-risk. Anyway, this was just me thinking out loud, both here and above.
The question in the title is good food for thought, and I'm not sure if one should aim for survival or flourishing from a reasoned point of view. If making the decision based on intuition, I'd choose flourishing, just because it's what excites me about the future.
Currently I'm not convinced that a near-best future is hard to achieve, after reading (some of) part 2 of 6 from the Forethought website. One view is that the value of the future should be judged by those in the future, and that as long as the world is changing we don't know the value of it yet [1]. So this future we're talking about could take a long time to arrive, when things become stable in the cosmic scale. Then, in this world of technological maturity and a stable reference class of moral patients, we ask if they find that the world is near-best (a eutopia).
Under this rubric a eutopia seems more likely because the close ancestors of the people of the future will have navigated to it. Although oppression and other ways of non-flourishing aren't ruled out, the assumption of a stable equilibrium rules out war and active conflict.
Another view is that the value of the future is not judged only by those living in that future, and that the people of the future may be oblivious to some value, or that in some cases our present judgment is better than those that of the inhabitants of the future. This is possible, but I don't think is likely, given that they will be so much smarter than we are, and understand themselves better. Also, it's logically possible that this future value should be judged by present people, or by people living between now and then, or there's some objective moral truths by which the future fails to flourish. But, these don't make much sense to me, and for the latter I don't think that moral truths exist. They may exist for the current human condition, and appear to be unchanging as parts of us stay immutable, but the future is for our ways of being to be highly liquid.
The _Common-sense utopia_ scenario cuts against my assumption that only the long term future matters, so maybe I should say more. At any time there could be ongoing moral catastrophes from the perspectives of the people living at that time, but what matters is the state of the long term, stable future.