I tend to think S-risk is just kind of a hard bar to get met. Since even if Sam Altman uses AI to torture his personal enemies forever, well that doesn't seem like it could ever be enough people to be an S-risk. If some small fraction of people get horrible outcomes and/or a bunch of mind crime happens to some simulated people I would describe that as more of a Omelas-esque middling scenario, since the vast majority of minds would probably still have amazing existences: Thus the median person's life would still be amazing and even the average would be pretty good.
It just seems very hard to imagine remotely plausible scenarios where there isn't like 100x as many luxurious post-scarcity societies as simulated hells.
In general I think middling post-singularity scenarios are somewhat underexamined. These have 2 big characteristics to think about compared to a good outcome: There's not enough centralized control and/or surveillance to enforce mind-crime, and so while the scenario is overwhelmingly good on net there's going to be a lot of mind crime happening that's infeasible to do anything about because nobody who would intervene knows when it's happening. Secondly without better coordination you end up with everyone not near the expansion frontier falling into a Malthusian trap. Since the rate at which one can collect more resource with which to grow is going to be less than exponential because you can't reach new mass-energy at an exponential rate (you're limited by the surface area of your feasibly accessible future light cone). Which is an issue because by default population growth is exponential (this is true both because current fertility is suppressed for economic reasons not applicable to a post-scarcity scenario, and because even low birth rates become exponential once aging is cured). This means that even in a best case scenario where you don't get insane growth from digital minds, people cloning themselves, people wasting resources on vanity projects, etc: It's still only going to be a few thousand years before your exponential growth reaches a point where it's literally impossible for accessible resources to support it because you'd need every single atom to support an entire civilization's worth of people (you can easily see why this is an issue doing some simple math with compound interest). Importantly while it might take a long time to become an issue (though it could also happen quite quickly due to things like EMs not following regular growth trends), getting some handle on population growth is something you need to do early on or you'll never be able to do it. Since you can't really coordinate anything once people have spread out too much to enforce rules on them after the fact (if they were all sent with aligned AI's from the outset that's different).
My fear is that if an AI has the tendency to generalize in that way, that that's a lot more likely to lead to generalizing in ways that are catastrophically bad for humans then it is to lead to it extending care to groups it wasn't programmed/trained to care about in the way this question seems to imagine.
Which is some technical sense might mean I should have put agree instead, but I don't think when you said substantial spillover that you meant the sort of spillover that might lead to outcomes like the repugnant conclusion, tiling the universe in hedonium, etc. I'm presuming you meant that it would still care about humans in a way that wouldn't abandon them because it decided for utilitarian/population ethics reasons to prioritize other types of entities to the exclusion of humans.
As I go into in another standalone comment, I tend to think any value you give an AI beyond caring about people's current preferences is on net probably not worth the risk. Whereas I think the downsides of just caring about people's current preferences are largely overrated, and many of those critiques depend upon taking seriously a notion of naive notion of moral progress (I think people underestimate how much "moral progress" is just people abandoning authoritarian social norms and reverting back to the comparatively more egalitarian norms we had for most of human existence as hunter gathers, once social/economic conditions suddenly are no longer applying the necessary pressure to keep those authoritarian norms in place).
I should note that extreme power concentration isn't an S-risk unless it's so over the top dystopian that it's somehow worse to live in said society than to be dead.
That being said I think a lot of differences in opinion when it comes to how people think about AI power concentration come down to which timelines you're imagining, how massive the abundance created by AI is/how zero sum you're imagining things, and lastly whether you think the people in control of AI would be so cartoonishly evil that they deliberately keep everyone impoverished despite no practical upside to doing that: https://www.astralcodexten.com/p/you-have-only-x-years-to-escape-permanent
I definitely had to change my answer on a question quite a lot because I realized from reading one of your comments that you didn't mean what I thought. Hopefully people's written justification helps more, since that probably tells you how they were interpreting it when they answered.
Glad I saw this comment because like many people I assumed this meant "do you expect aligned AI to on average turn out better than unaligned AI" which seems very different than what you actually meant.
>If you aren't extinct you can try to solve S risk
Not necessarily which is kind of the point. An S-risk scenario is something worse than death, so presumably at that point you aren't in a position to do anything to improve your situation. That being said I don't think S-risk is that serious a concern because I haven't seen any remotely plausible scenarios outlined. For instance wireheading seems like an obvious fail-mode but while pretty terrible I don't know that most people would say it's actively worse than death.
The kind of tech one expects after AGI has already been develop for some time (it says persist after all) can absolutely change biology, which is what I think the question is getting at.
For instance stick all the animals in utopian carefully AI managed environments that maximize their well being and then replace them with cybernetic faux animals that appear perfectly convincing but are actually just AI actors pretending to be animals (and who either can't suffer themselves or greatly enjoy their acting role).
You aren't thinking enough about the possibilities enabled by post singularity tech levels. You could eliminate animal suffering technologically in many different ways while still retaining something which looks superficially like nature.
For instance the animals might all get placed within carefully AI managed utopian environments (either rotating habitats or simulated) where their reproduction is controlled. Meanwhile on Earth you might just have cybernetic faux animals that are incapable of suffering or might even just be getting remotely controlled by AI who enjoy and are extremely proficient and perfectly acting out the role of animals so perfectly you couldn't tell the difference.
Alternatively you might keep the animals within nature but modify them so as to massively reduce their suffering and increase their enjoyment while altering its behavior or environment so as to avoid the negative ecological consequences that would otherwise have. Most simply one can imagine the animals being cyborgs (though in principle advanced biotech could replace anything doable with cybernetics) with an AI that takes possession of their body whenever anything really bad is going to happen (putting its mind in stasis, or into a simulated utopia for a brief while), then hands control back later if its still alive (while rewriting memories or affecting behavior to keep abnormal behavior from resulting) while of course causing the animal to act like it normally might from an injury, even though its actual suffering is minimal to nonexistent.
I could go on all day: My point is there's a lot of options.
Would really like to see a question on whether people are moral realists or not, and a way to filter results by how they answered. Since I'm very curious how people's answers differ based on whether they're a moral realist or a moral anti-realist.
Of course some care would have to be put into the phrasing of a question like that, but it seems pretty doable in a way that will produce useful and interesting results.
At the margin, S-risk work in AI is more important than x-risk work³
The scenarios I've seen proposed where an AGI traps us in a fate worse than death just don't seem remotely likely. Though this depends a little on what you consider to be a fate worse than death. For instance I wouldn't consider being trapped in a mindless bliss forever to be worse than death, so by definition it can't be an S-risk to me. Though I would certainly consider it to be an outcome we should strongly attempt to avoid, because I consider it only marginally better than death.
I tend to think S-risk is just kind of a hard bar to get met. Since even if Sam Altman uses AI to torture his personal enemies forever, well that doesn't seem like it could ever be enough people to be an S-risk. If some small fraction of people get horrible outcomes and/or a bunch of mind crime happens to some simulated people I would describe that as more of a Omelas-esque middling scenario, since the vast majority of minds would probably still have amazing existences: Thus the median person's life would still be amazing and even the average would be pretty good.
It just seems very hard to imagine remotely plausible scenarios where there isn't like 100x as many luxurious post-scarcity societies as simulated hells.
In general I think middling post-singularity scenarios are somewhat underexamined. These have 2 big characteristics to think about compared to a good outcome: There's not enough centralized control and/or surveillance to enforce mind-crime, and so while the scenario is overwhelmingly good on net there's going to be a lot of mind crime happening that's infeasible to do anything about because nobody who would intervene knows when it's happening. Secondly without better coordination you end up with everyone not near the expansion frontier falling into a Malthusian trap. Since the rate at which one can collect more resource with which to grow is going to be less than exponential because you can't reach new mass-energy at an exponential rate (you're limited by the surface area of your feasibly accessible future light cone). Which is an issue because by default population growth is exponential (this is true both because current fertility is suppressed for economic reasons not applicable to a post-scarcity scenario, and because even low birth rates become exponential once aging is cured). This means that even in a best case scenario where you don't get insane growth from digital minds, people cloning themselves, people wasting resources on vanity projects, etc: It's still only going to be a few thousand years before your exponential growth reaches a point where it's literally impossible for accessible resources to support it because you'd need every single atom to support an entire civilization's worth of people (you can easily see why this is an issue doing some simple math with compound interest). Importantly while it might take a long time to become an issue (though it could also happen quite quickly due to things like EMs not following regular growth trends), getting some handle on population growth is something you need to do early on or you'll never be able to do it. Since you can't really coordinate anything once people have spread out too much to enforce rules on them after the fact (if they were all sent with aligned AI's from the outset that's different).
My fear is that if an AI has the tendency to generalize in that way, that that's a lot more likely to lead to generalizing in ways that are catastrophically bad for humans then it is to lead to it extending care to groups it wasn't programmed/trained to care about in the way this question seems to imagine.
Which is some technical sense might mean I should have put agree instead, but I don't think when you said substantial spillover that you meant the sort of spillover that might lead to outcomes like the repugnant conclusion, tiling the universe in hedonium, etc. I'm presuming you meant that it would still care about humans in a way that wouldn't abandon them because it decided for utilitarian/population ethics reasons to prioritize other types of entities to the exclusion of humans.
As I go into in another standalone comment, I tend to think any value you give an AI beyond caring about people's current preferences is on net probably not worth the risk. Whereas I think the downsides of just caring about people's current preferences are largely overrated, and many of those critiques depend upon taking seriously a notion of naive notion of moral progress (I think people underestimate how much "moral progress" is just people abandoning authoritarian social norms and reverting back to the comparatively more egalitarian norms we had for most of human existence as hunter gathers, once social/economic conditions suddenly are no longer applying the necessary pressure to keep those authoritarian norms in place).
I should note that extreme power concentration isn't an S-risk unless it's so over the top dystopian that it's somehow worse to live in said society than to be dead.
That being said I think a lot of differences in opinion when it comes to how people think about AI power concentration come down to which timelines you're imagining, how massive the abundance created by AI is/how zero sum you're imagining things, and lastly whether you think the people in control of AI would be so cartoonishly evil that they deliberately keep everyone impoverished despite no practical upside to doing that: https://www.astralcodexten.com/p/you-have-only-x-years-to-escape-permanent
I definitely had to change my answer on a question quite a lot because I realized from reading one of your comments that you didn't mean what I thought. Hopefully people's written justification helps more, since that probably tells you how they were interpreting it when they answered.
Glad I saw this comment because like many people I assumed this meant "do you expect aligned AI to on average turn out better than unaligned AI" which seems very different than what you actually meant.
>If you aren't extinct you can try to solve S risk
Not necessarily which is kind of the point. An S-risk scenario is something worse than death, so presumably at that point you aren't in a position to do anything to improve your situation. That being said I don't think S-risk is that serious a concern because I haven't seen any remotely plausible scenarios outlined. For instance wireheading seems like an obvious fail-mode but while pretty terrible I don't know that most people would say it's actively worse than death.
The kind of tech one expects after AGI has already been develop for some time (it says persist after all) can absolutely change biology, which is what I think the question is getting at.
For instance stick all the animals in utopian carefully AI managed environments that maximize their well being and then replace them with cybernetic faux animals that appear perfectly convincing but are actually just AI actors pretending to be animals (and who either can't suffer themselves or greatly enjoy their acting role).
You aren't thinking enough about the possibilities enabled by post singularity tech levels. You could eliminate animal suffering technologically in many different ways while still retaining something which looks superficially like nature.
For instance the animals might all get placed within carefully AI managed utopian environments (either rotating habitats or simulated) where their reproduction is controlled. Meanwhile on Earth you might just have cybernetic faux animals that are incapable of suffering or might even just be getting remotely controlled by AI who enjoy and are extremely proficient and perfectly acting out the role of animals so perfectly you couldn't tell the difference.
Alternatively you might keep the animals within nature but modify them so as to massively reduce their suffering and increase their enjoyment while altering its behavior or environment so as to avoid the negative ecological consequences that would otherwise have. Most simply one can imagine the animals being cyborgs (though in principle advanced biotech could replace anything doable with cybernetics) with an AI that takes possession of their body whenever anything really bad is going to happen (putting its mind in stasis, or into a simulated utopia for a brief while), then hands control back later if its still alive (while rewriting memories or affecting behavior to keep abnormal behavior from resulting) while of course causing the animal to act like it normally might from an injury, even though its actual suffering is minimal to nonexistent.
I could go on all day: My point is there's a lot of options.
Would really like to see a question on whether people are moral realists or not, and a way to filter results by how they answered. Since I'm very curious how people's answers differ based on whether they're a moral realist or a moral anti-realist.
Of course some care would have to be put into the phrasing of a question like that, but it seems pretty doable in a way that will produce useful and interesting results.
The scenarios I've seen proposed where an AGI traps us in a fate worse than death just don't seem remotely likely. Though this depends a little on what you consider to be a fate worse than death. For instance I wouldn't consider being trapped in a mindless bliss forever to be worse than death, so by definition it can't be an S-risk to me. Though I would certainly consider it to be an outcome we should strongly attempt to avoid, because I consider it only marginally better than death.