I'm practically new to AI safety, so reading this post was a pretty intense crash course!
What I'm wondering though, even if we suppose that we can solve all the technical problems to create a completely beneficial, Gaia mother-like AGI which is both super-intelligent and genuinely really wants the best for humanity and the rest of earthlings (or even the whole universe), how can humans themselves even align on:
1. What should be the goals and priorities given limited resources and
2. What should be the reasonable contours of the solution space which isn't going to cause some harm, or since no harm is impossible, what would be acceptable harms for certain gains?
In other words, to my naïve understading it seems like the philosophical questions of what is "good" and what should an AGI even align to is the hardest bit?
I mean, obviously not obliterating life on Earth is a reasonable baseline but feels a bit low ambition? Or maybe this is just a completely different discussion?
Why not start from the other end and work backwards? Why wouldn't we treasure every living being and non-living thing?
Aren't insects (just to react to the article) worthy of protecting as an important part of the food chain (from a utilitarian standpoint), for biodiversity (resilience of the biosphere) or even just simply being? After all, there are numerous articles and studies about their numbers and species declining precipitously, see for example: https://www.theguardian.com/environment/2019/feb/10/plummeting-insect-numbers-threaten-collapse-of-nature
But let's stretch ourselves a bit further! What about non-living things? Why not give a bit more respect to objects, as a start by reducing waste? If we take a longtermist view, there will absolutely not be enough raw materials for people for even 100-200 more years – let alone a 800,000 – with our current (and increasing) global rates of resource extraction.
I'm not saying these should be immediate priorities over human beings, but I really miss these considerations from the article.
Thanks a lot, really appreciate these pointers!