(This post also has a Russian version, translated from the present original by K. Kirdan.)
Why this is worth researching
Humanity might develop artificial general intelligence (AGI), colonize space, and create astronomical amounts of things in the future (Bostrom 2003; MacAskill 2022; Althaus and Gloor 2016). But what things? How (dis)valuable? And how does this compare with things grabby aliens would eventually create if they colonize our corner of the universe? What does this imply for our work aimed at impacting the long-term future?
While this depends on many factors, a crucial one will likely be the values of our successors.
Here’s a position that might tempt us while considering whether it is worth researching this topic:
Our descendants are unlikely to have values that are both different from ours in a very significant way and predictable. Either they have values similar to ours or they have values we can’t predict. Therefore, trying to predict their values is a waste of time and resources.
While I see how this can seem compelling, I think this is very ill-informed.
First, predicting the values of our successors – what John Danaher (2021) calls axiological futurism – in worlds where these are meaningfully different from ours doesn’t seem intractable at all. Significant progress has already been made in this research area and there seems to be room for much more (see the next section and the Appendix).
Second, a scenario where the values of our descendants don’t significantly differ from ours appears quite unlikely to me. We should watch for things like the End of History illusion, here. Values seem to notably evolve through History, and there is no reason to assume we are so special that we should drop that prior.
Besides being tractable, I believe axiological futurism to be uncommonly important given its instrumentality in answering the crucial questions mentioned earlier. It therefore also seems unwarrantedly neglected as of today.
How to research this
Here are examples of broad questions that could be part of a research agenda on this topic:
- What are the best predictors of future human values? What can we learn from usual forecasting methods?
- How have people’s values changed throughout History? Why? What can we learn from this? (see, e.g., MacAskill 2022, Chapter 3; Harris 2019; Hopster 2022)
- Are there reasons to think we’ll observe less change in the future? Why? Value lock-in? Some form of moral convergence happening soon?
- Are there reasons to expect more change? Would that be due to the development of AGI, whole brain emulation, space colonization, and/or accelerated value drift? What if a global catastrophe occurs?
- More broadly, what impact will future technological progress have on values? (see Hanson 2016 for a forecast example.)
- Should we expect some values to be selected for? (see, e.g., Christiano 2013; Bostrom 2009, Tomasik 2017)
- Might a period of “long reflection” take place? If yes, can we get some idea of what could result from it?
- Does something like coherent extrapolated volition have any chance of being pursued and if so, what could realistically result from it?
- Are there futures – where humanity has certain values – that are unlikely but worth wagering on?
- Might our research on this topic affect the values we should expect our successors to have by, e.g., triggering a self-defeating or self-fulfilling prophecy effect? (Danaher 2021, section 2)
- What do/will aliens value and what does that tell us about ourselves?
- What about the values of a potential post-human-extinction civilization on Earth?
John Danaher (2021) gives examples of methodologies that could be used to answer these questions.
Also, my Appendix references examples and other relevant work, including subsequent posts in this sequence.
Acknowledgment
Thanks to Anders Sandberg for pointing me to the work of John Danaher (2021) and for our insightful discussion on this topic. Thanks to Elias Schmied for other recommendations. Thanks also to M. Victoria Calabrese for her stylistic suggestions. My work on this sequence so far has been funded by Existential Risk Alliance.
All assumptions/claims/omissions are my own.
Appendix: Relevant work
(This list is not exhaustive. More or less ranked by decreasing order of relevance.)
- Making forecasts – or raising relevant considerations – on what our successors might value
- On the possibility, likelihood, and implications of value lock-in / value drift
- On the incentives to colonize space
Could you point more specifically to what progress you think has been made? As this research area seems to have only existed since 2021 we can't have yet made successful predictions about future values so I'm curious what has been achieved.
Yeah so Danaher (2021) coined the term axiological futurism, but research on this topic has existed long before that. For instance, I find those two pieces particularly insightful:
They explore how compassionate values might be selected against because of evolutionary pressures, and be replaced by values more competitive for, e.g., space colonization races. In The Age of Em, Robin Hanson forecasts what would happen if whole brain emulation comes before de novo AGI, and arrives at similar conclusions.
I don't think we can say they made "successful predictions" and settled the debate, but it seems like they came up with quite important considerations.
I intend to elaborate more on this kind of work in future posts within this sequence. :)