Great post, my comments are responding to the longer linked document.

1. I had a thought about quasilinear utilities and moral preferences. There, you say "What’s more, even if, given massive increases in wealth, people switch to using most resources on satisfying moral preferences rather than self-interested preferences, there is no guarantee that those moral preferences are for what is in fact good. They could be misguided about what is in fact good, or have ideological preferences that are stronger than their preferences for what is in fact good; their approximately linear preferences could result in building endless temples to their favoured god, rather than promoting the good."

Still, I think there's an argument that can be run that outcomes caused by moral preferences tend to converge / be very similar to one another. Here, the first step is to 

distinguish de dicto and de re moral preferences. Second, if we imagine a group of agents with de dicto moral preferences, one argument for convergence is that each agent will be uncertain about what outcomes are morally best, and then agreement theorems suggest that once the various agents pool their information, they will tend to converge in their credence distribution over moral theories. But this means that their distribution over what outcomes are morally best will tend to converge. Connecting back to the main question, the point is that if we think that in the future most outcomes involve decision-makers being de dicto ethical, then we might expect concentration in the outcomes, if decision-makers tend to converge in their de dicto ethical credence distributions. Regardless of what is actually good, we should expect most outcomes in which agents want to be de dicto ethical to be similar, if these agents have similar credences across outcomes about the good. This only works for de dicto not de re moral preferences: if I want to help chickens and you want to help cows, no amount of pooling our information will lead to convergence in our preferences, because descriptive information won't resolve our dispute. 

2. I was a bit confused about the division of resources argument. I was thinking that if the division of resources is strong enough, that actually would support dichotomy, because then most futures are quite similar in total utility, because differences across territories will tend to be smoothed out. So the most important thing will just be to avoid extinction  before the point of division (which might be the point at which serious space colonisation happens). Again, after the point of division lots of different territories will have lots of different outcomes, so that the total value will be a function of the average value across territories, and the average will be smooth and hard to influence. I think maybe where I'm getting confused is you're imagining a division into a fairly small number of groups, the valorium-optimisers and the non-optimisers. By contrast, I'm imagining a division into vast numbers of different groups. So I think I agree that in your small-division case, dichotomy is less plausible, and pushing towards valorium-optimisation might be better than mitigating x-risk.

3. "if moral realism is false, then, morally, things in general are much lower-stakes — at least at the level of how we should use cosmic-scale resources" I think this argument is very suspicious. It fails according to quasi-realists, for example, who will reject counterfactual like "if moralism is false, then all actions are permissible." Rather, if one has the appropriate pro-attitudes towards helping the poor, quasi-realists will go in for strong counterfactuals like "I should have helped the poor even if I had had different pro-attitudes". Similarly, moral relativists don't have to be subjectivists, where subjectivists can be defined as those who accept the opposite counterfactuals: if I had had pro-attitudes towards suffering, suffering would have been good. Finally, I am not identifying another notion of "stakes" here beyond "what you should do." I also think the argument would fail for non-cognitivists who do not identify as quasi-realists, although I found those views harder to model. 

4. Even if AGI had values lock-in, what about belief shifts and concept shifts? It is hard for me to imagine an AGI that was not continuing to gather new evidence about the world. But new pieces of evidence could dramatically change at least its instrumental values. In addition, new pieces of evidence / new experiences could lead to concept shifts, which effectively  cause changes to its intrinsic values, by changing the meaning of the concepts involved. In one picture, for any concepts C and C*, C can shift over some amount of time and experience to have the intension currently assigned to C*.  If you have that strong shifty view, then maybe in the long run even a population of AGIs with 'fixed' intrinsic values will tend to move around vast amounts of value space. 

Also, regarding AGI lock-in, even if AGIs don't change their values, there are still several reasons why rational agents randomize their actions (https://web.stanford.edu/~icard/random.pdf). But chance behavior by the AGI dictator could over time allow for 'risks' of escaping lock-in. 

5.  Might be fruitful to engage with the very large literature on persistence effects in economics (https://economics.yale.edu/sites/default/files/understanding_persistence_ada-ns.pdf). They might have models of persistence that could be adapted to this case. 

Also curious what you guys would make of The Narrow Corridor by Acemoglu and Robinson, which gives a systematic model of how liberal democracies stably evolved through game theoretical competition between state and society. Seems like the kind of model that could be fruitfully projected forward to think about what it would take to preserve flourishing institutions. More generally, I'd be curious to see more examples of taking existing models of how successful institutions have stably evolved, and apply them to the far future.

6. I was a bit skeptical about the argument about Infernotopia. The issue is that even though utility is not bounded from below, I think it is very unlikely that the far future could have large numbers of lives with negative utility levels. Here is one argument for why: suppose utility is negative when you prefer to die than stay alive. In that case, as long as agents in the future have the ability to end their lives, it would be very unlikely to have large numbers of people whose utility was negative. The one caveat here is the specific kind of s-risk where suffering is adversarially created as a threat against altruists; I do think this is worth hedging against. But other than that, I'm having trouble seeing why there would be vast numbers of lives not worth living. 

You're definitely right that my original comment failed to explain the importance of redistribution! 

Hey Toby, thanks for your comment! I'm not sure we really disagree, because I agree that transfers are an important part of the picture, that's why I said "all of the interesting work is in designing anti-trust and tax measures that are robust to superintelligence". But I agree that my comment insufficiently emphasized the redistributive aspect. The overall point is that we may not need to design a brand new set of institutions to deal with the rise of superintelligence. At least one hypothesis worth considering is that markets plus avoiding market failure plus redistributive tax and transfer will be sufficient. We already have lots of political institutions that seek to remedy distributive problems associated with markets, and maybe these institutions will scale with growth in GDP. Or maybe not! But at least, I think this is a great place to start for the analysis, as opposed to from scratch 

Great post, here are three reactions:

1.  Markets as viatopia.

You say that "Without a positive vision, we risk defaulting to whatever emerges from market and geopolitical dynamics, with little reason to think that the result will be anywhere close to as good as it could be."

In fact, I think markets are potentially the best pathway for viatopia. The fundamental theorems of welfare economics suggest that without market failures, market dynamics will produce outcomes that maximize social welfare. One route to the best superintelligent future is simply to avoid market concentration (more than one superintelligence), and have governments impose Pigouvian taxes on externalities, the end. Then all of the interesting work is in designing anti-trust and tax measures that are robust to superintelligence. No easy task, but at least it is well-defined. Here, one more general question is whether superintelligence would destroy any of the fundamental conditions that allow markets to maximize social welfare. 

I think of this market-based approach as a version of viatopia as opposed to "utopia" in the narrow sense you defined. The point of markets is that rather than trying to define and control the best future directly, we allow that future to emerge through the market. 

2. Evolutionary mechanisms for viatopia.

Your post doesn't emphasize evolutionary perspectives on viatopia. One strategy could be to ensure diversity across civilizational types, along with mutation mechanisms, and try to create selection mechanisms which correlate fitness with welfare. In that case, we can expect welfare to increase in the long run. 

3. To promote viatopia, you focus on a series of societal primary goods. But these primary goods are plausibly also the focus of protopian efforts. What are some examples where protopianism and viatopianism recommend different actions? 

"I don't think that the Spanish flu made us more prepared against Covid-19" actually I'm betting our response to Covid-19 was better than it would have been without having had major pandemics in the past. For example, the response involved developing effective vaccines very quickly

Would be good more generally to have an updating record of the most important AI safety papers of each year 

Great post! A few reactions:

1. With space colonization, we can hopefully create causally isolated civilizations. Once this happens, the risk of a civilizational collapse falls dramatically, because of independence.

2. There are two different kinds of catastrophic risk: chancy, and merely uncertain. Compare flipping a fair coin (chancy) to flipping a coin that is either double headed or double tailed, but you don't know which (merely uncertain). If alignment is merely uncertain, then conditional on solving it once, we are in the double-headed case, and we will solve it again. Alignment might be like this: for example, one picture is that alignment might be brute forceable with enough data, but we just don't know whether this is so. At any rate, merely uncertain catastrophic risks do not have rerun risk, while chancy ones do. 

3. I'm a bit skeptical of demographic decline as a catastrophic risk, because of evolutionary pressure. If some groups stop reproducing, groups with high reproduction rates will tend to replace them. 

4. Regarding unipolar outcomes, you're suggesting a picture where unipolar outcomes have less catastrophic risk, but more lock-in risk. I'm unsure of this. First, unipolar world government might have higher risk of civil unrest. In particular, you might think that elites tend to treat residents better because of fear of external threats; without that threat, they may exploit residents more, leading to higher civil unrest. Second, unipolar AI outcomes may have higher risk of going rogue than multipolar, because in multipolar outcomes, humans may have extra value to AIs as a partner in competition against other AIs.