Comments
I previously wrote about "galactic x-risks" and cautioned on the long-term consequences of interstellar travel on the EA forum in a post called “Interstellar travel will probably doom the long-term future”, and this is a follow up to that post. The reason for the follow-up is that I outlined a tremendous threat and then left the solutions very open, and I was prompted to revisit the solutions as Betham’s Bulldog recently wrote a blog post about the issue and gave it a great name: “The Vulnerable Universe hypothesis”.
I’d recommend reading the post from Bentham’s Bulldog for a summary and then coming back here if you’re not familiar with the concepts, but I’ll present my own brief summary here in case you’re allergic to good writing.
In brief, in the last post, “Interstellar travel will probably doom the long-term future”, I outlined x-risks that could destroy a galactic civilisation. These “galactic x-risks” are highly theoretical and I guessed that at least half of them haven’t been discovered yet. The four I’m most worried about are:
Memetic hazards: Similar to mindset hazards, but for spacefaring civilisations, some extremely dangerous and contagious piece of information or an idea, like a joke so funny you die, but bad enough to topple civilisations or convince them to abandon their moral goal[1]. The trouble with this one is we might never prove its non-existence, maybe even with ASI.
Superluminal travel: If you can travel faster than the speed of light somehow (e.g. wormholes or warp drives), then you might be able to break causality[2] and destroy everyone across a galaxy in a short amount of time and there’s no defence because it’s impossible to detect.
If we send an interstellar spacecraft to colonise another star system, we will likely initiate a self-propagating expansion of human civilization (or we might just send probes to every star system at once[3]). This expansion will almost certainly de-correlate the paths of the long-term future of human civilization, likely initiating a series of events that leads to the existence of independent civilizations across the galaxy. And to quote Toby Ord on galactic x-risks:
settling more and more star systems doesn't just not help with these — it creates more and more opportunities for these to happen. If there were 100 billion settled systems then at least for the risks that can’t be defended against (such as vacuum collapse) a galactic-scale civilisation would undergo 100 billion centuries worth of this risk per century. So as well as an existential-risk-reducing effect for space settlement, there is also a systematic risk-increasing effect.
So, if we expand to many other star systems without a great plan, and galactic x-risks are real, then we are likely to increase the probability of a galaxy-wide existential catastrophe occurring in the long-term future. In this post, I’ll try to present a great plan to overcome this risk[4].
I want to make it very clear right away that I think the most tractable thing to work on in response to this threat is AI alignment and general interventions that get the world closer to viatopia.
To create a plan to mitigate the threat of galactic x-risks, multiple areas of currently unresolvable uncertainty need to be grappled with, which means any plan developed now is extremely likely to be very bad and, if we solve alignment, replaced by ASI. Some examples of the currently unresolvable uncertainties include:
Despite the above points, I do think coming up with grand plans has a lot of value in some areas, for example:
One approach to solving galactic x-risks is to focus on setting up defenses against them early on in the design of a galactic civilization, for example:
As star systems are colonised, equipment to detect and repulse (or even contain) strange matter[5] might be set up, e.g. through gravitational waves or particle beams.
The problem with this strategy is that some galactic x-risks may have no conceivable defence, like superluminal travel (hiding really well?) and vacuum decay (accelerating away at near light speed hopefully until the great separation?), or we might not know what they are in advance of expanding to other star systems.
Another approach to solving galactic x-risk is to investigate the features that cause actors[6] in space to inevitability initiate galactic x-risks, and avoid those features during cosmic expansion. The inevitability arises from actors being:
Divergent - indicating that their actions are taken independently of each other and they can diverge in their value systems and moral goals[7].
If any one of those 3 features are made false, then the threat of galactic x-risks diminishes.
Incidentally, I think a lack of divergence is one of the strongest cases against galactic x-risks being a threat, as if moral realism is correct and every star system is likely to converge on the same moral goals, then maybe galactic x-risks are much more unlikely. Also, maybe we send the same AI systems to every star system at the same time, so they all do ~the same thing with their stars. Divergence is also probably the feature that is most tractable to break.
In breaking any of these features, certain outcomes must be preserved to make sure we still take advantage of the cosmic endowment:
As many of the resources of the accessible universe as possible should be exploited for creating moral goodness[8]
So, solving galactic x-risk requires finding a grand plan that eliminates the risk whilst compromising least on the outcomes that control our ability to access the cosmic endowment. For each grand plan, I’ll delineate between (1) the feasibility of the grand plan and (2) compromises/trade-offs on the outcomes for civilization.
As a reminder, the grand plans I will outline solve just 1 challenge associated with large scale space expansion, but there are many more and harder challenges, like defining “moral good”. So the grand plans in this post shouldn’t be read as an overall architecture for grand space expansion but as potential features of a galactic civilization that might be required to solve one particular problem.
The first feature that leads to the inevitability of galactic x-risks occurring is the existence of powerful actors in space, which includes civilizations spanning star systems or individual superintelligences controlling vast resources. As a result, they would be capable of initiating galactic x-risks intentionally or accidentally, and to prevent this, they would have to be disempowered.
One method to disempower actors in space is to establish a limit on resource ownership based on the lowest threshold that would allow a civilization to initiate a galactic x-risk. This might be possible for preventing some galactic x-risks which might require huge quantities of energy to create, like strange matter[9], vacuum decay, wormholes, runaway subatomic particle decay[10], and interactions with other universes. In this scenario, other easier-to-initiate or non-energy-limited galactic x-risks are assumed to not pose a threat or not exist, like self-replicating spacecraft or memetic hazards, respectively. Once a resource ownership limit has been established, be it 1 star system or a million, it would have to be enforced. Limits on resource ownership enable easy enforcement of the resource limit as the enforcer would be the only actor with legal access to enough resources to eliminate any over-expanding actors. I think this works well practically as an overall architecture, but there are some hard questions about how the enforcer would distribute their power optimally to ensure rapid responses while maintaining enough concentrated power to enforce rules. Another practical challenge is that multiple actors might combine their resources to initiate a galactic x-risk, or might just want to combine their civilizations for cultural reasons, which would break the resource limit law. So, well specified laws would have to be designed around collaboration, likely with the enforcer overseeing communications and drawing borders around resources. On compromising on the cosmic endowment, the vast resources of the enforcer could be used for pre-defined moral goodness while they aren’t being used for enforcement, and the inhabitants of this universe would still have a lot of freedom with the ability to live blissfully should they choose. The enforcer would have to be carefully designed to ensure that it doesn’t use the powers for other less noble s-risky activities, potentially programmed and overseen by a council. Even so, it may still be susceptible to corruption, value drift, the enforcer itself initiating a galactic x-risk.
In general, I also worry that a galactic enforcer that monitors communications, draws borders, and can eliminate over-expanding actors is in deep tension with free will.
An alternative enforcement mechanism for the resource limit might be to rely on strong norms between actors in space, such that if one actor goes over the agreed upon resource limit, multiple other actors collaborate to contain their power. It would be in the interest of all of them to do that, not just to prevent galactic x-risk, but to balance power against dominance of one actor. This does introduce more points of failure though rather than just one with the enforcer, which makes it fall victim to the same dynamics that cause galactic x-risks in the first place. All it would take is a few civilizations to decide that the resource limit is dumb and combine their resources for galactic x-risks to become possible again. And if there are billions of civilizations with diverse goals and value systems, then it’s probably going to happen a lot.
If it turns out that the resource limits can’t be effectively enforced by strong norms or a stronger enforcer then it might be possible to integrate an oversight mechanism into the infrastructure of all civilizations in space. Quoting Jacob Watts on this idea:
Imagine, for example, a more distributed machine intelligence system. Perhaps it's really not all that invasive to monitor that you're not making a false vacuum or whatever. And it uses futuristic auto-secure hyper-delete technology to instantly delete everything it sees that isn't relevant. Also the system itself isn't all that powerful, but rather can alert others / draw attention to important things.
And system implementation as well as the actual violent / forceful enforcement that goes along with the system probably can and should also be implemented in a generally more cool, chill, and fair way than I associate with the centralized surveillance and control systems.
This idea sounds practical to me with highly sophisticated technology, and could likely draw a lot of attention to efforts to produce a galactic x-risk long before construction of the necessary equipment even begins.
Civilisations could become causally independent of each other by moving really fast away from each other. Then, due to the expansion rate of the universe, they would eventually be separated from each other forever. I think this works, at least on small scales (moving a whole galaxy has practical challenges and is also much too big to significantly change the dynamics of the vulnerable universe hypothesis). The problem is that this has major compromises on the desired outcomes. The resources required to move star systems for however long is required to reach near light speed and/or become separated from all other star systems is a lot of resources, and these resources could have been spent on cathedrals and happy experiences.
As Toby Ord notes, existential security rises with space expansion for a while but then decreases with very large scale expansion. Therefore, there is an optimal number of de-correlated space colonies to maximise existential security. If all civilisations are correlated, then one memetic hazard or a virus affecting overarching infrastructure could cripple the whole civilisation. But if civilisations are too de-correlated and too abundant, the probability of at least one of them initiating a galactic x-risk approaches certainty over long timescales. The sweet spot likely involves enough independent colonies to be resilient against local and correlated catastrophes (like asteroid impacts or infrastructure failures), but few enough that they can be meaningfully coordinated or monitored to prevent galactic x-risks. Finding this number depends heavily on unresolved questions, like how defense-dominant space is, how quickly values drift across isolated colonies, and which galactic x-risks actually turn out to be real, so it's probably something that would need to be continuously re-evaluated as our understanding of physics and the threat landscape develops.
Another approach is just to ensure that every new star system that is colonised will do a predictable thing, or at least, the same thing as all the other ones. There could also be updates to AIs controlling these star systems with briefings on new galactic x-risks that are discovered or updates on moral theories. There would be a centralised location that would act as the cultural and normative anchor. A centralised location does create a dangerous single point of failure though.
It's worth dwelling on the role of moral convergence here, as I think it may be one of the strongest reasons to be less alarmed by the vulnerable universe hypothesis overall. If moral realism is correct and sufficiently advanced civilizations tend to converge on the same moral truths (or even if something weaker holds, like convergence driven by shared evolutionary pressures, game-theoretic incentives, or the influence of similarly-designed AI systems) then the divergence problem largely dissolves. Civilizations that arrive at the same conclusions about what matters are unlikely to recklessly pursue experiments that risk destroying everything. This doesn't require perfect agreement, just enough convergence on the specific question of whether to do things that could trigger galactic x-risks. The degree to which you find the vulnerable universe hypothesis threatening should therefore be closely tied to your views on moral convergence: if you think advanced civilizations will reliably converge on not destroying the universe, then the vulnerable universe is much less vulnerable than it first appears. Conversely, if you think value divergence is deep and persistent, galactic x-risks become correspondingly harder to manage through any mechanism other than direct enforcement or disempowerment.
The most straightforward way to eliminate abundance is to simply not expand. Staying on Earth or within the Solar System forever obviously compromises the cosmic endowment, so it's not a serious option. The same is true for keeping all of civilization very small, like just a few star systems, and not expanding further.
Another answer is to expand into the universe for purely instrumental reasons, harvesting resources and energy without allowing independent powerful actors to exist beyond our home system. Star systems colonised this way would have no people, but would contribute to our goals in some other way, such as generating pure happiness through hedonic computation or robotic energy-harvesting for some later project. If there are no independent actors out there, there's nobody to initiate a galactic x-risk.
A more radical approach is to not have actors in space at all, not even instrumental outposts, and instead find valuable uses for the cosmos that don't involve expanding our civilization to new star systems. Post-ASI, it's possible that our value systems could entirely change and the concept of spreading a civilization to other star systems could be seen as outdated, weird, or immoral. After something like a long reflection, our civilizational goals might look nothing like what we currently imagine. There are many examples of human goals or expectations for the future changing in a huge way. Antarctica was expected to be used for mineral extraction, territorial expansion, and strategic control, but we decided it was more valuable to maintain it as a pristine environment for wildlife, scientific research, observatories, and tourism. The Moon was expected to be used as a military high ground for Cold War dominance, but instead it has served as a geopolitical symbol and a platform for scientific research. I can imagine a similarly dramatic shift in our expectations for the stars, but personally, I look forward to (a much wiser) human civilization expanding throughout the cosmos.
Another answer is to allow actors to expand in a digital world (which simultaneously disempowers actors and stops them being abundant in the universe). This way, it would be possible to “have people flourishing inside virtual worlds maintained by machines, where the people have no way of affecting the outside world”. Practically, some, like David Pearce, have argued that digital sentience is impossible. However, if it is possible (and I personally think it is[11]), I don’t see many practical challenges with this. As far as preserving the cosmic endowment goes, I think this could still be a candidate for a near-optimal future. Digital sentience is a more energy efficient way to produce positive experiences than using biological beings. However, if you believe the sentient experience is likely to be negative, then maybe you hate this plan.
It's also worth briefly revisiting the assumption I made earlier that aliens don't exist, because the abundance problem is where this assumption does the most work. If other civilizations exist elsewhere in the galaxy then abundance is not fully within our control. We cannot choose not to expand and thereby keep the number of powerful actors low, because other actors may already be out there or may emerge regardless of what we do. This changes the calculus. In a universe with aliens, the enforcer model becomes far harder to implement, since it would require either establishing dominance over civilizations we didn't create and don't share a history with, or negotiating enforcement agreements across potentially vast differences in values, biology, and technology. The strong norms approach faces even steeper obstacles, as norms require some shared cultural or game-theoretic foundation that may not exist between independently evolved species. Ironically, the existence of aliens might actually strengthen the case for rapid expansion, since a civilization that stays small and cautious could find itself unable to influence galactic governance at all or engage in acausal trade, leaving the management of galactic x-risks entirely in the hands of others who may not share our concern for them. This is one of the most uncomfortable implications of the vulnerable universe hypothesis: if aliens exist, many of the more cautious strategies become less viable, and the pressure to expand quickly and develop the capacity to coordinate with or defend against unknown actors increases substantially. The Fermi paradox therefore isn't just an interesting puzzle; our answer to the Fermi paradox is one of the most decision-relevant inputs into how we should approach the long-term future.
There are many options here to navigate the vulnerable universe, and each has some compromises to be made. The optimal future doesn’t mean that everything is perfect. There are many different options for our future even if the vulnerable universe hypothesis is true, which I think is very reassuring. It is, however, unclear which future is best.
I won’t pick a favourite plan because I don’t want to contribute to locking in any particular long-term goal for myself or anyone else. The important thing to navigate towards is achieving viatopia, so that we can reflect deeply on what future we want to create with the best possible tools and intentions. My hope is that this post updates all of us towards creating that future.
An interesting question here is how quickly could an ASI discover all of these memetic hazards? Intelligence and knowledge don’t seem to be limited in the sense that if you kept using more and more energy you could probably keep getting more intelligent and knowing more things indefinitely. But could there be something we could discover millions of years from now that could cripple a galactic civilization? Maybe we need to analyse data in-situ across the accessible universe to discover an important fundamental property of the universe, or a message from our creator.
So the attack could come in the past from your perspective, making it impossible to predict or defend against. ChatGPT explains how this might work here.
In six hours, to be precise
I did try to do this in the original post, but it wasn’t a very good plan.
Strange matter is an ultra stable form of matter that turns other matter into strange matter on contact. [Kurzgesagt YouTube video] [research article]
“Actors” can be anything from civilizations, an ASI system, a really powerful posthuman, or any other kind of agent that has goals.
I.e., if every star system was run by a computer running a program they would all do roughly the same thing, so the probability of a galactic x-risk being initiated would be very close to 0 or 1.
By this I mean that using 90% of the resources of every star system to protect them from the threat of self-replicating spacecraft and interstellar lasers would be a squandering of the cosmic endowment. It might be better to use those resources to simulate happy digital minds if we can.
Which requires extremely high densities to create e.g. neutron star mergers
Extremely speculative. I don’t think anyone has used those 4 words in sequence before.
I have been reading Ray Kurzwell recently
Executive summary: Assuming galactic-scale existential risks are real, the author argues that large-scale space expansion may increase long-run catastrophe risk unless we deliberately constrain power, divergence, or abundance, though which “grand plan” is best depends heavily on unresolved physics, moral convergence, and the existence of aliens.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.