14-word summary: What if weird stuff matters a lot but no one cares about weird stuff?

In the long term future, how much of a mismatch will there be between what matters most and what agents value? The hypothesis I want to consider is that the mismatch will be enormous, such that a tiny fraction of value is captured, perhaps as small a fraction as 10^-20.^[1]

The reason is that moral value seems to be a power law. A few things are extremely valuable and most things are not. More specifically, over all of the different things that humans like and find morally appealing (or would if that thing were detailed) a very small subset of those things are many orders of magnitude more important than others.

Suppose furthermore that it is not likely that the moral arc of the universe bends towards justice, such that our descendants ultimately converge on valuing the right things in the right ways.

Then there is a powerful argument that the expected value of the future is very low. If morality is profoundly power-law distributed across human motivations, then we’re likely to attain only a very small amount of total value if human motivations shape the long-term future. Given what people want to create, promote, and preserve, we should expect the vast majority of what exists to be of minimal value.^[2]

But if it’s not very power-law distributed, we should expect to attain most long-run value if human motivations shape the long-term future, since most of what humans want to promote is of roughly the same value.

The value of the future, relative to what is possible, is a direct function of the shape of the underlying distribution of value across state space.

Let’s formalise the hypothesis slightly further:

Moral Power Laws (MPL): Across all of the states of the world that humans like intrinsically and want to promote and preserve (weighted by the strength of human desires to do so), moral value is a power law.

To get the intuition for what this means, you can imagine a two-dimensional line plot.^[3] On the x-axis are properties that humans like. These properties get more space on the x-axis the more humans like them, and they’re arranged in order of moral value. On the y-axis is moral value. In linear scale, the line plotted is a hockey stick.

Here are some intuition pumps for buying the Moral Power Laws hypothesis:

1. There are a lot of ways to arrange 86 billion neurons. You could give them to one human, to 430 rats, or to 86 billion nematodes. Intuitions differ on the direction of the effect, but probably one of these is vastly more valuable than the other two.
2. Beyond biology, there are probably many ways to create conscious awareness, but some of these ways are vastly more efficient than others. Neurons are likely an inefficient medium for awareness.
3. For classical utilitarians, “hedonium” is likely many orders of magnitude more valuable than human brains (or the equivalent instantiated in silico). Bostrom calculates that, at the limit, computers could use matter about 10^20 times as efficiently as brains for whatever computations they do. And beyond that there is wire-heading, i.e. cutting away all of the computations in human brains inessential for pleasure. Given how many functions the brain performs, and how infrequently the pleasure centres are turned on, this should increase the total amount of pleasure at least a few orders of magnitude.

Benthamite utilitarians aren’t the only ones who look like they might want to buy into the Moral Power Laws hypothesis. Lots of non-utilitarian moral theories are fragile,^[4] where seemingly tiny and unimportant differences make massive differences in moral importance.

1. On views where diversity is valued intrinsically, the value of a thousand identical spheres is (say) 1/1,000 of the value of those spheres with a single electron varied in each.
2. Given various kinds of lexicality (e.g. Millian superiorities), a tiny amount of one object will be vastly better than a lot of that object slightly modified. (Consider the precise borderline between Mill’s higher and lower pleasures.)
3. If you value pretty much anything (e.g. consciousness, desire satisfaction), there’s likely to be a sharp line in phase space where a tiny change to the property makes an all-or-nothing difference to value. And these are ill-defined notions in folk theory, meaning that it may be very difficult to identify precisely the border where the phase change happens, such that we could get it wrong.

For moral theories where value is extremely fragile in this way, you could easily miss out on most of the value from getting small, seemingly unimportant details wrong.

So with the intuition pumps out of the way let’s evaluate the hypothesis.

Is the Moral Power Laws hypothesis true?

Let’s assess whether MPL seems true today and how that will change in the future, before moving on to specific objections. The key ideas here are twofold:

1. In the future we’re going to have numerous options and pursue a wider variety of possible goals as a consequence, leading to more divergence between humanity’s values and what matters.
2. From a number of moral viewpoints, it looks like there are going to be things that matter many orders of magnitude more than anything else, and which almost no one is very interested in making happen.

Return to the semi-metaphorical line plot constructed in the opening. The x-axis is what humans value, arranged by their importance, with more space going to things that humans value more. The y-axis is value. How hockey-stick shaped is the line that we would plot on that graph today, and how will that plot change in the future as technology makes more things possible?

Whether MPL holds today depends substantially on how much you value nonhuman animals. If you don’t value nonhuman animals at all, then you might think human values are pretty well correlated with what matters. Sure, humans value intrinsically and want to promote much that isn’t intrinsically very valuable (like Notre Dame — sorry), and could care more about their outgroup. But in general, we put most of our moral weight on humans’ welfare, and in a fairly universalist way, which is where about all of the value is. We don’t care that much about pleasure, and could desire to optimize it more, but we are pretty constrained by technology and nature in what we can do, so we wouldn’t gain that much more value if we did.

But if nonhuman animals are valued significantly, the graph quickly becomes a hockey stick. Humans in the round care little about farmed animals or wild animals (let alone insects), who may account for around 99.99999% of total value.

So depending on your values you might think that our graph is a sloped line or a sideways L. Even if it’s a sideways L, though, the fact that humans care fairly universally about human welfare is a good start. If animals account for around 99.9999% of total value, then human values are still correlated with about 1 in a millionth of total value, which could be worse.

The direction of history seems overall pretty good regardless of your perspective: however much suffering there is in the world today, we care a lot more than we did in years past about outsiders’ welfare, including that of nonhuman animals.

However, in the future I expect this trend to reverse, and for value to be much more power-law distributed as our option space changes drastically. As humanity wins victory over nature, the world will increasingly be much less selected for by biological necessity and much more selected for by human desires and the physical laws. In this situation we won’t be constrained to a relatively small range of valuable objects (animals) but will instead be able to create any conceivable thing in value space.

So the option space is going to get much bigger. We’ll have to make choices about whether to preserve habitable solar systems for biological life or turn them into Dyson swarms; or about whether to turn each Dyson swarm into insentient digital slaves to learn and accrue power, 10^20 human minds in silico, a single God-like mind fully consuming all of the star’s energy, or 10^40 disintegrated momentary pleasure blips — and thousands of nearby computational variants of each of these.

In addition to increasing option space, I expect there to be increasing possibilities for variation in what we value. In a slow takeoff scenario, where there is no value lock-in and humans gradually settle the stars and develop their own local cultures and tastes, I’d expect humans to come to value a very wide variety of things. If you think the value differences between Islam, Hinduism, Catholicism, Republicans, leftists, environmentalists, rationalists, transhumanists, utilitarians, Kantians, and Aristotelians are large, they’d grow vastly larger again with a much wider set of possibilities for us to actualize as well as intellectual and aesthetic currents to explore. In this world I would expect massively greater divergence by default.

(I don’t know if my folk history is accurate, but my understanding is that early religions and cultures had a lot in common with each other — e.g. big gods, animism — because these groups had a lot in common economically and geographically. Things seem vastly more ideologically varied today due to some combination of time, information, and possibility.)

While I don’t expect slow takeoff like this (I instead expect eternity in 6 hours), the slow takeoff scenario is an intuition pump for how much divergence we could see in different fast takeoff futures. If a slow takeoff would see this much divergence across space, then at a first pass one would expect to see this much divergence as well across alternative future trajectories in lock-in worlds.

So in the future we’re going to have numerous options and pursue a wider variety of possible goals as a consequence.

A key question, then — echoing back to our dilemma around nonhuman animals — is whether these widely diverging goals are fairly similar in moral value, or whether some of these goals are vastly more important than the others.

The simplest way to argue that MPL will hold over our descendants’ future option space is to point to some things that humans won’t want to create a lot of but which are overwhelmingly valuable. Then MPL follows.

On classical utilitarianism this is straightforward. If what you want to maximize is the aggregate amount of total pleasure in the universe, you are going to have to pursue something wholly bizarre. Let’s go back to 86 billion neurons. There’s currently very little that can convincingly be said for or against allocating those neurons to one human, 430 rats, or 86 billion nematodes. In this Earthly case, we might already find that the utilitarian is going to make some odd judgments that fly in the face of what most humans care about. Now let’s ratchet up the neuron count to 86 trillion. Will the utilitarian want to create 1,000 humans, or instead create a much larger mind consisting of all of these neurons? It would be very surprising if, between a single archangel’s brain, 1,000 human brains, 430,000 rat brains, and 86 trillion nematode brains, the function from neurons to value was such that the human brains sat directly in the moral optimum.^[5] The human brain is probably not the global optimum for hedonic value. But that’s unfortunate, because humans really like human brains, and don’t much care for hedonium.

Suppose you find it boring to launch the same hedonic shockwave over and over again, and instead value diversity intrinsically. Then you might want to take great care to ensure that we have a future mirroring Thomas Aquinas’s ontology,^[6] where every creature that is possible under Heaven has been created, and ensure that we don't lock in a small set of possibilities, destroying most value in the process.

And this argument generalises. Probably our evolved biological substrates are nowhere near the global optimum for anything. So if due to our ancestral environment we place disproportionate weight on the Earthly properties that we nostalgically hold dear, we’ll miss a large amount of value.

Overall, I think there is a good prima facie case that morality will be power-law distributed in the distant future on a range of ethical views.

I want to now consider a couple of specific objections from metaethics and ethics, and then finally consider a realistic future in which MPL is somewhat less relevant to the case for moral trajectory change than it may seem.

How value might not be power-law distributed part I: Metaethics

Here’s a dilemma for the Moral Power Laws hypothesis:

1. Either moral realism is true or moral antirealism is true.^[7]

2. If moral realism is true, MPL is false.
3. If moral antirealism is true, MPL is false.
4. Therefore, MPL is false.

The intuition is this: if moral realism is true, we’ll converge on the best world. If moral antirealism is true, then bizarre edge cases like hedonium don’t matter, because humans don’t care about bizarre edge cases like hedonium.

My main reply is that this is much too quick, and it depends on a lot of choice points. The key reply to the moral realist is that morality is not like gravity, some universal attraction that pulls everything together over infinite time. Whether to expect convergence on the best outcomes is highly sensitive to initial conditions. The key reply to the moral antirealist is that valuers can greatly value some strange things, even if these things aren’t written in the stars. But both objections have some bite.

Moral realism

Philosophers are generally attracted to three kinds of moral realism.^[8]

1. Non-naturalist moral realism
2. Naturalist moral realism
3. Kantian constructivism

On non-naturalist moral realism, morality is like a Platonic Form. There’s a truthmaker for moral statements independent of any physical property or any human desire. However, for non-naturalists morality has no causal power. The standard view is that humans just happened to get lucky by stumbling upon the right moral view. Morality doesn’t pull us towards it, it’s just out there and some of us happen to be lucky enough to agree with it. If this view is true, then we should very simply not expect convergence by default, since morality is not an attractor state.

On naturalist moral realism, morality is a natural property like mass or charge which we detect with our senses. For example, some naturalists think we detect moral properties through empathy, or through introspection, instead of, say, through vision and touch. If this is true then we should expect some convergence towards morality since humans can’t help but sense it. But we better hope that our descendants share the same humanistic senses, otherwise they will not themselves be attracted towards it.

On Kantian constructivism, there are attractor states in reasoning. Once you get in a certain kind of reasoning game, you’re forced by your own lights to a particular moral conclusion. Here again, we need to ask what sorts of reasoning properties you need to access this space of moral reasons. Do next-token prediction algorithms have the right architecture and foundational normative concepts to find their way into this attractor state? If not, then we should worry that our digital descendants will not find their way to the best world.

So on all of these popular realist metaethical views, there’s an open question whether to expect convergence, or how much convergence to expect. All realist metaethical views seem variously sensitive to the initial conditions of moral deliberation. Which makes sense — if you’re detecting a property you need to make sure your property detector is in good order.

Moral antirealism

Moral antirealism as I am articulating it here is the view that moral realism is false — faultless disagreement among ideal moral reasoners is possible. Classically, antirealists tend to think that morality is reducible to our values.

There’s two types of objections the moral antirealist might make to MPL. One is more convincing than the other.

First, the antirealist might complain that MPL is false by definition — things can’t be more valuable than how much we value them, since morality reduces to our values. So we’ll value things exactly as much as we should.

This objection falters for two reasons. One is moral disagreement. Perhaps in aggregate humans value things exactly as much as we should, but we don’t value hedonium (say) as much as you do. So you have a strong reason to change that fact. The second is that antirealism allows for moral error and learning. Someone who isn’t deeply in touch with what they care about can come to be better in touch with what they care about and thereby come to value things as much as they, in an important sense, should. So the analytic objection doesn’t work.

Second, the antirealist might doubt that anyone cares about very strange properties like maximal hedonic value. All of us grew up on Earth and evolved from Earthlings, so we care about Earthly things. Some of us might think we value heavenly things like hedonium, but if we thought carefully enough and learned enough cognitive science we would realize that we actually don’t have values that are an exponential function of aggregate pleasure. Perhaps we’d find that we are indifferent between lots of different ways of organizing matter, and value hedonium and shmedonium (its close cousin) and all of its other variants equally, even if they have vastly different computational properties.

I think this objection is a good one. If your values are just not very determinate over things not of this Earth, then you might not care much at all whether we turn every Dyson swarm into a single God-like mind or 10^40 momentary pleasure blips — or any of their nearby computational variants. And if you encountered all of the arguments for and against each alternative, and learned everything you could about the physical world, you still wouldn’t move an inch, or sway slightly one way or the other.

I think it’s a good objection, but I just don’t know if this is true. Certainly, I find myself swayed towards specific views on how much to value various insect minds. I came around to thinking that the whole of insect life is intrinsically more valuable than the whole of the human species. Is this just bad faith, and I should really be speciesist or indifferent, or is this getting in touch with what really matters to me? I find it difficult to say. But the objection is clearly not a knock-down, nor can it be dismissed.

Summing up

Moral realists still have to be worried about missing out on convergence, and moral antirealists can seemingly still value some strange things highly. So whichever horn you take on the dilemma, you should be open (at least on this basis) to the possibility that moral value is power-law distributed.

How value might not be power-law distributed part II: Ethics

There do however seem to be ethical views on which MPL is false. Here are a few:

1. You could accept diminishing returns to value in utility (with average utilitarianism as the most extreme version of this) so that happiness doesn’t matter much after you’ve already got a lot of it — but you’re unlikely to be a longtermist, laser focused on extinction risk if you do.
2. You could think that we are just on a razor-thin lucky edge where human brains are really good for making value, or that an axiologically serendipitous form of functionalism is true where the most computationally efficient world is also the happiest world.
3. You could accept deep incommensurability in value, as with the antirealist considered above, such that many futures can simply not be compared to one another, and you’d be indifferent towards them.

These views seem striking in their unattractiveness, but there are probably other, more attractive ethical views that do not imply MPL, or imply it less extremely.

A lower bound? How we might still capture most value given MPL

I’ve argued that MPL is quite plausible. A natural inference to draw is that there’s a good chance that our descendants attain only a power-law-small amount of total value, since by implication their values will be mostly uncorrelated with what matters most. But there are some arguments that we could nonetheless attain a large portion of total value, even if human values in the round are broadly uncorrelated with what matters.

The best argument I’ve heard for how we might capture a large portion of value, even if MPL is true, is the implicit argument in Will MacAskill’s afterwords to What We Owe the Future. (Spoilers ahead, you could first read the short story here.) In the story, humanity develops ASI and plans to settle the universe. Before doing so, they split up all of the stars in the affectable universe evenly amongst all of humanity and allow humanity to trade and bargain for stars amongst each other. The environmentalists trade for the habitable star systems (a small portion of the cosmos), so that they can preserve biological life. The fictional stand-in for classical utilitarians trade for a large number of the very far-away star systems – as they are patient and can wait a long time to promote moral value — and eventually use it to make hedonium. As a result, the environmentalists capture most of moral value by their lights, preserving all of the habitable planets. And classical utilitarians also capture most moral value by their lights, bargaining for an outsized proportion of the universe and the majority of usable stars, and maximizing utility there.

If the universe is settled via ideal trade and positive-sum cooperation, then most people might be able to capture most of what they value, by their own lights, even if morality is power-law distributed and even if moral disagreement is widespread. While I very much don’t expect this to happen by default, it shows the possibility of attaining most moral value even under MPL. Even if people’s values are in general highly uncorrelated with what matters most, outcomes like trade and cooperation can still allow most value to be attained. I think this is a very powerful argument for trade-based futures, and in general advancing futures where the cardinality in humanity’s utility function is well-preserved by judgment aggregation instead of using coarser methods of judgment aggregation like first-past-the-post voting.

If MPL is true, then the likelihood that an ideal form of trade and cooperation happens, multiplied by the amount of resources allocated to agents whose values are aligned with the best outcomes, forms one kind of lower bound for the value of the future.

On the other hand, If we don’t achieve ideal trade and positive-sum cooperation, or you can’t personally steer the world in the direction of your values, then you should take the MPL hypothesis as an argument that the expected total value of the future is extremely low compared to its possible ceiling.

On the positive side, it may also imply that there is still a role for effective altruism in 202500.