Comments
Really cool thought, this is persuasive to me.
I’m curious, what more specifically do you find it persuasive of? I generally feel that people do not easily want to bite the bullets that experiences have no independent positive value (only interdependent value), and that value doesn’t aggregate, and that outweighing “does not compute” in a physical balancing kind of sense. (I haven’t yet read the 2016 Oxford Handbook of Hypo-egoic Phenomena, but I expect that hypo-egoic systems, like many or most kinds of Buddhism, may be an angle from which it’d be easier to bite these bullets, though I don’t know much about the current intersection of Buddhism and EA.)
If I can try to rephrase your beliefs: economic rationality tells us that tradeoffs do in fact exist, and therefore rational agents must be able to make a comparison in every case. There has to be some amount of every value that you'd trade for another amount of every other value, otherwise you'll end up paralyzed and decisionless.
You're saying that, although we'd like to have this coherent total utility function, realistically it's impossible to do so. We run into the theoretical problems you mention, and more fundamentally, some of our goals simply are not maximizing goals, and there is no rule that can accurately describe the relationship between those goals. Do we end up paralyzed and decisionless, with no principled way to tradeoff between the different goals? Yes, that's unavoidable.
And one clarification: Would you say that this non-comparability is a feature more of human preferences, where we biologically have desires that aren't integrated into a single utility function, or morality, where there are independent goals with independent moral worth?
Yes, we’ll unavoidably face quantitative resource splits between goals, none of which we can fully satisfy as long as we have even one infinitely hungry goal like “minimize suffering”, “maximize happiness”, or “maximize survival”. In practice, we can resolve conflicts between these goals by coming up with a common language to mediate trade between them, but how could they settle on an agreement if they were all independent and infinite goals? (My currently preferred solution is that happiness and survival are not such goals, compared to compassion.)
Alternatively, they could split from being a unified agent into being three agents, each independently unified, but they’d eventually run into conflicts again down the line (if they’re competing over control, space, energy, etc.). I’m interested in internal unity from the “minimize suffering” perspective, because violent competition from non-negotiating splitting causes suffering. In other words, I suppose my self-compassion wants my goals to play in harmony, and “self-compassion aligned with omnicompassion” is the unification that results in that harmony in the most robust way I can imagine.
More biological needs are more clearly in the domain of self-compassion, while omnicompassion is the theoretical attractor, asymptotically approximated ideal, or “gravity” that pulls self-compassion towards extended self-compassion, which is like a process of gradually importing more and more needs of others as my own, which increases expected harmony so long as it’s done slowly enough to maintain the harmony within the initial small self. For people with a chaotic life situation or who are working on a lot of unmet needs, this domain of self-compassion could occupy a lot of their attention for years while still being aligned with eventually extending self-compassion, which means that a seemingly non-helping person may be working on precisely the need areas that are long-term aligned with their role in minimizing suffering (or maximizing harmony or however one likes to think of it).
(I may have sidestepped your question of morality and moral worth, because I prefer to think in terms of needs and motivating tensions, and to see if the theoretical implications of unified consequentialism could be derived from our needs instead of abstract shoulds, obligations, imperatives, or duties.)
I used to think that things have independent positive value and that this value would aggregate and intercompare over time and space.
In other words, I used to believe in some sort of Cosmic Scoreboard where I could weigh, for example, someone’s lifetime happiness against their, or someone else’s, moment of suffering.
I now think that this clinging to things as independently valuable and aggregable contributes to theoretical problems like utility monsters, wireheading, the repugnant conclusion (and other intuitively grotesque outweighing scenarios), infinitarian paralysis, and disagreements in cause prioritization.
I now feel that my previous beliefs in independent positive value, intercomparable aggregates of experiences, and the Cosmic Scoreboard more generally were convenient fictions that helped me avoid ‘EA guilt’ from not helping to prevent the suffering I could—by believing that there could be more important things, or that the suffering could be outweighed instead of prevented.
I’d now say that no kind of outweighing helps the suffering, because the suffering is separate in spacetime; outweighing is a tool of thought we use to prioritize our decisions so as to not regret them later, not a physical process like mixing red and green liquids to see which color wins. We have limited attention, and each moment of suffering is worth preventing for its own sake.
We don’t minimize statistics of aggregate suffering on the Cosmic Scoreboard except as a tool of thought, while in actuality we arrange ourselves and the world so as to prevent as many moments of suffering as we can. Suffering is not a property of lives, populations, or worlds, but of phenomenally bound moments, and those moments are what (I think) we ultimately care about, are moved by, and want to long-term equalize and minimize, from behind the Veil of ignorance.
For more on the internal process that is leading me to let go of independent positive values, replacing them with their interdependent value (in terms of their causal relationships to preventing suffering), here is my comment on the recent post, You have more than one goal, and that's fine:
I don’t see a way to ultimately resolve conflicts between an (infinite) optimizing (i.e., maximizing or minimizing) goal and other goals if they’re conceptualized as independent from the optimizing goal. Even if we consider the independent goals as something to only suffice (i.e., take care of “well enough”) instead of optimize as much as possible, it’ll be the case that our optimizing goal, by its infinite nature, wants to negotiate to itself as much resources as possible, and its reasons for earning its living within me are independently convincing (that’s why it’s an infinite goal of mine in the first place).
So an infinite goal of preventing suffering wants to understand why my conflicting other goals require a certain amount of resources (time, attention, energy, money) for them to be sufficed, and in practice this feels to me like an irreconcilable conflict unless they can negotiate by speaking a common language, i.e., one which the infinite goal can understand.
In the case of my other goals wanting resources from an {infinite, universal, all-encompassing, impartial, uncompromising} compassion, my so-called other goals start to be conceptualized through the language of self-compassion, which the larger, universal compassion understands as a practical limitation worth spending resources on – not for the other goals’ independent sake, but because they play a necessary and valuable role in the context of self-compassion aligned with omnicompassion. In practice, it also feels most sustainable and long-term wise to usually if not always err on the side of self-compassion, and to only gradually attempt moving resources from self-compassionate sub-goals and mini-games towards the infinite goal. Eventually, omnicompassion may expect less and less attachment to the other goals as independent values, acknowledging only their interdependent value in terms of serving the infinite goal, but it is patient and understands human limitations and growing pains and the counterproductive nature of pushing its infinite agenda too much too quickly.
If others have found ways to reconcile infinite optimizing goals with sufficing goals without a common language to mediate negotiations between them, I’d be very interested in hearing about them, although this already works for me, and I’m working on becoming able to write more about this, because it has felt like an all-around unified “operating system”, replacing utilitarianism. :)
This would be my practical question as well, for the following reasons!
I don’t see a way to ultimately resolve conflicts between an (infinite) optimizing (i.e., maximizing or minimizing) goal and other goals if they’re conceptualized as independent from the optimizing goal. Even if we consider the independent goals as something to only suffice (i.e., take care of “well enough”) instead of optimize as much as possible, it’ll be the case that our optimizing goal, by its infinite nature, wants to negotiate to itself as much resources as possible, and its reasons for earning its living within me are independently convincing (that’s why it’s an infinite goal of mine in the first place).
So an infinite goal of preventing suffering wants to understand why my conflicting other goals require a certain amount of resources (time, attention, energy, money) for them to be sufficed, and in practice this feels to me like an irreconcilable conflict unless they can negotiate by speaking a common language, i.e., one which the infinite goal can understand.
In the case of my other goals wanting resources from an {infinite, universal, all-encompassing, impartial, uncompromising} compassion, my so-called other goals start to be conceptualized through the language of self-compassion, which the larger, universal compassion understands as a practical limitation worth spending resources on – not for the other goals’ independent sake, but because they play a necessary and valuable role in the context of self-compassion aligned with omnicompassion. In practice, it also feels most sustainable and long-term wise to usually if not always err on the side of self-compassion, and to only gradually attempt moving resources from self-compassionate sub-goals and mini-games towards the infinite goal. Eventually, omnicompassion may expect less and less attachment to the other goals as independent values, acknowledging only their interdependent value in terms of serving the infinite goal, but it is patient and understands human limitations and growing pains and the counterproductive nature of pushing its infinite agenda too much too quickly.
If others have found ways to reconcile infinite optimizing goals with sufficing goals without a common language to mediate negotiations between them, I’d be very interested in hearing about them, although this already works for me, and I’m working on becoming able to write more about this, because it has felt like an all-around unified “operating system”, replacing utilitarianism. :)
By “doesn’t have to work on reducing x-risk”, do you mean that they don’t want to?
I’d expect that negative utilitarians (NUs) do want to reduce x-risk, because
(1) x-risk is rarely an either/or risk of 100% extinction; instead, more x-risk probably correlates with more risk of extreme suffering (from non-total pandemics/disasters/wars/etc., and all of their after-effects)
(2) even facing a 100% human extinction, we’d want to account for our epistemic uncertainty of the conditions from which suffering can evolve (re-evolve on Earth, or be found elsewhere within the reach of our descendants)
NUs don’t necessarily jump to suicide as a solution, because helping others is an infinite game to live for, especially after accounting for the epistemic uncertainty of all possible forms of suffering and their evolution. There is further basic research on suffering to be done before turning off the lights and hoping that all the billions of exoplanets would have their own guardians.
It is a straw man argument that NUs don’t value life or positive states, because NUs value them instrumentally, which may translate into substantial practical efforts (compared even with someone who claims to be terminally motivated by them).
Thanks for the guidelines.
Is there any policy (or existing discussion) on commenting on old posts? Is anything relevant enough to be worth adding to posts that are years old, or do you prefer that replies (across years) become their own posts linking to the original ones?
This is not the case when there's a well-defined procedure for resolving such conflicts.
Yes, but there isn’t. The theoretical case for terminal value monism is strong because monism doesn’t need such a procedure. All forms of terminal value pluralism run into the problem of incommensurability; monism doesn’t. With monism, we can evaluate and compare x-risks and positive psychology states & traits (vs. suffering) by their instrumental effects for minimizing suffering (which may be empirically difficult, but not theoretically impossible). What more do we want from a unified theory?
Do we want the slightest (epsilon) increase in x-risk to end up weighing more than any suffering? If this is our pre-decided definition, are we going to give up on suffering as a terminal value, caring about suffering only if it increases x-risk?
For example, you can map several terminal values onto a numerical "utility" scale.
I can’t. Who can? (In a non-arbitrary way we could agree on from behind the veil of ignorance.) The analogy of a scale requires a common dimension by which the comparator can sort the two (mass, in the analogy of a scale). What is a common dimension for intercomparing suffering & x-risk, or suffering & positive states? An arbitrary numerical assignment?
To arrive at an impartial theory that doesn’t sanctify our self-serving intuitions, we’d want to formulate our terminal value pluralism, “behind the veil”, by agreeing on independent numerical utility values for different terminal value-grounded currencies, such as:
(1) +epsilon probability of human extinction
(2) +epsilon probability that someone undergoes, e.g., a cluster headache episode (or equivalent)
(3) +epsilon probability that someone instantiates/deepens a positive psychology state [requiring a common dimension for “positivity”, unlike monism]
With monism, we don’t need to agree on definitions & values for multiple such currencies. Instead, we want to ground the (dis)value of other values in their relationships to extreme suffering, which everyone already finds terminally motivating in their own case (unlike x-risk reduction or positivity-production, worth noting). I wouldn’t agree to any theory where extreme suffering can be outweighed by enough positivity elsewhere, because outweighing “does not compute”: positivity can only outweigh suffering if it reduces even more suffering, but not by itself, because a positive fantasy of infinite utility is not an antidote to suffering, because the aggregate terminal positivity physically exists only as a fantasy [an imaginary spreadsheet cell] that never interacts with our terminal suffering.
Without monism, how do we agree on the pluralist numerical values if we could end up undergoing the cluster headache-equivalent suffering ourselves (i.e., simulating an impartial compassion)? Are we to trust that cluster headaches aren’t so bad, when they’re outweighed according to a formula that some (not all) people agreed on?
Multiple terminal values will always lead to irreconcilable conflicts.
(1) Do you hold suffering to be a terminal value motivating its minimization for its own sake?
(2) Do you also hold that there is some positive maximand that does not ultimately derive its value from its instrumental usefulness for minimizing suffering?
Anyone who answers yes to both (1) and (2) is not a unified entity playing one infinite game with one common currency (infinite optimand), but contains at least two infinite optimands, because with limited resources, we will never fully satisfy any single terminal value (e.g., its probability of being minimized or maximized throughout space & time).
I’ve been working on a compassion-centric motivation unification (as an improvement of existing formulations of negative utilitarianism) because I find it the most consistent & psychologically realistic theory that solves all these theoretical problems with no ultimately unacceptable implications. To arrive at practical answers from thought experiments, we want to account for all possibly relevant externalities of our scenarios. For example, the practical situations of {killing children} vs. {not having children} do not have “roughly the same outcome” in any scenario I can think of (due to all kinds of inescapable interdependencies). Similarly, compassion for all sentient beings does not necessarily imply attempting to end Earth (do you see Buddhists researching that?), because technocivilization might reach more & more exoplanets the longer it survives, or at least want to remain to ensure that suffering won’t re-evolve here.
To further explore intuitions between terminal value monism vs. terminal value pluralism, can you order the following motivations by your certainty of holding them as absolutes?
(A) You want to minimize suffering moments.
(B) You want to minimize the risk of extinction (i.e., prolong the survival of life/consciousness).
(C) You want to maximize happy moments.
I sometimes imagine I’m a Dyson sphere of near-infinite resources splitting my budget between these goals. I find that B is often instrumental for A on a cosmic scale, but that C derives its budget entirely from the degree to which it helps A: equanimity, resilience, growth, learning, awe, gratitude, and other phenomena of positive psychology are wonderful tools for compassionate actors for minimizing suffering, but I would not copy/boost them beyond the degree to which they are the minimizing way. In other words, I could not tell my Exoplanet Rescue Mission Department why I wanted to spend their resources on creating more ecstatic meditators on Mars, because A is only interested in instruments for minimizing suffering. Besides, I wouldn’t undergo surgery without anaesthesia for any number of meditators on Mars, because they wouldn’t help my suffering; in a world where anaesthetics opportunity-cost a monastery on Mars, what would you do? Is “outweighing” between terminal values an actual physical computation taking place anywhere outside an ethicist’s head, or a fiction?
As a parallel comment, here is more (from a previous discussion) of why I am gravitating towards suffering as the only independent (dis)value and everything else as interdependently valuable in terms of preventing suffering:
––––
I experience all of the things quoted in Complexity of value,
but I don’t know how to ultimately prioritize between them unless they are commensurable. I make them commensurable by weighing their interdependent value in terms of the one thing we all(?) agree is an independent motivation: preventable suffering. (If preventable suffering is not worth preventing for its own sake, what is it worth preventing for, and is this other thing agreeable to someone undergoing the suffering as the reason for its motivating power?) This does not mean that I constantly think of them in these terms (that would be counterproductive), but in conflict resolution I do not assign them independent positive numerical values, which pluralism would imply one way or another.
Any pluralist theory begs the question of outweighing suffering with enough of any independently positive value. If you think about it for five minutes, aggregate happiness (or any other experience) does not exist. If our first priority is to prevent preventable suffering, that alone is an infinite game; it does not help to make a detour to boost/copy positive states unless this is causally connected to preventing suffering. (Aggregates of suffering do not exist either, but each moment of suffering is terminally worth preventing, and we have limited attention, so aggregates and chain-reactions of suffering are useful tools of thought for preventing as many as we can. So are many other things without requiring our attaching them independent positive value, or else we would be tiling Mars with them whenever it outweighed helping suffering on Earth according to some formula.)
My experience so far with this kind of unification is that it avoids many (or even all) of the theoretical problems that are still considered canonical challenges for pluralist utilitarianisms that assign both independent negative value to suffering and independent positive value to other things. I do not claim that this would be simple or intuitive – that would be analogous to reading about some Buddhist system, realizing its theoretical unity, and teleporting past its lifelong experiential integration – but I do claim that a unified theory with grounding in a universally accepted terminal value might be worth exploring further, because we cannot presuppose that any kind of CEV would be intuitive or easy to align oneself with.
[...]
People also differ along their background assumptions on whether AGI makes the universally life-preventing button a relevant question, because for many, the idea of an AGI represents an omnipotent optimizer that will decide everything about the future. If so, we want to be careful about assigning independent positive value to all the things, because each one of those invites this AGI to consider {outweighing suffering} with {producing those things}, since pluralist theories do not require a causal connection between the things being weighed.