Linkpost: Bracketing violates the sure-thing principle
I'd been meaning to turn this into a proper post, but didn't get around to it. So I've just linked to this Google doc, which shows that the "bracketing" decision rule violates a version of the sure-thing principle. I personally don't think this is actually much of a problem (see the conclusion of this post, for one), but it might be of interest.
Quotes: Recent discussions of backfire risks in AI safety
Some thinkers in AI safety have recently pointed out various backfire effects that attempts to reduce AI x-risk can have. I think pretty much all of these effects were known before,[1] but it's helpful to have them front of mind. In particular, I'm skeptical that we can weigh these effects against the upsides precisely enough to say an AI x-risk intervention is positive or negative in expectation, without making an arbitrary call. (Even if our favorite intervention doesn't have these specific downsides, we should ask if we're pricing in the downsides (and upsides) we haven't yet discovered.)
(Emphasis mine, in all the quotes below.)
Holden's Oct 2025 80K interview:
Holden: I mean, take any project. Let’s just take something that seems really nice, like alignment research. You’re trying to detect if the AI is scheming against you and make it not scheme against you. Maybe that’ll be good. But maybe the thing you’re doing is something that is going to get people excited, and then they’re going to try it instead of doing some other approach. And then it doesn’t work, and the other approach would have worked. Well, now you’ve done tremendous harm. Maybe it will work fine, but it will give people a false sense of security, make them think the problem is solved more than it is, make them move on to other things, and then you’ll have a tremendous negative impact that way.
Rob Wiblin: Maybe it’ll be used by a human group to get more control, to more reliably be able to direct an AI to do something and then do a power grab.
Holden: [...] Maybe it would have been great if the AIs took over the world. Maybe we’ll build AIs that are not exactly aligned with humans; they’re actually just much better — they’re kind of like our bright side, they’re the side we wish we were. [...]
[... M]aybe alignment is just a really… What it means is that you’re helping make sure that someone who’s intellectually unsophisticated — that’s us, that’s humans — remains forever in control of the rest of the universe and imposes whatever dumb ideas we have on it forevermore, instead of having our future evolve according to things that are much more sophisticated and better reasoners following their own values.
[...]
Holden: I just think AI is too multidimensional, and there’s too many considerations pointing in opposite directions. I’m worried about AIs taking over the world, but I’m also worried about the wrong humans taking over the world. And a lot of those things tend to offset each other, and making one better can make the other worse. [...]
[T]here’s also a lot of micro ways in which you could do harm. Just literally working in safety and being annoying, you might do net harm. You might just talk to the wrong person at the wrong time, get on their nerves. I’ve heard lots of stories of this. Just like, this person does great safety work, but they really annoyed this one person, and that might be the reason we all go extinct.
[...]
Option value in the policy world is kind of a bad concept anyway. A lot of times when you’re at a nonprofit or a company and you don’t know what to do, you try and preserve option value. But giving the government the option to go one way or the other, that’s not a neutral intervention — it’s just like you don’t know what they’re going to do with that option. Giving them the option could have been bad. … you don’t know who’s going to be in power when, and whether they’re going to have anything like the goals that you had when you put in some power that they had. I know people have been excited at various points about giving government more power and then at other points giving government less power.
And all this stuff, I mean, this one axis you’re talking about: centralisation of power versus decentralisation. Most things that touch policy at all in any way will move us along that spectrum in one direction or another, so therefore have a high chance of being negative [...]
And then most things that you can do in AI at all will have some impact on policy. Even just alignment research: policy will be shaped by what we’re seeing from alignment research, how tractable it looks, what the interventions look like.
[... I]n AI, it’s easier to annoy someone and polarise them against you, because whatever it is you’re trying to do, there’s some coalition that’s trying to do the exact opposite. In certain parts of global health and farm animal welfare, there’s certainly people who want to prioritise it less, but it doesn’t have the same directional ambiguity.
Helen Toner’s Nov 2025 80K interview:
Helen: And I think there’s a natural tension here as well among some people who are very concerned about existential risk from AI, really bad outcomes, and AI safety: there’s this sense that it’s actually helpful if there’s only a smaller number of players. Because, one, they can coordinate better — so maybe if racing leads to riskier outcomes, if you just have two top players, they can coordinate more directly than if you have three or four or 10 — and also a smaller number of players is going to be easier for an outside body to regulate, so if you just have a small number of companies, that’s going to be easier to regulate.
[...] But the problem is then the “Then what?” question of, if you do manage to avoid some of those worst-case outcomes, and then you have this incredibly powerful technology in the hands of a very small number of people, I think just historically that’s been really bad. It’s really bad when you have small groups that are very powerful, and typically it doesn’t result in good outcomes for the rest of the world and the rest of humanity.
[...]
Rob: I feel like we’re in a very difficult spot, because so many of the obvious solutions that you might have, or approaches you might take to dealing with loss of control do make the concentration of power problem worse and vice versa. So what policies you favour and disfavour depends quite sensitively on the relative risk of these two things, the relative likelihood of things going negatively in one way versus the other way.
And at least on the loss of control thing, people disagree so much on the likelihood. People who are similarly informed, know about everything there is to know about this, go all the way from thinking it’s a 1-in-1,000 chance to it’s a 1-in-2 chance — a 0.1% likelihood to 50% chance that we have some sort of catastrophic loss of control. And discussing it leads sometimes to some convergence, but people just have not converged on a common sense of how likely this outcome is.
So the people who think it’s 50% likely that we have some catastrophic loss-of-control event, it’s understandable that they think, “Well, we just have to make the best of it. Unfortunately, we have to concentrate. It’s the only way. And the concentration of power stuff is very sad and going to be a difficult issue to deal with, but we have to bear that cost.” And people who think it’s one in 1,000 are going to say, “This is a terrible move that you’re making, because we’re accepting much more risk, we’re creating much more risk than we’re actually eliminating.”
Wei Dai, “Legible vs. Illegible AI Safety Problems”:
Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to). But some problems are illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved. (Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)
From an x-risk perspective, working on highly legible safety problems has low or even negative expected value. Similar to working on AI capabilities, it brings forward the date by which AGI/ASI will be deployed, leaving less time to solve the illegible x-safety problems.
Musings on non-consequentialist altruism under deep unawareness
(This is a reply to a comment by Magnus Vinding, which ended up seeming like it was worth a standalone Quick Take.)
From Magnus:
For example, if we walk past a complete stranger who is enduring torment and is in need of urgent help, we would rightly take action to help this person, even if we cannot say whether this action reduces total suffering or otherwise improves the world overall. I think that's a reasonable practical stance, and I think the spirit of this stance applies to many ways in which we can and do benefit strangers, not just to rare emergencies.
The intuition here seems to be, "trying to actively do good in some restricted domain is morally right (e.g., virtuous), even when we're not justified in thinking this will have net-positive consequences[1] according to impartial altruism". Let's call this intuition Local Altruism is Right (LAR). I'm definitely sympathetic to this. I just think we should be cautious about extending LAR beyond fairly mundane "common sense" cases, especially to longtermist work.
For one, the reason most of us bothered with EA interventions was to do good "on net" in some sense. We weren't explicitly weighing up all the consequences, of course, but we didn't think we were literally ignoring some consequences — we took ourselves to be accounting for them with some combination of coarse-grained EV reasoning, heuristics, "symmetry" principles, discounting speculative stuff, etc. So it's suspiciously convenient if, once we realize that that reason was confused, we still come to the same practical conclusions.
Second, for me the LAR intuition goes away upon reflection unless at least the following hold (caveat in footnote):[2]
Some examples:
None of which is to say I have a fleshed-out theory! I'm keen to think more about what non-consequentialist altruism under unawareness might look like.
I mean to include Clifton's Option 3 as a possible operationalization of "net-positive consequences according to impartial altruism".
In the definition of LAR, "trying to actively do good" is the key phrase. I find it pretty intuitive that we don't need conditions nearly as strong as (1)+(2) below when we're asking, "Should you refrain from doing [intuitively evil thing]?"
Maybe the most promising angle is to show that it's normatively relevant that our beliefs about the more distant moral patients are (qualitatively?) less grounded in good reasons (see Clifton).
To clarify, the context of the quoted remark was that, just as we can care for those we love in the face of cluelessness, we can likewise care for and benefit strangers.
Specifically in relation to this:
we still have reason to respect other values we hold dear — those that were never grounded purely in the impartial good in the first place. Integrity, care for those we love, and generally not being a jerk, for starters. Beyond that, my honest answer is: I don’t know.
I think the "other values we hold dear" can and should also include a strong focus on helping strangers as well, even if challenges from cluelessness and the like were to prove devastating to standard versions of consequentialism. I mostly responded to what I saw as an overly exclusive focus on those we love versus others.
I'm not sure about this, though. As I wrote in a previous comment:
The reasons to do various parochial things, or respect deontological constraints, aren't like this. They aren't grounded in something like "this thing out there in the world is horrible, and should be prevented wherever/whenever it is [or whoever causes it]".
The concern I've tried to convey in our discussion so far is: Insofar as our moral reasons for action are grounded in "this thing out there in the world is horrible, and should be prevented wherever/whenever it is [or whoever causes it]", then shining the spotlight of our active altruism on beings who happen to be salient/near to us is arbitrary. To me, active "altruism" per se[1] is pretty inextricable from anti-arbitrariness.
And I'm saying, suppose for a moment we're no longer trying to be actively altruistic, and instead consider normative reasons that aren't grounded in the above. Then, prioritizing those whom you actually have special relationships with isn't arbitrary in the relevant sense. Because those relationships give you a reason to prioritize them. (Of course, if we started from an impartial altruistic perspective, this reason would be dwarfed by our duty to reduce large-scale suffering overall, insofar as that's tractable! But the worry is that it's not.)
Is your position something like, "We also have special relationships with strangers who are near to us"? I might be sympathetic to that, but it seems like it needs more unpacking.
Like I said, I do share the LAR intuition in some limited contexts, and it would be pretty sad if there's no non-arbitrary way to make sense of active altruism at all. I find this situation unsettling. But I currently feel confused as to how much I honestly endorse LAR.
As opposed to deontological(-ish) prohibitions against harming strangers.
Thanks for writing this, Anthony!
I find myself wondering what counts as "speculative" vs not. Here are some guesses at sufficient conditions for speculativeness:
An effect is speculative if it is highly sensitive to:
I'm also very interested in this question, because it isn't obvious to me where to draw the line in fields like wild animal welfare. I think I know as little about nematode sentience + welfare, for example, as I do about possible far future beings.
Maybe one difference is that it at least feels possible in theory to get more information about nematodes, but not really possible to get more information about far future beings? Although I'm skeptical of my intuitions here, since maybe it's easier than I think to get information about far future beings and harder than I think to get information about nematode sentience.
vegan advocacy [...] has a lot of ambiguous effects on wild animal suffering (which don't seem more "speculative", given the fairly robust-seeming arguments Tomasik has written about)
I'd be curious to hear more about this. What are the robust-seeming arguments?
The "lower meat production" ⇒ "higher net primary productivity" ⇒ "higher wild animal suffering" connection seems robust to me. Or not that much less robust than the intended benefit, at least.
This might not be the place for a discussion of this, but I personally don't feel that the "robustness" of Tomasikian chain of reasoning you note here is similar to the "robustness" of the idea that factory farms contain a crazy amount of suffering.
In the first instance, the specific chain of arrows above seems quite speculative, since we really have no idea how land use would change in a world with no factory farming. Are we that confident net primary productivity will increase? I'm aware there are good arguments for it, but I'd be surprised if someone couldn't come up with good arguments against if they tried.
More importantly, I don't think that's a sufficient reasoning chain to demonstrate that wild animal effects dominate? You'd need to show that wild+farmed animal welfare on post-factory farmed land uses is lower than wild+farmed animal welfare on current land uses, and that seems very sensitive to specific claims about moral weights, weights between types of suffering, empirical information about wild animal quality of life, what it means for a life to be net-negative, etc.
Or am I misunderstanding what you mean by robustness? I've just finished reading your unawareness sequence and mostly feel clueless about everything, including what it could mean for a reasoning chain to be robust.
Sorry this wasn't clear! I wasn't thinking about the choice between fully eliminating factory farming vs. the status quo. I had in mind marginal decreased demand for animal products leading to marginal decreased land use (in expectation), which I do think we have a fairly simple and well-evidenced mechanism for.
I also didn't mean to say the wild animal effects dominate, just that they're large enough to be competitive with the farmed animal effects. I agree the tradeoffs between e.g. cow or chicken suffering vs. wild insect suffering seem ambiguous. (And yep, from a non-suffering-focused perspective, it would also plausibly be ambiguous whether increased wild insect populations are bad.)
(I think when I wrote the above comment, I was thinking of pretty coarse-grained buckets of "robustness" vs "speculativeness".)
No, longtermism is not redundant
I’m not keen on the recent trend of arguments that persuading people of longtermism is unnecessary, or even counterproductive, for encouraging them to work on certain cause areas (e.g., here, here). This is for a few reasons:
Why expect "heuristics" to be robust to unknown unknowns?
I often read/hear claims that, if we're worried that our evaluations of interventions won't hold up under unknown unknowns, we should follow (simple) heuristics. But what precisely is the argument for this? This isn't a rhetorical question — I'm just noting my confusion and want to understand this view better.
Interested to hear more from those who endorse this view!
Out of curiosity, any particular examples of such claims?
I partly had in mind personal communications, but some public examples (and very brief summaries of my reactions, not fleshed out counterarguments):
Permissive epistemology doesn't imply precise credences / completeness / non-cluelessness
(Many thanks to Jesse Clifton and Sylvester Kollin for discussion.)
My arguments against precise Bayesianism and for cluelessness appeal heavily to the premise “we shouldn’t arbitrarily narrow down our beliefs”. This premise is very compelling to me (and I’d be surprised if it’s not compelling to most others upon reflection, at least if we leave “arbitrary” open to interpretation). I hope to get around to writing more about it eventually.
But suppose you don’t care much about avoiding arbitrariness, for any interesting interpretation of “arbitrariness”. Suppose you have a highly permissive epistemology. E.g., you might think that as long as your beliefs satisfy some “coherence” conditions, that’s as far as rationality can take you, and you’re free to have whichever coherent beliefs you’re disposed to. Or you might think “beliefs” are nothing more or less than constraints on preferences, and it’s fine if preferences are arbitrary.
If you have such a view, does that imply your credences are precise, or your preferences are complete, or the like?
No.[1] You might simply introspect on what your response to your evidence, intuitions, etc. is, and find that the most honest representation of that response is imprecise/incomplete. You might try weighing up various considerations about the consequences of some pair of actions, and find that your disposition is, “I have no clue. Even after noticing that some precise number popped into my head, all things considered I have no preference either way (but I also don’t feel indifferent, because my preferences are insensitive to mild sweetening).”
Of course, if you have a permissive epistemology and you don’t have such introspective reactions, there’s nothing more I can say to you. But it’s important to acknowledge that precision, completeness, and non-cluelessness are not some privileged default for the permissivist.
Why you should consider trying SSRIs
I was initially hesitant to post this, out of some vague fear of stigma and stating the obvious, and not wanting people to pathologize my ethical views based on the fact that I take antidepressants. This is pretty silly for two reasons. First, I think that if my past self had read something like this, he could have been spared years of suffering, and there are probably several readers in his position. EAs are pretty open about mental illness anyway. Second, if anything the fact that I am SFE "despite" currently not being depressed at all (indeed quite consistently happy), thanks to SSRIs, should make readers less likely to attribute my views to a mental illness.[1]
I claim that even if you don't feel so bad as to qualify as capital-D depressed, you might feel noticeably less bad on a daily basis if you try SSRIs.[2] That has been my experience, and I can honestly say this has likely been the cheapest sustainable boost in my well-being I've ever found. Being happier has also probably made me more effective/productive, though this is harder to assess.
(Obviously, my experience is not universal, I'm probably just lucky to an extent, this is not expert medical advice, and you might either find that SSRIs are ineffective for you or that the side effects are less tolerable than I have found them. You should definitely do your own research!)
In the months (at least) prior to SSRIs, my level of depression was "mild" according to the Burns checklist. I felt rather lonely during downtime, and like a bit of a failure for not having an exciting social life. I didn't endorse the latter judgment, and felt pretty fulfilled by my altruistic work, but that dissatisfaction persisted even when I tried to reason myself out of it (or tried taking up new hobbies). This wasn't debilitating by any means—so much so that I didn't really feel like I "deserved" to receive treatment intended for depression, and yes I realize how dumb that sounds in hindsight—but it was something of a pall hanging over my life all the same.
SSRIs just dispelled those feelings.
Waiting so long to give these things a try was a mistake. I made that mistake out of a combination of the aforementioned suspicion that I wasn't depressed enough to need them, and overestimation of how difficult it would be to get a prescription.[3] Just because my suffering wasn't as deep as others', that didn't mean it needed to exist.
This medication isn't magic; my life isn't perfect, and I still have some other dissatisfactions I'm working on. But, for the amount of difference this has made for me, it seemed negligent not to share my experience and hopefully encourage others in an analogous position to show themselves a bit of compassion.
[1] Yes, I have seen people do this before—not to me personally, but to other SFEs.
[2] This probably holds for other antidepressants too. I'm just focusing on SSRIs here because I have experience with them, and they incidentally have a worse reputation than, e.g., Wellbutrin.
[3] At least in the U.S., there are online services where you can share your symptoms with a doctor and just get a prescription at a pretty low price. For some reason, I expected a lot more awkward bureaucracy and mandatory therapy than this. I won't get specific here because I don't want to be a shill, but if you're curious, feel free to PM me.
SlateStarCodex has a long post on SSRIs and their side-effects (from 2014); including sexual side-effects. (Here is a 2016 paper which also reports on sexual side-effects.) I don't have expertise in this topic, however.
In Defense of Aiming for the Minimum
I’m not really sympathetic to the following common sentiment: “EAs should not try to do as much good as feasible at the expense of their own well-being / the good of their close associates.”
It’s tautologically true that if trying to hyper-optimize comes at too much of a cost to the energy you can devote to your most important altruistic work, then trying to hyper-optimize is altruistically counterproductive. I acknowledge that this is the principle behind the sentiment above, and evidently some people’s effectiveness has benefited from advice like this.
But in practice, I see EAs apply this principle in ways that seem suspiciously favorable to their own well-being, or to the status quo. When you find yourself trying to justify on the grounds of impact the amounts of self-care people afford themselves when they don’t care about being effectively altruistic, you should be extremely suspicious.
Some examples, which I cite not to pick on the authors in particular—since I think many others are making a similar mistake—but just because they actually wrote these claims down.
1. “Aiming for the minimum of self-care is dangerous”
I felt a bit suspicious, looking at how I spent my time. Surely that long road trip wasn’t necessary to avoid misery? Did I really need to spend several weekends in a row building a ridiculous LED laser maze, when my other side project was talking to young synthetic biologists about ethics?
I think this is just correct. If your argument is that EAs shouldn’t be totally self-effacing because some frivolities are psychologically necessary to keep rescuing people from the bottomless pit of suffering, then sure, do the things that are psychologically necessary. I’m skeptical that “psychologically necessary” actually looks similar to the amount of frivolities indulged by the average person who is as well-off as EAs generally are.
Do I live up to this standard? Hardly. That doesn’t mean I should pretend I’m doing the right thing.
Minimization is greedy. You don’t get to celebrate that you’ve gained an hour a day [from sleeping seven instead of eight hours], or done something impactful this week, because that minimizing urge is still looking at all your unclaimed time, and wondering why you aren’t using it better, too.
How important is my own celebration, though, when you really weigh it against what I could be doing with even more time? (This isn’t just abstract impact points; there are other beings whose struggles matter no less than mine do, and fewer frivolities for me could mean relief for them.)
I think where I fundamentally disagree with this post is that, for many people, I don’t think aiming for the minimum puts you close to less than the minimum. Getting to the minimum, much less below it, can be very hard, such that people who aim at it just aren’t in much danger of undershooting. If you find this is not true for yourself, then please do back off from the minimum. But remember that in the counterfactual where you hadn’t tested your limits, you probably would not have gotten close to optimal.
This post includes some saddening anecdotes about people ending up miserable because they tried to optimize all their time for altruism. I don’t want to trivialize their suffering. Yet I can conjure anecdotes in the opposite direction (and the kind of altruism I care about reduces more suffering in expectation). Several of my colleagues seem to work more than the typical job entails, and I don’t have any evidence of the quality of their work being the worse for this. I’ve found that the amount of time I can realistically devote to altruistic efforts is pretty malleable. No, I’m not a machine; of course I have my limits. But when I gave myself permission to do altruistic things for parts of weekends, or into later hours of weekdays, well, I could. “My happiness is not the point,” as Julia said in this post, and while she evidently doesn’t endorse that statement, I do. That just seems to be the inevitable consequence of taking the sentience of other beings besides yourself (or your loved ones) seriously.
See also this comment:
Personally have been trying to think of my life only as a means to an end. Will my life technically might have value, I am fairly sure it is rather minuscule compared to the potential impact can make. I think it's possible, though probably difficult, to intuit this and still feel fine / not guilty, about things. … I'm a bit wary on this topic that people might be a bit biased to select beliefs based on what is satisfying or which ones feel good.
I do think Tessa's point about slack has some force—though in a sense, this merely shifts the “minimum” up by some robustness margin, which is unlikely to be large enough to justify the average person’s indulgences.
2. “You have more than one goal, and that’s fine”
If I donate to my friend’s fundraiser for her sick uncle, I’m pursuing a goal. But it’s the goal of “support my friend and our friendship,” not my goal of “make the world as good as possible.” When I make a decision, it’s better if I’m clear about which goal I’m pursuing. I don’t have to beat myself up about this money not being used for optimizing the world — that was never the point of that donation. That money is coming from my “personal satisfaction” budget, along with money I use for things like getting coffee with friends.
It puzzles me that, as common as concerns about the utility monster—sacrificing the well-being of the many for the super-happiness of one—are, we seem to find it totally intuitive that one can (passively) sacrifice the well-being of the many for one’s own rather mild comforts. (This is confounded by the act vs. omission distinction, but do you really endorse that?)
The latter conclusion is basically the implication of accepting goals other than “make the world as good as possible.” What makes these other goals so special, that they can demand disproportionate attention (“disproportionate” relative to how much actual well-being is at stake)?
Due to the writing style, it’s honestly not clear to me what exactly this post was claiming. But the author does emphatically say that devoting all of their time to the activity that helps more people per hour would be “premature optimization.” And they celebrate an example of a less effective thing they do because it consistently makes a few people happy.
I don’t see how the post actually defends doing the less effective thing. To the extent that you impartially care about other sentient beings, and don’t think their experiences matter any less because you have fewer warm fuzzy feelings about them, what is the justification for willingly helping fewer people?
Some reasons not to primarily argue for veganism on health/climate change grounds
I've often heard animal advocates claim that since non-vegans are generally more receptive to arguments from health benefits and reducing climate impact, we should prioritize those arguments, in order to reduce farmed animal suffering most effectively.
On its face, this is pretty reasonable, and I personally don't care intrinsically about how virtuous people's motivations for going vegan are. Suffering is suffering, no matter its sociological cause.
But there are some reasons I'm nervous about this approach, at least if it comes at the opportunity cost of moral advocacy. None of these are original to me, but I want to summarize them here since I think this is a somewhat neglected point:
Quick comment. With respect to your first point, this has always struck me as one of the better points as to why non ethical arguments should primarily avoided when it comes to making the case for veganism. However, after reading Tobias Leenaert's 'How to Create a Vegan World: A Pragmatic Approach', I've become a bit more agnostic on this notion. He notes a few studies from The Humane League that show that red-meat reducers/avoiders tend to eat less chicken than your standard omnivore. He also referenced a few studies from Nick Cooney's book, Veganomics, which covers some of this on p. 107-111. Combined with the overall impact non-ethical vegans could have on supply/demand for other vegan products (and their improvement in quality), I've been a bit less worried about this reason.
I think your other reasons are all extremely important and underrated, though, so still lean overall that the ethical argument should be relied on when possible :)
Wow, that's promising news! Thanks for sharing.
The Repugnant Conclusion is worse than I thought
At the risk of belaboring the obvious to anyone who has considered this point before: The RC glosses over the exact content of happiness and suffering that are summed up to the quantities of “welfare” defining world A and world Z. In world A, each life with welfare 1,000,000 could, on one extreme, consist purely of (a) good experiences that sum in intensity to a level 1,000,000, or on the other, (b) good experiences summing to 1,000,000,000 minus bad experiences summing (in absolute value) to 999,000,000. Similarly, each of the lives of welfare 1 in world Z could be (a) purely level 1 good experiences, or (b) level 1,000,001 good experiences minus level 1,000,000 bad experiences.
To my intuitions, it’s pretty easy to accept the RC if our conception of worlds A and Z is the pair (a, a) from the (of course non-exhaustive) possibilities above, even more so for (b, a). However, the RC is extremely unpalatable if we consider the pair (a, b). This conclusion, which is entailed by any plausible non-negative[1] total utilitarian view, is that a world of tremendous happiness with absolutely no suffering is worse than a world of many beings each experiencing just slightly more happiness than those in the first, but along with tremendous agony.
To drive home how counterintuitive that is, we can apply the same reasoning often applied against NU views: Suppose the level 1,000,001 happiness in each being in world Z is compressed into one millisecond of some super-bliss, contained within a life of otherwise unremitting misery. There doesn’t appear to be any temporal ordering of the experiences of each life in world Z such that this conclusion isn’t morally absurd to me. (Going out with a bang sounds nice, but not nice enough to make the preceding pure misery worth it; remember this is a millisecond!) This is even accounting for the possible scope neglect involved in considering the massive number of lives in world Z. Indeed, multiplying these lives seems to make the picture more horrifying, not less.
Again, at the risk of sounding obvious: The repugnance of the RC here is that on total non-NU axiologies, we’d be forced to consider the kind of life I just sketched a “net-positive” life morally speaking.[2] Worse, we're forced to consider an astronomical number of such lives better than a (comparatively small) pure utopia.
[1] “Negative” here includes lexical and lexical threshold views.
[2] I’m setting aside possible defenses based on the axiological importance of duration. This is because (1) I’m quite uncertain about that point, though I share the intuition, and (2) it seems any such defense rescues NU just as well. I.e. one can, under this principle, maintain that 1 hour of torture-level suffering is impossible to morally outweigh, but 1 millisecond isn’t.
This conclusion, which is entailed by any plausible non-negative[1] total utilitarian view, is that a world of tremendous happiness with absolutely no suffering is worse than a world of many beings each experiencing just slightly more happiness than those in the first, but along with tremendous agony.
It seems to me that you're kind of rigging this thought experiment when you define an amount of happiness that's greater than an amount of suffering, but you describe the happiness as "slight" and the suffering as "tremendous", even though the former is larger than the latter.
I don't call the happiness itself "slight," I call it "slightly more" than the suffering (edit: and also just slightly more than the happiness per person in world A). I acknowledge the happiness is tremendous. But it comes along with just barely less tremendous suffering. If that's not morally compelling to you, fine, but really the point is that there appears (to me at least) to be quite a strong moral distinction between 1,000,001 happiness minus 1,000,000 suffering, and 1 happiness.
Linkpost: "Tranquilism Respects Individual Desires"
I wrote a defense of an axiology on which an experience is perfectly good to the extent that it is absent of craving for change. This defense follows in part from a reductionist view of personal identity, which is usually considered in EA circles to be in support of total symmetric utilitarianism, but I argue that this view lends support to a form of negative utilitarianism.
Some things I liked about What We Owe the Future, despite my disagreements with the treatment of value asymmetries:
A Parfitian Veil of Ignorance
[Edit: I would be very surprised if I were the first person to have proposed this; it probably exists somewhere else, I just don't know of a source.]
Prompted by Holden’s discussion of the veil of ignorance as a utilitarian intuition pump (contra Rawls), I thought about an alternative to the standard veil. My intuitions about tradeoffs of massive harms for a large number of small benefits—at least for some conceptions of "benefit"—diverge from those in his post, when considering this version.
The standard veil of ignorance asks you to imagine being totally ignorant as to which person you will be in a population. (Assume we’re only considering fixed population sizes, so there’s no worry that this exercise sneaks in average utilitarianism, etc.)
But the many EA fans of Parfit (or Buddha) know that this idea of a discrete person is metaphysically problematic. So we can look at another approach, inspired by empty individualism.
Imagine that when evaluating two possible worlds, you don’t know which slice of experience in each world you would be. To make things easy enough to grasp, take a “slice” to be just the longest amount of time necessary for a sentient being to register an experience, but not much longer. Let’s say one second.
These worlds might entail probabilities of experiences as well. So, since it’s hard to intuitively grasp probabilities as effectively as frequencies, suppose each world is “re-rolled” a large enough times that each outcome happens at least once, in proportion to its probability. e.g., In Holden’s example of a 1 in 100 million chance of someone dying, the experiences of that person are repeated 100 million times, and one of those experience streams is cut short by death.
So now a purely aggregative and symmetric utilitarian offers me a choice, from behind this veil of ignorance, between two worlds. Option 1 consists of a person who lives for one day with constantly neutral experiences—no happiness, no suffering (including boredom). In option 2, that person instead spends that day relaxing on a nice beach, with a 1 in 100 million chance of ending that day by spiraling into a depression (instead of dying peacefully in their sleep).
I imagine, first, rescaling things so in #1 the person lives 100 million days of neutrality, and in #2, they live 99,999,999 peaceful beach-days—suspend your disbelief and imagine they never get bored—followed by a beach-day that ends in depression. Then I imagine I don’t know which moment of experience in either of these options I’ll be.
Choosing #1 seems pretty defensible to me, from this perspective. Several of those experience-moments in #2 are going to consist purely of misery. They won’t be comforted by the fact that they’re rare, or that they’re in the context of a “person” who otherwise is quite happy. They’ll just suffer.
I’m not saying the probabilities don’t matter. Of course they do; I’d rather take #2 than a third option where there’s a 1 in 100 thousand chance of depression. I’m also pretty uncertain where I stand when we modify #1 so that the person’s life is a constant mild itch instead of neutrality. The intuition this thought experiment prompts in me is the lexical badness of at least sufficiently intense suffering, compared with happiness or other goods. And I think the reason it prompts such an intuition is that in this version of the veil of ignorance, discrete “persons” don’t get to dictate what package of experiences is worth it, i.e., what happens to the multitude of experience-moments in their life. Instead, one has to take the experience-moments themselves as sovereign, and decide how to handle conflicts among their preferences. (I discuss this more here.)
I find the framing of "experience slices" definitely pushes my intuitions in the same direction.
One question I like to think about is whether I'd choose to gain either
(a) a neutral experience
or
(b) flipping a coin and reliving all the positive experience slices of my life if heads, and reliving all the negative ones if tails
My life feels highly net positive but I'd almost certainly not take option (b). I'd guess there's likely risk aversion intuition also being snuck here too though.
Some vaguely clustered opinions on metaethics/metanormativity
I'm finding myself slightly more sympathetic to moral antirealism lately, but still afford most of my credence to a form of realism that would not be labeled "strong" or "robust." There are several complicated propositions I find plausible that are in tension:
1. I have a strong aversion to arbitrary or ad hoc elements in ethics. Practically this cashes out as things like: (1) rejecting any solutions to population ethics that violate transitivity, and (2) being fairly unpersuaded by solutions to fanaticism that round down small probabilities or cap the utility function.
2. Despite this, I do not intrinsically care about the simplicity of a moral theory, at least for some conceptions of "simplicity." It's quite common in EA and rationalist circles to dismiss simple or monistic moral theories as attempting to shoehorn the complexity of human values into one box. I grant that I might unintentionally be doing this when I respond to critiques of the moral theory that makes most sense to me, which is "simple." But from the inside I don't introspect that this is what's going on. I would be perfectly happy to add some complexity to my theory to avoid underfitting the moral data, provided this isn't so contrived as to constitute overfitting. The closest cases I can think of where I might need to do this are in population ethics and fanaticism. I simply don't see what could matter morally in the kinds of things whose intrinsic value I reject: rules, virtues, happiness, desert, ... When I think of these things, and the thought experiments meant to pump one's intuitions in their favor, I do feel their emotional force. It's simply that I am more inclined to think of them as just that: emotional, or game theoretically useful constructs that break down when you eliminate bad consequences on conscious experience. The fact that I may "care" about them doesn't mean I endorse them as relevant to making the world a better place.
3. Changing my mind on moral matters doesn't feel like "figuring out my values." I roughly know what I value. Many things I value, like a disproportionate degree of comfort for myself, are things I very much wish I didn't value, things I don't think I should value. A common response I've received is something like: "The values you don't think you 'should' have are simply ones that contradict stronger values you hold. You have meta-preferences/meta-values." Sure, but I don't think this has always been the case. Before I learned about EA, I don't think it would have been accurate to say I really did "value" impartial maximization of good across sentient beings. This was a value I had to adopt, to bring my motivations in line with my reasons. Encountering EA materials did not feel at all like "Oh, you know what, deep down this was always what I would've wanted to optimize for, I just didn't know I would've wanted it."
4. The question "what would you do if you discovered the moral truth was to do [obviously bad thing]?" doesn't make sense to me, for certain inputs of [obviously bad thing], e.g. torturing all sentient beings as much as possible. For extreme inputs of that sort, the question is similar to "what would you do if you discovered 2+2=5?" For less extreme inputs, such that it's plausible to me I simply have not thought through ethics enough that I could imagine that hypothetical but merely find it unlikely right now, the question does make sense, and I see nothing wrong with saying "yes." I suspect many antirealists do this all the time, radically changing their minds on moral questions due to considerations other than empirical discoveries, and they would not be content saying "screw the moral truth" by retaining their previous stance.
5. I do not expect that artificial superintelligence would converge on The Moral Truth by default. Even if it did, the convergence might be too slow to prevent catastrophes. But I also doubt humans will converge on this either. Both humans and AIs are limited by our access only to our "own" qualia, and indeed our own present qualia. The kind of "moral realism" I find plausible with respect to this convergence question is that convergence to moral truth could occur for a perfectly rational and fully informed agent, with unlimited computation and - most importantly - subjective access to the hypothetical future experiences of all sentient beings. These conditions are so idealized that I am probably as pessimistic about AI as any antirealist, but I'm not sure yet if they're so idealized that I functionally am an antirealist in this sense.
Poll: Is this one of your cruxes for cluelessness?
There's a cluster of responses to arguments for cluelessness I've encountered, which I'm not yet sure I understand but maybe is important. Here's my attempted summary:[1]
If you're skeptical of cluelessness, I'd find it super helpful if you'd agree-vote if you agree with the above response or something very similar to it, and disagree-vote otherwise. ETA: added a poll widget below, please use that instead (thanks to Toby Tremlett for suggesting this). (Please don't vote if you aren't skeptical of cluelessness.) And feel free to comment with some different version of the above you'd agree with, if that difference is important for you. Thanks!
Some examples of sentiments that, IIUC, this summary encapsulates (emphasis mine):
* Greaves: "I think most of us feel like we’re really just making up arbitrary numbers, but that’s really uncomfortable because precisely which arbitrary numbers we make up seems to make a difference to what we ended up doing." See also Greaves' discussion of the "decision discomfort" involved in complex cluelessness.
* Soares: "Now, I agree that this scenario is ridiculous. And that it sucks. And I agree that picking a precise minute feels uncomfortable. And I agree that this is demanding way more precision than you are able to generate. But if you find yourself in the game, you'd best pick the minute as well as you can. When the gun is pressed against your temple, you cash out your credences."
A non-trivial reason for this is that precise numbers expose ideological assumptions, and a whole of people do not like this.
It's easy to lie with numbers, but it's even easier to lie without a number.
I understand my vote to be consistent with us never in fact reaching 'principled' precise EV numbers though.
Is there a way to retract a vote? I accidentally voted (I'm not skeptical of cluelessness).
There should be a little x if you hover over your icon, clicking that should remove your vote.
One of the only good scenarios that I can think of where this response to cluelessness makes sense, is if a person subscribes to moral realism. And even then, the arguments for moral uncertainty seem too compelling to me. Do you know of any person that is not skeptical of cluelessness?
My take is that, if you really try to look at it from first principles, most will arrive at the conclusion that it is not possible to calculate the EV of any action for us humans. The rest is just cope because one is not willing to give up on ones EA identity and all of the sacrifices you have done. Sunk-cost fallacy is just too big of an obstacle.
Yes, if there is a moral theory that is objectively correct, then one should in principle be able to translate that moral theory into a framework that helps us calculate EV. But since we are not omniscient, that just does seem impossible. But in combination with the principle that we can't understand the downstream effect of our actions in the long-term, I don't understand how somebody can be skeptical of cluelessness.
I know this comment does not directly address your initial prompt, but I thought I'd rather post it than not. Thank you for sharing!
(You might find this discussion helpful.)