antimonyanthony's Quick takes

Linkpost: Bracketing violates the sure-thing principle

I'd been meaning to turn this into a proper post, but didn't get around to it. So I've just linked to this Google doc, which shows that the "bracketing" decision rule violates a version of the sure-thing principle. I personally don't think this is actually much of a problem (see the conclusion of this post, for one), but it might be of interest.

Quotes: Recent discussions of backfire risks in AI safety

Some thinkers in AI safety have recently pointed out various backfire effects that attempts to reduce AI x-risk can have. I think pretty much all of these effects were known before,^[1] but it's helpful to have them front of mind. In particular, I'm skeptical that we can weigh these effects against the upsides precisely enough to say an AI x-risk intervention is positive or negative in expectation, without making an arbitrary call. (Even if our favorite intervention doesn't have these specific downsides, we should ask if we're pricing in the downsides (and upsides) we haven't yet discovered.)

(Emphasis mine, in all the quotes below.)

Holden's Oct 2025 80K interview:

Holden: I mean, take any project. Let’s just take something that seems really nice, like alignment research. You’re trying to detect if the AI is scheming against you and make it not scheme against you. Maybe that’ll be good. But maybe the thing you’re doing is something that is going to get people excited, and then they’re going to try it instead of doing some other approach. And then it doesn’t work, and the other approach would have worked. Well, now you’ve done tremendous harm. Maybe it will work fine, but it will give people a false sense of security, make them think the problem is solved more than it is, make them move on to other things, and then you’ll have a tremendous negative impact that way.

Rob Wiblin: Maybe it’ll be used by a human group to get more control, to more reliably be able to direct an AI to do something and then do a power grab.

Holden: [...] Maybe it would have been great if the AIs took over the world. Maybe we’ll build AIs that are not exactly aligned with humans; they’re actually just much better — they’re kind of like our bright side, they’re the side we wish we were. [...]

[... M]aybe alignment is just a really… What it means is that you’re helping make sure that someone who’s intellectually unsophisticated — that’s us, that’s humans — remains forever in control of the rest of the universe and imposes whatever dumb ideas we have on it forevermore, instead of having our future evolve according to things that are much more sophisticated and better reasoners following their own values.

[...]

Holden: I just think AI is too multidimensional, and there’s too many considerations pointing in opposite directions. I’m worried about AIs taking over the world, but I’m also worried about the wrong humans taking over the world. And a lot of those things tend to offset each other, and making one better can make the other worse. [...]

[T]here’s also a lot of micro ways in which you could do harm. Just literally working in safety and being annoying, you might do net harm. You might just talk to the wrong person at the wrong time, get on their nerves. I’ve heard lots of stories of this. Just like, this person does great safety work, but they really annoyed this one person, and that might be the reason we all go extinct. 

[...]

Option value in the policy world is kind of a bad concept anyway. A lot of times when you’re at a nonprofit or a company and you don’t know what to do, you try and preserve option value. But giving the government the option to go one way or the other, that’s not a neutral intervention — it’s just like you don’t know what they’re going to do with that option. Giving them the option could have been bad. … you don’t know who’s going to be in power when, and whether they’re going to have anything like the goals that you had when you put in some power that they had. I know people have been excited at various points about giving government more power and then at other points giving government less power.

And all this stuff, I mean, this one axis you’re talking about: centralisation of power versus decentralisation. Most things that touch policy at all in any way will move us along that spectrum in one direction or another, so therefore have a high chance of being negative [...] 

And then most things that you can do in AI at all will have some impact on policy. Even just alignment research: policy will be shaped by what we’re seeing from alignment research, how tractable it looks, what the interventions look like.

[... I]n AI, it’s easier to annoy someone and polarise them against you, because whatever it is you’re trying to do, there’s some coalition that’s trying to do the exact opposite. In certain parts of global health and farm animal welfare, there’s certainly people who want to prioritise it less, but it doesn’t have the same directional ambiguity.

Helen Toner’s Nov 2025 80K interview:

Helen: And I think there’s a natural tension here as well among some people who are very concerned about existential risk from AI, really bad outcomes, and AI safety: there’s this sense that it’s actually helpful if there’s only a smaller number of players. Because, one, they can coordinate better — so maybe if racing leads to riskier outcomes, if you just have two top players, they can coordinate more directly than if you have three or four or 10 — and also a smaller number of players is going to be easier for an outside body to regulate, so if you just have a small number of companies, that’s going to be easier to regulate.

[...] But the problem is then the “Then what?” question of, if you do manage to avoid some of those worst-case outcomes, and then you have this incredibly powerful technology in the hands of a very small number of people, I think just historically that’s been really bad. It’s really bad when you have small groups that are very powerful, and typically it doesn’t result in good outcomes for the rest of the world and the rest of humanity.

[...]

Rob: I feel like we’re in a very difficult spot, because so many of the obvious solutions that you might have, or approaches you might take to dealing with loss of control do make the concentration of power problem worse and vice versa. So what policies you favour and disfavour depends quite sensitively on the relative risk of these two things, the relative likelihood of things going negatively in one way versus the other way.

And at least on the loss of control thing, people disagree so much on the likelihood. People who are similarly informed, know about everything there is to know about this, go all the way from thinking it’s a 1-in-1,000 chance to it’s a 1-in-2 chance — a 0.1% likelihood to 50% chance that we have some sort of catastrophic loss of control. And discussing it leads sometimes to some convergence, but people just have not converged on a common sense of how likely this outcome is.

So the people who think it’s 50% likely that we have some catastrophic loss-of-control event, it’s understandable that they think, “Well, we just have to make the best of it. Unfortunately, we have to concentrate. It’s the only way. And the concentration of power stuff is very sad and going to be a difficult issue to deal with, but we have to bear that cost.” And people who think it’s one in 1,000 are going to say, “This is a terrible move that you’re making, because we’re accepting much more risk, we’re creating much more risk than we’re actually eliminating.”

Wei Dai, “Legible vs. Illegible AI Safety Problems”:

Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to). But some problems are illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved. (Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)

From an x-risk perspective, working on highly legible safety problems has low or even negative expected value. Similar to working on AI capabilities, it brings forward the date by which AGI/ASI will be deployed, leaving less time to solve the illegible x-safety problems.

1. ^{^}

   Among other sources, see my compilations of backfire effects here and here, and discussion of downside risks of capacity-building / aiming for "option value" or "wiser futures" here.

Poll: Is this one of your cruxes for cluelessness?

There's a cluster of responses to arguments for cluelessness I've encountered, which I'm not yet sure I understand but maybe is important. Here's my attempted summary:^[1]

Sure, maybe assigning each action a precise EV feels arbitrary. But that feeling merely reflects the psychological difficulty of generating principled numbers, for non-ideal agents like us. It's not a problem for the view that even non-ideal agents should, ultimately, evaluate actions as more or less rational based on precise EV.

If you're skeptical of cluelessness, I'd find it super helpful if you'd agree-vote if you agree with the above response or something very similar to it, and disagree-vote otherwise. ETA: added a poll widget below, please use that instead (thanks to Toby Tremlett for suggesting this). (Please don't vote if you aren't skeptical of cluelessness.) And feel free to comment with some different version of the above you'd agree with, if that difference is important for you. Thanks!

1. ^{^}

   Some examples of sentiments that, IIUC, this summary encapsulates (emphasis mine):
   * Greaves: "I think most of us feel like we’re really just making up arbitrary numbers, but that’s really uncomfortable because precisely which arbitrary numbers we make up seems to make a difference to what we ended up doing." See also Greaves' discussion of the "decision discomfort" involved in complex cluelessness.
   * Soares: "Now, I agree that this scenario is ridiculous. And that it sucks. And I agree that picking a precise minute feels uncomfortable. And I agree that this is demanding way more precision than you are able to generate. But if you find yourself in the game, you'd best pick the minute as well as you can. When the gun is pressed against your temple, you cash out your credences."

Musings on non-consequentialist altruism under deep unawareness

(This is a reply to a comment by Magnus Vinding, which ended up seeming like it was worth a standalone Quick Take.)

From Magnus:

For example, if we walk past a complete stranger who is enduring torment and is in need of urgent help, we would rightly take action to help this person, even if we cannot say whether this action reduces total suffering or otherwise improves the world overall. I think that's a reasonable practical stance, and I think the spirit of this stance applies to many ways in which we can and do benefit strangers, not just to rare emergencies.

The intuition here seems to be, "trying to actively do good in some restricted domain is morally right (e.g., virtuous), even when we're not justified in thinking this will have net-positive consequences^[1] according to impartial altruism". Let's call this intuition Local Altruism is Right (LAR). I'm definitely sympathetic to this. I just think we should be cautious about extending LAR beyond fairly mundane "common sense" cases, especially to longtermist work.

For one, the reason most of us bothered with EA interventions was to do good "on net" in some sense. We weren't explicitly weighing up all the consequences, of course, but we didn't think we were literally ignoring some consequences — we took ourselves to be accounting for them with some combination of coarse-grained EV reasoning, heuristics, "symmetry" principles, discounting speculative stuff, etc. So it's suspiciously convenient if, once we realize that that reason was confused, we still come to the same practical conclusions.

Second, for me the LAR intuition goes away upon reflection unless at least the following hold (caveat in footnote):^[2]

1. The "restricted domain" isn't too contrived in some sense, rather it's some natural-seeming category of moral patients or welfare-relevant outcome.
   1. (How we delineate "contrived" vs. "not contrived" is of course rather subjective, which is exactly why I'm suspicious of LAR as an impartial altruistic principle. I'm just taking the intuition on its own terms.)
2. I'm at least justified in (i) expecting my intervention to do good overall in that domain, and (ii) expecting not to have large off-target effects of indeterminate net sign in domains of similar "speculativeness" (see "implementation robustness").
   1. ("Speculativeness", too, is subjective. And while I definitely find it intuitive that our impacts on more robustly foreseeable moral patients are privileged,^[3] I haven't found a satisfying way of making sense of this intuition. But if we want to respect this intuition, condition (ii) seems necessary. If the set of moral patients A seems no less robustly foreseeable than set B, why would I morally privilege B? Cf. my discussion here.)

Some examples:

• Trying to reduce farmed animal suffering: Depends on the intervention.
  • I have the LAR intuition for things like donating to humane invertebrate slaughter research, which doesn't seem to have large backfire risks on other animals in the near term. (Even if it does plausibly have large backfire risks for future digital minds, say.)
  • For me LAR is considerably weaker for things like vegan advocacy, which has a lot of ambiguous effects on wild animal suffering (which don't seem more "speculative", given the fairly robust-seeming arguments Tomasik has written about).
• Trying to prevent digital suffering (even pre-space colonization): All the interventions I'm aware of are so apparently non-implementation-robust that I don't have the LAR intuition here. Example here.
  • If we instead said something like "this intervention is implementation-robust w.r.t. helping some specific subset of digital minds," that subset feels contrived, so I still wouldn't have the intuition in favor of the intervention.
• Trying to prevent extinction: (Assuming a non-suffering-focused view for the sake of argument, because lots of EAs seem to think trying to prevent extinction is common-sensically good/right.) As I've argued here and here, the interventions EAs have proposed to reduce extinction risk don't seem to satisfy condition (i) of implementation robustness above. Even if they did, off-target effects on animal suffering arguably undermine condition (ii) (see e.g. this post).

None of which is to say I have a fleshed-out theory! I'm keen to think more about what non-consequentialist altruism under unawareness might look like.

1. ^{^}

   I mean to include Clifton's Option 3 as a possible operationalization of "net-positive consequences according to impartial altruism".

2. ^{^}

   In the definition of LAR, "trying to actively do good" is the key phrase. I find it pretty intuitive that we don't need conditions nearly as strong as (1)+(2) below when we're asking, "Should you refrain from doing [intuitively evil thing]?"

3. ^{^}

   Maybe the most promising angle is to show that it's normatively relevant that our beliefs about the more distant moral patients are (qualitatively?) less grounded in good reasons (see Clifton).

Permissive epistemology doesn't imply precise credences / completeness / non-cluelessness

(Many thanks to Jesse Clifton and Sylvester Kollin for discussion.)

My arguments against precise Bayesianism and for cluelessness appeal heavily to the premise “we shouldn’t arbitrarily narrow down our beliefs”. This premise is very compelling to me (and I’d be surprised if it’s not compelling to most others upon reflection, at least if we leave “arbitrary” open to interpretation). I hope to get around to writing more about it eventually.

But suppose you don’t care much about avoiding arbitrariness, for any interesting interpretation of “arbitrariness”. Suppose you have a highly permissive epistemology. E.g., you might think that as long as your beliefs satisfy some “coherence” conditions, that’s as far as rationality can take you, and you’re free to have whichever coherent beliefs you’re disposed to. Or you might think “beliefs” are nothing more or less than constraints on preferences, and it’s fine if preferences are arbitrary.

If you have such a view, does that imply your credences are precise, or your preferences are complete, or the like?

No.^[1] You might simply introspect on what your response to your evidence, intuitions, etc. is, and find that the most honest representation of that response is imprecise/incomplete. You might try weighing up various considerations about the consequences of some pair of actions, and find that your disposition is, “I have no clue. Even after noticing that some precise number popped into my head, all things considered I have no preference either way (but I also don’t feel indifferent, because my preferences are insensitive to mild sweetening).”

Of course, if you have a permissive epistemology and you don’t have such introspective reactions, there’s nothing more I can say to you. But it’s important to acknowledge that precision, completeness, and non-cluelessness are not some privileged default for the permissivist.

1. ^{^}

   As argued here and here, imprecision and incompleteness are consistent with all the usual coherence conditions that don’t straightforwardly beg the question against incompleteness.

Why expect "heuristics" to be robust to unknown unknowns?

I often read/hear claims that, if we're worried that our evaluations of interventions won't hold up under unknown unknowns, we should follow (simple) heuristics. But what precisely is the argument for this? This isn't a rhetorical question — I'm just noting my confusion and want to understand this view better.

Interested to hear more from those who endorse this view!

No, longtermism is not redundant

I’m not keen on the recent trend of arguments that persuading people of longtermism is unnecessary, or even counterproductive, for encouraging them to work on certain cause areas (e.g., here, here). This is for a few reasons:

• It’s not enough to believe that extinction risks within our lifetimes are high, and that extinction would constitute a significant moral problem purely on the grounds of harms to existing beings. Arguments for the tractability of reducing those risks, sufficient to outweigh the nearterm good done by focusing on global human health or animal welfare, seem lacking in the arguments I’ve seen for prioritizing extinction risk reduction on non-longtermist grounds.
• Take the AI alignment problem as one example (among the possible extinction risks, I’m most familiar with this one). I think it’s plausible that the collective efforts of alignment researchers and people working on governance will prevent extinction, though I’m not prepared to put a number on this. But as far as I’ve seen, there haven’t been compelling cost-effectiveness estimates suggesting that the marginal dollar or work-hour invested in alignment is competitive with GiveWell charities or interventions against factory farming, from a purely neartermist perspective. (Shulman discusses this in this interview, but without specifics about tractability that I would find persuasive.)
• More importantly, not all longtermist cause areas are risks that would befall currently existing beings. MacAskill discusses this a bit here, including the importance of shaping the values of the future rather than (I would say “complacently”) supposing things will converge towards a utopia by default. Near-term extinction risks do seem likely to be the most time-sensitive thing that non-downside-focused longtermists would want to prioritize. But again, tractability makes a difference, and for those who are downside-focused, there simply isn’t this convenient convergence between near- and long-term interventions. As far I can tell, s-risks affecting beings in the near future fortunately seem highly unlikely.

Why you should consider trying SSRIs

I was initially hesitant to post this, out of some vague fear of stigma and stating the obvious, and not wanting people to pathologize my ethical views based on the fact that I take antidepressants. This is pretty silly for two reasons. First, I think that if my past self had read something like this, he could have been spared years of suffering, and there are probably several readers in his position. EAs are pretty open about mental illness anyway. Second, if anything the fact that I am SFE "despite" currently not being depressed at all (indeed quite consistently happy), thanks to SSRIs, should make readers less likely to attribute my views to a mental illness.[1]

I claim that even if you don't feel so bad as to qualify as capital-D depressed, you might feel noticeably less bad on a daily basis if you try SSRIs.[2] That has been my experience, and I can honestly say this has likely been the cheapest sustainable boost in my well-being I've ever found. Being happier has also probably made me more effective/productive, though this is harder to assess.

(Obviously, my experience is not universal, I'm probably just lucky to an extent, this is not expert medical advice, and you might either find that SSRIs are ineffective for you or that the side effects are less tolerable than I have found them. You should definitely do your own research!)

In the months (at least) prior to SSRIs, my level of depression was "mild" according to the Burns checklist. I felt rather lonely during downtime, and like a bit of a failure for not having an exciting social life. I didn't endorse the latter judgment, and felt pretty fulfilled by my altruistic work, but that dissatisfaction persisted even when I tried to reason myself out of it (or tried taking up new hobbies). This wasn't debilitating by any means—so much so that I didn't really feel like I "deserved" to receive treatment intended for depression, and yes I realize how dumb that sounds in hindsight—but it was something of a pall hanging over my life all the same.

SSRIs just dispelled those feelings.

Waiting so long to give these things a try was a mistake. I made that mistake out of a combination of the aforementioned suspicion that I wasn't depressed enough to need them, and overestimation of how difficult it would be to get a prescription.[3] Just because my suffering wasn't as deep as others', that didn't mean it needed to exist.

This medication isn't magic; my life isn't perfect, and I still have some other dissatisfactions I'm working on. But, for the amount of difference this has made for me, it seemed negligent not to share my experience and hopefully encourage others in an analogous position to show themselves a bit of compassion.

[1] Yes, I have seen people do this before—not to me personally, but to other SFEs.

[2] This probably holds for other antidepressants too. I'm just focusing on SSRIs here because I have experience with them, and they incidentally have a worse reputation than, e.g., Wellbutrin.

[3] At least in the U.S., there are online services where you can share your symptoms with a doctor and just get a prescription at a pretty low price. For some reason, I expected a lot more awkward bureaucracy and mandatory therapy than this. I won't get specific here because I don't want to be a shill, but if you're curious, feel free to PM me.

In Defense of Aiming for the Minimum

I’m not really sympathetic to the following common sentiment: “EAs should not try to do as much good as feasible at the expense of their own well-being / the good of their close associates.”

It’s tautologically true that if trying to hyper-optimize comes at too much of a cost to the energy you can devote to your most important altruistic work, then trying to hyper-optimize is altruistically counterproductive. I acknowledge that this is the principle behind the sentiment above, and evidently some people’s effectiveness has benefited from advice like this.

But in practice, I see EAs apply this principle in ways that seem suspiciously favorable to their own well-being, or to the status quo. When you find yourself trying to justify on the grounds of impact the amounts of self-care people afford themselves when they don’t care about being effectively altruistic, you should be extremely suspicious.

Some examples, which I cite not to pick on the authors in particular—since I think many others are making a similar mistake—but just because they actually wrote these claims down.

1. “Aiming for the minimum of self-care is dangerous”

I felt a bit suspicious, looking at how I spent my time. Surely that long road trip wasn’t necessary to avoid misery? Did I really need to spend several weekends in a row building a ridiculous LED laser maze, when my other side project was talking to young synthetic biologists about ethics?

I think this is just correct. If your argument is that EAs shouldn’t be totally self-effacing because some frivolities are psychologically necessary to keep rescuing people from the bottomless pit of suffering, then sure, do the things that are psychologically necessary. I’m skeptical that “psychologically necessary” actually looks similar to the amount of frivolities indulged by the average person who is as well-off as EAs generally are.

Do I live up to this standard? Hardly. That doesn’t mean I should pretend I’m doing the right thing.

Minimization is greedy. You don’t get to celebrate that you’ve gained an hour a day [from sleeping seven instead of eight hours], or done something impactful this week, because that minimizing urge is still looking at all your unclaimed time, and wondering why you aren’t using it better, too.

How important is my own celebration, though, when you really weigh it against what I could be doing with even more time? (This isn’t just abstract impact points; there are other beings whose struggles matter no less than mine do, and fewer frivolities for me could mean relief for them.)

I think where I fundamentally disagree with this post is that, for many people, I don’t think aiming for the minimum puts you close to less than the minimum. Getting to the minimum, much less below it, can be very hard, such that people who aim at it just aren’t in much danger of undershooting. If you find this is not true for yourself, then please do back off from the minimum. But remember that in the counterfactual where you hadn’t tested your limits, you probably would not have gotten close to optimal.

This post includes some saddening anecdotes about people ending up miserable because they tried to optimize all their time for altruism. I don’t want to trivialize their suffering. Yet I can conjure anecdotes in the opposite direction (and the kind of altruism I care about reduces more suffering in expectation). Several of my colleagues seem to work more than the typical job entails, and I don’t have any evidence of the quality of their work being the worse for this. I’ve found that the amount of time I can realistically devote to altruistic efforts is pretty malleable. No, I’m not a machine; of course I have my limits. But when I gave myself permission to do altruistic things for parts of weekends, or into later hours of weekdays, well, I could. “My happiness is not the point,” as Julia said in this post, and while she evidently doesn’t endorse that statement, I do. That just seems to be the inevitable consequence of taking the sentience of other beings besides yourself (or your loved ones) seriously.

See also this comment:

Personally have been trying to think of my life only as a means to an end. Will my life technically might have value, I am fairly sure it is rather minuscule compared to the potential impact can make. I think it's possible, though probably difficult, to intuit this and still feel fine / not guilty, about things. … I'm a bit wary on this topic that people might be a bit biased to select beliefs based on what is satisfying or which ones feel good.

I do think Tessa's point about slack has some force—though in a sense, this merely shifts the “minimum” up by some robustness margin, which is unlikely to be large enough to justify the average person’s indulgences.

2. “You have more than one goal, and that’s fine”

If I donate to my friend’s fundraiser for her sick uncle, I’m pursuing a goal. But it’s the goal of “support my friend and our friendship,” not my goal of “make the world as good as possible.” When I make a decision, it’s better if I’m clear about which goal I’m pursuing. I don’t have to beat myself up about this money not being used for optimizing the world — that was never the point of that donation. That money is coming from my “personal satisfaction” budget, along with money I use for things like getting coffee with friends.

It puzzles me that, as common as concerns about the utility monster—sacrificing the well-being of the many for the super-happiness of one—are, we seem to find it totally intuitive that one can (passively) sacrifice the well-being of the many for one’s own rather mild comforts. (This is confounded by the act vs. omission distinction, but do you really endorse that?)

The latter conclusion is basically the implication of accepting goals other than “make the world as good as possible.” What makes these other goals so special, that they can demand disproportionate attention (“disproportionate” relative to how much actual well-being is at stake)?

3. “Ineffective Altruism”

Due to the writing style, it’s honestly not clear to me what exactly this post was claiming. But the author does emphatically say that devoting all of their time to the activity that helps more people per hour would be “premature optimization.” And they celebrate an example of a less effective thing they do because it consistently makes a few people happy.

I don’t see how the post actually defends doing the less effective thing. To the extent that you impartially care about other sentient beings, and don’t think their experiences matter any less because you have fewer warm fuzzy feelings about them, what is the justification for willingly helping fewer people?

Some reasons not to primarily argue for veganism on health/climate change grounds

I've often heard animal advocates claim that since non-vegans are generally more receptive to arguments from health benefits and reducing climate impact, we should prioritize those arguments, in order to reduce farmed animal suffering most effectively.

On its face, this is pretty reasonable, and I personally don't care intrinsically about how virtuous people's motivations for going vegan are. Suffering is suffering, no matter its sociological cause.

But there are some reasons I'm nervous about this approach, at least if it comes at the opportunity cost of moral advocacy. None of these are original to me, but I want to summarize them here since I think this is a somewhat neglected point:

1. Plausibly many who are persuaded by the health/CC arguments won't want to make the full change to veganism, so they'll substitute cows for chickens and fish. Both of which are evidently less bad for one's health and CC risk, but because these animals are so small and have fewer welfare protections, this switch causes a lot more suffering per calorie. More speculatively, there could be a switch to insect consumption.
2. Health/CC arguments don't apply to reducing wild animal suffering, and indeed emphasizing environmental motivations for going vegan might exacerbate support for conservation for its own sake, independent of individual animals' welfare. (To be fair, moral arguments can also backfire if the emphasis is on general care for animals, rather than specifically preventing extreme suffering.)
3. Relatedly, health/CC arguments don't motivate one to oppose other potential sources of suffering in voiceless sentient beings, like reckless terraforming and panspermia, or unregulated advanced simulations. This isn't to say all anti-speciesists will make that connection, but caring about animals themselves rather than avoiding exploiting them for human-centric reasons seems likely to increase concern for other minds.
4. While the evidence re: CC seems quite robust, nutrition science is super uncertain and messy. Based on both this prior about the field and suspicious convergence concerns, I'd be surprised if a scientific consensus established veganism as systematically better for one's health than alternatives. That said, I'd also be very surprised about a consensus that it's worse, and clearly even primarily ethics-based arguments for veganism should also clarify that it's feasible to live (very) healthily on a vegan diet.

Some things I liked about What We Owe the Future, despite my disagreements with the treatment of value asymmetries:

• The thought experiment of imagining that you live one big super-life composed of all sentient beings’ experiences is cool, as a way of probing moral intuitions. (I'd say this kind of thought experiment is the core of ethics.)
• It seems better than e.g. Rawls’ veil of ignorance because living all lives (1) makes it more salient that the possibly rare extreme experiences of some lives still exist even if you're (un)lucky enough not to go through them, and (2) avoids favoring average-utilitarian intuitions.
• Although the devil is very much in the details of what measure of (dis)value the total view totals up, the critiques of average, critical level, and symmetric person-affecting views are spot-on.
• There's some good discussion of avoiding lock-in of bad (/not-reflected-upon) values as a priority that most longtermists can get behind.
• I was already inclined to think dominant values can be very contingent on factors that don't seem ethically relevant, like differences in reproduction rates (biological or otherwise) or flukes of power imbalances. So I didn't update much from reading about this. But I have the impression that many longtermists are a bit too complacent about future people converging to the values we'd endorse with proper reflection (strangely, even when they're less sympathetic to moral realism than I am). And the vignettes about e.g. Benjamin Lay were pretty inspiring.
• Relatedly, it's great that premature space settlement is acknowledged as a source of lock-in / reduction of option value. Lots of discourse on longtermism seems to gloss over this.

The Repugnant Conclusion is worse than I thought

At the risk of belaboring the obvious to anyone who has considered this point before: The RC glosses over the exact content of happiness and suffering that are summed up to the quantities of “welfare” defining world A and world Z. In world A, each life with welfare 1,000,000 could, on one extreme, consist purely of (a) good experiences that sum in intensity to a level 1,000,000, or on the other, (b) good experiences summing to 1,000,000,000 minus bad experiences summing (in absolute value) to 999,000,000. Similarly, each of the lives of welfare 1 in world Z could be (a) purely level 1 good experiences, or (b) level 1,000,001 good experiences minus level 1,000,000 bad experiences.

To my intuitions, it’s pretty easy to accept the RC if our conception of worlds A and Z is the pair (a, a) from the (of course non-exhaustive) possibilities above, even more so for (b, a). However, the RC is extremely unpalatable if we consider the pair (a, b). This conclusion, which is entailed by any plausible non-negative[1] total utilitarian view, is that a world of tremendous happiness with absolutely no suffering is worse than a world of many beings each experiencing just slightly more happiness than those in the first, but along with tremendous agony.

To drive home how counterintuitive that is, we can apply the same reasoning often applied against NU views: Suppose the level 1,000,001 happiness in each being in world Z is compressed into one millisecond of some super-bliss, contained within a life of otherwise unremitting misery. There doesn’t appear to be any temporal ordering of the experiences of each life in world Z such that this conclusion isn’t morally absurd to me. (Going out with a bang sounds nice, but not nice enough to make the preceding pure misery worth it; remember this is a millisecond!) This is even accounting for the possible scope neglect involved in considering the massive number of lives in world Z. Indeed, multiplying these lives seems to make the picture more horrifying, not less.

Again, at the risk of sounding obvious: The repugnance of the RC here is that on total non-NU axiologies, we’d be forced to consider the kind of life I just sketched a “net-positive” life morally speaking.[2] Worse, we're forced to consider an astronomical number of such lives better than a (comparatively small) pure utopia.

[1] “Negative” here includes lexical and lexical threshold views.

[2] I’m setting aside possible defenses based on the axiological importance of duration. This is because (1) I’m quite uncertain about that point, though I share the intuition, and (2) it seems any such defense rescues NU just as well. I.e. one can, under this principle, maintain that 1 hour of torture-level suffering is impossible to morally outweigh, but 1 millisecond isn’t.

A Parfitian Veil of Ignorance

[Edit: I would be very surprised if I were the first person to have proposed this; it probably exists somewhere else, I just don't know of a source.]

Prompted by Holden’s discussion of the veil of ignorance as a utilitarian intuition pump (contra Rawls), I thought about an alternative to the standard veil. My intuitions about tradeoffs of massive harms for a large number of small benefits—at least for some conceptions of "benefit"—diverge from those in his post, when considering this version.

The standard veil of ignorance asks you to imagine being totally ignorant as to which person you will be in a population. (Assume we’re only considering fixed population sizes, so there’s no worry that this exercise sneaks in average utilitarianism, etc.)

But the many EA fans of Parfit (or Buddha) know that this idea of a discrete person is metaphysically problematic. So we can look at another approach, inspired by empty individualism.

Imagine that when evaluating two possible worlds, you don’t know which slice of experience in each world you would be. To make things easy enough to grasp, take a “slice” to be just the longest amount of time necessary for a sentient being to register an experience, but not much longer. Let’s say one second.

These worlds might entail probabilities of experiences as well. So, since it’s hard to intuitively grasp probabilities as effectively as frequencies, suppose each world is “re-rolled” a large enough times that each outcome happens at least once, in proportion to its probability. e.g., In Holden’s example of a 1 in 100 million chance of someone dying, the experiences of that person are repeated 100 million times, and one of those experience streams is cut short by death.

So now a purely aggregative and symmetric utilitarian offers me a choice, from behind this veil of ignorance, between two worlds. Option 1 consists of a person who lives for one day with constantly neutral experiences—no happiness, no suffering (including boredom). In option 2, that person instead spends that day relaxing on a nice beach, with a 1 in 100 million chance of ending that day by spiraling into a depression (instead of dying peacefully in their sleep).

I imagine, first, rescaling things so in #1 the person lives 100 million days of neutrality, and in #2, they live 99,999,999 peaceful beach-days—suspend your disbelief and imagine they never get bored—followed by a beach-day that ends in depression. Then I imagine I don’t know which moment of experience in either of these options I’ll be.

Choosing #1 seems pretty defensible to me, from this perspective. Several of those experience-moments in #2 are going to consist purely of misery. They won’t be comforted by the fact that they’re rare, or that they’re in the context of a “person” who otherwise is quite happy. They’ll just suffer.

I’m not saying the probabilities don’t matter. Of course they do; I’d rather take #2 than a third option where there’s a 1 in 100 thousand chance of depression. I’m also pretty uncertain where I stand when we modify #1 so that the person’s life is a constant mild itch instead of neutrality. The intuition this thought experiment prompts in me is the lexical badness of at least sufficiently intense suffering, compared with happiness or other goods. And I think the reason it prompts such an intuition is that in this version of the veil of ignorance, discrete “persons” don’t get to dictate what package of experiences is worth it, i.e., what happens to the multitude of experience-moments in their life. Instead, one has to take the experience-moments themselves as sovereign, and decide how to handle conflicts among their preferences. (I discuss this more here.)

Linkpost: "Tranquilism Respects Individual Desires"

I wrote a defense of an axiology on which an experience is perfectly good to the extent that it is absent of craving for change. This defense follows in part from a reductionist view of personal identity, which is usually considered in EA circles to be in support of total symmetric utilitarianism, but I argue that this view lends support to a form of negative utilitarianism.

Some vaguely clustered opinions on metaethics/metanormativity

I'm finding myself slightly more sympathetic to moral antirealism lately, but still afford most of my credence to a form of realism that would not be labeled "strong" or "robust." There are several complicated propositions I find plausible that are in tension:

1. I have a strong aversion to arbitrary or ad hoc elements in ethics. Practically this cashes out as things like: (1) rejecting any solutions to population ethics that violate transitivity, and (2) being fairly unpersuaded by solutions to fanaticism that round down small probabilities or cap the utility function.

2. Despite this, I do not intrinsically care about the simplicity of a moral theory, at least for some conceptions of "simplicity." It's quite common in EA and rationalist circles to dismiss simple or monistic moral theories as attempting to shoehorn the complexity of human values into one box. I grant that I might unintentionally be doing this when I respond to critiques of the moral theory that makes most sense to me, which is "simple." But from the inside I don't introspect that this is what's going on. I would be perfectly happy to add some complexity to my theory to avoid underfitting the moral data, provided this isn't so contrived as to constitute overfitting. The closest cases I can think of where I might need to do this are in population ethics and fanaticism. I simply don't see what could matter morally in the kinds of things whose intrinsic value I reject: rules, virtues, happiness, desert, ... When I think of these things, and the thought experiments meant to pump one's intuitions in their favor, I do feel their emotional force. It's simply that I am more inclined to think of them as just that: emotional, or game theoretically useful constructs that break down when you eliminate bad consequences on conscious experience. The fact that I may "care" about them doesn't mean I endorse them as relevant to making the world a better place.

3. Changing my mind on moral matters doesn't feel like "figuring out my values." I roughly know what I value. Many things I value, like a disproportionate degree of comfort for myself, are things I very much wish I didn't value, things I don't think I should value. A common response I've received is something like: "The values you don't think you 'should' have are simply ones that contradict stronger values you hold. You have meta-preferences/meta-values." Sure, but I don't think this has always been the case. Before I learned about EA, I don't think it would have been accurate to say I really did "value" impartial maximization of good across sentient beings. This was a value I had to adopt, to bring my motivations in line with my reasons. Encountering EA materials did not feel at all like "Oh, you know what, deep down this was always what I would've wanted to optimize for, I just didn't know I would've wanted it."

4. The question "what would you do if you discovered the moral truth was to do [obviously bad thing]?" doesn't make sense to me, for certain inputs of [obviously bad thing], e.g. torturing all sentient beings as much as possible. For extreme inputs of that sort, the question is similar to "what would you do if you discovered 2+2=5?" For less extreme inputs, such that it's plausible to me I simply have not thought through ethics enough that I could imagine that hypothetical but merely find it unlikely right now, the question does make sense, and I see nothing wrong with saying "yes." I suspect many antirealists do this all the time, radically changing their minds on moral questions due to considerations other than empirical discoveries, and they would not be content saying "screw the moral truth" by retaining their previous stance.