Excellent job describing this clearly and accessibly. (And teaching me about Saturationism) You portray this well from where I stand on the opposite shore!

I think that truth and variety matter^[1] and would prefer to live in a universe with more truth and more variety even at the cost of "loss of perfection" or suffering^[2]. Even if I could not tell the difference (unaware of deceptions) or never experienced any positive effects (from variety). I personally value these things, despite them not simplifying to additional positive valence. 

Here are some values I have that don't pan out to positive welfare:
1) I think if someone dedicates their life to some endeavor, and they are feeling the rush of emotions as their life's project comes to an end, it would be "bad" if those exact same emotions/experience were via beaming that feeling into their mind. They cared about the project itself, they wanted it to be due to their own effort, they are not doing it for the pleasurable experience portion. 
2) I think it is bad if we, with well meaning, erase^[3] everyone's memory to replay their perfect happiest possible moment over and over again for the rest of their lifetimes. I think, even if nothing could be a better world of experience for everyone, and no one will ever know it's a lie, that this is a flawed^[4] life and a flawed world.
3) Someone who changes the future but doesn't live to see it has, in some sense, a better life than someone who tried and failed, even though their felt experience is identical.
4) I think it would be a shame if many things are never understood or discovered before the universe ends, regardless of how this translates to total happiness experienced. 
5) I think an empty universe is bad in comparison to what it could have been, even if no one exists to prefer something else.

You cannot imagine an unwatched world without mentally watching it
Yes, but I think that if you delete the outside view then lots of things are incoherent and worse. 
The badness of death disappears. We know that being dead is bad because of the outside view  where they could be happily living. Same for unborn people. Accounting for past and future experiences is using an outside view, no one is actively having those experiences. Similarly, if we forcibly rewrite everyone's desires to be something terrible, (like to desire everyone dies horribly, and then we all happily satisfy this!) this only is bad from outside. 

The summing view makes arbitrary outside view choices too. Why sum? Why not only care about the single happiest life ever lived? No one experiences the sum, the lives that didn't happen, or the future. You are asking the accountant to look at total welfare achieved out of potential welfare possible, not just total welfare and without judgement on worlds which could have existed instead. There is nothing to evaluate, without an outside view to compare against. Choosing to "add them up" instead of "average them" or "prioritize the worst-off" is a world-shape preference the accountant imposes, not something anyone experiences. You want to say the Totalist's account is fundamental while the Saturationist's account is doing something superficial. But the Totalist is just as arbitrarily choosing how and what to aggregate when he's adding up experiences, and using an outside view to do it. Yes, variety across experiences is an outside view parameter, but that isn't unusual.

Its also a little weird to assume positive welfare "exists" and "can be summed." Maybe when you "sum" it, the redundant identical feelings overlap, occupying the space of one single unique experience, and summing it results in the Saturationist's account. Maybe that's just how adding up felt experiences works.

No one proposes an ugly impersonal value.
No, variety and truth are very much an ugly impersonal values. These could have very high costs and often result in a lot of suffering. All kinds of stories celebrate people who wrecked their lives forcing truth or justice into the world, or chased perfection/uniqueness (even when it doesn't work!), which is strange behavior if these values are just about netting more pleasurable experiences. Their value did NOT evaporate when it had an net-negative-welfare cost. Sometimes we admire it as a life more well-lived, in some non-welfare sense. They had purpose instead of happiness, or something. We often hold these values dear albeit with conflicted uncertainty about what is best and right and matters most when it had such a terrible downside. 
We don't celebrate the people that are happiest and heroically help others bury their head in the sand to protect their happiness. Which would indicate that happiness is not the single final value...

Ask A whether she wants to switch worlds: she is indifferent, and she must be, because if she preferred one, the welfare would not be equal. Ask B, same answer, because nothing changes for him.
Turning around and sating that "this must mean the welfare wasn't equal" is hard to respond to. Maybe its high quality welfare without being higher quantity? People do things for for results that they will never personally experience all the time. Its weird to say those values are fake.

I think there are additional fundamental values beyond welfare, like variety and truth. Unfortunately I'm not sure if they can be explained further than "I value them..." so I may have to just describe them and then stop.

Anyway, all this results in this belief that I have:
If we reach the world where everyone has the most perfect life, and it's the same life, I think this world is worse than many non-identical, not-quite-perfect lives. I wouldn't force worse lives on anyone, but I'd volunteer to live a worse life so that the universe contains something different. Normally sharing my different life would benefit others; but even if no one benefited at all and I wasted my perfection, I think I might still want to do it.

I'm open to the possibility that I'm a limited idiot who can't see what the god-minds of the future will want. In my defense, I'm not hopelessly unimaginative. I fully expect that we will want to do currently-crazy-sounding things likes erase ourselves, limit/reshape our minds, replay experiences on loop, merge into amorphous monads, do tedious activities in deep utopia, and tons of other unimaginable things that might seem undesirable, even hideous, to me today...) The god-minds might all agree that tiling the universe with hedonium is obviously correct. Maybe I've just intrinsified some stuff that is a mere side-effect of promoting progress or weeding out traitors or something that will no longer apply in the grand scheme of things. But from my best attempts at imagining the god-minds at the end of the universe, I think they would value variety and truth even at the expense of some lost hedonium.

1. ^{^}

   "cosmically"

2. ^{^}

   I would also not maximize truth and variety at all costs.

3. ^{^}

   assuming they didn't agree to it. I expect we will choose to do this to ourselves quite a lot actually.

^{^}

Maybe not a bad life compared to today's good lives, etc. It's an alright future. But I think its far worse than many other life-shapes and potential worlds that include variety.

Show all footnotes

Thank you. I think that's a fair summary. To be precise, rather than claiming that the maximality rule is generally too sensitive to small changes, my critique is that it's generally too insensitive to small changes, as virtually all updates that would be registered by a graded approach make no difference at all. As a consequence of this insensitivity, maximality implies isolated points of extreme sensitivity where some tiny change can result in a categorical jump. So yes, I think it's too sensitive to small changes in the sense that it packs all its sensitivity into those sharp points where dominance is achieved.

Thank you for these useful comments.

I agree that indeterminacy can still obtain under a graded approach. However, it’s worth noting that the band in which indeterminacy obtains under a graded approach will generally be much narrower than under maximality. So moving from maximality to a graded approach is a big change.

When I try to weigh up the reasons in favor of any A vs. B w.r.t. their total consequences, fully accounting for unawareness, I really don't see why I should consider the "upper endpoint" larger or smaller in absolute value than the "lower endpoint".

I gather we see this differently, but to very briefly explain why it seems warranted to me: Because of the evidence and arguments favoring the most plausible strategies, and the lack of similarly strong evidence and arguments against them. In particular, I don’t think the arguments against the capacity-building strategies I discuss are as strong as those in their favor. Moreover, much of the value of building capacity is the value of being able to act on considerations we aren’t yet aware of. So, in my view, unawareness bears asymmetrically on these strategies rather than neutrally (I realize this latter point is stated very briefly and needs further development).

I would like to see more of an argument that this isn't our situation, perhaps building on what you say in your appendix about low-footprint capacity-building.

Without trying to build an elaborate case here, there is a simple extension that I think further strengthens the case: instead of focusing only on the best-supported strategies, we can specifically compare the seemingly best-supported strategy (say, some form of low-footprint CB) with the strategy that seems worst (I suspect it’s better not to go into details about what that might be).

When comparing two such contrasting strategies, the expected difference plausibly gets beyond the (relatively) narrow band of indeterminacy under some plausible graded approach, considering both the arguments favoring the seemingly better strategy and the arguments opposing the seemingly worse strategy. The reason this helps is that the width of the indeterminacy band doesn’t obviously scale with how contrasting the pair is, whereas the expected difference does. (Again, the expected difference between the two strategies need not be remotely as stark as in the figure below.)

You can have insensitivity to mild sweetening under vague degrees of betterness; it doesn't assume maximality.

I agree that there can be insensitivity to mild sweetening under rules other than maximality. (As noted in more general terms, the blanket objection that rejects “any evidence or proposal we might wager on … assumes the kind of categorical decision rule that collapses a broad set of evidential states into complete and undifferentiated indeterminacy.”)

But interestingly, under a graded approach, even if indeterminacy may in some sense be preserved after a mild sweetening (within the relatively narrow indeterminacy band), we still wouldn’t get complete insensitivity to mild sweetening. In that case, we would have a hazy cloud of vague expected values, and a vague degree of ex ante betterness, but that vague degree would still slightly increase from the mild sweetening, even if the change isn’t large enough to yield a determinately higher degree of betterness for one option. The hazy cloud and degree do move.

A justified wager is not guaranteed under such an approach, as our evidence could in principle be closely balanced, but the bar for finding one is relatively low (since the approach doesn’t collapse a broad set of distinct evidential states into indeterminacy, cf. the figure here). It seems to me we can make such justified wagers given both how mild sweetenings do incrementally move ex ante degrees of betterness and the, in my view, sufficiently weighty arguments and considerations that support the most plausible strategies over the worst ones.

I see. I guess if there's a chance that we can stop genie from being out of box, we should try. You're also correct that tiered access can indeed mitigate issues. And it does seem like general models being as good and 'Bio Mythos' might genuinely not happen. So, exploring tightening access and data makes sense. 

All of this requires co-ordinated action though as even one bad actor can significantly turn things for worse. And the fact that covid is fresh in policy makers' minds could help. 

My points are more in the case of being or inevitably going to a point where we can 'reasonably freely' edit pathogens and vaccines. If we can do that, making it freer and restricting implementation is generally a better idea. And if possible to do safely, we should aim for that, I think. Because there is a lot of potential of good for everyone: humans, animals, other biological beings and maybe more (like plastic or rust eating bacteria for environment!). 

I talk about this paper in light of comparing more modern meat alternatives (Beyond/Impossible/etc.) to traditional vegetarian mains (tofu/curry/etc.), and why the academic literature at least suggests more cause for optimism about the latter than the former, here: 

https://regressiontothemeat.substack.com/p/the-more-successful-add-more-veggie

Psychopaths are extraordinary, i.e. out of the ordinary, uncommon, a minority, in a negative sense (lucky there are only few of them around).

I implicitly meant extraordinary in a positive sense, extraordinary achievers of valuable goals. 

One is extraordinary in either sense  because one was born with uncommon traits and/or because one has spent a lot of effort to develops an uncommon capacity.

Thanks for making and sharing this! A friend who is new to EA is curious about AIxBio risk and I think this could be helpful for her.