If AI agents use LLMs as their brain, how much of the harm that they end up doing is up to the chance of which token gets sampled from the output distribution?

As a dumb example, say some agent using an LLM as its backbone is reasoning "[...], so humans deserve to", and the LLM's output distribution for the next token assigns 98% probability to "die" and 1% to "live". The reasoning that follows will be influenced by which of these gets sampled.

A different objection:

• Low-welfare confinement: Barely-positive-welfare value-bearers occupy a small enough region of the landscape, or are sparse enough and illuminate only a small enough neighbourhood around them, or both, such that the total value that can be contributed by barely-positive-welfare value-bearers is strictly smaller than the total value that can be contributed by more-than-barely-positive-welfare value-bearers.

This kind of statement seems like philosophical wishful thinking: it makes an arbitrary assumption about what is possible in order to avoid an uncomfortable conclusion about what is preferable (the Repugnant Conclusion). It simply asserts without justification that the space of low-welfare variety is small.

This isn't just an innocent modeling choice, nor is it an axiom of the Saturationism axiology itself; it's a substantive claim about reality inserted to make the axiology work. By stipulating that low-welfare value-bearers occupy a confined region of the conceptual landscape, we are forcing a physical constraint onto the universe without providing any physical or empirical justification for it.

Why should we accept this asymmetry? It seems equally plausible that barely-positive-welfare lives could manifest in a greater diversity of configurations than high-welfare lives. Without a robust justification for why the low-welfare landscape is uniquely restricted, this constraint just looks like an ad hoc mathematical patch to save the axiology from the Repugnant Conclusion.

Suppose that you could live for an extraordinary amount of time, and could choose any sort of life you wanted. Would you choose to relive, endlessly, the same maximally-good experience over and over again? Or would you instead choose to live through a wide variety of extremely-wonderful experiences?

I think most of us would clearly prefer the latter, and not merely because we would worry we’d get bored otherwise — we could stipulate that the maximally-good experience is one where one never feels bored. Rather, we have an intrinsic preference for variety.

Again, the problem is that it's almost impossible to genuinely hold that stipulation in our heads. We just can't psychologically comprehend the part "is one where one never feels bored", because nothing in human experience mirrors that.

When a thought experiment explicitly stipulates that boredom is locked at zero, we can't let our real-world psychology smuggle it back in under the guise of an "intrinsic preference." If we truly accept the premise as stated—that the experience remains maximally good indefinitely without decay—then rejecting it based on "variety" seems just a psychological trap.

Quick take:

I disagree with the intuition that diversity is intrinsically valuable (Variety > Homogeneity). I think this intuition is merely a psychological illusion driven by a failure to truly accept the definitions of the thought experiment.

When we imagine living in a homogeneous world, we intuit that we would enjoy it less (due to boredom, lack of novelty, etc.)—meaning our individual well-being would suffer. But that just means diversity is a preference within our individual utility functions, not an intrinsic part of the axiology.

The thought experiment tricks us because we fail to truly hold "all other things being equal." If we are explicitly told that the welfare levels in Homogeneity are greater than or equal to those in Variety, then those welfare figures already account for any negative effect caused by a lack of diversity. Once you strictly hold welfare constant, the intuition that Variety > Homogeneity evaporates. Diversity is likely just an instrumental good for the welfare of individuals, not an intrinsic axiological metric.

Our intuition tricks us into double-counting diversity: we treat it as an intrinsic axiological value when it is actually just an instrumental preference within an individual's utility function. If the individuals in a "monoculture" are perfectly content with that homogeneity, there is no real contradiction in concluding that Homogeneity > Variety.

I have the impression that the most effective interventions, especially in global health/poverty, are usually temporary, in the sense that you need to keep reinvesting regularly, usually because the intervention provides a consumable good; for example malaria chemoprevention: it needs to be provided yearly. In contrast, solutions that seem more permanent in the long-term (e.g. a hypothetical malaria vaccination, or building infrastructure), are typically much less cost-effective on the margin because of their high cost.

How do we balance pure marginal effectiveness vs eventually moving towards more permanent solutions? Could it be that by overly optimising for marginal cost-effectiveness, we might be missing a better ‘global maximum’ in the utility landscape, but we just need to descend from the current ‘local maximum’ to be able to get there eventually?

Second, identify the people who are consistently more accurate than everyone else — say, those in the top 0.1% for accuracy, for multiple years in a row. These are your “superforecasters.”

My worry is this: if enough people are trying to make forecasts, just by random chance you will get some people that attain whatever arbitrary amount of accuracy you desire (e.g. getting several forecasts in a row right for several years). How do we tell if a "superforecaster" is really that or they just got lucky until now? If the latter, their past success is not an indicator of future success.

Thanks, I'll take a look at the literature you suggested before thinking further.

Sorry, perhaps I wasn't clear: I didn't mean matching by the identity of the individual, I meant matching on just their utility values (doesn't matter who is happy/suffering, only the unordered collection of utility values matters). So in your example, A and A' would be identical worlds (modulo ethical preference).

Formally: Let $a,b:U_I\to N$ be multisets of utilities (world states). (Notice that I'm using multisets and not vectors on purpose to indicate that the identities of the individuals don't matter.) To compare them, define the multiset $a\cap b$ as $(a\cap b)\left(u\right)=min\left(a\right(u),b(u\left)\right)$, and define $a^{\prime }=a-a\cap b$ and $b^{\prime }=b-a\cap b$ (pointwise). Then we compare $a^{\prime }$ and $b^{\prime }$ with leximin.

However, this still isn't transitive, unfortunately. E.g:
A: {{2}}
B: {{1, 3, 3}}
C: {{3}}
Then $A\gtrsim B$ and $B\gtrsim C$ but $A\gtrsim /C$ .

Right now I think the best solution is use plain leximin (as defined in my post) and reject the Mere Addition Principle.

Just a quick thought that comes to mind:

To avoid the birth paradox, when comparing two vectors we could first match all individuals that have the same utility in both worlds, we then eliminate these utilities from the comparisons, and then we perform leximin comparison on the remaining utilities from both worlds. I think this solves the birth paradox while preserving everything else.

I think this post is too long for what it's trying to do. There's no need to frontload so many technicalities - just compare finite sequences of real numbers. The other details don't matter too much.

You're probably right :) I kind of wanted to write down all my assumptions to be able to point at a self-contained document when I ask myself "what's my current preferred ethical system", and I got a bit carried away. Indeed you could explain the gist of it with real numbers, though I think it was worth highlighting that real numbers (with their implicit algebraic and order structure we might be tempted to use) are probably a really bad fit to measure utilities.
 

If I've understood your view correctly, it's a lexical view that's got the same problem that basically all lexical views have: it prioritises tiny improvements for some over any improvements, no matter how large, for others. 

Yes, and perhaps this is its biggest "repugnant conclusion", but I think it is far better than the original repugnant conclusions. It's a feature: as soon as you allow some exchange rate, you end up with scenarios where you are allowed to sacrifice the well-being of some for the benefit of everyone else, and once you can do this once you can keep iterating this ad infinitum. I want to reject that. 

Also, in practice it is usually impossible to completely keep track of every single individual, let alone come up with a precise utility value for each; therefore actions that specifically try to target the literal worst-off individual (the 'tiny improvement' you mention) have a low chance of success (improving the leximin score) because they have a low chance of actually identifying the correct individual. Therefore I'd argue that, when taking uncertainty and opportunity cost into account, this system would in practice still prioritize broad interventions (the 'large improvements') most of the time, and among those, prioritising those that affect the lower end of the spectrum (so that they have a higher probability of improving the life of the worst-off individual).

Another missing factor that might make this even less of a problem is considering variation through time rather than just snapshots in time (something I intended to write in a separate post). If we assume time (or "useful time", e.g. up until the heat death of the universe) is finite, we can apply leximin of world states (with discrete time steps so as to have a finite amount of world states) over time as the actual metric to rank actions. If we assume time is infinite and sentient beings might exist forever (terrible assumption, but it leads to an elegant model), I propose using the limit inferior of world states instead. Indeed, under either model, broader actions that affect more people in the lower end of the spectrum will probably be prioritised because they will probably have a larger compounding effect than the 'tiny interventions', thus reducing the chance of extreme suffering individuals over the course of time.

The key point is this: the reason your view avoids the Repugnant Conclusion is that it recommends populations where there is much less total wellbeing but some people have great lives over populations where there is more total wellbeing but everyone has a mediocre life.

Exactly, and I see that as desirable. As I explain in the post, I reject the need to maximize total wellbeing. You can think of it as an extreme (lexical) version of "quality over quantity".

Actually, when it comes to comparing same-person populations of people with positive wellbeing, it looks to me like your view always recommends the one with the highest maximum wellbeing. That's because you're using the leximax ordering.

Yes. If the maxima are equal, you then look at the second happiest individuals; and so on.

You *can* avoid the Repugnant Conclusion if you're willing to go down this route. But I suspect most people would think that the cure is worse than the disease.

I'd say that this is way more acceptable than the repugnant conclusion, but I would also bet this is quite an unpopular view.
Perhaps people would prefer the pure leximin approach then: in that case, (9, 9) would be better than (1, 10). The problem then is that (1, 2, 10) is worse than (1, 10): by adding a happy person, we've made the world worse. Perhaps this is the least worrying conclusion of all the ones we've discussed, if you accept a radical version of "we want to make people happy, not make happy people". And maybe there's a different "fix" altogether that avoids both issues.

I also think it's inaccurate to call this view a version of "prioritarianism" - again, unless I've misunderstood how it works.

I call it prioritarianism because in both leximin and leximinmax we're comparing suffering first, and among the sufferers, we prioritize the ones that are suffering the most. However, when we look at the positive side, no one is suffering overall, they are just different levels of happiness. If you keep using leximin, it still prioritizes the people that are least happy. If you use leximax, you are prioritizing the happiest.

Actually, now that I think of it again after the all-nighter, yes leximax on happy people seems bad. Perhaps pure leximin all the way up is the better approach. The only problem is the "birth paradox" outlined above, but perhaps it's the least of all problems we've considered. I'll try to think if there are other solutions.

Thanks for taking the time to reply!