I think smaller beings will produce welfare more efficiently. For example, I guess bees produce it 5 k times as effectively as humans.

I just don't think it makes any sense to have an aggregated total measure of "welfare". We can describe what is the distribution of welfare across the sentient beings of the universe, but to simply bunch it all up has essentially no meaning. In what way is a world with a billion very happy people any worse than a world with a trillion merely okay ones? I know which one I'd rather be born into! How can a world be worse for everyone individually yet somehow better, if the only meaning of welfare is that it is experienced by sentient beings to begin with?

There is a terrorist who is going to release a very infectious virus which will infect all humans on Earth. The virus makes people infertile forever, thus effectively leading to human extinction, but it also makes people fully lose their desire to have children, and have much better self-assessed lives. Would it make sense to kill the terrorist? Killing the terrorist would worsen the lives of all humans alive, but it would also prevent human extinction.

It's moral because the terrorist is infringing the wishes of those people right now, and violating their self-determination. If the people decided to infect themselves, then it would be ok.

Genocide involves suffering, and suffering is bad, so I assume there would be a better option to maximise impartial welfare. For example, ASI could arguably persuade humans that their extinction was for the better, or just pass everyone to a simulation without anyone noticing, and then shutting down the simulation in a way that no suffering is caused in the process.

I disagree that the genocide is made permissible by making the death a sufficiently painless euthanasia. Sure, the suffering is an additional evil, but the killing is an evil unto itself. Honestly, consider where these arguments could lead in realistic situations and consider whether you would be okay with that, or if you feel like relying on a circumstantial "well but actually in reality this would always come out negative net utility due to the suffering" is protection enough. If you get conclusions like these from your ethical framework it's probably a good sign that it might have some flaws.

For example, replacing rocks by humans is fine, and so might be replacing humans by digitals minds. However, the replacement process itself should maximise welfare, and I am very sceptical that "brutal colonization efforts" would be the most efficient way for ASI to perform the replacement.

Rocks aren't sentient, they don't count. And your logic still doesn't work. What if you can instantly vaporize everyone with a thermonuclear bomb, as they are all concentrated within the radius of the fireball? Death would then be instantaneous. Would that make it acceptable? Very much doubt it.

I am not sure I understand your point. Are you suggesting we should not maximise impartial welfare because this principle might imply humans being a small fraction of the overall number of beings?

I just don't think total sum utilitarianism maps well with the kind of intuitions I'd like a functional moral system to match. I think ideally a good aggregation system for utility should not be vulnerable to being gamed via utility monsters. I lean more towards average utility as a good index, though that too has its flaws and I'm not entirely happy with it. I've written a (very tongue-in-cheek) post about it on Less Wrong.

Whether suffering is good or bad depends on the situation, including the person assessing it.

Sure. So that actually backs my point that it's all relative to sentient subjects. There is no fundamental "real morality", though there are real facts about the conscious experience of sentient beings. But trade-offs between these experiences aren't obvious and can't be settled empirically.

The idea sounds bad to me too! The reason is that, in the real world, killing rarely brings about good outcomes. I am strongly against violence, and killing people.

But more so, killing people violates their own very strong preference towards not being killed. That holds for an ASI too.

Genociding a population is almost always a bad idea, but I do not think one should reject it in all cases. Would you agree that killing a terrorist to prevent 1 billion human deaths would be good? If so, would you agree that killing N terrorists to prevent N^1000 billion human deaths would also be good?

I mean, ok, one can construct these hypothetical scenarios, but the one you suggested wasn't about preventing deaths, but ensuring the existence of more lives in the future. And those are very different things.

Total utilitarianism only says one should maximise welfare. It does not say killing weaker beings is a useful heuristic to maximise welfare. My own view is that killing weaker beings is a terrible heuristic to maximise welfare (e.g. it may favour factory-farming, which I think is pretty bad).

But obviously if you count future beings too - as you are - then it becomes inevitable that this approach does justify genocide. Take the very real example of the natives of the Americas. By this logic, the same exact logic that you used for an example of why an ASI could be justified in genociding us, the colonists were justified in genociding the natives. After all, they lived in far lower population densities that the land could support with advanced agricultural techniques, and they lived hunter-gatherer or at best bronze-age style lives, far less rich of pleasures and enjoyments than a modern one. So killing a few millions of them to allow eventually for over 100 million modern Americans to make full use of the land would have been a good thing.

See the problem with the logic? As long as you have better technology and precommit to high population densities you can justify all sorts of brutal colonization efforts as a net good, if not maximal good. And that's a horrible broken logic. It's the same logic that the ASI that kills everyone on Earth just so it can colonize the galaxy would follow. If you think it's disgusting when applied to humans, well, the same standards ought to apply to ASI.

In the sense of increasing expected total hedonistic utility, where hedonistic utility can be thought of as positive conscious experiences.

I don't think total sum utilitarianism is a very sensible framework to me. I think it can work as a guideline within certain boundaries, but it breaks down as soon as you admit the potential for things like utility monsters, which ASI as you're describing it effectively is. Everyone only experiences one life, their own, regardless of how many other conscious entities are out there.

I would say it comes from the Laws of Physics, like everything else. While I am being tortured, the particles and fields in my body are such that I have a bad conscious experience.

That just kicks the metaphysical can down the road. Ok, suffering is physical. Who says that suffering is good or bad? Or that it is always good or bad? Who says that what's important is total rather than average utility, or some other more complex function? Who says how can we compare the utility of subjects A and B when their subjective qualia are incommensurate? None of these things can be answered by physics or really empirical observation of any kind alone, that we know of.

if humans in 2100 determined they wanted to maintain forever the energy utilization of humans and AIs below 2100 levels, and never let humans nor AIs leave Earth, I would be happy for advanced AIs to cause human extinction (ideally in a painless way) in order to get access to more energy to power positive conscious experiences of digital minds.

I disagree, personally. The idea that it's okay to kill some beings today to allow more to exist in the future does not seem good to me at all, for several reasons:

1. at a first order level, because I don't subscribe to total sum utilitarianism, or rather I don't think it's applicable so wildly out of domain - otherwise you could equally justify e.g. genociding the population of an area that you believe can be "better utilized" to allow a different population to grow in it and use its resources fully. I hope we can agree that is in fact a bad thing; not merely worse than somehow allowing for coexistence, but just bad. We should not in fact do this kind of thing today, so I don't think AI should do it to us in the future;
2. at a higher order, because if you allow such things as good you create terrible incentives in which basically everyone who thinks they can do better is justified in trying to kill anyone else.

So I think if your utility function returns these results as good, it's a sign your utility function is wrong; fix it. I personally think that the free choices of existing beings are supreme here; having a future is worth it insofar as present existing beings desire that there is a future. If humanity decided to go voluntarily extinct (assuming such a momentous decision could genuinely be taken by everyone in synchrony - bit of a stretch), I'd say they should be free to, without feeling bound by either the will of their now dead ancestors nor the prospect of their still hypothetical descendants. It's not that I can't imagine a situation in which in a war between humans and ASI I couldn't think, from my present perspective, that the ASI could be right, though I'd still hope such a war does not turn genocidal (and in fact any ASI that I agree with would be one that doesn't resort to genocide as long as it has the option to). But if the situation you described happened, I'd side with the humans, and I'd definitely say that we shouldn't build the kind of ASI that wouldn't either. Any ASI that can arbitrarily decide "I don't like what these humans are doing, better to kill them all and start afresh" is in fact a ticking time bomb, a paperclipper that is only happy to suffer us live as long as we also make paperclips.

good to the universe

Good in what sense? I don't really buy the moral realist perspective - I don't see where could this "real morality" possibly come from. But on top of that, I think we all agree that disempowerment, genocide, slavery, etc. are bad; we also frown upon our own disempowerment of non-human animals. So there are two options:

1. either the "real morality" is completely different and alien from the morality we humans currently tend to aspire to as an ideal (at least in this corner of the world and of the internet), or
2. a SAI that would disempower us despite being vastly more smart and capable and having no need to would be pretty evil.

If it's 1, then I'm even more curious to know what this real morality is, why should I care, or why would the SAI understand it while we seem to be drifting further away from it. And if it's 2, then obviously unleashing an evil SAI on the universe is a horrible thing to do, not just for us, but for everyone else in our future light cone. Either way, I don't see a path to "maybe we should let SAI disempower us because it's the greater good". Any sufficiently nice SAI would understand well enough we don't want to be disempowered.

We also don't plan to ask the kids for "permission" to move.

A comment on this: as a child, my parents moved twice in the space of a few years (basically, I never did more than two years in a row in the same elementary school). I never really even thought I should have been consulted, but in hindsight, those two quick moves probably contributed significantly to my attachment issues, as they both broke some nice friendships I had found myself in and pretty much taught me a sort of unconscious "don't get too attached to people" lesson. Between that and the bad luck that the last school I ended up in was not quite as nice an environment for me as the first two, this probably had a big impact on me that I still feel the effects of. So, what I mean here is, it's easy to consider the needs of children secondary to those of the big, important, money-making adults, but when it comes to things like these, a few key moments during development can have big impacts downstream.

Wouldn't lots of infections even for flu happen through e.g. faeces or such? If the birds are densely packed it might be hard to achieve much with UVC from the ceiling.

Also, about the idea of this being eventually transferable to humans, I wonder if birds have different sensitivity to it (due to being covered in feathers).

But I don't see a case for climate change risk specifically approaching anywhere near those levels, especially on timescales less than 100 years or so. 

I think the thing with climate change is that unlike those other things it's not just a vague possibility, it's a certainty. The uncertainty lies in the precise entity of the risk. At the higher end of warming it gets damn well dangerous (not to mention, it can be the trigger for other crises, e.g. imagine India suffering from killer heatwaves leading to additional friction with Pakistan, both nuclear powers). So it's a baseline of merely "a lot dead people, a lot lost wealth, a lot things to somehow fix or repair", and then the tail outcomes are potentially much much worse. They're considered unlikely but of course we may have overlooked a feedback loop or tipping point too much. I honestly don't feel as confident that climate change isn't a big risk to our civilization when it's likely to stress multiple infrastructures at once (mainly, food supply combined with a need to change our energy usage combined with a need to provide more AC and refrigeration as a matter of survival in some regions combined with sea levels rising which may eat on valuable land and cities).

I'm often tempted to have views like this. But as my friend roughly puts it, "once you apply the standard of 'good person' to people you interact with, you'd soon find yourself without any allies, friends, employers, or idols." 

I'm not saying "these people are evil and irredeemable, ignore them". But I'm saying they are being fundamentally irrational about it. "You can't reason a person out of a position they didn't reason themselves in". In other words, I don't think it's worth worrying about not mentioning climate change merely for the sake of not alienating them when the result is it will alienate many more people on other sides of the spectrum. Besides, those among those people who think like you might also go "oh well these guys are wrong about climate change but I can't hold it against them since they had to put together a compromise statement". I think as of now many minimizing attitudes towards AI risk are also irrational, but it's still a much newer topic and a more speculative one, with less evidence behind it. I think people might still be in the "figuring things out" stage for that, while for climate change, opinions are very much fossilized, and in some cases determined by things other than rational evaluation of the evidence. Basically, I think in this specific circumstance, there is no way of being neutral: either mentioning or not mentioning climate change gets read as a signal. You can only pick which side of the issue to stand on, and if you think you have a better shot with people who ground their thinking in evidence, then the side that believes climate change is real has more of those.

My impression is that on the left there is a strong current that tries to push the idea of EAs being one and the same with longtermists, and both being lost after right-wing coded worries and ignoring the real threats (climate change). Statements about climate change not being an existential threat are often misinterpreted (even if technically true, they come off as dismissal). In practice, to most people, existential (everyone dies) and civilizational (some people survive, but they're in a Mad Max post-apocalyptic state) risks are both so awful that they count as negative infinity, and warrant equal effort to be averted.

I'm going to be real, I don't trust much the rationality of anyone who right now believes that climate change is straight up fake, as some do - that is a position patently divorced from reality. Meanwhile there's plenty of people who are getting soured up on AI safety because they see it presented as something that is being used to steal attention from it. I don't know precisely how can one assess the relative impacts of these trends, but I think it would be very urgent to determine it.

I think not mentioning climate change, though technically it's correct to not consider it a likely extinction risk, was probably the biggest problem with the statement. It may have given it a tinge or vibe of right-wing political polarization, as it feels like it's almost ignoring the elephant in the room, and that puts people on the defensive. Perhaps a broader statement could have mentioned "risks to our species or our civilization such as nuclear war, climate change and pandemics", which broadens the kind of risks included. After all, some very extreme climate change scenarios could be an X-risk, much like nuclear war could "only" cause a civilization collapse, but not extinction. Plus, these risks are correlated and entangled (climate change could cause new pandemics due to shifting habitats for pathogens, AI could raise international tensions and trigger a nuclear war if put in charge of defense, and so on). An acknowledgement of that is important.

There is unfortunately some degree of "these things just sound like Real Serious Problems and that thing sounds like a sci-fi movie plot" going on here too, and I don't think you can do much about that. The point of the message should not be compromised on that - part of the goal is exactly to make people think "this might not be sci-fi any more, but your reality".

My second entry to the contest:

The tale of the Lion and the Boy

This is more of a general allegory of trying to manage a superintelligence to your advantage. In keeping with the original literary tradition of fables, I've used sentient animals for this one.