Thanks for the post, this is SO CRUCIAL!

Sounds like a great idea to me!

Do we need a yearly strategy paper of the EA movement?

I think Ethical co-evolution of humanity and AI is a very interesting concept, that on the one hand points out to humans staying in control and not handing over power and knowldge of what is "right" to AI, but on the other hand willing to learn and develop, understanding we are shortsighted when it comes to morality, and also may have a lot to learn from AI!

@Beyond Singularity Another issue is how you calculate positive vs negative valence. David Pearce thinks ethics is only about reducing negative experiences, and although it is good to make beings have more positive ones, it is not within ethics. So his view is, I think, the more extreme side of negative utilitarianism. 

I think positive experiences are within the ethical realm, and although reducing suffering is more important than increasing happiness, I would still try calculate how much beings are happy as well and how to optimize that. 

I don't think this large-scale cooperation or society or groups function is morality. It is linked to morality but it is fundamentally something else. A society can "function" well with having part of it suffer tremendously for the benefit of another group. There is nothing objective with longing for a world with less suffering, it is basically in another realm, not in the one of math or rational, though it is tied to rationality in some way

I agree when thinking about the long future we have to have better more robust moral frameworks and the time and moral circle expansion are very crucial factors that have to be taken into consideration

I tend to think the word objective doesn't fit morality from a philosophical standpoint. "objective" truths are arguments that we can decide upon who is right by checking predictions, each of them is evidence for the validity of a claim. If I say earth is round, we can check this claim by talking to experts or flying to space and looking at earth, etc. All of which are prediction of subjective experiences we will have.

So "objective" argument means I  am guessing something about the future world, that it will look one way and not another. Morality is in a totally different domian. I would like the future world to be with less suffering and not more, but this is a longing, not a prediction that can be refuted. So with morality there is some sense of logic and being systematic, but at the core it is not an objective question becasue it can't be decided upon with predictions

Very interesting idea that could improve the worldwide interconnectedness of the movement. I like it very much. 

Thanks. I think as time horizon gets bigger, uncertainty may also get bigger, and this should be calculated into the expected value formula.