Even my grocery shopping list? 😳 That's a bit embarrassing but I hope fellow EAs can help me optimize it for impact

Climate change is going pretty well, I've heard carbon emissions are up!

Also, humans are carbon-based creatures so having more carbon around seems plausibly good 😊

Are we using the old 12 signs astrological chart, or the updated one with Ophiuchus 13th astrological sign?

Fair! I agree to that, at least until this point of time.

But I think there could be a time where we could have picked most of the "social low-hanging fruit" (cases like the abolition of slavery, universal suffrage, universal education), so there's not a lot for easy social progress left to do. At least comparatively, then investing on the "moral philosophy low-hanging fruit" will look more worthwhile.

Some important cases of philosophical moral problems that might have great axiological moral importance, at least under consequentialism/utilitarianism could be population ethics (totalism vs averagism), our duties towards wild animals, and the moral status of digital beings.

I think figuring them out could have great importance. Of course, if we always just keep them as just an interesting philosophical thought experiment and we don't do anything about promoting any outcomes, they might not matter that much. But I'm guessing people in the year 2100 might want to start implementing some of those ideas.

Same! Seems like a fascinating, although complicated topic. You might enjoy Oded Galor's "The Journey of Humanity", if you haven't read it. :)

Sure! So I think most of our conceptual philosophical moral progress until now has been quite poor. If looked under the lens of moral consistency reasoning I outlined in point (3), cosmopolitanism, feminism, human rights, animal rights, and even longtermism all seem like slight variations on the same argument ("There are no morally relevant differences between Amy and Bob, so we should treat them equally").

In contrast, I think the fact that we are starting to develop cases like population ethics, infinite ethics, complicated variations of thought experiments (there are infinite variations of the trolley problem we could conjure up), that really test our limits of our moral sense and moral intuitions, hints at the fact that we might need a more systematic, perhaps computerized approach to moral philosophy. I think the likely path is that most conceptual moral progress in the future (in the sense of figuring out new theories and thought experiments) will happen with the assistance of AI systems.

I can't point to anything very concrete, since I can't predict the future of moral philosophy in any concrete way, but I think philosophical ethics might become very conceptually advanced and depart heavily from common-sense morality. I think this has been an increasing gap since the enlightenment. Challenges to common-sense morality have been slowly increasing. We might be at the early beginning of that exponential takeoff.

Of course, many of the moral systems that AIs will develop we will consider to be ridiculous. And some might be! But in other cases, we might be too backwards or morally tied to our biologically and culturally shaped moral intuitions and taboos to realize that it is in fact an advancement. For example, the Repugnant Conclusion in population ethics might be true (or the optimal decision in some sense, if you're a moral anti-realist), even if it goes against many of our moral intuitions. 

The effort will take place in separating the wheat from the chaff. And I'm not sure if it will be AI or actual moral philosophers doing this effort of discriminating good from bad ethical systems and concepts.

Outside of Marxism and continental philosophy (particularly the Frankfurt School and some Foucault), I think this idea has lost a lot of grip! So it has actually become a minority view or even awareness among current academic philosophers, particularly in the anglosphere.

However, I think it's a very useful idea that should make us look at our social arrangements (institutions, beliefs, morality...) with some level of initial suspicion. Luckily, some similar arguments (often called "debunking arguments" or "genealogical arguments") are starting to gain traction within philosophy again. 

I hadn't! Thanks for bringing this to my attention, I will take a look in the coming months.

Good! I think I mostly agree with this and I should probably flag it somewhere in the main post. 

I do agree with you, and I think it also shows what is a central point of the later parts of my thesis, when I will talk about the empirical ideas rather than philosophical ideas: that technologies (from shipbuilding, to the industrial revolution, to factory farming, to future AI) are more of a factor in moral progress or regress than ideologies. So many moral philosophers might have the wrong focus. 

(Although many of those things I would call "social" progress rather than "moral" strictly speaking, because it was triggered by external factors (economic and technological change) rather than moral reflection. It's not that we became more cruel to animals in terms of our intentions, it's that we gained more power over them.)

I agree with you this is very important, and I'd like to see more work on it. Sadly I don't have much concrete to say on this topic. The following is my opinion as a layman on AI:

I've found Toby Ord's framework here https://www.youtube.com/watch?v=jb7BoXYTWYI to be useful for thinking about these issues. I guess I'm an advocate for differential progress, like Ord. That is, prioritizing safety advancements relative to technical advancements. Not stopping work on AI capabilities, but right now shifting the current balance from capabilities work to safety work. And then in some years/decades once we have figured out alignment, shift the focus on capabilities again.

My very rough take on things is that as long as we manage to develop advanced LLMs (e.g. GPT5, 6, 7... and Copilots) slowly and carefully before dangerous AGI, we should use those LLMs to help us with technical alignment work. I think technical alignment work is the current bottleneck of the whole situation. There are either not enough people or we're not smart enough to figure it out on our own (but maybe computers could help!).

So, to your points, I think right now (1) Runaway AI Risk is higher than (2) Malicious actors catching up. I don't know by how much, since I don't know how well Chinese labs are doing regarding AI, and if they could come to big breakthroughs on their own. (And I don't know how to compare those points (1) and (2) to (3) and (4).)

Perhaps somebody could do a BOTEC calculation or a rough model with some very rough numbers to see what's a good tradeoff, and put it up for discussion. I'd like to see some work on this.