One reason is that longtermists are largely philosophers, who have no particular expertise on the details of aligning AI.

Another reason worth taking into consideration is if the true moral view is "fussy", rather than "easygoing". If you're "easygoing" in what you consider utopia, then, conditional on survival, most achievable value gets realized by default (we get great human lives), and extinction is the one really action-relevant lock-in event. But if you're fussy about realizing the best possible utopia, then, conditional on survival, we're still likely to miss most achievable value across a huge swathe of futures (we don't tile the universe with happy digital minds, say, or whatever crazy future might be the best utopia). The space of "didn't go extinct but missed most of the value" turns out to be enormous, and some of the features determining where in that space we land (early decisions about digital minds, population-ethics, allocation of resources during space settlement, which value-systems get amplified during the AI transition) are themselves plausibly locked-in, even if they don't feel as salient as extinction.

(But then again, maybe I'm recency biased because I just re-read Better Futures for the discussion week here on the Forum)

That doesn't settle the prioritization, and, like, the people are Forethought and 80.000 Hours are directly and explicitly working on the AI transition? So it's not like x-risk is off the table. My vibe for highly-engaged EAs is that perhaps it just feels that the main arguments about x-risk have already been made.

That's really great! You posed my question better than I did, and answered it more detail than I was expecting. Thanks a lot. :)

RE: "my hope is that aiming for truly great futures, and just trying to improve incrementally on 'default' futures, recommend quite similar and compatible courses."

For what it's worth, it might be worth your time to check Gerald Gauss' "The Tyranny of the Ideal", particularly his framework around page 82, with what he calls "The Choice": that sometimes we might face scenarios in which the local optimum is in the opposite direction from where the global optimum is. I discovered it recently and I thought it was pretty good for thinking about these kinds of issues.

Hi Fin, sorry I'm a bit late with my question, I was rereading parts of the Better Futures series. First of all, I have to say it's one of my favorite article series I've ever read, and I'll be citing it in my own work going forward. The easygoing-versus-fussy distinction in particular is something I'm finding really interesting to dig into. :) Would love to discuss it in more detail at some point.

I wanted to push on the metaphor of sailing to an island, which appears at the start of No Easy Eutopia, but my question is going to take some preamble explanation (sorry!).

I find myself preferring a slightly different picture. Rather than thinking of eutopia as an island we're navigating to, I tend to think of society as the ship itself, drifting through a sea of value over time (a topography of better and worse regions we're already moving through). Societal change feels to me more like a search through uncharted moral territories than an expedition to a specific destination. On that picture, the priority seems more likely to be "how do we improve the ship, so that society reliably moves toward better regions of the sea?"

A couple of clarifications. First, I grant fussiness, I agree most plausible axiologies locate near-best futures in a very narrow region (I lean towards total hedonistic utilitarianism, myself). Second, I'm not a a quietist, in my own work I'm defending what I call moral niche construction, a fairly interventionist view on which we should actively reshape institutions, technologies, and even our own moral psychology (through things like AI moral decisionmakers or bioenhancement) to push society toward better regions. So the disagreement isn't really about ambition, either.

Where I want to press is the following. In the ship-improvement picture, I can grant openly that we probably will never reach eutopia. We end up in a high-value region of the sea (in a local optima), much better than where we are now, plausibly very good in absolute terms, but not the narrow island. 

That sounds like a concession, but on rereading Convergence and Compromise, it looks to me like the target-pursuit picture probably doesn't reach the island either: you mention how WAM-convergence is unlikely, partial convergence plus trade faces serious obstacles, value-destroying threats can eat most of the value... So the comparison isn't "guaranteed eutopia versus probably-not-eutopia", since you yourself seem pretty pessimistic. So it's two orientations that both probably miss the island, where one delivers reliable improvements to our current region of the sea along the way, and the other keeps optimizing toward a target it probably won't hit. And, well, if you miss the moon, you don't really land upon the stars... you drift in empty space and die, haha.

(There are similar points on Jerry Gaus' The Tyranny of the Ideal, and on recent debates between ideal theory and non-ideal theory in moral and political philosophy)

So, finally, my question is: given that target-pursuit probably doesn't reach eutopia either, on the series' own analysis, why is the practical orientation toward the narrow target rather than toward improving our current region of the sea (e.g. pursuing very high + plausibly easy to reach and resilient local optima)? What's the case for target-pursuit as a practical orientation, once we factor in that we will probably fail? Is it a case akin to fanaticism, where, if we land in the island, the payoff would be huge?

(Apologies in advance if this is addressed somewhere in the series, my memory context window isn't large enough to hold the whole essay series at once!)

Ow my eyes

For what it's worth, here's some bibliography in case anyone is interested in researching (moral) intuitions in philosophy.

An excerpt from my MA thesis:

"There are several possible characterizations of what intuitions are precisely supposed to be. Exceptionalists (e.g. Sosa, Ludwig) argue that intuitions are analytic or conceptual truths, a priori, and/or dealing with conceptual competence. Particularists (e.g. Bealer, Huemer, Schwitzgebel, Kagan) argue that intuitions have a distinct phenomenology, such as being snap judgments that are not consciously inferred from any other belief, or are a sui generis faculty. Minimalists (e.g. Machery, Lewis) argue that intuitions are not different from the application of concepts in ordinary life. (Machery, 2017, Ch. 2)"

I borrowed this terminology from Chapter 2 of Edouard Machery's book, Philosophy Within Its Proper Bounds (2017).

How much confidence would you give to the statement: "Whiteleg shrimps are sentient"?

Justification for my vote: High credence given that I also believe many insects of similar size are sentient too. But my credence is highly volatile to further evidence, given that we lack a lot of empirical information and we don't know much the correct philosophy of mind either.

See the paper "Can insects feel pain? A review of the neural and behavioural evidence" by Gibbons et al. here for the Birch framework applied to insects. (Feel free to skip to the conclusion if you just want the quick takeaway).

Thank you for recording the talks! I couldn't attend but will be watching them

Morality is Objective

(Vote Explanation) Morality is objective in the sense that, under strong conditions of ideal deliberation (where everyone affected is exposed to all relevant non-moral facts and can freely exchange reasons and arguments) we would often converge on the same basic moral conclusions. This kind of agreement under ideal conditions gives morality its objectivity, without needing to appeal to abstract and mind-independent moral facts. This constructivist position avoids the metaphysical and epistemological problems of robust moral realism, while still grounding moral claims in terms of justification.

(Although their views are not exactly the same, I take this view to be aligned with the metaethical views of philosophers Christine Korsgaard, Sharon Street, Philip Kitcher, and Jürgen Habermas. https://plato.stanford.edu/entries/constructivism-metaethics/ )

RE: "I am curious, why do you care about Big Things without small things? Are Big Things not underpinned by values of small everyday things?"

Perhaps it has to do with the level of ambition. Let's talk about a particular value to narrow down the discussion. Some people see "caring for all sentient beings" as an extension of empathy. Some others see it as a logical extension of a principle of impartiality or equality for all. I think I am more in this second camp. I don't care about invertebrate welfare, for example, because I am particularly empathetic towards them. Most people find bugs to be a bit icky, particularly under a magnifying glass, which turns off their empathy.

Rather, they are suffering sentient beings, which means that the same arguments for why we should care about people (and their wellbeing/interests/preferences) also apply to these invertebrates. And caring about, say, invertebrate welfare, requires a use of reason towards impartiality that might sometimes make you de-prioritize friends and family.

Secondly, I also have a big curiosity about understanding the universe, society, etc. which makes me feel like I'm wasting my time in social situations of friends and family when the conversation topics are a bit trivial.

As I repeat a bit throughout the post, I realize I might be a bit of an psychological outlier here, but I hope people can also see why this perspective might be appealing. Most people are compartimenalizing their views on AI existential risk to a level that I'm not sure makes sense.

To answer the two questions: For me as a philosopher, I think this is where I can have greatest impact, compared to writing technical stuff on very niche subjects, which might probably not matter much. Think how the majority of the impact that Peter Singer, Will MacAskill, Toby Ord, Richard Chappell, or Bentham's Bulldog have been a mix of new ideas and public advocacy for them. I could say similar thing about other types of intellectuals like Eliezer Yudkowsky, Nick Bostrom, or Anders Sandberg.

I think polymathy is also where the comparative advantage often lies for a philosopher. Particularly for me, I'm not so good at technical topics that I would greatly excel at a niche thing such as population ethics. I can, however, draw from other fields and learn how particular moral intuitions might be unreliable, for example. And what might feel like a advocating for a relatively small change in moral beliefs (e.g. what we do about insect suffering, or the potential suffering of digital minds) could change future societies greatly.

Yet I don't disregard specializing into one thing. I'm currently working on my PhD, which a very specialized project.

And I would give very different advice if I was working on AI safety directly. If that were the case, maybe digging deep into a topic to become a world expert or have a breakthrough might be the best way to go.