FWIW my impression of the EA community's position is that we need to build safe AI, not that we need to stop AI development altogether (although some may hold this view).

Stopping AI development altogether misses out on all the benefits from AI, which could genuinely be extensive and could include helping us with other very pressing problems (global health, animal welfare etc.).

I do think one can do a tremendous amount of good at OpenAI, and a tremendous amount of harm. I am in favor of roles at AI companies being on the 80,000 Hours job board so that the former is more likely.

My view is these roles are going to filled regardless. Wouldn't you want someone who is safety-conscious in them?

It’s only intuitive to me not to eat cars because it isn’t good for wellbeing!

In a world in which cars are tasty and healthy to eat I imagine we wouldn’t find it so irrational to eat them. Unless of course you’d be losing a method of transportation by eating it and can get other options that are just as healthy and tasty for cheaper — in which case we’re just resorting to wellbeing arguments again.

This just seems to be question-begging. It just seems to me you're saying "axiological realism gives rise to normative realism because surely axiological realism gives rise to normative realism".

But this means that moral anti-realists must think that you can never have a reason to care about something independent of what you actually do care about. This is crazy as shown by the following cases:

1. A person wants to eat a car. They know they’d get no enjoyment from it—the whole experience would be quite painful and unpleasant. On moral anti-realism, they’re not being irrational. They have no reason to take a different action.

I think the person wanting to eat a car is irrational because they will not be promoting their wellbeing by doing so and their wellbeing is what they actually care about.

So the reason not to eat the car isn't stance-independent—it's based on their own underlying values.

What is the reason not to eat the car that isn’t grounded in concern for wellbeing or some other value the person already holds?

Thanks for highlighting the relative lack of attention paid to cause prioritization and cross-cause prioritization. 

I have also written about how important it is to enable EAs to become familiar with existing cause prioritization findings. It's not just about how much research is done but also that EAs can take it into account and act on it.

You’re basically saying happier machines will be more productive and so we are likely to make them to be happy?

Firstly we don’t necessarily understand consciousness enough to know if we are making them happy, or even if they are conscious.

Also, I’m not so sure if happier means more productive. More computing power, better algorithms and more data will mean more productive. I’m open to hearing arguments why this would also mean the machine is more likely to be happy. 

Maybe the causality goes the other way - more productive means more happy. If machines achieve their goals they get more satisfaction. Then maybe happiness just depends on how easy the goals we give it is. If we set AI on an intractable problem and it never fulfills it maybe it will suffer. But if AIs are constantly achieving things they will be happy. 

I’m not saying you’re wrong just that it seems there’s a lot we still don’t know and the link between optimization and happiness isn’t straightforward to me.

Well the closest analogue we have today is factory farmed animals. We use them in a way that causes tremendous suffering. We don't really mean to cause the suffering, but it's a by product of how we use them.

And another, perhaps even better, analogue is slavery. Maybe we'll end up essentially enslaving digital minds because it's useful to do so - if we were to give them too much freedom they wouldn't as effectively do what we want them to do.

Creating digital minds just so that they can live good lives is a possibility, but I'd imagine if you would ask someone on the street if we should do this, they'd look at you like you were crazy.

Again, I'm not sure how things will pan out, and I would welcome strong arguments that suffering is unlikely, but it's something that does worry me.

Do you agree that the experience of digital minds likely dominates far future calculations?

This leads me to want to prioritize making sure that if we do create digital minds, we do so well. This could entail raising the moral status of digital minds, improving our ability to understand sentience and consciousness, and making sure AI goes well and can help us with these things.

Extinction risk becomes lower importance to me. If we go extinct we get 0 value from digital minds which seems bad, but it also means we avoid the futures where we create them and they suffer. It’s hard to say if we are on track to creating them to flourish or suffer - I think there are arguments on both sides. The futures where we create digital minds may be the ones where we wanted to “use” them, which could mean them suffering. Alternatively, we have seen our moral circle expand over time and this may continue, so there is a real possibility we could create them to flourish. I don’t have a clear view which side wins here, so overall going extinct doesn’t seem obviously terrible to me.

This is a question I could easily change my mind on.

The experience of digital minds seems to dominate far future calculations. We can get a lot of value from this, a lot of disvalue, or anything in between.

If we go extinct then we get 0 value from digital minds. This seems bad, but we also avoid the futures where we create them and they suffer. It’s hard to say if we are on track to creating them to flourish or suffer - I think there are arguments on both sides. The futures where we create digital minds may be the ones where we wanted to “use” them, which could mean them suffering. Alternatively, we have seen our moral circle expand over time and this may continue, so there is a real possibility we could create them to flourish. I don’t have a clear view which side wins here, so overall going extinct doesn’t seem obviously terrible to me.

We could instead focus on raising the moral status of digital minds, our ability to understand sentience and consciousness, improve societal values, making sure AI goes well and helps us with these things. These robustly increase the expected value of digital sentience in futures where we survive. 

So because reducing extinction risk is close to 0 expected value to me, and increasing the value of futures where we survive is robustly positive in expected value, I lean towards increasing the value of futures where we survive.