All of Jack Malde's Comments + Replies

That is fair. I still think the idea that aligned superintelligent AI in the wrong hands can be very bad may be under-appreciated. The implication is that something like moral circle expansion seems very important at the moment to help mitigate these risks. And of course work to ensure that countries with better values win the race to powerful AI.

Well I'm assigning extinction a value of zero and a neutral world is any world that has some individuals but also has a value of zero. For example it could be a world where half of the people live bad (negative) lives and the other half live equivalently good (positive) lives. So the sum total of wellbeing adds up to zero. 

A dystopia is one which is significantly negative overall. For example a world in which there are trillions of factory farmed animals that live very bad lives. A world with no individuals is a world without all this suffering.

Could it be more important to improve human values than to make sure AI is aligned?

Consider the following (which is almost definitely oversimplified):

For clarity, let’s assume dystopia is worse than extinction. This could be a scenario where factory farming expands to an incredibly large scale with the aid of AI, or a bad AI-powered regime takes over the world. Let's assume neutral world is equivalent to extinc... (read more)

• We can think of this stance as analogous to: 
  • The utilitarian parent: “I care primarily about doing what’s best for humanity at large, but I wouldn’t want to neglect my children to such a strong degree that all defensible notions of how to be a decent parent state that I fucked up.”

I wonder if we don't mind people privileging their own children because:

1. People love their kids too damn much and it just doesn't seem realistic for people to neglect their children to help others.
2. A world in which it is normalised to neglect your children to "focus on humanit

Looking forward to reading this. A quick note: in 3. Tomi’s argument that creating happy people is good your introductory text doesn't match what is in the table.

Not sure I entirely agree with the second paragraph. The white paper outlines how philanthropy in this area is quite neglected, and there are organisations like LaMP which could certainly use more funding. Page 5 of the white paper also outlines bottlenecks in the process - even if firms do have strong incentives to acquire talent there can be informational gaps that prevent them from finding the best individuals, and similar informational gaps exist for the individuals that prevent them from actively utilising the best pathways. 

Having said that I'm not claiming this is the best use of EA dollars - just posting for people's information.

Yeah I have a feeling that the best way to argue for this on EA grounds might surprisingly be on the basis of richer world economic growth, which is kind of antithetical with EA's origins, but has been argued to be of overwhelming importance e.g.:

• Economist Tyler Cowen says our overwhelming priorities should be maximising economic growth and making civilisation more stable. Is he right?
• The Moral Consequences of Economic Growth

Thank you, good to flag these points. 

Regarding the AI Safety point, I want to think through this more, but I note that the alignment approach of OpenAI is very capabilities-driven, requiring talent and compute to align AI using AI. I think one's belief of the sign of immigration on x-risk here might depend on how much you think top labs like OpenAI actually take the safety risks seriously. If they do, more immigration can help them make safe AI.

Regarding the meat-eater problem, I think the possibility of an animal Kuznets curve is relevant. If such a... (read more)

Thank you!

Will this be recorded?

Please do! I'm fascinated by the idea that we can accelerate moral progress by focusing on economic growth.

Hey, thanks for this list! I was wondering if you have come across Benjamin Friedman's Moral Consequences of Economic Growth and, if so, what you thought. I thought I might see it on this list.

Matthew is right that uncertainty over the future is the main justification for discount rates

I don't think this is true if we're talking about Ramsey discounting. Discounting for public policy: A survey and Ramsey and Intergenerational Welfare Economics don't seem to indicate this. 

Also, am I missing something, or would a zero discount rate make this analysis impossible?

I don't think anyone is suggesting a zero discount rate? Worth noting though that that former paper I linked to discusses a generally accepted argument that the discount rate should f... (read more)

Do economists actually use discount rates to account for uncertainty? My understanding was that we are discounting expected utilities, so uncertainty should be accounted for in those expected utilities themselves.

Maybe it’s easier to account for uncertainty via an increasing discount rate, but an exponential discount rate seems inappropriate. For starters I would think our degree of uncertainty would moderate over time (e.g. we may be a lot more uncertain about effects ten years from now than today, but I doubt we are much more uncertain about effects 1,000,010 years from now compared to 1,000,000 or even 500,000 years from now).

Nice research!

In my preferred model, investing in science has a social impact of 220x, as measured in Open Philanthropy’s framework.

How does this compare to other top causes that Open Phil funds? Is there a summary somewhere?

You might consider testing your ideas a few times to see if they would be effective before you suggest them.

Thank you Arvo, I really appreciate it! I look forward to seeing more work from you and the team.

Well you might need a reasoned response I.e. it seems that when I do X a bad thing happens to me therefore I should endeavor not to do X.

Here is the quote from Richard Dawkins:

“If you think about what pain is for biologically speaking, pain is a warning to the animal, ‘don’t do that again’.

“If the animal does something which results in pain, that is a kind of ritual death – it is telling the animal, ‘if you do that again you might die and fail to reproduce’. That’s why natural selection has built the capacity to feel pain into our nervous systems.

“You coul... (read more)

In fact, it has been suggested by Richard Dawkins that less intelligent animals might experience greater suffering, as they require more intense pain to elicit a response. The evolutionary process would have ensured they feel sufficient pain.

I’m a bit confused by this. Presumably GiveWell doesn’t need that much money to function. Less Open Phil money probably won’t affect GiveWell, instead it will affect GiveWell’s recommended charities which will of course still receive money from other sources, in part due to GiveWell’s recommendation.

Strong upvoted. I think this is correct, important and well-argued, and I welcome the call to OP to clarify their views. 

This post is directed at OP, but this conclusion should be noted by the EA community as a whole which still prioritises global poverty over all else.

The only caveat I would raise is that we need to retain some focus on global poverty in EA for various instrumental reasons: it can attract more people into the movement, allows us to show concrete wins etc.  

Nice paper!

One interesting result of the paper is that neglectedness seems is key to whether a policy change matters for a long time. For policies that can be expected to attract more interest after the referendum passes, I see less persistence. It is not a hugely dramatic effect, but it could make a difference on the margin or in extreme cases. This seems to lend some support to the EA practice of paying attention to neglectedness.

This is probably me being stupid, but I'm not sure I understand this. Are you saying more neglected areas are those that would... (read more)

I'm sorry, this doesn't engage with the main point(s) you are trying to make, but I'm not sure why you use the term "existential risk" (which you define as risks of human extinction and undesirable lock-ins that don’t involve s-risk-level suffering) when you could have just used the term "extinction risk".

You say:

If you’re uncertain whether humanity’s future will be net positive, and therefore whether existential risk^[1] reduction is good, you might reason that we should keep civilization going for now so we can learn more and, in the future, make a b

Previous work has referred to such a risk as 'existential risk'. But this is a misnomer. Existential risk is technically broader and it encompasses another case: the risk of an event that drastically and permanently curtails the potential of humanity. For the rest of this report we characterise the risk as that of extinction where previous work has used 'existential'. 

I was happy to see this endnote, but then I noticed several uses of "existential risk" in this abridged report when I think you should have said "extinction risk". I'd recommend going through to check this.

In theory people will always prefer cash because they can spend it on whatever they want (unless of course it is difficult to buy what they most want). This isn’t really up for debate.

What is up for debate is if people actually spend money in a way that most effectively improves their welfare. It sounds paternalistic to say, but I suspect they don’t for the reasons Nick and others have given.

Yeah I feel that sometimes theories get really convoluted and ad hoc in an attempt to avoid unpalatable conclusions. This seems to be one of those times.

I can give Scanlon a free pass when he says under his theory we should save two people from certain death rather than one person from certain death because the 'additional' person would have some sort of complaint. However when the authors of this post say, for a similar reason, that the theory implies it's better to do an intervention that will save two people with probability 90% rather than one person w... (read more)

I do worry about future animal suffering. It's partly for that reason that I'm less concerned about reducing risks of extinction than I am about reducing other existential risks that will result in large amounts of suffering in the future. This informed some of my choices of interventions for which I am 'not clueless about'. E.g. 

• Technical AI alignment / AI governance and coordination research: it has been suggested that misaligned AI could be a significant s-risk.
• Expanding our moral circle: relevance to future suffering should be obvious.
• Global prior

A relevant GPI paper is Longtermism, aggregation, and catastrophic risk by Emma J. Curran.

I briefly summarised it here, also pasted below:

The bottom line: If one is sceptical about aggregative views, where one can be driven by sufficiently many small harms outweighing a smaller number of large harms, one should also be sceptical about longtermism.

My brief summary:

• Longtermists generally prefer reducing catastrophic risk to saving lives of people today. This is because, even though you would be reducing probability of harm by a small amount if focusing on ca

Contractualism doesn't allow aggregation across individuals. If each person has 0.3% chance of averting death with a net, then any one of those individual's claims is still less strong than the claim of the person who will die with probability ~=1. Scanlon's theory then says save the one person.

I don't buy the argument that AI safety is in some way responsible for dangerous AI capabilities. Even if the concept of AI safety had never been raised I'm pretty sure we would still have had AI orgs pop up.

Also yes it is possible that working on AI Safety could limit AI and be a catastrophe in terms of lost welfare, but I still think AI safety work is net positive in expectation given the Bostrom astronomical waste argument and genuine concerns about AI risk from experts. 

The key point here is that cluelessness doesn't arise just because we can thin... (read more)

Thanks for your response, I'm excited to see your sequence. I understand you can't cover everything of interest, but maybe my comments give ideas as to where you could do some further work.

I'll need to find some time to read that Holden post.

I'm happy that the flowchart was useful to you! I might consider working on it in the future, but I think the issues are that I'm not convinced many people would use it and that the actual content of the flowchart might be pretty contentious - so it would be easy to be accused of being biased. I was using my karma score as a signal of if I should continue with it, and karma wasn't impressive.

I'll direct you to my response to Arepo

I think you might be right that Greaves is too strong on this and I'll admit I'm still quite uncertain about exactly how cluelessness cashes out. However, I know I have difficulties funding GHD work (and would even if there was nothing else to fund), but that I don't have similar difficulties for certain longtermist interventions. I'll try to explain.

I don't want to fund GHD work because it's just very plausible that the animal suffering might outweigh the human benefit. Some have called for development economists to consider the welfare of non-human anima... (read more)

I’m very excited to read this sequence! I have a few thoughts. Not sure how valid or insightful they are but thought I’d put them out there:

On going beyond EVM / risk neutrality

• The motivation for investigating alternatives to EVM seems to be that EVM has some counterintuitive implications. I'm interested in the meta question of how much we should be swayed by counterintuitive conclusions when EVM seems to be so well-motivated (e.g. VNM theorem), and the fact that we know we are prone to biases and cognitive difficulties with large numbers.
• Would alternative

I might have gone for "Utilitarianism may be irrational or self-undermining" rather than "Utilitarianism is irrational or self-undermining".

Personally, I would have kept the original title. Titles that are both accurate and clickbaity are the best kind - they get engagement without being deceptive. 

I don't think karma is always a great marker of a post's quality or appropriateness. See an earlier exchange we had.

Ah ok I actually used the word “attack”. I probably shouldn’t have, I feel no animosity at all towards Michael. I love debating these topics and engaging with arguments. I wish he’d had more room to expand on his person-affecting leanings. In a sense he is “attacking” longtermism but in a way that I welcome and enjoy responding to.

I happen to think the level of attention Will gave to population ethics and the concepts of the non-identity problem, repugnant conclusion, and person-affecting intuition is fairly admirable for a book intended for a general non-... (read more)

I'm not sure I have framed the review as an attack? I don't think it is. I have no problem with Michael writing the review, I just disagree with the points he made.

It was a while since I read the book in its entirety, but I will just leave a quote from the introduction which to me doesn't read as "disquietingly polemical" (bold emphasis mine):

For those who want to dig deeper into some of my claims, I have compiled extensive supplementary materials, including special reports I commissioned as background research, and made them available at whatweowethefutur

OK. I think Will intended WWOTF to be a persuasive piece so I’m not sure if this is a valid criticism. He wasn’t writing a textbook.

Hey, thanks for you comment! To be honest my addendum is a bit speculative and I haven't thought about a huge amount. I think I may have been a little extreme and that factoring moral uncertainty would soften some of what I said.

Would 100 percent of philosophies working on the question of the far future really be the best way to improve the field, with other important philosophical professions neglected?

When you say "improve the field" I'm not sure what you mean. Personally I don't think there is intrinsic value in philosophical progress, only instrumental... (read more)

That said, off the top of my head, philosophers who have written sympathetically about person-affecting views include Bader, Narveson (two classic articles here and here), Roberts (especially here, but she's written on it a few times), Frick (here and in his thesis), Heyd, Boonin, Temkin (here and probably elsewhere). There are not 'many' philosophers in the world, and population ethics is a small field, so this is a non-trivial number of authors! For an overview of the non-identity problem in particular, see the SEP.

I agree we should be more swayed by arg... (read more)

Fourth, I'm not sure why you think I've misrepresented MacAskill (do you mean 'misunderstood'?). In the part you quote, I am (I think?) making my own assessment, not stating MacAskill's view at all.

You say the following in the summary of the book section (bold part added by me):

If correct, this [the intuition of neutrality] would present a severe challenge to longtermism

By including it in the 'summary' section I think you implicitly present this as a view Will espoused in the book - and I don't agree that he did.

But the cause du jour of longtermism is prev

I lean towards thinking the following is unfair.

Third, the thrust of my article is that MacAskill makes a disquietingly polemical, one-sided case for longtermism.

If one were just to read WWOTF they would come away with an understanding of:

• The intuition of neutrality - what it is, the fact that some people hold it, the fact that if you accept it you shouldn't care about losing future generations.
• The non-identity problem - what it is and why some see it as an argument against being able to improve the future.
• The repugnant conclusion - what it is, how some fi

Thanks for this reply Michael! I'll do a few replies and understand that you don't want to get in a long back and forth so will understand if you don't reply further.

Firstly, the following is all very useful background so I appreciate these clarifications:

First, the piece you're referring to is a book review in an academic philosophy journal. I'm writing primarily for other philosophers who I can expect to have lots of background knowledge (which means I don't need to provide it myself).

Second, book reviews are, by design, very short. You're even discourag

Fair point, but I would still disagree his analysis implies that human extinction would be good. He discusses digital sentience and how, on our current trajectory, we may develop digital sentience with negative welfare. An implication isn't necessarily that we should go extinct, but perhaps instead that we should try to alter this trajectory so that we instead create digital sentience that flourishes. 

So it's far too simple to say that his analysis "concludes that human extinction would be a very good thing". It is also inaccurate because, quite literally, he doesn't conclude that. 

So I agree with your choice to remove that wording.

If I were you I would remove that part altogether. As Kyle has already said his analysis might imply that human extinction is highly undesirable.

For example, if animal welfare is significantly net negative now then human extinction removes our ability to help these animals, and they may just suffer for the rest of time (assuming whatever killed us off didn’t also kill off all other sentient life).

Just because total welfare may be net negative now and may have been decreasing over time doesn’t mean that this will always be the case. Maybe we can do something about it and have a flourishing future.

I agree that alignment research would suffer during a pause, but I've been wondering recently how much of an issue that is. The key point is that capabilities research would also be paused, so it's not like AI capabilities would be racing ahead of our knowledge on how to control ever more powerful systems. You'd simply be delaying both capabilities and alignment progress.

You might then ask - what's the point of a pause if alignment research stops? Isn't the whole point of a pause to figure out alignment?

I'm not sure that's the whole point of a pause. A pau... (read more)

Fair enough. A more precise question could be, would it be beneficial to slow progress from the current trend?

Or another question could be, would it be desirable or undesirable to give more compute and talent to top AI labs?