On talking about this publicly

A number of people have asked why there hasn’t been more communication around FTX. I’ll explain my own case here; I’m not speaking for others. The upshot is that, honestly, I still feel pretty clueless about what would have been the right decisions, in terms of communications, from both me and from others, including EV, over the course of the last year and a half. I do, strongly, feel like I misjudged how long everything would take, and I really wish I’d gotten myself into the mode of “this will all take years.” 

Shortly a... (read more)

Elon Musk

Stuart Buck asks:

“[W]hy was MacAskill trying to ingratiate himself with Elon Musk so that SBF could put several billion dollars (not even his in the first place) towards buying Twitter? Contributing towards Musk's purchase of Twitter was the best EA use of several billion dollars? That was going to save more lives than any other philanthropic opportunity? Based on what analysis?”

Sam was interested in investing in Twitter because he thought it would be a good investment; it would be a way of making more money for him to give away, rather than a way... (read more)

How I publicly talked about Sam 

Some people have asked questions about how I publicly talked about Sam, on podcasts and elsewhere. Here is a list of all the occasions I could find where I publicly talked about him.  Though I had my issues with him, especially his overconfidence, overall I was excited by him. I thought he was set to do a tremendous amount of good for the world, and at the time I felt happy to convey that thought. Of course, knowing what I know now, I hate how badly I misjudged him, and hate that I at all helped improve his re... (read more)

What I heard from former Alameda people 

A number of people have asked about what I heard and thought about the split at early Alameda. I talk about this on the Spencer podcast, but here’s a summary. I’ll emphasise that this is me speaking about my own experience; I’m not speaking for others.

In early 2018 there was a management dispute at Alameda Research. The company had started to lose money, and a number of people were unhappy with how Sam was running the company. They told Sam they wanted to buy him out and that they’d leave if he didn’t accept the... (read more)

Lessons and updates

The scale of the harm from the fraud committed by Sam Bankman-Fried and the others at FTX and Alameda is difficult to comprehend. Over a million people lost money; dozens of projects’ plans were thrown into disarray because they could not use funding they had received or were promised; the reputational damage to EA has made the good that thousands of honest, morally motivated people are trying to do that much harder. On any reasonable understanding of what happened, what they did was deplorable. I’m horrified by the fact that I was Sam’s... (read more)

Hi Yarrow (and others on this thread) - this topic comes up on the Clearer Thinking podcast, which comes out tomorrow. As Emma Richter mentions, the Clearer Thinking podcast is aimed more at people in or related to EA, whereas Sam Harris's wasn't; it was up to him what topics he wanted to focus on. 

Thanks! Didn't know you're sceptical of AI x-risk. I wonder if there's a correlation between being a philosopher and having low AI x-risk estimates; it seems that way anecdotally. 

Thanks so much for those links, I hadn't seen them! 

(So much AI-related stuff coming out every day, it's so hard to keep on top of everything!)

This is a quick post to talk a little bit about what I’m planning to focus on in the near and medium-term future, and to highlight that I’m currently hiring for a joint executive and research assistant position. You can read more about the role and apply here! If you’re potentially interested, hopefully the comments below can help you figure out whether you’d enjoy the role. 

Recent advances in AI, combined with economic modelling (e.g. here), suggest that we might well face explosive AI-driven growth in technological capability in the next d... (read more)

I'm really excited about Zach coming on board as CEA's new CEO! 

Though I haven't worked with him a ton, the interactions I have had with him have been systematically positive: he's been consistently professional, mission-focused and inspiring. He helped lead EV US well through what was a difficult time, and I'm really looking forward to seeing what CEA achieves under his leadership!

Thank you so much for your work with EV over the last year, Howie! It was enormously helpful to have someone so well-trusted, with such excellent judgment, in this position. I’m sure you’ll have an enormous positive impact at Open Phil.

And welcome, Rob - I think it’s fantastic news that you’ve taken the role!

I mentioned a few months ago that I was planning to resign from the board of EV UK: I’ve now officially done so.

Since last November, I’ve been recused from the board on all matters associated with FTX and related topics, which has ended up being a large proportion of board business. (This is because the recusal affected not just decisions that were directly related to the collapse of FTX, but also many other decisions for which the way EV UK has been affected by the collapse of FTX was important context.) I know I initially said that I’d wait for ther... (read more)

(My personal views only, and like Nick I've been recused from a lot of board work since November.)

Thank you, Nick, for all your work on the Boards over the last eleven years. You helped steward the organisations into existence, and were central to helping them flourish and grow. I’ve always been impressed by your work ethic, your willingness to listen and learn, and your ability to provide feedback that was incisive, helpful, and kind.

Because you’ve been less in the limelight than me or Toby, I think many people don’t know just how crucial a role you playe... (read more)

Hey,

I’m really sorry to hear about this experience. I’ve also experienced what feels like social pressure to have particular beliefs (e.g. around non-causal decision theory, high AI x-risk estimates, other general pictures of the world), and it’s something I also don’t like about the movement. My biggest worries with my own beliefs stem around the worry that I’d have very different views if I’d found myself in a different social environment. It’s just simply very hard to successfully have a group of people who are trying to both figure out what’s corr... (read more)

This was an extremely helpful post, thank you! 

This is a good point, and it's worth pointing out that increasing $\overline{v}$ is always good whereas increasing $\tau$ is only good if the future is of positive value. So risk aversion reduces the value of increasing $\tau$ relative to increasing $\overline{v}$, provided we put some probability on a bad future.

Agree this is worth pointing out! I've a draft paper that goes into some of this stuff in more detail, and I make this argument. 

Another potential argument for trying to improve $\overline{v}$ is that, plausibly at least, the value lost as a r... (read more)

Existential risk, and an alternative framework

One common issue with “existential risk” is that it’s so easy to conflate it with “extinction risk”. It seems that even you end up falling into this use of language. You say: “if there were 20 percentage points of near-term existential risk (so an 80 percent chance of survival)”. But human extinction is not necessary for something to be an existential risk, so 20 percentage points of near-term existential risk doesn’t entail an 80 percent chance of survival. (Human extinction may also not be sufficient for exis... (read more)

Flow vs fixed resources

In footnote 14 you say: “It has also been suggested (Sandberg et al 2016, Ord 2021) that the ultimate physical limits may be set by a civilisation that expands to secure resources but doesn’t use them to create value until much later on, when the energy can be used more efficiently. If so, one could tweak the framework to model this not as a flow of intrinsic value over time, but a flow of new resources which can eventually be used to create value.”

This feels to me that it would really be changing the framework considerably, rather t... (read more)

Humanity

Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.

One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:

• A species that Homo sapiens evolves into 
• “Uploaded” humans
• “Aligned” AI systems
• Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes

I’m not sure where you draw the line, or if there is a principled place to draw the ... (read more)

Humanity

Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.

One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:

• A species that Homo sapiens evolves into 
• “Uploaded” humans
• “Aligned” AI systems
• Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes

I’m not sure where you draw the line, or if there is a principled place to draw the ... (read more)

Humanity

Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.

One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:

• A species that Homo sapiens evolves into 
• “Uploaded” humans
• “Aligned” AI systems
• Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes

I’m not sure where you draw the line, or if there is a principled place to draw the ... (read more)

Enhancements

I felt like the paper gave enhancements short shrift. As you note, they are the intervention most plausibly competes with existential risk reduction, as they scale with $\overline{v}$ $\tau$ .

You say: “As with many of these idealised changes, they face the challenge of why this wouldn’t happen eventually, even without the current effort. I think this is a serious challenge for many proposed enhancements.”

I agree that this is a serious challenge, and that one should have more starting scepticism about the persistence of enhancements compared with ... (read more)

Gains

“While the idea of a gain is simple — a permanent improvement in instantaneous value of a fixed size — it is not so clear how common they are.”

I agree that gains aren’t where the action is, when it comes to longterm impact. Nonetheless, here are some potential examples:

• Species loss / preservation of species
• Preservation of historical information
• Preservation of Earth’s ecosystem
• Preservation of areas of natural importance, like Olympus Mons

These plausibly have two sources of longterm value. The first is that future agents might have slightly better lives... (read more)

Speed-ups

You write: “How plausible are speed-ups? The broad course of human history suggests that speed-ups are possible,” and, “though there is more scholarly debate about whether the industrial revolution would have ever happened had it not started in the way it did. And there are other smaller breakthroughs, such as the phonetic alphabet, that only occurred once and whose main effect may have been to speed up progress. So contingent speed-ups may be possible.”

This was the section of the paper I was most surprised / confused by. You seemed open to speed-... (read more)

Advancements 

I broadly agree with the upshots you draw, but here are three points that make things a little more complicated:

Continued exponential growth

As you note: (i) if v(.) continues exponentially, then advancements can compete with existential risk reduction; (ii) such continued exponential growth seems very unlikely.

However, it seems above 0 probability that we could have continued exponential growth in v(.) forever, including at the end point (and perhaps even at a very fast rate, like doubling every year).  And, if so, then the total val... (read more)

Hi Toby,

Thanks so much for doing and sharing this! It’s a beautiful piece of work - characteristically clear and precise.

Remarkably, I didn’t know you’d been writing this, or had an essay coming out that volume! Especially given that I’d been doing some similar work, though with a different emphasis. 

I’ve got a number of thoughts, which I’ll break into different comments.

Strong upvote on this - it’s an issue that a lot of people have been discussing, and I found the post very clear!

There’s lots more to say, and I only had time to write something quickly but one consideration is about division of effort with respect to timelines to transformative AI. The longer AI timelines are, the more plausible principles-led EA movement-building looks.

Though I’ve updated a lot in the last couple of years on transformative-AI-in-the-next-decade, I think we should still put significant probability mass on “long” timelines (e.g. more than ... (read more)

Thanks!  I agree that we are already (kind of) doing most of these things. So the question is whether further centralisation is tractable (and desirable). Like I say, it seems to me the big thing is if there’s someone, or some group of people, who really wants to make that further centralisation to happen. (E.g. I don’t think I’d be the right person even if I wanted to do it.)

Some things I didn't understand from your bullet-point list:

Having most of the resources come from one place

By “resources” do you primarily mean funding?  (I'll assume ... (read more)

Maybe what’s going on here is vagueness, and me being unclear.

Jeff’s clarification is helpful. I could have just dropped “part of the EA movement or” and the sentence would have been clearer and better. 

The key thing I was meaning in this context is: “Is a project engaging in EA movement-building, such that it would make sense that they at least potentially have obligations or responsibilities towards the EA movement as a whole?” The answer is clearly “no” for LEEP (for example), and “yes” for CEA. On that question, I would say “no” for GovAI, Lo... (read more)

“​to the extent that the text above is breaking down centralization into sub-dimensions, and then impliedly taking something like the mean score of sub-domains to generate an overall centralization score.”

Thanks for pointing this out! I didn't intend my post to be taking the mean score across sub-domains; I agree that of the dimensions I list, decision-making power is the most important sub-dimension. (Though the dimensions are interrelated: If you can’t tightly control group membership, or if there isn’t legal ownership, that limits decision-making power ... (read more)

Thanks for this comment, it’s very inspiring!

One thought I had is that do-ocracy (as opposed to “someone will have got this covered, right?”) describes other areas, as well as EA. On the recent 80k podcast, Lennart Heim describes a similar dynamic within AI governance:

“at some point, I would discover that compute seems really important as an input to these AI systems — so maybe just understanding this seems useful for understanding the development of AI. And I really saw nobody working on this. So I was like, “I guess I must be wrong if nobody’s worki... (read more)

Honestly, it does seem like it might be challenging, and I welcome ideas on things to do. (In particular, it might be hard without sacrificing lots of value in other ways. E.g. going on big-name podcasts can be very, very valuable, and I wouldn’t want to indefinitely avoid doing that - that would be too big a cost. More generally, public advocacy is still very valuable, and I still plan to be “a” public proponent of EA.)

The lowest-hanging fruit is just really hammering the message to journalists / writers I speak to; but there’s not a super tight corr... (read more)

Thanks! And I agree re comparative advantage!

Hey - I’m starting to post and comment more on the Forum than I have been, and you might be wondering about whether and when I’m going to respond to questions around FTX. So here’s a short comment to explain how I’m currently thinking about things:

The independent investigation commissioned by EV is still ongoing, and the firm running it strongly preferred me not to publish posts on backwards-looking topics around FTX while the investigation is still in-progress. I don’t know when it’ll be finished, or what the situation will be like for communicating on th... (read more)

Sorry - done, thanks! 

Given the TIME article, I thought I should give you all an update. Even though I have major issues with the piece, I don’t plan to respond to it right now.

Since my last shortform post, I’ve done a bunch of thinking, updating and planning in light of the FTX collapse. I had hoped to be able to publish a first post with some thoughts and clarifications by now; I really want to get it out as soon as I can, but I won’t comment publicly on FTX at least until the independent investigation commissioned by EV is over. Unfortunately, I think that’s a minimum of 2 m... (read more)

Thanks for asking! Still not entirely determined - I’ve been planning some time off over the winter, so I’ll revisit this in the new year.

I’ve been thinking hard about whether to publicly comment more on FTX in the near term. Much for the reasons Holden gives here, and for some of the reasons given here, I’ve decided against saying any more than I’ve already said for now.

I’m still in the process of understanding what happened,  and processing the new information that comes in every day. I'm also still working through my views on how I and the EA community could and should respond.

I know this might be dissatisfying, and I’m really sorry about that, but I think it’s the rig... (read more)

Hi Eli, thank you so much for writing this! I’m very overloaded at the moment, so I’m very sorry I’m not going to be able to engage fully with this. I just wanted to make the most important comment, though, which is a meta one: that I think this is an excellent example of constructive critical engagement — I’m glad that you’ve stated your disagreements so clearly, and I also appreciate that you reached out in advance to share a draft. 

Hi - thanks for writing this! A few things regarding your references to WWOTF:

The following is, as far as I can tell, the main argument that MacAskill makes against the Asymmetry (p. 172)

I’m confused by this sentence. The Asymmetry endorses neutrality about bringing into existence lives that have positive wellbeing, and I argue against this view for much of the population ethics chapter, in the sections “The Intuition of Neutrality”,  “Clumsy Gods: The Fragility of Identity”, and “Why the Intuition of Neutrality is Wrong”.

this still leaves open the qu

It’s because we don’t get to control the price - that’s down to the publisher.

I’d love us to set up a non-profit publishing house or imprint that could mean that we would have control over the price.

It would be a very different book if the audience had been EAs. There would have been a lot more on prioritisation (see response to Berger thread above), a lot more numbers and back-of-the-envelope calculations, a lot more on AI, a lot more deep philosophy arguments, and generally more of a willingness to engage in more speculative arguments. I’d have had more of the philosophy essay “In this chapter I argue that..” style, and I’d have put less effort into “bringing the ideas to life” via metaphors and case studies. Chapters 8 and 9, on population ethics a... (read more)

Thanks! 

Yes, we got extensive advice on infohazards from experts on this and other areas, including from people who have both domain expertise and thought a lot about how to communicate about key ideas publicly given info hazard concerns. We were careful not to mention anything that isn’t already in the public discourse.

To be clear - these are a part of my non-EA life, not my EA life!  I’m not sure if something similar would be a good idea to have as part of EA events - either way, I don’t think I can advise on that!

Some sorts of critical commentary are well worth engaging with (e.g. Keiran Setiya’s review of WWOTF); in other cases, where criticism is clearly misrepresentative or strawmanning, I think it’s often best not to engage.

I think it’s a combination of multiplicative factors. Very, very roughly:

• Prescribed medication and supplements: 2x improvement
• Understanding my own mind and adapting my life around that (including meditation, CBT, etc): 1.5x improvement 
• Work and personal life improvements (not stressed about getting an academic job, doing rewarding work, having great friends and a great relationship): 2x improvement 

To illustrate quantitatively (with normal weekly wellbeing on a +10 to -10 scale) with pretty made-up numbers, it feels like an average week used to b... (read more)

Huge question, which I’ll absolutely fail to do proper justice to in this reply! Very briefly, however:  

• I think that AI itself (e.g. language models) will help a lot with AI safety.
• In general, my perception of society is that it’s very risk-averse about new technologies, has very high safety standards, and governments are happy to slow down the introduction of new tech. 
• I’m comparatively sceptical of ultra-fast takeoff scenarios, and of very near-term AGI (though I think both of these are possible, and that’s where much of the risk lies), w