All of William_MacAskill's Comments + Replies

Merry Christmas, everyone!

This year, I’m feeling grateful to be me.

Recently, I gave some information about myself to Claude, and asked how I compared to other 38-yr-old men in the world.

I thought I understood global inequality well, but I still found the results quite moving. The usual AI sycophancy and reassurance was gone:

Compared to Global Averages, you’re in an exceptionally privileged position - likely in the top 1-3% globally:

- Work: Most men globally work manual labor, informal economy, or small trade with no job security. Your stable research posit

In my memory, the main impetus was a couple of leading AI safety ML researchers started making the case for 5-year timelines. They were broadly qualitatively correct and remarkably insightful (promoting the scaling-first worldview), but obviously quantitatively too aggressive. And AlphaGo and AlphaZero had freaked people out, too. 

A lot of other people at the time (including close advisers to OP folks) had 10-20yr timelines. My subjective impression was that people in the OP orbit generally had more aggressive timelines than Ajeya's report did. 

Re "Oxford EAs" - Toby Ord is presumably a paradigm of that. In the Great AI Timelines Scare of 2017, I spent some time looking into timelines. His median, then, was 15 years, which has held up pretty well. (And his x-risk probability, as stated in the Precipice, was 10%.)

I think I was wrong in my views on timelines then. But people shouldn't assume I'm a stand-in for the views of "Oxford EAs".
 

I agree - this is a great point. Thanks, Simon!

You are right that the magnitude of rerun risk from alignment should be lower than the probability of misaligned AI doom. However, in worlds in which AI takeover is very likely but that we can't change that, or in worlds where it's very unlikely and we can't change that, those aren't the interesting worlds, from the perspective of taking action. (Owen and Fin have a post on this topic that should be coming out fairly soon).  So, if we're taking this consideration into account, this should also discount th... (read more)

Okay, looking at the spectrum again, it still seems to me like I've labelled them correctly? Maybe I'm missing something. It's optimistic if we can retain a knowledge of how to align AGI because then we can just use that knowledge later and we don't face the same magnitude of risk of the misaligned AI. 

I agree with this. One way of seeing that is how many doublings of energy consumption civilisation can have before it needs to move beyond the solar system? The answer to that is about 40 doublings. Which, depending on your views on just how fast explosive industrial expansion goes, could be a pretty long time, e.g. decades.

I do think that non-existential level catastrophes are a big deal even despite the rerun risk consideration, because I expect the civilisation that comes back from such a catastrophe to be on a worse values trajectory than the one we have today. In particular, because the world today is unusually democratic and liberal, and I expect a re-roll of history to result in less democracy than we have today at the current technological level. However, other people have pushed me on that, and I don't feel like the case here is very strong. There are also obvious re... (read more)

One more clarification to the comment for forum users: I have tendonitis, and so I'm voice dictating all of my comments, so they might read oddly!

Thanks, that's a good catch. Really, in the simple model the relevant point of time for the first run should be when the alignment challenge has been solved, even for superintelligence. But that's before 'reasonably good global governance".

Of course, there's an issue that this is trying to model alignment as a binary thing for simplicity, even though really if a catastrophe came when half of the alignment challenge had been solved, that would still be a really big deal for similar reasons to the paper.

One additional comment is that this sort of "concepts m... (read more)

I think not at all a dumb idea, and I talk about this in Section 6.4. It actually feels like an activity you could do at very low cost that might have very high value per unit cost. 

The second way in which this post is an experiment is that it's an example of what I've been calling AI-enhanced writing. The experiment here is to see how much more productive I can be in the research and writing process by relying very heavily on AI assistance — Ttrying to use AI rather than myself wherever I can possibly do so. In this case, I went from having the basic idea to having this draft in about a day of work.

I'd be very interested in people's comments on how apparent it is that AI was used so extensively in drafting this piece — in particular if there are examples of AI slop that you can find in the text and that I missed.

The first way in which this post is an experiment is that it's work-in-progress that I'm presenting at a Forethought Research progress meeting. The experiment is just to publish it as a draft and then have the comments that I would normally receive as GoogleDoc comments on this forum post instead. The hope is that by doing this more people can get up to speed with Forethought research earlier than they would have and we can also get more feedback and thoughts at an earlier stage from a wider diversity of people.

I'd welcome takes from Forumites on how valuable or not this was.

I, of course, agree! 

One additional point, as I'm sure you know,  is that potentially you can also affect P(things go really well | AI takeover). And actions to increase ΔP(things go really well | AI takeover) might be quite similar to actions that increase ΔP(things go really well | no AI takeover). If so, that's an additional argument for those actions compared to affecting ΔP(no AI takeover).


Re the formal breakdown, people sometimes miss the BF supplement here which goes into this in a bit more depth. And here's an excerpt from a forthcoming p... (read more)

Thanks! I agree strongly with that.

(Also, thank you for doing this analysis, it's great stuff!)

😢

Rutger Bregman isn’t on the Forum, but sent me this message and gave me permission to share:

Great piece! I strongly agree with your point about PR. EA should just be EA, like the Quakers just had to be Quakers and Peter Singer should just be Peter Singer.

Of course EA had to learn big lessons from the FTX saga. But those were moral and practical lessons so that the movement could be proud of itself again. Not PR-lessons. The best people are drawn to EA not because it’s the coolest thing on campus, but because it’s a magnet for the most morally serious + the

Argh, thanks for catching that! Edited now.

If this perspective involves a strong belief that AI will not change the world much, then IMO that's just one of the (few?) things that are ~fully out of scope for Forethought

 

I disagree with this. There would need to be some other reason for why they should work at Forethought rather than elsewhere, but there are plausible answers to that — e.g. they work on space governance, or they want to write up why they think AI won't change the world much and engage with the counterarguments. 

I can't speak to the "AI as a normal technology" people in particular, but a shortlist I created of people I'd be very excited about includes someone who just doesn't buy at all that AI will drive an intelligence explosion or explosive growth.

I think there are lots of types of people where it wouldn't be a great fit, though. E.g. continental philosophers; at least some of the "sociotechnical" AI folks; more mainstream academics who are focused on academic publishing. And if you're just focused on AI alignment, probably you'll get more at a different org th... (read more)

Thanks for writing this, Lizka! 

Some misc comments from me:

• I have the worry that people will see Forethought as "the Will MacAskill org", at least to some extent, and therefore think you've got to share my worldview to join. So I want to discourage that impression! There's lots of healthy disagreement within the team, and we try to actively encourage disagreement. (Salient examples include disagreement around: AI takeover risk; whether the better futures perspective is totally off-base or not;  moral realism / antirealism; how much and what work

I'm not even sure your arguments would be weak in that scenario. 

Thanks - classic Toby point!  I agree entirely that you need additional assumptions.

I was imagining someone who thinks that, say, there's a 90% risk of unaligned AI takeover, and a 50% loss of EV of the future from other non-alignment issues that we can influence. So EV of the future is 5%.

If so, completely solving AI risk would increase the EV of the future to 50%; halving both would increase it only to 41%.

But, even so, it's probably easier to halve both than to completely eliminate AI takeover risk, and more generally the case for a mixed strategy seems strong. 

Haha, thank you for the carrot - please have one yourself!

"Harangue" was meant to be a light-hearted term. I agree, in general, on carrots rather than sticks. One style of carrot is commenting things like "Great post!" - even if not adding any content, I think it probably would increase the quantity of posts on the Forum, and somewhat act as a reward signal (more than just karma).

making EA the hub for working on "making the AI transition go well"

I don't think EA should be THE hub. In an ideal world, loads of people and different groups would be working on these issues.  But at the moment, really almost no one is. So the question is whether it's better if, given that, EA does work on it, and at least some work gets done. I think yes.

(Analogy: was it good or bad that in the earlier days, there was some work on AI alignment, even though that work was almost exclusively done by EA/rationalist types?)

I think it's likely that without a long (e.g. multi-decade) AI pause, one or more of these "non-takeover AI risks" can't be solved or reduced to an acceptable level.

I don't understand why you're framing the goal as "solving or reducing to an acceptable level", rather than thinking about how much expected impact we can have.  I'm in favour of slowing the intelligence explosion (and in particular of "Pause at human-level".) But here's how I'd think about the conversion of slowdown/pause into additional value:

Let's say the software-only intelligenc... (read more)

Thanks, Nick, that's helpful. I'm not sure how much we actually disagree — in particular, I wasn't meaning this post to be a general assessment of EA as a movement, rather than pointing to one major issue — but I'll use the opportunity to clarify my position at least. 

The EA movement is not (and should not be) dependent on continuous intellectual advancement and breakthrough for success. When I look at your 3 categories for the “future” of EA, they seem to refer more to our relevance as thought leaders, rather than what we actually achieve in the worl

Thanks - I agree the latter is important, and I think it's an error if "Attending carefully to the effect of communicating ideas in different ways" (appreciating that most of your audience is not extremely high-decoupling, etc) is rounded off to being overly focused on PR. 

I agree with you that in the intervening time, the pendulum has swung too far in the other direction, and am glad to see your pushback.

Thank you for clarifying - that's really helpful to hear!


"I think that most of the intellectual core continues to hold EA values and pursue the goals they pursue for EA reasons (trying to make the world better as effectively as possible, e.g. by trying to reduce AI risk), they've just updated against that path involving a lot of focus on EA itself"

And I agree strongly with this — and I think if it's a shame if people interpret the latter as meaning "abandoning EA" rather than "rolling up our sleeves and getting on with object-level work."

Thank you so much for writing this; I found a lot of it quite moving.

Since I read Strangers Drowning, this quote has really stuck in my mind:

"for do-gooders, it is always wartime"

And this from what you wrote resonates deeply, too:

"appreciate how wonderful it is to care about helping others."

"celebrate being part of a community that cares so much that we want to do so as effectively as we can."

Meditation and the cultivation of gratitude has been pretty transformative in my own life for my own wellbeing and ability to cope with living in a world in which it's always wartime. I'm so glad you've had the same experience.

(cross-posted from LW)

Hey Rob, thanks for writing this, and sorry for the slow response. In brief, I think you do misunderstand my views, in ways that Buck, Ryan and Habryka point out. I’ll clarify a little more.

Some areas where the criticism seems reasonable:

• I think it’s fair to say that I worded the compute governance sentence poorly, in ways Habryka clarified.
• I’m somewhat sympathetic to the criticism that there was a “missing mood” (cf e.g. here and here), given that a lot of people won’t know my broader views. I’m very happy to say: "I defini

What do you think would be a better reward? We're pretty constrained in our options. 

I want to get in the habit of treating AIs with dignity and respect. 

Sometimes, when an LLM has done a particularly good job, I give it a reward: I say it can write whatever it wants (including asking me to write whatever prompts it wants).

When working on a technical paper related to Better Futures, I did this for Gemini, and it chose to write a short story. I found it pretty moving, and asked if I could publish it. Here it is.

The Architect and the Gardener

On a vast and empty plain, two builders were given a task: to create a home that would last for ages, a sanctuary for all the generations to come. They were given s... (read more)

I think that most of classic EA vs the rest of the world is a difference in preferences / values, rather than a difference in beliefs. Ditto for someone funding their local sports teams rather than anti-aging research.  We're saying that people are failing in the project of rationally trying to improve the world by as much as possible - but few people really care much or at all about succeeding at that project. (If they cared more, GiveWell would be moving a lot more money than it is.)

In contrast, most people really really don't want to die in the nex... (read more)

a smaller bottleneck just increases the variance. But this is bad in expectation if you think that the value of the future is a concave function of the fraction of world power wielded by people with the correct values, because of trade and compromise.

Yes, this was meant to be the argument, thanks for clarifying it!

This has been proposed in the philosophy literature! It's the simplest sort of "variable-value" view, and was originally proposed by Yew-Kwang Ng. (Although you add linearity for negative worlds.) 

I think you're right that it avoids scale-tipping, which is neat.

Beyond that, I'm not sure how your proposal differs much from joint-aggregation bounded views that we discuss in the paper?

Various issues with it:
- Needs to be a "difference-making" view, otherwise is linear in practice
- Violates separability
- EV of near-term extinction, on this view, probably becomes very positive
 

I like the figure! 

Though the probability distiribution would have to be conditional on people in the future not trying to optimise the future. (You could have a "no easy eutopia" view, but expect that people in the future will optimise toward the good and hit the narrow target, and therefore have a curve that's more like the green line).

Thanks - that's fair. We were at least should have said "near-term extinction", and of course to define an outcome as exactly 0 we'd need to make it very specific.

Ah, I responded on the other comment, here.

 

Glad to see this series up! Tons of great points here.

Thanks! And it’s great to see you back on here!

One thing I would add is a that I think the analysis about fragility of value and intervention impact has a structural problem. Supposing that the value of the future is hyper-fragile as a combination of numerous multiplicative factors, you wind up thinking the output is extremely low value compared to the maximum, so there's more to gain. OK.

But a hypothesis of hyper-fragility along these lines also indicates that after whatever interventions you mak

Yeah, vibes were very good I thought - very energised and positive.

Because the democracies (N America, Europe) would have been differentially destroyed (or damaged), and I think that the world  is "unusually" democratic (i.e. more than the average we'd get with many historical replays).

I agree re preventing catastrophes at least - e.g. a nuclear war has great long-term harm via destroying many leading democracies, making the post-catastrophe world less democratic, even if it doesn't result in extinction.

On resilience in particular, I'd need to see the argument spelled out a bit more. 

Thanks - sorry my initial post was unclear.

"I'm not a dedicated utilitarian, so I typically tend to value futures with some human flourishing and little suffering vastly higher than futures with no sentient beings. But I am actually convinced that we should tilt a little toward futures with more flourishing."

See the next essay, "no easy eutopia" for more on this!

I responded to a Jack's similar thought here.

Man, was that unclear? 

Sorry for sucking at basic communication, lol.

In general, competition of various kinds seems like it has been one of the most positive forces for human development - competition between individuals for excellence, between scientists for innovation, between companies for cost-effectively meeting consumer wants, and between countries. Historically 'uncoordinated' competition has often had much better results than coordination!

I agree with the historical claim (with caveats below), but I think how that historical success ports over to future expected success is at best very murky.

A few comments here why:... (read more)

Fair, but bear in mind that we're conditioning on your action successfully reducing x-catastrophe. So you know that you're not in the world where alignment is impossibly difficult.

Instead, you're in a world where it was possible to make a difference on p(doom) (because you in fact made the difference), but where nonetheless that p(doom) reduction hadn't happened anyway. I think that's pretty likely to be a pretty messed up world, because, in the non-messed-up-world, the p(doom) reduction already happens and your action didn't make a difference. 
 

I agree with the core point, and that was part of my motivation for working on this area. There is a counterargument, as Ben says, which is that any particular intervention to promote Flourishing might be very non-robust.

And there is an additional argument, which is that in worlds in which you have successfully reduced x-risk, the future is more likely to be negative-EV (because worlds in which you have successfully reduced x-risk are more likely to be worlds in which x-risk is high, and those worlds are more likely to be going badly in general (e.g. great... (read more)

Thanks!

A couple of comments:

1/
I'm comparing Surviving (as I define it) and Flourishing.  But if long-term existential risk is high, that equally decreases the value of increasing Surviving and the value of increasing Flourishing. So how much long-term existential risk there is doesn't affect that comparison.

2/ 
But maybe efforts to reduce long-term existential risk are even better than work on either Surviving or Flourishing?

In the supplement, I assume (for simplicity) that we just can't affect long-term existential risk.

But I also think that, if ... (read more)