Hi EarlyVelcro,

I’m happy to see more debate of how much we should prioritise AI safety. We intend to debate some of these issues on the podcast, and have already started recording with Ben Garfinkel.

However, I think you’re misrepresenting how much the key idea series recommends working on AI safety. We feature a range of other problem areas prominently and I don’t think many readers will come away thinking that our position is that “EA should focus on AI alone”.

We list 9 priority career paths, of which only 2 are directly related to AI safety, recommend a variety of other options, and say that there are many good options we don’t list.

Elsewhere on the page, we also discuss the importance of personal fit and coordination, which can make it better for an individual to enter different problem areas from those we most highlight.

The most relevant section is short, so I’d encourage readers of this thread to read the section and make up their own mind.

Top AI safety researchers are now saying that they expect AI to be safe by default, without further intervention from EA. See here and here.

Two points:

• "Probably safe by default" doesn't mean "we shouldn't work on it". My estimate of 90% that you quote still leaves a 10% chance of catastrophe, which is worth reducing. (Though the 10% is very non-robust.) It also is my opinion before updating on other people's views.
• Those posts were published because AI Impacts was looking to have conversations with people who had safe-by-default views, so there's a strong selection bias. If you looked for people with doom-by-default views, you could find them.

Hi EarlyVelcro,

Howie from 80k here.

As Ben said in his comment, the key ideas page, which is the most current summary of 80k’s views, doesn't recommend that “EA should focus on AI alone”. We don't think the EA community's focus should be anything close to that narrow.

That said, I do see how the page might give the impression that AI dominates 80k’s recommendations since most of the other paths/problems talked about are ‘meta’ or ‘capacity building’ paths. The page mentions that “we’d be excited for people to explore [our list of problems we haven’t yet investigated] as well as other areas that could foreseeably have a positive effect on the long-term future” but it doesn’t say anything about what those problems are (other than a link to our problem profiles page, which has a list).

I think it makes sense that people end up focusing on the areas we mention directly and the page could do a better job of communicating that our priorities are more diverse.

The good news is that we’re currently putting together a more thorough list of areas that we think might be very promising but aren't among our priority paths/problems.[1] Unfortunately, it didn’t quite get done in time to add it to this version of key ideas.

More generally, I think 80k’s content was particularly heavy on AI over the last year and, while it will likely remain our top priority, I expect it will make up a smaller portion of our content over the next few years.

[1] Many of these will be areas we haven't yet investigated or areas that are too niche to highlight among our priority paths.

There seems to be a large variance in researchers' estimates about timelines and takeoff-speed. Pointing to specific writeups that lean one way or another can't give much insight about the distribution of estimates. Also, I think that at least some researchers are less likely to discuss their estimates publicly if they're leaning towards shorter timelines and a discontinuous takeoff, which subjects the public discourse on the topic to a selection bias.

So I'm skeptical about the claim that "Most researchers seem to be moving away from a fast takeoff view of AI safety, and are now opting for a softer takeoff view".

Top AI safety researchers are now saying that they expect AI to be safe by default, without further intervention from EA. See here and here.

Again, there seems to be a large variance in researchers' views about this. Pointing to specific writeups can't give much insight about the distribution of views.

I think another large part of the focus comes from their views on population ethics. For example, in the article, you can "save" people by ensuring they're born in the first place:

Let’s explore some hypothetical numbers to illustrate the general concept. If there’s a 5% chance that civilisation lasts for ten million years, then in expectation, there are 5000 future generations. If thousands of people making a concerted effort could, with a 55% probability, reduce the risk of premature extinction by 1 percentage point, then these efforts would in expectation save 28 future generations. If each generation contains ten billion people, that would be 280 billion lives saved. If there’s a chance civilisation lasts longer than ten million years, or that there are more than ten billion people in each future generation, then the argument is strengthened even further.

(bold mine)

I discuss this further in my section "Implications for EA priorities" in this post of mine. I recommend trying this tool of theirs.

Note that 80k sometimes takes a softer tone, eg here:

An individual can only focus on one or two areas at a time, but a large group of people working together should most likely spread out over several.

When this happens, there are additional factors to consider when choosing a problem area. Instead of aiming to identify the single most pressing issue at the margin, the aim is to work out:

1. The ideal allocation of people over issues, and which direction that allocation should move in.

2. Where your comparative advantage lies compared to others in the group.

We call this the ‘portfolio approach’.

"It’s not clear that advanced artificial intelligence is going to arrive any time within the next several decades" - On the other hand, it's seems, at least to me, most likely that it will. Even if several more breakthroughs would be required to reach general intelligence, those may still come relatively fast as deep learning has now finally become useful enough in a wide enough array of applications that there is far more money and talent in the field than there ever was before by orders of magnitude. Now this by itself wouldn't necessarily guarantee fast advancement in a field, but AI research is still the kind of area where a single individual can push the research forward significantly just by themselves. And governments are beginning to realise the strategic importance of AI, so even more resources are flooding the field.

"One of the top AI safety organizations, MIRI, has now gone private so now we can’t even inspect whether they are doing useful work." - this is not an unreasonable choice and we have their past record to go on. Nonetheless, there are more open options if this is important to you.

"Productive AI safety research work is inaccessible to over 99.9% of the population, making this advice almost useless to nearly everyone reading the article." - Not necessarily. Even if becoming good enough to be a researcher is very hard, it probably isn't nearly as hard to become good enough at a particular area to help mentor other people.