Rohin Shah

3595Joined May 2015


Hi, I'm Rohin Shah! I work as a Research Scientist on the technical AGI safety team at DeepMind. I completed my PhD at the Center for Human-Compatible AI at UC Berkeley, where I worked on building AI systems that can learn to assist a human user, even if they don't initially know what the user wants.

I'm particularly interested in big picture questions about artificial intelligence. What techniques will we use to build human-level AI systems? How will their deployment affect the world? What can we do to make this deployment go better? I write up summaries and thoughts about recent work tackling these questions in the Alignment Newsletter.

In the past, I ran the EA UC Berkeley and EA at the University of Washington groups.


If we bracket the timelines part and just ask about p(doom), I think and makes it quite easy to reach extremely dire forecasts about AGI. Getting extremely novel software right on the first try is just that hard.

Surely not. Neither of those make any arguments about AI, just about software generally. If you literally think those two are sufficient arguments for concluding "AI kills us with high probability" I don't see why you don't conclude "Powerpoint kills us with high probability".

I think this is generally false, but might benefit from some examples or more specifics.

(Referring to the OP, not the comment)

The argument here emphatically cannot be merely summarized as "AGI soon [is] a very contrarian position [and market prices are another indication of this]".

Can you describe in concrete detail a possible world in which:

  1. "AGI in 30 years" is a very contrarian position, including amongst hedge fund managers, bankers, billionaires, etc
  2. Market prices indicate that we'll get AGI in 30 years

It seems to me that if you were in such a situation, all of the non-contrarian hedge fund managers, bankers, billionaires would do the opposite of all of the trades that you've listed in this post, which would then push market prices back to rejecting "AGI in 30 years"; they have more money so their views dominate. What, concretely, prevents that from happening?

This post's thesis is that markets don't expect AGI in the next 30 years. I'll make a stronger claim: most people don't expect AGI in the next 30 years; it's a contrarian position. Anyone expecting AGI in that time is disagreeing with a very large swath of humanity.

(It's a stronger claim because "most people don't expect AGI" implies "markets don't expect AGI", but the reverse is not true. (Not literally so -- you can construct scenarios like "only investors expect AGI while others don't" where most people don't expect AGI but the market does expect AGI -- but these seem like edge cases that clearly don't apply to reality.))

Personally I feel okay disagreeing with the rest of humanity on this, because (a) the arguments seem solid to me, while the counterarguments don't, and (b) the AGI community has put in much more serial thought into the question than the rest of humanity.

If you already knew that belief in AGI soon was a very contrarian position (including amongst the most wealthy, smart, and influential people), I don't think you should update at all on the fact that the market doesn't expect AGI.

If you didn't know that, consider this your wake up call to reflect on the fact that most people disagree with you. You don' t need to think about financial markets or real interest rates or what trades would make you rich; all of those effects are downstream of the fact that most people disagree with you.

(Separately, I do expect you can bet on belief in AGI through financial markets, since I do expect that if AGI is coming soon the rest of the world will eventually figure that out. But it's not clear when you can expect to make money; that depends on when exactly the rest of humanity figures it out.)

My view on this is rather that there seem to be several key technologies and measures of progress that have very limited room for further growth, and the ~zero-to-one growth that occurred along many of these key dimensions seems to have been low-hanging fruit that coincided with the high growth rates that we observed around the mid-1900s. And I think this counts as modest evidence against a future growth explosion.

Hmm, it seems to me like these observations are all predicted by the model I'm advocating, so I don't see why they're evidence against that model. (Which is why I incorrectly thought you were instead saying that there wasn't room for much growth, sorry for the misunderstanding.)

(I do agree that declining growth rates are evidence against the model.)

At any given point in time, I expect that progress looks like "taking the low-hanging fruit"; the reason growth goes up over time anyway is because there's a lot more effort looking for fruit as time goes on, and it turns out that effect dominates.

For example, around 0 AD you might have said "recent millennia have had much higher growth rates because of the innovations of agriculture, cities and trade, which allowed for more efficient food production and thus specialization of labor. The zero-to-one growth on these key dimensions was low-hanging fruit, so this is modest evidence against further increases in growth in the future"; that would have been been an update in the wrong direction.

You're trying to argue for "there are no / very few important technologies with massive room for growth" by giving examples of specific things without massive room for growth.

In general arguing for "there is no X that satisfies Y" by giving examples of individual Xs that don't satisfy Y is going to be pretty rough and not very persuasive to me, unless there's some reason that can be abstracted out of the individual examples that is likely to apply to all Xs, which I don't see in this case. I don't care much whether the examples are technologies or measures (though I do agree measures are better).

(I'm also less convinced because I can immediately think of related measures where it seems like we have lots of room to grow, like  "the speed at which we can cost-effectively transmit matter around Earth" or "the efficiency with which we can harvest fusion energy".)

For similar reasons I don't update much on empirical trends in hardware progress (there's still tons of progress to be made in software, and still tons of progress to be made in areas other than computing).

I agree that explosive growth looks unlikely without efficiency gains; "no efficiency gains" means that the positive feedback loop that drives hyperbolic growth isn't happening. (But for this to move me I need to believe "no/limited efficiency gains".)

I think the decline in innovations per capita is the strongest challenge to this view; I just don't really see the others as significant evidence one way or the other.

I don't disagree with anything you've written here, but I'm not seeing why the limits they impose are anywhere close to where we are today.

I think most of the arguments I present in the section on why I consider Model 2 most plausible are about declining growth along various metrics.

Yes, sorry, I shouldn't have said "most".

especially those presented in the section “Many key technologies only have modest room for further growth

Yeah, I mostly don't buy the argument (sorry for not noting that earlier). It's not the case that there are N technologies and progress consists solely of improving those technologies; progress usually happens by developing new technologies. So I don't see the fact that some technologies are near-perfect as all that relevant. For example:

“Electric motors, pumps, battery charging, hydroelectric power, electricity transmission — among many other things — operate at near perfect efficiency (often around 90%).”

Even if we get literally no improvement in any of these technologies, we could still see huge growth in this sector by developing new technologies for energy generation that generate much more power than we can currently generate.

Load More