Thoughts on short timelines

Tobias_Baumann

Thoughts on short timelines

Tobias_Baumann

6 min readOct 23, 2018

Comments 14

Sorted by

New & upvoted

Daniel_Eth

I think you're right about AGI being very unlikely within the next 10 years. I would note, though, that the OpenPhil piece you linked to predicted at least 10% chance within 20 years, not 10 years (and I expect many people predicting "short timelines" would consider 20 years to be "short"). If you grant 1-2% chance to AGI in 10 years, perhaps that translates to 5-10% within 20 years.

Milan Griffes

I'm guessing the people you have mind would say that 1-2% of AGI within 10 years is also non-negligible, so the argument for focusing on it still holds (given the enormity of the expected impact).

Daniel_Eth

I think this line of reasoning may be misguided, at least if taken in a particular direction. If the AI Safety community loudly talks about there being a significant chance of AGI within 10 years, then this will hurt the AI Safety community's reputation when 10 years later we're not even close. It's important that we don't come off as alarmists. I'd also imagine that the argument "1% is still significant enough to warrant focus" won't resonate with a lot of people. If we really think the chances in the next 10 years are quite small, I think we're better off (at least for PR reasons) talking about how there's a significant chance of AGI in 20-30 years (or whatever we think), and how solving the problem of safety might take that long, so we should start today.

Milan Griffes

Makes sense – I think the optics question is pretty separate from the "what's our actual best-guess?" question.

Buck

Alternatively, one could look at the opinions of smart people in the effective altruism community (“EA experts”), who tend to assign a non-negligible probability to short timelines. But this (vaguely defined) set of people is subject to a self-selection bias – if you think AGI is likely to happen soon, you’re much more likely to spend years thinking and talking about that – and has little external validation of their “expert” status.

One way of countering this bias is only looking at the opinion of EAs who have thought about this a lot and got into EA from some cause area other than AI safety. My impression is that that group has roughly similar timelines to EAs who were initially focused on AI safety.

Lukas Finnveden

Seems like there's still self-selection going on, depending on how much you think 'a lot' is, and how good you are at finding everyone who have thought about it that much. You might be missing out on people who thought about it for, say, 20 hours, decided it wasn't important, and moved on to other cause areas without writing up their thoughts.

On the other hand, it seems like people are worried about and interested in talking about AGI happening in 20 or 30 or 50 years time, so it doesn't seem likely that everyone who thinks 10-year timelines are <10% stops talking about it.

PeterMcCluskey

I disagree with your analysis of "are we that ignorant?".

For things like nuclear war or financial meltdown, we've got lots of relevant data, and not too much reason to expect new risks. For advanced nanotechnology, I think we are ignorant enough that a 10% chance sounds right (I'm guessing it will take something like $1 billion in focused funding).

With AGI, ML researchers can be influenced to change their forecast by 75 years by subtle changes in how the question is worded. That suggests unusual uncertainty.

We can see from Moore's law and from ML progress that we're on track for something at least as unusual as the industrial revolution.

The stock and bond markets do provide some evidence of predictability, but I'm unsure how good they are at evaluating events that happen much less than once per century.

[deleted]

Re your first point that it doesn't seem like we're making very fast progress:

People have very different views of how fast we're progressing. There's ultimately no objective measure because for any measure you could log or exponentiate it which also gives a measure of where we are but a very different picture. The evolutionary perspective suggests that if we're moving from insect to lizard abilities in a matter of years, we're going fast. From the 80k podcast with Paul Christiano:

"look at what evolution was able to do with varying amounts of compute. If you look at what each order of magnitude buys you in nature, you’re going from insects to small fish to lizards to rats to crows to primates to humans. Each of those is one order of magnitude, roughly, so you should be thinking of there are these jumps. It is the case that the different between insect and lizard feels a lot smaller to us and is less intuitive significance than the difference between primate and human or crow and primate"

Although this is usually used to argue for moderate to fast takeoff, it also favors of ignorance, i.e. at best, just don't know how fast we're progressing because this contradicts our intuitive sense.

FYI I can't confirm your observation that AI researchers don't believe in short timelines.

John_Maxwell

Greg Brockman makes a case for short timelines in this presentation: Can we rule out near-term AGI? My prior is that deep neural nets are insufficient for AGI, and the increasing amounts of compute necessary to achieve breakthroughs reveals limits of the paradigm, but Greg's outside view argument is hard to dismiss.

Davidmanheim_duplicate0.2207304259636873

A few points.

First, market predictions for the intermediate term are mostly garbage. (Intermediate being 3-5 years.) It gets much much less predictive after that. The market's ability to predict is constrained by the investment time-frame of most investors, fundamental constraints on how noisy the market is, and other factors. But given all of that, the ridiculous valuation of tech firms - uber, twitter, etc. not to mention the crazy P/E rations for google, amazon, etc. seems to imply that the market thinks something important will happen there,

Second, I don't think you're defining the timelines question clearly. (Neither is anyone else.) One version is; "conditional on a moderately fast takeoff, when would we need to have solved the alignment problem before in order to prevent runaway value misalignment." Another is "regardless of takeoff speed, when will a given AI surpass the best performance by all humans across every domain." A third is "when will all AI systems overall be able to do more than any one single person." A fourth is "when will a specific AI be able to do more than one average person." And lastly, "when will people stop finding strange edge cases to argue that AI isn't yet more capable than humans, despite outperforming it on nearly every task"

I could see good arguments for 10 years as 10% probable for questions one and three. I think that most AI experts are thinking of something akin to questions two and four when they say 50 years. And I see good arguments that there is epsilon probability of question five in the next century.

Larks

the crazy P/E rations for google, amazon, etc. seems to imply that the market thinks something important will happen there,

Google's forward PE is 19x, vs the S&P500 on 15x. What's more, this is overstated, because it involves the expensing of R&D, which logically should be capitalised. Facebook is even cheaper at 16x, though if I recall correctly that excludes stock-based-comp expense.

I agree that many other tech firms have much more priced into their valuations, and that fundamental analysts in the stock market realistically only look 0-3 years out.

turchin

What is the actionable difference between "1-2 per cent" and "10 per cent" predictions? If we knew that an asteroid is coming to Earth and it will hit the Earth with one of these probabilities, how our attempts to diverge it would depend on the probability of the impact?

Should we ignore 1 per cent probability, but go all-in in preventing 10 per cent probability?

If there is no difference in actions, the difference in probability estimates is rather meaningless.

Jeff Kaufman 🔸

You can extend your argument to even smaller probabilities: how much effort should go into this if we think the chance is 0.1%? 0.01? Or in the other direction, 50%, 90%, etc. In extremes it's very clear that this should affect how much focus we put into averting it, and I don't think there's anything special about 1% vs 10% in this regard.

Another way of thinking about it is that AI is not the only existential risk. If your estimate for AI is 1% in the next ten years but pandemics is 10%, vs 10% for AI and 1% for pandemics, then that should also affect where you think people should focus.

turchin

Yes, it is clear. My question was: "Do we have any specific difference in mind about AI strategies for 1 per cent in 10 years vs. 10 per cent in 10 years cases?" If we going to ignore the risk in both cases, there is no difference is it 1 per cent or 10 per cent.

I don't know any short-term publically available strategy for the 10 years case, no matter what is the probability.

Comments