TLDR; no one talks about how far we are towards safe AGI and the focus is on when AGI doom arrives. We want to readjust this focus and measure progress, guide and facilitate research, and evaluate projects in AI safety for impact. We also ask you to add your views to this survey.

Prelude

It was a hot London Summer day and fourteen people had gathered to discuss the future of AI safety. To ascertain the mission criticality of our endeavor, we drew everyone’s AGI timelines on a whiteboard. The product of our estimates was close to the Metaculus estimate and some expressed concern at the stress of impending doom.

We then proceeded to draw our expected timelines for when alignment would be solved and very few had ever analyzed this. The few that expressed their timelines ended at 15 years hence, in 2037. And I thought this was a remarkably hopeful message!

A case for hope

Let us be optimistic and expect the median arrival for the solution to alignment is 2037 estimated as a Gaussian with a standard deviation of 10 years^[1]. If we estimate the probability that the solution will come before AGI based on sampling the probability mass of the Metaculus’ forecast for AGI and our “Safety Timeline” (or “alignment solution timeline”), we can calculate the probability that alignment will be solved before an AGI is released:

$P (s o l u t i o n < A G I) = M e a n (S a m p l e (N o r m a l (2037, 10)) < S a m p l e (M e t a c u l u s (weak AGI))) = 58.73$

We can then insert our conditional prior for the probability that the solution will also lead to human flourishing $P (human flourishing | solution)$ and the prior probability of doom from AGI to calibrate P(AGI-caused doom)^[2]. In this case, we set $P (human flourishing | solution)$ at a 90% effectiveness probability and the $P (d o o m) = M e a n (P (d o o m))$ from 24 researchers in AI safety (see the list in the Appendix). From this, we can calculate the probability mass for the calibrated $P (d o o m)$ , $C P (d o o m)$ .

$M e a n (P (d o o m)) = 28.73$

Now we introduce the previous graph with the updated $C P (d o o m)$ for any year on the AGI timelines. We calculate the uncalibrated $P (d o o m)$ by multiplying $P (A G I | y e a r)$ by the mean of $P (d o o m)$ estimates while $C P (d o o m)$ is $P (d o o m) \cdot (1 - C D F (P (s o l u t i o n | y e a r) \cdot P (AGI safe | s o l u t i o n))$ , i.e. P(doom) is multiplied by the complement of the cumulative probability distribution for safe AGI ...

Thomas Steinthal

Posts 2

Posts
2