(Crossposted to LessWrong)
Abstract
The linked paper is our submission to the Open Philanthropy AI Worldviews Contest. In it, we estimate the likelihood of transformative artificial general intelligence (AGI) by 2043 and find it to be <1%.
Specifically, we argue:
- The bar is high: AGI as defined by the contest—something like AI that can perform nearly all valuable tasks at human cost or less—which we will call transformative AGI is a much higher bar than merely massive progress in AI, or even the unambiguous attainment of expensive superhuman AGI or cheap but uneven AGI.
- Many steps are needed: The probability of transformative AGI by 2043 can be decomposed as the joint probability of a number of necessary steps, which we group into categories of software, hardware, and sociopolitical factors.
- No step is guaranteed: For each step, we estimate a probability of success by 2043, conditional on prior steps being achieved. Many steps are quite constrained by the short timeline, and our estimates range from 16% to 95%.
- Therefore, the odds are low: Multiplying the cascading conditional probabilities together, we estimate that transformative AGI by 2043 is 0.4% likely. Reaching >10% seems to require probabilities that feel unreasonably high, and even 3% seems unlikely.
Thoughtfully applying the cascading conditional probability approach to this question yields lower probability values than is often supposed. This framework helps enumerate the many future scenarios where humanity makes partial but incomplete progress toward transformative AGI.
Executive summary
For AGI to do most human work for <$25/hr by 2043, many things must happen.
We forecast cascading conditional probabilities for 10 necessary events, and find they multiply to an overall likelihood of 0.4%:
Event | Forecastby 2043 or TAGI, |
| We invent algorithms for transformative AGI | 60% |
| We invent a way for AGIs to learn faster than humans | 40% |
| AGI inference costs drop below $25/hr (per human equivalent) | 16% |
| We invent and scale cheap, quality robots | 60% |
| We massively scale production of chips and power | 46% |
| We avoid derailment by human regulation | 70% |
| We avoid derailment by AI-caused delay | 90% |
| We avoid derailment from wars (e.g., China invades Taiwan) | 70% |
| We avoid derailment from pandemics | 90% |
| We avoid derailment from severe depressions | 95% |
| Joint odds | 0.4% |
If you think our estimates are pessimistic, feel free to substitute your own here. You’ll find it difficult to arrive at odds above 10%.
Of course, the difficulty is by construction. Any framework that multiplies ten probabilities together is almost fated to produce low odds.
So a good skeptic must ask: Is our framework fair?
There are two possible errors to beware of:
- Did we neglect possible parallel paths to transformative AGI?
- Did we hew toward unconditional probabilities rather than fully conditional probabilities?
We believe we are innocent of both sins.
Regarding failing to model parallel disjunctive paths:
- We have chosen generic steps that don’t make rigid assumptions about the particular algorithms, requirements, or timelines of AGI technology
- One opinionated claim we do make is that transformative AGI by 2043 will almost certainly be run on semiconductor transistors powered by electricity and built in capital-intensive fabs, and we spend many pages justifying this belief
Regarding failing to really grapple with conditional probabilities:
- Our conditional probabilities are, in some cases, quite different from our unconditional probabilities. In particular, we assume that a world on track to transformative AGI will…
- Construct semiconductor fabs and power plants at a far faster pace than today (our unconditional probability is substantially lower)
- Have invented very cheap and efficient chips by today’s standards (our unconditional probability is substantially lower)
- Have higher risks of disruption by regulation
- Have higher risks of disruption by war
- Have lower risks of disruption by natural pandemic
- Have higher risks of disruption by engineered pandemic
Therefore, for the reasons above—namely, that transformative AGI is a very high bar (far higher than “mere” AGI) and many uncertain events must jointly occur—we are persuaded that the likelihood of transformative AGI by 2043 is <1%, a much lower number than we otherwise intuit. We nonetheless anticipate stunning advancements in AI over the next 20 years, and forecast substantially higher likelihoods of transformative AGI beyond 2043.
For details, read the full paper.
About the authors
This essay is jointly authored by Ari Allyn-Feuer and Ted Sanders. Below, we share our areas of expertise and track records of forecasting. Of course, credentials are no guarantee of accuracy. We share them not to appeal to our authority (plenty of experts are wrong), but to suggest that if it sounds like we’ve said something obviously wrong, it may merit a second look (or at least a compassionate understanding that not every argument can be explicitly addressed in an essay trying not to become a book).
Ari Allyn-Feuer
Areas of expertise
I am a decent expert in the complexity of biology and using computers to understand biology.
- I earned a Ph.D. in Bioinformatics at the University of Michigan, where I spent years using ML methods to model the relationships between the genome, epigenome, and cellular and organismal functions. At graduation I had offers to work in the AI departments of three large pharmaceutical and biotechnology companies, plus a biological software company.
- I have spent the last five years as an AI Engineer, later Product Manager, now Director of AI Product, in the AI department of GSK, an industry-leading AI group which uses cutting edge methods and hardware (including Cerebras units and work with quantum computing), is connected with leading academics in AI and the epigenome, and is particularly engaged in reinforcement learning research.
Track record of forecasting
While I don’t have Ted’s explicit formal credentials as a forecaster, I’ve issued some pretty important public correctives of then-dominant narratives:
- I said in print on January 24, 2020 that due to its observed properties, the then-unnamed novel coronavirus spreading in Wuhan, China, had a significant chance of promptly going pandemic and killing tens of millions of humans. It subsequently did.
- I said in print in June 2020 that it was an odds-on favorite for mRNA and adenovirus COVID-19 vaccines to prove highly effective and be deployed at scale in late 2020. They subsequently did and were.
- I said in print in 2013 when the Hyperloop proposal was released that the technical approach of air bearings in overland vacuum tubes on scavenged rights of way wouldn’t work. Subsequently, despite having insisted they would work and spent millions of dollars on them, every Hyperloop company abandoned all three of these elements, and development of Hyperloops has largely ceased.
- I said in print in 2016 that Level 4 self-driving cars would not be commercialized or near commercialization by 2021 due to the long tail of unusual situations, when several major car companies said they would. They subsequently were not.
- I used my entire net worth and borrowing capacity to buy an abandoned mansion in 2011, and sold it seven years later for five times the price.
Luck played a role in each of these predictions, and I have also made other predictions that didn’t pan out as well, but I hope my record reflects my decent calibration and genuine open-mindedness.
Ted Sanders
Areas of expertise
I am a decent expert in semiconductor technology and AI technology.
- I earned a PhD in Applied Physics from Stanford, where I spent years researching semiconductor physics and the potential of new technologies to beat the 60 mV/dec limit of today's silicon transistor (e.g., magnetic computing, quantum computing, photonic computing, reversible computing, negative capacitance transistors, and other ideas). These years of research inform our perspective on the likelihood of hardware progress over the next 20 years.
- After graduation, I had the opportunity to work at Intel R&D on next-gen computer chips, but instead, worked as a management consultant in the semiconductor industry and advised semiconductor CEOs on R&D prioritization and supply chain strategy. These years of work inform our perspective on the difficulty of rapidly scaling semiconductor production.
- Today, I work on AGI technology as a research engineer at OpenAI, a company aiming to develop transformative AGI. This work informs our perspective on software progress needed for AGI. (Disclaimer: nothing in this essay reflects OpenAI’s beliefs or its non-public information.)
Track record of forecasting
I have a track record of success in forecasting competitions:
- Top prize in SciCast technology forecasting tournament (15 out of ~10,000, ~$2,500 winnings)
- Top Hypermind US NGDP forecaster in 2014 (1 out of ~1,000)
- 1st place Stanford CME250 AI/ML Prediction Competition (1 of 73)
- 2nd place ‘Let’s invent tomorrow’ Private Banking prediction market (2 out of ~100)
- 2nd place DAGGRE Workshop competition (2 out of ~50)
- 3rd place LG Display Futurecasting Tournament (3 out of 100+)
- 4th Place SciCast conditional forecasting contest
- 9th place DAGGRE Geopolitical Forecasting Competition
- 30th place Replication Markets (~$1,000 winnings)
- Winner of ~$4200 in the 2022 Hybrid Persuasion-Forecasting Tournament on existential risks (told ranking was “quite well”)
Each finish resulted from luck alongside skill, but in aggregate I hope my record reflects my decent calibration and genuine open-mindedness.
Discussion
We look forward to discussing our essay with you in the comments below. The more we learn from you, the more pleased we'll be.
If you disagree with our admittedly imperfect guesses, we kindly ask that you supply your own preferred probabilities (or framework modifications). It's easier to tear down than build up, and we'd love to hear how you think this analysis can be improved.
I don't think I understand the structure of this estimate, or else I might understand and just be skeptical of it. Here are some quick questions and points of skepticism.
Starting from the top, you say:
This section appears to be an estimate of all-things-considered feasibility of transformative AI, and draws extensively on evidence about how lots of things go wrong in practice when implementing complicated projects. But then in subsequent sections you talk about how even if we "succeed" at this step there is still a significant probability of failing because the algorithms don't work in a realistic amount of time.
Can you say what exactly you are assigning a 60% probability to, and why it's getting multiplied with ten other factors? Are you saying that there is a 40% chance that by 2043 AI algorithms couldn't yield AGI no matter how much serial time and compute they had available? (It seems surprising to claim that even by 2023!) Presumably not that, but what exactly are you giving a 60% chance?
(ETA: after reading later sectio... (read more)
Excellent comment; thank you for engaging in such detail. I'll respond piece by piece. I'll also try to highlight the things you think we believe but don't actually believe.
Section 1: Likelihood of AGI algorithms
Yes, we assign a 40% chance that we don't have AI algorithms by 2043 capable of learning to do nearly any human task with realistic amounts of time and compute. Some things we probably agree on:
- Progress has been promising and investment is rising.
- Obviously the development of AI that can do AI research more cheaply than humans could be a huge accelerant, with the magnitude depending on the value-to-cost ratio. Already GPT-4 is accelerating my own software productivity, and future models over the next twenty years will no doubt be leagues better (as well as more efficient).
- Obviously slow progre
... (read more)I would guess that more or less anything done by current ML can be done by ML from 2013 but with much more compute and fiddling. So it's not at all clear to me whether existing algorithms are sufficient for AGI given enough compute, just as it wasn't clear in 2013. I don't have any idea what makes this clear to you.
Given that I feel like compute and algorithms mostly trade off, hopefully it's clear why I'm confused about what the 60% represents. But I'm happy for it to mean something like: it makes sense at all to compare AI performance vs brain performance, and expect them to be able to solve a similar range of tasks within 5-10 orders of magnitude of the same amount of compute.
... (read more)C'mon Paul - please extend some principle of charity here. :)
You have repeatedly ascribed silly, impossible beliefs to us and I don't know why (to be fair, in this particular case you're just asking, not ascribing). Genuinely, man, I feel bad that our writing has either (a) given the impression that we believe such things or (b) given the impression that we're the type of people who'd believe such things.
Like, are these sincere questions? Is your mental model of us that there's a genuine uncertainty over whether we'll say "Yes, a war precludes AGI" vs "No, a war does preclude AGI."
To make it clear: No, of course a war between China and Taiwan does not make it impossible to build AGI by 2043. As our essay explicitly says.
To mak... (read more)
Am I really the only person who thinks it's a bit crazy that we use this blobby comment thread as if it's the best way we have to organize disagreement/argumentation for audiences? I feel like we could almost certainly improve by using, e.g., a horizontal flow as is relatively standard in debate.[1]
With a generic example below:
To be clear, the commentary could still incorporate non-block/prose text.
Alternatively, people could use something like Kialo.com. But surely there has to be something better than this comment thread, in terms of 1) ease of determining where points go unrefuted, 2) ease of quickly tracing all responses in specific branches (rather than having to skim through the entire blob to find any related responses), and 3) seeing claims side-by-side, rather than having to scroll back and forth to see the full text. (Quoting definitely helps with this, though!)
(Depending on the format: this is definitely standard in many policy debate leagues.)
I put little weight on this analysis because it seems like a central example of the multiple stage fallacy. But it does seem worth trying to identify clear example of the authors not accounting properly for conditionals. So here are three concrete criticisms (though note that these are based on skimming rather than close-reading the PDF):
The third criticism is more involved; I'll summarize it as "the authors are sometimes treating the different events as sequential in time, and sometimes sequential in logical flow". For example, the authors assign around 1% to events 1-5 happening before 2043. If they're correct, then conditioning on events 1-5 happening before 2043, they'll very likely only happen just be... (read more)
Got it. As mentioned I disagree with your 0.7 war derailment. Upon further thought I don't necessarily disagree with your 0.7 "regulation derailment", but I think that in most cases where I'm talking to people about AI risk, I'd want to factor this out (because I typically want to make claims like "here's what happens if we don't do something about it").
Anyway, the "derailment" part isn't really the key disagreement here. The key disagreement is methodological. Here's one concrete alternative methodology which I think is better: a more symmetric model which involves three estimates:
By "rerailed" here I mean roughly "something as extreme as a derailment happens, but in a way which pushes us over the threshold to be on track toward... (read more)
This is a really impressive paper full of highly interesting arguments. I am enjoying reading it. That said, and I hope I'm not being too dismissive here, I have a strong suspicion that the central argument in this paper suffers from what Eliezer Yudkowsky calls the multiple stage fallacy,
I think canonicalizing this as a Fallacy was very premature: Yudkowsky wrote his post based on two examples:
Nate Silver's Trump's Six Stages of Doom (which got a very-clearly-wrong-in-hindsight answer)
My Breaking Down Cryonics Probabilities (which is harder to evaluate since it's about something farther in the future)
I wrote a response at the time, ending with:
In the discussion people gave a few other examples of people using this sort of model:
-
-
-
... (read more)Robin Hanson in Break Cryonics Down
Forecasters in the Good Judgement project (comment)
Animal Charity Evaluators trying to predict the power of a planned behavior chan
Related to this, you can only multiply probabilities if they're independent, but I think a lot of the listed probabilities are positively correlated, which means the joint probability is higher than their product. For example, it seems to me that "AGI inference costs drop below $25/hr" and "We massively scale production of chips and power" are strongly correlated.
Not reading the paper, and not planning to engage in much discussion, and stating beliefs without justification, but briefly commenting since you asked readers to explain disagreement:
I think this framework is bad and the probabilities are far too low, e.g.:
Separately, note that "AI that can quickly and affordably be trained to perform nearly all economically and strategically valuable tasks at roughly human cost or less" is a much higher bar than the-thing-we-should-be-paying-attention-to (which is more like takeover ability; see e.g. Kokotajlo).
I mean, I don't think all of your conditions are necessary (e.g. "We invent a way for AGIs to learn faster than humans" and "We massively scale production of chips and power") and I think together they carve reality quite far from the joints, such that breaking the AGI question into these subquestions doesn't help you think more clearly [edit: e.g. because compute and algorithms largely trade off, so concepts like 'sufficient compute for AGI' or 'sufficient algorithms for AGI' aren't useful].
Thank you for the clarification. To me, it is not 100.0% guaranteed that AGIs will be able to rapidly parallelize all learning and it is not 100.0% guaranteed that we'll have enough chips by 2043. Therefore, I think it helps to assign probabilities to them. If you are 100.0% confident in their likelihood of occurrence, then you can of course remove those factors. We personally find it difficult to be so confident about the future.
I agree that the success of AlphaZero and GPT-4 are promising notes, but I don't think they imply a 100.0% likelihood that AGI, whatever it looks like, will learn just as fast on every task.
With AlphaZero in particular, fast reinforcement training is possible because (a) the game state can be efficiently modeled by a computer and (b) the reward can be efficiently computed by a computer.
In contrast, look at a task like self-driving. Despite massive investment, our self-driving AIs are learning more slowly than human teenagers. Part of the reason for this is that conditions (a) and (b) no longer hold. First, our simulations of reality are imperfect, and therefore fleets must be deployed to drive millions of miles. Second, calculating reward functions (i.e., ... (read more)
You start off saying that existing algorithms are not good enough to yield AGI (and you point to the hardness of self-driving cars as evidence) and fairly likely won't be good enough for 20 years. And also you claim that existing levels of compute would be a way too low to learn to drive even if we had human-level algorithms. Doesn't each of those factors on its own explain the difficulty of self-driving? How are you also using the difficulty of self-driving to independently argue for a third conjunctive source of difficulty?
Maybe another related question: can you make a forecast about human-level self-driving (e.g. similar accident rates vs speed tradeoffs to a tourist driving in a random US city) and explain its correlation with your forecast about human-level AI overall? If you think full self-driving is reasonably likely in the next 10 years, that superficially appears to undermine the way you are using it as evidence for very unlikely AGI in 20 years. Conversely, if you think self-driving is very unlikely in the next 10 years, then it would be easier for people to update their overall views about your forecasts after observing (or failing to observe) full self-driving.
I think ... (read more)
Here are my forecasts of self-driving from 2018: https://www.tedsanders.com/on-self-driving-cars/
Five years later, I'm pretty happy with how my forecasts are looking. I predicted:
Today I regularly take Cruises around SF and it seems decently likely that self-driving taxis are on track to be widely deployed across the USA by 2030. Feels pretty probable, but still plenty of ways that it could be delayed or heterogenous (e.g., regulation, stalling progress, unit economics).
Plus, even wide robotaxi deployment doesn't mean human taxi drivers are rendered obsolete. Seems very plausible we operate for many many years with a mixed fleet, where... (read more)
I like that you can interact with this. It makes understanding models so much easier.
Playing with the calculator, I see that the result is driven to a surprising degree by the likelihood that "Compute needed by AGI, relative to a human brain (1e20-1e21 FLOPS)" is <1/1,000x (i.e. the bottom two options).[1]
I think this shows that your conclusion is driven substantially by your choice to hardcode "1e20-1e21 FLOPS" specifically, and then to treat this figure as a reasonable proxy for what computation an AGI would need. (That is, you suggest ~~1x as the midpoint for "Compute needed by AGI, relative to... 1e20-1e21 FLOPS").
I think it's also a bit of an issue to call the variable "relative to a human brain (1e20-1e21 FLOPS)". Most users will read it as "relative to a human brain" while it's really "relative to 1e20-1e21 FLOPS", which is quite a specific take on what a human brain is achieving.
I value the fact that you argue for choosing this figure here. However, it seems like you're hardcoding in confidence that isn't warranted. Even from your own perspective, I'd guess that including your uncertainty over this figure would bump up the probability by a factor of 2-3, while it looks l... (read more)
My quick rebuttal is the flaw you seem to also acknowledge. These different factors that you calculate are not separate variables. They all likely influence the probabilities of each other. (greater capabilities can give rise to greater scaling of manufacturing, since people will want more of it. Greater intelligence can find better forms of efficiency, which means cheaper to run, etc.) This is how you can use probabilities to estimate almost anything is extremely improbable, as you noted.
I'd have to think more carefully about the probabilities you came up with and the model for the headline number, but everything else you discuss is pretty consistent with my view. (I also did a PhD in post-silicon computing technology, but unlike Ted I went right into industry R&D afterwards, so I imagine I have a less synoptic view of things like supply chains. I'm a bit more optimistic, apparently—you assign <1% probability to novel computing technologies running global-scale AI by 2043, but I put down a full percent!)
The table "Examples transisto... (read more)
I appreciate the "We avoid derailment by…" sections – I think some forecasts have implicitly overly relied on a "business as usual" frame, and it's worth thinking about derailment.
TSMC is obviously a market leader, but it seems weird to assume that TAI is infeasible without them?... (read more)
One thing I find deeply unconvincing about such a low probability (< 1%) and that does not require expert knowledge is that other ways to slice this would yield much higher estimates.
E.g. it seems difficult to justify a less than 10% probability that there will be really strong pressures to develop AGI and it seems similarly difficult to justify a less than 10% success probability given such an effort and what we now know.
Thanks for posting this, Ted, it’s definitely made me think more about the potential barriers and the proper way to combine probability estimates.
One thing I was hoping you could clarify: In some of your comments and estimates, it seems like you are suggesting that it’s decently plausible(?)[1] we will “have AGI“ by 2043, it’s just that it won’t lead to transformative AGI before 2043 because the progress in robotics, semiconductors, and energy scaling will be too slow by 2043. However, it seems to me that once we have (expensive/physically-limited) AG... (read more)
Your probabilities are not independent, your estimates mostly flow from a world model which seem to me to be flatly and clearly wrong.
The plainest examples seem to be assigning
despite current models learning vastly faster than humans (training time of LLMs is not a human lifetime, and covers vastly more data) and the current nearing AGI and inference being dramatically cheaper and plummeting with algorithmic improvements. There is a general... (read more)
I don't think this is necessarily the right metric, for the same reason that I think the following statement doesn't hold:
Basically, while the contest rules do say, "By 'AGI' we mean something like 'AI that can quickly and affordably be trained to perform nearly all economically and strategically valuable tasks at roughly human cost or less'" they then go on to clarify, "Wha... (read more)
It feels like you're double counting a lot of the categories of derailment at first glance? There's a highly conjunctive story of each of the derailments that makes me suspicious of multiplying them together as if they're conjunctive. I'm also confused as to how you're calculating the disjunctive probabilities because on page 78 you put "Conditional on being on a trajectory to transformative AGI, we forecast a 40% chance of severe war erupting by 2042". However, this doesn't seem to be an argument for derailment, it seems more likely it'd be an argument for race dynamics increasing?
Cool! I mostly like your decomposition/framing. A major nitpick is that robotics doesn't matter so much: dispatching to human actuators is probably cheap and easy, like listing mturk jobs or persuasion/manipulation.
Agreed. AGI can have great influence in the world just by dispatching humans.
But by the definition of transformative AGI that we use - i.e., that AGI is able to do nearly all human jobs - I don't think it's fair to equate "doing a job" with "hiring someone else to do the job." To me, It would be a little silly to say "all human work has been automated" and only mean "the CEO is an AGI, but yeah everyone still has to go to work."
Of course, if you don't think robotics is necessary for transformative AGI, then you are welcome to remove the factor (or equivalently set it to 100%). In that case, our prediction would still be <1%.
While I agree with many of the object-level criticisms of various priors that seem to be out of touch of current state of ML, I would like to instead make precise a certain obvious flaw in the methodology of the paper which was pointed out several times and which you seem to be unjustifiably dismissive of.
tldr: when playing Baysian inference it is crucial to be cognizant that regardless of how certain your priors are the more conditional steps involved in your model the less credence you should give to the overall prediction.
As for the case at hand, it is ... (read more)
Model error higher than 1%?