Holden Karnofsky

Wiki Contributions


Forecasting transformative AI: what's the burden of proof?

Most of your post seems to be arguing that current economic trends don't suggest a coming growth explosion.

If current economic trends were all the information I had, I would think a growth explosion this century is <<50% likely (maybe 5-10%?) My main reason for a higher probability is AI-specific analysis (covered in future posts).

This post is arguing not "Current economic trends suggest a growth explosion is near" but rather "A growth explosion is plausible enough (and not strongly enough contraindicated by current economic trends) that we shouldn't too heavily discount separate estimates implying that transformative AI will be developed in the coming decades." I mostly don't see the arguments in the piece you linked as providing a strong counter to this claim, but if you highlight which you think provide the strongest counters, I can consider more closely.

The one that seems initially like the best candidate for such an argument is "Many of our technologies cannot get orders of magnitude more efficient." But I'm not arguing that e.g. particular energy technologies will get orders of magnitude more efficient; I'm arguing we'll see enough acceleration to be able to quickly develop something as transformative as digital people. There may be an argument that this isn't possible due to key bottlenecks being near their efficiency limits, but I don't think the case in your piece is at that level of specificity.

Forecasting transformative AI: what's the burden of proof?

Thanks! This post is using experimental formatting so I can't fix this myself, but hopefully it will be fixed soon.

Forecasting transformative AI: what's the burden of proof?

Agreed. This is similar in spirit to the "My cause is most important" part.

Forecasting transformative AI: what's the burden of proof?

It seems to me like "transformative AI is coming this century" and "this century is the most important century" are very different claims which you tend to conflate in this sequence.

I agree they're different claims; I've tried not to conflate them. For example, in this section I give different probabilities for transformative AI and two different interpretations of "most important century."

This post contains a few cases where I think the situation is somewhat confusing, because there are "burden of proof" arguments that take the basic form, "If this type of AI is developed, that will make it likely that it's the most important century; there's a burden of proof on arguing that it's the most important century because ___." So that does lead to some cases where I am defending "most important century" within a post on AI timelines.

More generally, I think that claims which depend on the specifics of our long-term trajectory after transformative AI are much easier to dismiss as being speculative (especially given how much pushback claims about reaching TAI already receive for being speculative). So I'd much rather people focus on the claim that "AI will be really, really big" than "AI will be bigger than anything else which comes afterwards". But it seems like framing this sequence of posts as the "most important century" sequence pushes towards the latter.

I struggled a bit with this; you might find this page helpful, especially the final section, "Holistic intent of the 'most important century' phrase." I ultimately decided that relative to where most readers are by default, "most important century" is conveying a more accurate high-level message than something like "extraordinarily important century" - the latter simply does not get across the strength of the claim - even though it's true that "most important century" could end up being false while the overall spirit of the series (that this is a massively high-stakes situation) ends up being right.

I also think it's the case that the odds of "most important century" being literally true are still decently high (though substantially lower than "transformative AI this century"). A key intuition behind this claim is the idea that PASTA could radically speed things up, such that this century ends up containing as much eventfulness as we're used to from many centuries. (Some more along these lines in the section starting "To put this possibility in perspective, it's worth noting that the world seems to have 'sped up'" from the page linked above.)

Oh, also, depending on how you define "important", it may be the case that past centuries were more important because they contained the best opportunities to influence TAI - e.g. when the west became dominant, or during WW1 and WW2, or the cold war. Again, that's not very action-guiding, but it does make the "most important century" claim even more speculative.

I address this briefly in footnote 1 on the page linked above: "You could say that actions of past centuries also have had ripple effects that will influence this future. But I'd reply that the effects of these actions were highly chaotic and unpredictable, compared to the effects of actions closer-in-time to the point where the transition occurs."

This Can't Go On

Thanks for all the thoughts on this point! I don't think the comparison to currency is fair (the size of today's economy is a real quantity, not a nominal one), but I agree with William Kiely that the "several economies per atom" point is best understood as an intuition pump rather than an airtight argument. I'm going to put a little thought into whether there might be other ways of communicating how astronomically huge some of these numbers are, and how odd it would be to expect 2% annual growth to take us there and beyond.

One thought: it is possible that there's some hypothetical virtual world (or other configuration of atoms) with astronomical value compared to today's economy. But if so, getting to that probably involves some sort of extreme control and understanding of our environments, such as what might be possible with digital people. And I'd expect the path to such a thing to look more like "At some point we figure out how to essentially escape physical constraints and design an optimal state [e.g., via digital people], causing a spike (not necessarily instantaneous, but quite quick) in the size of the economy" than like "We get from here to there at 2% growth per year."

Digital People Would Be An Even Bigger Deal

I think this depends on empirical questions about the returns to more compute for a single mind. If the mind is closely based on a human brain, it might be pretty hard to get much out of more compute, so duplication might have better returns. If the mind is not based on a human brain, it seems hard to say how this shakes out.

All Possible Views About Humanity's Future Are Wild

I'm not sure I'm fully following, but I think the "almost exactly the same time" point is key (and I was getting at something similar with "However, note that this doesn't seem to have happened in ~13.77 billion years so far since the universe began, and according to the above sections, there's only about 1.5 billion years left for it to happen before we spread throughout the galaxy"). The other thing is that I'm not sure the "observation selection effect" does much to make this less "wild": anthropically, it seems much more likely that we'd be in a later-in-time, higher-population civilization than an early-in-time, low-population one.

Digital People FAQ

If we have advanced AI that is capable of constructing a digital human simulation, wouldn't it also by proxy be advanced enough to be conscious on its own, without the need for anything approximating human beings? I can imagine humans wanting to create copies of themselves for various purposes but isn't it much more likely for completely artificial silicon-first entities to take over the galaxy? Those entities wouldn't have the need for any human pleasures and could thus conquer the universe much more efficiently than any "digital humans" ever could.

It does seem likely to me that advanced AI would have the capabilities needed to spread through the galaxy on its own. Where digital people might come in is that - if advanced AI systems remain "aligned" / under human control - digital people may be important for steering the construction of a galaxy-wide civilization according to human-like (or descended-from-human-like) values. It may therefore be important for digital people to remain "in charge" and to do a lot of work on things like reflecting on values, negotiating with each other, designing and supervising AI systems, etc.

If we get to a point where "digital people" are possible, can we expect to be able to tweak the underlying circuitry to eliminate the concept of pain and suffering altogether, creating "humans" incapable of experiencing anything but joy, no matter what happens to them? Its really hard to imagine from a biological human perspective but anything is possible in a digital world and this wouldn't necessarily make these "humans" any less productive.

"Tweaking the underlying circuitry" wouldn't automatically be possible just as a consequence of being able to simulate human minds. But I'd guess the ability to do this sort of tweak would follow pretty quickly.

As a corollary, do we have a reason to believe that "digital humans" will want to experience anything other than 24/7 heroin-like euphoria in their "down time", rather than complex experiences like zero-g? Real-life humans cannot do that as our bodies quickly break down from heroin exposure, but digital ones won't have such arbitrary limitations.

I think a number of people (including myself) would hesitate to experience "24/7 heroin-like euphoria" and might opt for something else.

Load More