AGI and Lock-In

Lukas Finnveden; Jess_Riedel; CarlShulman

AGI and Lock-In

Lukas Finnveden,

Comments 20

Sorted by

New & upvoted

Lizka

I'm curating this (although I wish it had a more skimmable summary).

It's an important topic (and a weak point in the classic most important century discussion) and a lot of the considerations^[1] seem important and new (at least to me!). I like that the post and document make a serious attempt at clarifying what isn't being said (like some claims about likelihood), flag different levels of uncertainty in the various claims, and clarify what is meant by "AGI"^[2].

Here's a quick attempt at a restructured/slightly paraphrased summary — please correct me if I got something wrong:

Assuming AGI, it's relatively possible^[3] to stabilize/lock in many features of society — both good and bad — for a long time (millions or trillions of years). This is because:
- AGIs can be faithful to a specific goal or a set of goals for a long time
- with sufficient resources, institutions can be created that will pursue these values until an external source (like foreign intervention, the death of an authoritarian leader, or internal rebellion) stops them
- current economic and military powers could come together and use AGI to make an institution of this kind, which would be able to defend itself against external sources
  - Meaning that such an institution could pursue its agenda for millions or trillions of years.
Why this matters:
- Stabilizing features like this can be bad (an existential risk) if their values or goals are poorly chosen or insufficiently flexible.
- Stable institutions could be important to ensuring good values do persist.
- The feasibility of the above is evidence that "significant influence over the long-run future is possible."

^{^}
e.g. what kinds of information/values could be stored for a long time and how
^{^}
See here.
^{^}
For the different levels of confidence the authors have in these arguments, you can look at this section in the document.

Lukas Finnveden

Thanks Lizka. I think about section 0.0 as being a ~1-page summary (in between the 1-paragraph summary and the 6-page summary) but I could have better flagged that it can be read that way. And your bullet point summary is definitely even punchier.

Wei Dai

Consider a civilization that has "locked in" the value of hedonistic utilitarianism. Subsequently some AI in this civilization discovers what appears to be a convincing argument for a new, more optimal design of hedonium, which purports to be 2x more efficient at generating hedons per unit of resources consumed. Except that this argument actually exploits a flaw in the reasoning processes of the AI (which is widespread in this civilization) such that the new design is actually optimized for something different from what was intended when the "lock in" happened. The closest this post comes to addressing this scenario seems to be "An extreme version of this would be to prevent all reasoning that could plausibly lead to value-drift, halting progress in philosophy." But even if a civilization was willing to take this extreme step, I'm not sure how you'd design a filter that could reliably detect and block all "reasoning" that might exploit some flaw in your reasoning process.

Maybe in order to prevent this, the civilization tried to locked in "maximize the quantity of this specific design of hedonium" as their goal instead of hedonistic utilitarianism in the abstract. But 1) maybe the original design of hedonium is already flawed or highly suboptimal, and 2) what if (as an example) some AI discovers an argument that they should engage in acausal trade in order to maximize the quantity of hedonium in the multiverse, except that this argument is actually wrong.

This is related to the problem of metaphilosophy, and my hope that we can one day understand "correct reasoning" well enough to design AIs that we can be confident are free from flaws like these, but I don't know how to argue that this is actually feasible.

Lukas Finnveden

I broadly agree with this. For the civilizations that want to keep thinking about their values or the philosophically tricky parts of their strategy, there will be an open question about how convergent/correct their thinking process is (although there's lots you can do to make it more convergent/correct — eg. redo it under lots of different conditions, have arguments be reviewed by many different people/AIs, etc).

And it does seem like all reasonable civilizations should want to do some thinking like this. For those civilizations, this post is just saying that other sources of instability could be removed (if they so chose, and insofar as that was compatible with the intended thinking process).

Also, separately, my best guess is that competent civilizations (whatever that means) that were aiming for correctness would probably succeed (at least in areas were correctness is well defined). Maybe by solving metaphilosophy and doing that, maybe because they took lots of precautions like mentioned above, maybe just because it's hard to get permanently stuck at incorrect beliefs if lots of people are dedicated to getting things right, have all the time and resources in the world, and are really open-minded. (If they're not open-minded but feel strongly attached to keeping their current views, then I become more pessimistic.)

But even if a civilization was willing to take this extreme step, I'm not sure how you'd design a filter that could reliably detect and block all "reasoning" that might exploit some flaw in your reasoning process.

By being unreasonably conservative. Most AIs could be tasked with narrowly doing their job, a few with pushing forward technology/engineering, none with doing anything that looks suspiciously like ethics/philosophy. (This seems like a bad idea.)

Jess_Riedel

Just to be clear: we mostly don’t argue for the desirability or likelihood of lock-in, just its technological feasibility. Am I correctly interpreting your comment to be cautionary, questioning the desirability of lock-in given the apparent difficulty of doing so while maintaining sufficiently flexibility to handle unforeseen philosophical arguments?

Wei Dai

To take a step back, I'm not sure it makes sense to talk about "technological feasibility" of lock-in, as opposed to say its expected cost, because suppose the only feasible method of lock-in causes you to lose 99% of the potential value of the universe, that seems like a more important piece of information than "it's technologically feasible".

(On second thought, maybe I'm being unfair in this criticism, because feasibility of lock-in is already pretty clear to me, at least if one is willing to assume extreme costs, so I'm more interested in the question of "but can it be done at more acceptable costs", but perhaps this isn't true of others.)

That aside, I guess I'm trying to understand what you're envisioning when you say "An extreme version of this would be to prevent all reasoning that could plausibly lead to value-drift, halting progress in philosophy." What kind of mechanism do you have in mind for doing this? Also, you distinguish between stopping philosophical progress vs stopping technological progress, but since technological progress often requires solving philosophical questions (e.g., related to how to safely use the new technology), do you really see much distinction between the two?

trammell

Thanks, great post!

You say that "using digital error correction, it would be extremely unlikely that errors would be introduced even across millions or billions of years. (See section 4.2.) " But that's not entirely obvious to me from section 4.2. I understand that error correction is qualitatively very efficient, as you say, in that the probability of an error being introduced per unit time can be made as low as you like at the cost of only making the string of bits a certain small-seeming multiple longer (and my understanding is that multiple shrinks the longer the original string was?). But for any multiple, there's some period of time long enough that the probability of faithfully maintaining some string of bits for that long is low. Is there any chance you could offer an estimate of, say, how much longer you'd have to make a petabyte in order to get the probability of an error over a billion years below 1%?

Lukas Finnveden

This is a great question. I think the answer depends on the type of storage you're doing.

If you have a totally static lump of data that you want to encode in a harddrive and not touch for a billion years, I think the challenge is mostly in designing a type of storage unit that won't age. Digital error correction won't help if your whole magnetism-based harddrive loses its magnetism. I'm not sure how hard this is.

But I think more realistically, you want to use a type of hardware that you regularly use, regularly service, and where you can copy the information to a new harddrive when one is about to fail. So I'll answer the question in that context.

As an error rate, let's use the failure rate of 3.7e-9 per byte per month ~= 1.5e-11 per bit per day from this stack overflow reply. (It's for RAM, which I think is more volatile than e.g. SSD storage, and certainly not optimised for stability, so you could probably get that down a lot.)

Let's use the following as an error correction method: Each bit is represented by N bits; for any computation the computer does, it will use the majority vote of the N bits; and once per day,^[1] each bit is reset to the majority vote of its group of bits.

If so...

for N=1, the probability that a bit is stable for 1e9 years is ~exp(-1.5e-11*365*1e9)=0.4%. Yikes!
for N=3, the probability that 2 bit flips happen in a single day is ~3*(1.5e-11)^2 and so the probability that a group of bits is stable for 1e9 years is ~exp(-3*(1.5e-11)^2*365*1e9)=1-2e-10. Much better, but there will probably still be a million errors in that petabyte of data.
for N=5, the probability that 3 bit flips happen in a single day is ~(5 choose 2)*(1.5e-11)^3 and so the probability that the whole petabyte of data is safe for 1e9 years is ~99.99%. And so on this scheme, it seems that 5 petabytes of storage is enough to make 1 petabyte stable for a billion years.

Based on the discussion here, I think the errors in doing the majority-voting calculations are negligible compared to the cosmic ray calculations. At least if you do it cleverly so that you don't get too many correlations and ruin your redundance (which there are ways to do according to results on error correcting computations — though I'm not sure if they might require some fixed amount of extra storage space to do this, in which case you might need N somewhat greater than 5).

Now this scheme requires that you have a functioning civilization that can provide electricity for the computer, that can replace the hardware when it starts failing, and stuff — but that's all things that we wanted to have anyway. And any essential component of that civilization can run on similarly error-corrected hardware.

And to account for larger-scale problems than cosmic rays (e.g. local earthquake throws harddrive to the ground and shatters it, or you accidentally erase a file when you were supposed to make a copy of it), you'd probably want backup copies of the petabyte on different places across the Earth, which you replaced each time something happened to one of them. If there's an 0.1% chance of that happening in any one day (corresponding to once/3 years, which seems like an overestimate if you're careful), and you immediately notice it and replace the copy within a day, and you have 5 copies in total, the probability that one of them keeps working at all times is ~exp(-(0.001)^5*365*1e9)~=99.96%. So combined with the previous 5, that'd be a multiple of 5*5=25.

This felt enlightening. I'll add a link to this comment from the doc.

^{^}
Using a day here rather than an hour or a month isn't super-motivated. If you reset things very frequently, you might interfere with normal use of the computer, and errors in the resetting-operation might start to dominate the errors from cosmic rays. But I think a day should be above the threshold where that's much of an issue.

trammell

Cool, thanks for thinking this through!

This is super speculative of course, but if the future involves competition between different civilizations / value systems, do you think having to devote say 96% (i.e. 24/25) of a civilization's storage capacity to redundancy would significantly weaken its fitness? I guess it would depend on what fraction of total resources are spent on information storage...?

Also, by the same token, even if there is a "singleton" at some relatively early time, mightn't it prefer to take on a non-negligible risk of value drift later in time if it means being able to, say, 10x its effective storage capacity in the meantime?

(I know your 24/25 was a conservative estimate in some ways; on the other hand it only addresses the first billion years, which is arguably only a small fraction of the possible future, so hopefully it's not too biased a number to anchor on!)

Lukas Finnveden

Depends on how much of their data they'd have to back up like this. If every bit ever produced or operated on instead had to be be 25 bits — that seems like a big fitness hit. But if they're only this paranoid about a few crucial files (e.g. the minds of a few decision-makers), then that's cheap.

And there's another question about how much stability contributes to fitness. In humans, cancer tends to not be great for fitness. Analogously, it's possible that most random errors in future civilizations would look less like slowly corrupting values and more like a coordinated whole splintering into squabbling factions that can easily be conquered by a unified enemy. If so, you might think that an institution that cared about stopping value-drift and an institution that didn't would both have a similarly large interest in preventing random errors.

Also, by the same token, even if there is a "singleton" at some relatively early time, mightn't it prefer to take on a non-negligible risk of value drift later in time if it means being able to, say, 10x its effective storage capacity in the meantime?

The counter-argument is that it will be super rich regardless, so it seems like satiable value systems would be happy to spend a lot on preventing really bad events from happening with small probability. Whereas instabiable value systems would notice that most resources are in the cosmos, and so also be obsessed with avoiding unwanted value drift. But yeah, if the values contain a pure time preference, and/or doesn't care that much about the most probable types of value drift, then it's possible that they wouldn't deem the investment worth it.

MichaelPlant

Just skimmed this, but I notice there seems to be something inconsistent between this and the usual AI dooomerism stuff. For instance, above you claim that we should be worried about values lock-in because we will be able to align AI - cf doomerism that says alignment won't work; equally, above you state the value drift could be prevented by 'turning the AGI off and on again' - which is, again, at odds with the doomerist claim that we can't do this. I'm unsure what to make of this tension.

Lukas Finnveden

Quoting from the post:

Thus, we suspect that an adequate solution to AI alignment could be achieved given sufficient time and effort. (Though whether that will actually happen is a different question, not addressed since our focus is on feasibility rather than likelihood.)

AI doomers tend to agree with this claim. See e.g. Eliezer in list of lethalities:

None of this is about anything being impossible in principle. The metaphor I usually use is that if a textbook from one hundred years in the future fell into our hands, containing all of the simple ideas that actually work robustly in practice, we could probably build an aligned superintelligence in six months. (...) What's lethal is that we do not have the Textbook From The Future telling us all the simple solutions that actually in real life just work and are robust; we're going to be doing everything with metaphorical sigmoids on the first critical try. No difficulty discussed here about AGI alignment is claimed by me to be impossible - to merely human science and engineering, let alone in principle - if we had 100 years to solve it using unlimited retries, the way that science usually has an unbounded time budget and unlimited retries. This list of lethalities is about things we are not on course to solve in practice in time on the first critical try; none of it is meant to make a much stronger claim about things that are impossible in principle.

N N

Stipulate, for the sake of the argument, that Lukas et al. actually disagree with the doomers about various points. What would follow from that?

Lukas Finnveden

For instance we might get WBEs only in hypothetical-2080 but get superintelligent LLMs in 2040, and the people using superintelligent LLMs make the world unrecognisably different by 2042 itself.

I definitely don't just want to talk about what happens / what's feasible before the world becomes unrecognisably different. It seems pretty likely to me that lock-in will only become feasible after the world has become extremely strange. (Though this depends a bit on details of how to define "feasible", and what we count as the start-date of lock-in.)

And I think that advanced civilizations that tried could eventually become very knowledgable about how to create AI with a wide variety of properties, which is why I feel ok with the assumption that AIs could be made similar to humans in some ways without being WBEs.

(In particular, the arguments in this document are not novel suggestions for how to succeed with alignment in a realistic scenario with limited time! That still seems like a hard problem! C.f. my response to Michael Plant.)

Lukas Finnveden

Chaos theory is about systems where tiny deviations in initial conditions cause large deviations in what happens in the future. My impression (though I don't know much about the field) is that, assuming some model of a system (e.g. the weather), you can prove things about how far ahead you can predict the system given some uncertainty (normally about the initial conditions, though uncertainty brought about by limited compute that forces approximations should work similarly). Whether the weather corresponds to any particular model isn't really susceptible to proofs, but that question can be tackled by normal science.

Lukas Finnveden

Yeah, I agree that multipolar dynamics could prevent lock-in from happening in practice.

I do think that "there is a non-trivial probability that a dominant institution will in fact exist", and also that there's a non-trivial probability that a multipolar scenario will either

(i) end via all relevant actors agreeing to set-up some stable compromise institution(s), or
(ii) itself end up being stable via each actor making themselves stable and their future interactions being very predictable. (E.g. because of an offence-defence balance strongly favoring defence.)

...but arguing for that isn't really a focus of the doc.

(And also, a large part of why I believe they might happen is that they sound plausible enough, and I haven't heard great arguments for why we should be confident in some particular alternative. Which is a bit hard to forcefully argue for.)

Lukas Finnveden

If re-running evolution requires simulating the weather and if this is computationally too difficult then re-running evolution may not be a viable path to AGI.

There are many things that prevent us from literally rerunning human evolution. The evolution anchor is not a proof that we could do exactly what evolution did, but instead an argument that if something as inefficient as evolution spit out human intelligence with that amount of compute, surely humanity could do it if we had a similar amount of compute. Evolution is very inefficient — it has itself been far less optimized than the creatures it produces.

(I'd have more specific objections to the idea that chaos-theory-in-weather in particular would be an issue: I think that a weather-distribution approximated with a different random generation procedure would be as likely to produce human intelligence as a weather distribution generated by Earth's precise chaotic behavior. But that's not very relevant, because there would be far bigger differences between Earthly evolution and what-humans-would-do-with-1e40-FLOP than the weather.)

Jim Buhler

Insightful! Thanks for writing this.

> Perhaps it will be possible to design AGI systems with goals that are cleanly separated from the rest of their cognition (e.g. as an explicit utility function), such that learning new facts and heuristics doesn’t change the systems’ values.

In that case, value lock-in is the default (unless corrigibility/uncertainty is somehow part of what the AGI values), such that there's no need for the "stable institution" you keep mentioning, right?

> But the one example of general intelligence we have — humans — instead seem to store their values as a distributed combination of many heuristics, intuitions, and patterns of thought. If the same is true for AGI, it is hard to be confident that new experiences would not occasionally cause their values to shift.

Therefore, it seems to me that most of your doc assumes we're in this scenario? Is that the case? Did I widely misunderstand something?

Lukas Finnveden

If AGI systems had goals that were cleanly separated from the rest of their cognition, such that they could learn and self-improve without risking any value drift (as long as the values-file wasn't modified), then there's a straightforward argument that you could stabilise and preserve that system's goals by just storing the values-file with enough redundancy and digital error correction.

So this would make section 6 mostly irrelevant. But I think most other sections remain relevant, insofar as people weren't already convinced that being able to build stable AGI systems would enable world-wide lock-in.

Therefore, it seems to me that most of your doc assumes we're in this scenario [without clean separation between values and other parts]?

I was mostly imagining this scenario as I was writing, so when relevant, examples/terminology/arguments will be taylored for that, yeah.

Lukas Finnveden

Thanks!

You've assumed from the get go that AIs will follow similar reinforcement-learning like paradigms like humans and converge on similar ontologies of looking at the world as humans. You've also assumed these ontologies will be stable - for instance a RL agent wouldn't become superintelligent, use reasoning and then decide to self modify into something that is not an RL agent.

Something like that, though I would phrase it as relying on the claim that it's feasible to build AI systems like that, since the piece is about the feasibility of lock-in. And in that context, the claim seems pretty safe to me. (Largely because we know that humans exist.)

You've assumed laws of physics as we know them today are constraints on things like computation and space colonization and oversight and alignment processes for other AIs.

Yup, sounds right.

Does this assume a clean separation between two kinds of processes - those that can be predicted and those that can't?

That's a good question. I wouldn't be shocked if something like this was roughly right, even if it's not exactly right. Let's imagine the situation from the post, where we have an intelligent observer with some large amount of compute that gets to see the paths of lots of other civilizations built by evolved species. Now let's imagine a graph where the x-axis has some increasing combination of "compute" and "number of previous examples seen", and the y-axis has something like "ability to predict important events". At first, the y-value would probably go up pretty fast with greater x, as the observer get a better sense of what the distribution of outcomes are. But on our understanding of chaos theory, it's ability to predict e.g. the weather years in advance would be limited even at astoundingly large values of compute+knowledge of what the distribution is like. And since chaotic processes affect important real-world events in various ways (e.g. the genes of new humans seem similarly random as the weather, and that has huge effects), it seems plausible that our imagined graph would asymptote towards some limit of what's predictable.

And that's not even bringing up fundamental quantum effects, which are fundamentally unpredictable from our perspective. (With a many-worlds interpretation, they might be predictable in the sense that all of them will happen. But that still lets us make interesting claims about "fractions of everett branches", which seems pretty interchangeable with "probabilities of events".)

In any case, I don't think this impinges much on the main claims in the doc. (Though if I was convinced that the picture above was wildly wrong, I might want to give a bit of extra thought to what's the most convenient definition of lock-in.)

Comments

AGI and Lock-In

AGI and Lock-In

0.0 The claim

0.1 Preserving information

0.2 Executing intentions

0.3 Preventing disruption

0.4 Some things we don’t argue for

0.5 Structure of the document

Contributions