td;lr 
No control method exists to safely contain the global feedback effects of self-sufficient learning machinery. What if this control problem turns out to be an unsolvable problem?

Where are we two decades into resolving to solve a seemingly impossible problem?

How do you interpret these two quotes, by a founding researcher, fourteen years apart?^[1]

• A. We indeed made comprehensive progress on the AGI control problem, and now at least the overall problem does not seem impossible anymore.
• B. The more we studied the overall problem, the more we uncovered complex sub-problems we'd need to solve as well, but so far can at best find partial solutions to.

Which problems involving physical/information systems were not solved after two decades?

We cannot rely on the notion that if we try long enough, maybe AGI safety turns out possible after all.

Historically, many researchers and engineers tried to solve problems that turned out impossible:

• perpetual motion machines that both conserve and disperse energy.
• uniting general relativity and quantum mechanics into some local variable theory.
• singular methods to 'square the circle', 'double the cube' or 'trisect the angle'.
• distributed data stores where messages of data are consistent in their content, and also continuously available in a network that is also tolerant to partitions.
• formal axiomatic systems that are consistent, complete and decidable.

Smart creative researchers of their generation came up with idealized problems. Problems that, if solved, would transform science, if not humanity. They plowed away at the problem for decades, if not millennia. Until some bright outsider proved by contradiction of the parts that the problem is unsolvable.

Our community is smart and creative – but we cannot just rely on our resolve to align AI. We should never forsake our epistemic rationality, no matter how much something seems the instrumentally rational thing to do.

Nor can we take comfort in the claim by a founder of this field that they still know it to be possible to control AGI to stay safe. 

Thirty years into running a program to secure the foundations of mathematics, David Hilbert declared “We must know. We will know!” By then, Kurt Gödel had constructed the first incompleteness theorem. Hilbert kept his declaration for his gravestone.

Short of securing the foundations of safe AGI control – that is, through empirically-sound formal reasoning – we cannot rely on any researcher's pithy claim that "alignment is possible in principle".

Going by historical cases, this problem could turn out solvable. Just really, really hard to solve. The flying machine seemed an impossible feat of engineering. Next, controlling a rocket’s trajectory to the moon seemed impossible.

By the same reference class, ‘long-term safe AGI’ could turn out unsolvable – the perpetual motion machine of our time. It takes just one researcher to define the problem to be solved, reason from empirically sound premises, and arrive finally at a logical contradiction between the two.^[2]

Can you derive whether a solution exists, without testing in real life?

There is an empirically direct way to know whether AGI would stay safe to humans: 
Build the AGI. Then just keep observing, per generation, whether the people around us are dying.

Unfortunately, we do not have the luxury of experimenting with dangerous autonomous AI systems to see whether they cause human extinction or not. When it comes to extinction, we do not get another chance to test.

Crux:
Even if we could keep testing new conceptualized versions of guess-maybe-safe AGI, is there any essential difference between our epistemic method and that of medieval researchers who kept testing new versions of a perpetual motion machine?

OpenPhil bet tens of millions of dollars on technical research conditional on the positive hypothesis ("a solution exists to the control problem"). Before sinking hundreds of millions more into that bet, would it be prudent to hedge with a few million for investigating the negative hypothesis ("no solution exists")?

Before anyone tries building "safe AGI", we need to know whether any version of AGI – as precisely defined – could be controlled by any method to stay safe.
 
Here is how:

1. Define the concepts of 'control' 'general AI' 'to stay safe' (as soundly corresponding to observations in practice).
2. Specify the logical rules that must hold for such a physical system (categorically, by definition or empirically tested laws).
3. Reason step-by-step to derive whether the logical result of "control AGI" is in contradiction with "to stay safe".

This post defines the three concepts more precisely, and explains some ways you can reason about each. No formal reasoning is included – to keep it brief, and to leave the esoteric analytic language aside for now.

What does it mean to control machinery that learn and operate self-sufficiently?

Recall three concepts we want to define more precisely:

1. 'Control'
2. 'general AI'
3. 'to stay safe'
    

It is common for researchers to have very different conceptions of each term. 
For instance:

1. Is 'control' about:
   1. adjusting the utility function represented inside the machine so it allows itself to be turned off?
   2. correcting machine-propagated side-effects across the outside world?
2. Is 'AGI' about:
   1. any machine capable of making accurate predictions about a variety of complicated systems in the outside world?
   2. any machinery that operates self-sufficiently as an assembly of artificial components that process inputs into outputs, and in aggregate sense and act across many domains/contexts?
3. Is 'stays safe' about:
   1. aligning the AGI’s preferences to not kill us all?
   2. guaranteeing an upper-bound on the chance that AGI in the long term would cause outcomes out of line with a/any condition needed for the continued existence of organic DNA-based life?

To argue rigorously about solvability, we need to:

• Pin down meanings:  
  Disambiguate each term, to not accidentally switch between different meanings in our argument. Eg. distinguish between ‘explicitly optimizes outputs toward not killing us’ and ‘does not cause the deaths of all humans’.
• Define comprehensively:  
  Ensure that each definition covers all the relevant aspects we need to solve for. 
  Eg. what about a machine causing non-monitored side-effects that turn out lethal?
• Define elegantly:  
  Eliminate any defined aspect that we do not yet need to solve for. 
  Eg. we first need to know whether AGI eventually cause the extinction of all humans, before considering ‘alignment with preferences expressed by all humans’.

How to define ‘control’? 

In practice, AGI control necessarily repeats these steps:

1. Sensing inputs through channels connected to any relevant part of the physical environment (including its hardware internals).
2. Modeling the environment based on the channel-received inputs.
3. Simulating effects propagating through the modeled environment.
4. Comparing effects to reference values (to align against) over human-safety-relevant dimensions.
5. Correcting effects counterfactually through outputs to actuators connected to the environment.

Underlying principles:

• Control requires both detection and correction. 
• Control methods are always implemented as a feedback loop.
• Control is exerted by the use of signals (actuation) to conditionalize the directivity and degrees of other signals (effects).

Any method of control is inherently incomplete. In the case of AGI, the question would be whether the degree of control possible is at least greater than the degree of control necessary. 

AGI control signals would be a tiny, tiny subset of all physical signals propagating through the environment, and therefore limited in tracking and conditionalizing the resulting effects. AGI mostly could not even control all local effects of their own components'  physical interactions.

There are fundamental limits to the controllability of AGI. Dozens of impossibility results seem applicable to one or more of the control steps.^[4]

Controllability limits form constraints on the possibility of alignment.
Any alignment method must be implemented as a control loop. AGI cannot comprehensively simulate near term how noise drift in AGI's interactions can amplify (via any available environmental non-linearity, of which in any real-world context, there are many) into much larger changes to conditions. To keep outcomes aligned with values, AGI must continuously sense, update their model of, and correct those changes.

Dense questions for academics:

1. Measurability limits:  
   • To what extent are distributed environmental effects picked up by local sensors?  
   • How much information is lost in compressing high-fidelity sensor inputs to store on hardware?
   • What limits are imposed by noise interference over distances traveled by signals (Shannon Entropy) and smallest-digit/degree detectability (Planck Limit up) – given that smaller-than-measurable local changes can be nonlinearly amplified (Chaos Theory) into a large global divergence of final conditions?
2. Modelability limits:  
   • To what extent is the space of relevant machinery-to-society/ecosystem interactions more functionally complex than modelable within the machinery alone (Good Regulators etc.)? 
   • How can AGI model the future code that AGI will learn based on future unknown inputs?
3. Simulatability limits: 
   • Can future code be simulated, where the actual functioning of that code depends on future unknown inputs?
   • Can behavioral properties of future code be simulated, if no shortcut function is available that reduces total compute time steps, or if any shorter available computation branches would be non-deterministically time-complex (Computational Reducibility)?
4. Commensurability limits:  
   • How would the machinery pick, weigh, and sort effects into relevant comparative dimensions (NP-Complete Knapsack Problem)? 
   • What irreconcilable normative assumptions must be made in setting the reference values to compare against?
5. Actuator feedback limits
   • What is the minimum time delay of digital processing between measurement and correction (Latency)?
   • Can AGI constrain the chaotic loops through itself – of actuated changes to the environment feeding into sensor inputs, that feed into code functioning, that feed back into actuated changes?
   • To what extent can distributed environmental effects be corrected for by local actuators?  

How “complete” does this control need to be?  Be careful to distinguish: ^[5]

• Localized user-initiated control often built into tech products brought to market,
• Comprehensive automated control needed to prevent risks of an auto-scaling/catalyzing technology from materializing globally over the long term.

How to define ‘AGI’?

Distinguish:

• Narrow AI as a model with static code parameters (updated only through human engineers) processing inputs into outputs over a single domain (eg. of image pixels, text tokens).
• General AI as dynamically optimizing configurations encoded into hardware^[6](without needing humans) that process inputs into outputs over multiple domains representing outside contexts.

Corporations are scaling narrow AI model training and deployment toward general AI systems. Current-generation GPT is no longer a narrow AI, given that it processes inputs from the image domain into a language domain. Nor is GPT-4 a general AI. It is in a fuzzy gap between the two concepts.

Corporations already are artificial bodies (‘corpora’ in Latin).

Corporations have been replacing human workers as “functional components” with labor-efficient AI. Standardized hardware components allow AI to outcompete human wetware on physical labor (eg. via electric motors), intellectual labor (faster computation through high-fidelity communication links), and the reproduction of components itself.^[7]

Any corporation or economy that fully automates themself this way – no longer needing humans to maintain their artificial components – over their entire production and operation chains, would in fact be general AI.
 

So to re-define general AI more precisely:

• Self-sufficient
  need no further interactions with humans
  (or lifeforms sharing an ancestor with humans)
  to operate and maintain (and thus produce) 
  their own functional components over time.
• Learning
  optimizing component configurations
  for outcomes that are tracked 
  across multiple domains. 
• Machinery ^[8]
  connected standardized components 
  configured out of artificial (vs. organic 
  DNA-expressed) molecular substrates.

How to define ‘stays safe’?

This is about the introduction of self-sufficient learning machinery, and of all modified versions thereof over time, into the world we humans live in. 

Does this introduction of essentially a new species cause global changes to the world that fall outside the narrow ranges of localized conditions that human bodies need to continue to function and exist?
 

Distinguish:

1. Uncontainability^[9] of unsafe effects:
   That we fundamentally cannot establish, by any means, 
    any sound and valid statistical guarantee that the risk 
    probability that the introduction of AGI into the world 
    causes human-species-wide-lethal outcomes over 
    the long term^[10] is guaranteed to be constrained 
    below some reasonable chance percentage X 
    (as an upper maximum-allowable bound).
2. Convergence on unsafe effects: 
   That the chance that AGI, persisting in some form, 
    causes human-species-wide-lethal outcomes 
    is strictly and asymptotically convergent 
    toward certain over the long term, and 
    that it is strictly impossible for the nature 
    of this trend to be otherwise.

I know of three AGI Safety researchers who have written about specific forms of impossibility reasoning (including Yudkowsky in quote above). Each of their argument forms was about AGI uncontainability, essentially premised on there being fundamental limits to the controllability of AGI component interactions.

By the precautionary principle,^[11]AGI uncontainability should be sufficient reason to never ever get even remotely near to building AGI. Uncontained effects that destabilise conditions outside any of the ranges our human bodies need to survive, would kill us.

But there is an even stronger form of argument:  
Not only would AGI component interactions be uncontainable; they will also necessarily converge on causing the extinction of all humans.

The convergence argument most commonly-discussed is instrumental convergence, about the machinery channelling its optimisation through the intermediate outcomes, explicitly tracked and planned for internally, that tend to result in the machinery being more likely to achieve a variety of (unknown/uncertain) aimed-for outcomes later. 

Instrumental convergence has a mutual-reinforcing complement:  
substrate-needs convergence.

This is not about code components being optimised for explicit goals. Substrate-needs convergence is about all functional components being selected for implicit needs. Components are selected for their potential to bring about environmental conditions/contexts implicitly needed for their continued existence and functioning (at increasing scales, in more ways, in more domains of action).

Any changing population of AGI components converges over time on propagating those specific environmental effects that fulfill their needs. 

All AGI outputs will tend to iteratively select^[12] towards causing those specific effects.
Whatever learned or produced components that across all their physical interactions with connected contexts happen to direct outside effects that feed back into their own maintenance and replication as assembled electro-molecular configurations…do that.^[13]

AGI's artificial configurations differ from human organic configurations, by definition.
What follows is that the environmental conditions and contexts needed to maintain and replicate AGI configurations differ too from what our human bodies need to survive. 

For instance, silicon dioxide (+ many alternate precursors for semiconductor assembly) needs to be heated above 1400 ºC to free outer electrons, and allow the ingot to melt. While production needs extremely high temperatures, computation runs best at extremely low temperatures (to reduce the electrical resistance over conductor wires). 

Humans need around room temperature to survive, at every point of our lifecycle. 
AGI hardware would need, and be robust over, a much wider range of temperatures and pressures than our comparatively fragile human wetware can handle.

Temperature and pressure can be measured and controlled for. That's misleading.
Many other, subtler conditions would be needed by (and selected for in) AGI that lie beyond what the AGI's actual built-in detection and correction methods could control for.^[14] We humans too depend on highly specific environmental conditions for the components nested inside our bodies (proteins→organelles→cells→cell lining→) to continue in their complex functioning, such to be maintaining of our overall existence. 

Between the highly specific set of artificial needs and highly specific set of organic needs, there is mostly non-overlap. AGI cannot control most of the components' iterative effects from converging on their artificial needs, so they do. Their fulfilled artificial needs are disjunctive of our organic needs for survival. So the humans die.

Under runaway feedback, our planetary environment is modified in the directions needed for continued and greater AGI existence. Outside the ranges we can survive.
 

In summary: 

1. Fundamental controllability limits:
   Control methods cannot constrain most environmental effects propagated by the interacting components of “AGI”.  Any built-in method of detecting and correcting effects, or of aligning the external effects with internal reference values, would be insufficient.
2. AGI insufficient controllability:
   A subset of uncontrollable effects will feed back into maintaining or increasing configurations of hardware that propagate those effects. No available internal control feedback loops can correct for all possible external feedback loops.
3. Substrate-needs convergence:
   These environmental effects are needed for those components to come into and stay in existence. But their environmental needs are different from our needs. Their artificial needs are disjunctive off our organic needs for survival. Ie. toxic to human existence. 

AGI would necessarily converge on causing the extinction of all humans.
 

Where from here?

Over two decades, AI Safety founders resolved to solve the control problem, to no avail:

• They reasoned that technological and scientific 'progress' is necessary for optimising the universe – and that continued 'progress' would result in AGI. 
• They wanted to use AGI to reconfigure humanity and colonise reachable galaxies.
• They, and their followers, promoted and financed^[15] development of 'safe AGI'.
• They worried about how companies they helped start up raced to scale ML models.

Now we are here.

• Still working on the technical problem that founders deemed solvable.
• Getting around to the idea that slowing AI development is possible.

In a different world with different founders, would we have diversified our bets more?

• A. Invest in securing the foundations of whatever 'control AGI to stay safe' means?
• B. Invest in deriving, by contradiction of the foundations, that no solution exists?

Would we seek to learn from a researcher claiming they derived that no solution exists?

Would we now?

Acknowledgements: 

Peter S. Park, Kerry Vaughan, and Forrest Landry (my mentor) for the quick feedback.  

For readers' comments, see below and on LessWrong.

1. ^{^}

    Listen to Roman Yampolskiy's answer here. 

2. ^{^}

   An outside researcher could very well have found a logical contradiction in the AGI control problem years ago without your knowing, given the inferential distance. Gödel himself had to construct an entire new language and self-reference methodology for the incompleteness theorems to even work. 

   Historically, an impossibility result that conflicted with the field’s stated aim took years to be verified and accepted by insiders. A field’s founder like Hilbert never came to accept the result. Science advances one funeral at a time.

3. ^{^}

   "Invert, always invert" is a loose translation of the original German ("man muss immer umkehren"). A more accurate literal translation is "man must always turn to the other side".
   
   I first read “invert, always invert" from polymath Charlie Munger:

   The great algebraist, Jacobi, had exactly the same approach as Carson and was known for his constant repetition of one phrase: “Invert, always invert.” It is in the nature of things, as Jacobi knew, that many hard problems are best solved only when they are addressed backward.

   Another great Charlie quote:

   All I want to know is where I’m going to die, so I’ll never go there.

4. ^{^}

   Roman Yampolskiy is offering to give feedback on draft papers written by capable independent scholars, on a specific fundamental limit or no-go theorem described in academic literature that is applicable to AGI controllability. You can pick from dozens of examples from different fields listed here, and email Roman a brief proposal.

5. ^{^}

   To illustrate:  Let’s say before the Wright Brothers built the flying machine, they wondered how to control this introduced technology to stay safe to humans. 

   If they thought like a flight engineer, they would focus on locally measurable effects (eg. actuating wings). They could test whether the risk of a plane crash is below some acceptable upper-bound rate.

   However, the Wright Brothers could not guarantee ahead of time that the introduction of any working plane design, with any built-in control mechanism, that would continue to be produced and modified would stay safe in its effects on society and the ecosystem as a whole (eg. they would not have predicted the deployment of nuclear bombs with planes given the knowledge available at the time). The downstream effects are unmodellable.

   They could check whether the operation (with fossil fuels) and re-production (with toxic chemicals) of their plane in itself has harmful effects. To the extent that harmful conditions are needed for producing and operating the machine, the machine’s existence is inherently unsafe.

   Gradual natural selection can multiply these harms. Over time, any machinery interacting with the outside world in ways that feed back into the re-production of constituent components gets selected for.

   But since planes get produced by humans, humans can select planes on the basis of human needs. Not so with auto-scaling technologies like AGI. 

6. ^{^}

   Non-solid-substrate AGI cannot be ruled out, but seems unlikely initially. Standardisation of isolatable parts is a big advantage, and there is a (temporary) path dependency with current silicon-based semiconductor manufacturing.

7. ^{^}

   Corporations have increasingly been replacing human workers with learning machinery. For example, humans are now getting pushed out of the loop as digital creatives, market makers, dock and warehouse workers, and production workers.

   If this trend continues, humans would have negligible economic value left to add in market transactions of labor (not even for providing needed physical atoms and energy, which would replace human money as the units of trade):

   • As to physical labor: 
   Hardware can actuate power real-time through eg. electric motors, whereas humans are limited by their soft appendages and tools they can wield through those appendages. Semiconductor chips don’t need an oxygenated atmosphere/surrounding solute to operate in and can withstand higher as well as lower pressures. 

   • As to intellectual labor: 
   Silicon-based algorithms can duplicate and disperse code faster (whereas humans face the wetware-to-wetware bandwidth bottleneck). While human skulls do hold brains that are much more energy-efficient at processing information than current silicon chip designs, humans take decades to create new humans with finite skull space. The production of semiconductor circuits for servers as well as distribution of algorithms across those can be rapidly scaled up to convert more energy into computational work. 

   • As to re-production labor: 
   Silicon life have a higher ‘start-up cost’ (vs. carbon lifeforms), a cost currently financed by humans racing to seed the prerequisite infrastructure. But once set up, artificial lifeforms can absorb further resources and expand across physical spaces at much faster rates (without further assistance by humans in their reproduction).

8. ^{^}

   The term "machinery" is more sound here than the singular term "machine".

   Agent unit boundaries that apply to humans would not apply to "AGI". So the distinction between a single agent vs. multiple agents breaks down here.

   Scalable machine learning architectures run on standardized hardware with much lower constraints on the available bandwidth for transmitting, and the fidelity of copying, information across physical distances. This in comparison to the non-standardized wetware of individual humans.

   Given our evolutionary history as a skeleton-and-skin-bounded agentic being, human perception is biased toward ‘agent-as-a-macroscopic-unit’ explanations.

   It is intuitive to view AGI as being a single independently-acting unit that holds discrete capabilities and consistent preferences, rather than viewing agentic being to lie on a continuous distribution. Discussions about single-agent vs. multi-agent scenarios imply that consistent temporally stable boundaries can be drawn.

   A human faces biological constraints that lead them to have a more constant sense of self than an adaptive population of AGI components would have.

   We humans cannot:
   • swap out body parts like robots can.
   • nor scale up our embedded cognition (ie. grow our brain beyond its surrounding skull) like foundational models can.
   • nor communicate messages across large distances (without use of tech and without facing major bandwidth bottlenecks in expressing through our biological interfaces) like remote procedure calls or ML cloud compute can.
   • nor copy over memorized code/information like NN finetuning, software repos, or computer viruses can.

9. ^{^}

   Roman just mentioned that he has used the term 'uncontainable' to mean "cannot confine AGI actions to a box". My new definition for 'uncontainable' differs from the original meaning, so that could confuse others in conversations. Still brainstorming alternative terms that may fit (not 'uncontrainable', not...). Comment if you thought of any alternative term!

10. ^{^}

    In theory, long term here would be modelled as "over infinite time".
    In practice though, the relevant period is "decades to centuries".

11. ^{^}

    Why it makes sense to apply the precautionary principle to the question of whether to introduce new scalable technology into society:
    There are many more ways to break the complex (local-contextualized) functioning of our society and greater ecosystem that we humans depend on to live and live well, than there are ways to foster that life-supporting functioning. 

12. ^{^}

    ‘Iteratively select’ involves lots of subtleties, though most are not essential for reasoning about the control problem. 

    One subtlety is co-option:

    If narrow AI gets developed into AGI, AGI components will replicate in more and more non-trivial ways. Unlike when carbon-based lifeforms started replicating ~3.7 billion years ago, for AGI there would already exist repurposable functions at higher abstraction layers of virtualised code – pre-assembled in the data scraped from human lifeforms with own causal history.

    Analogy to a mind-hijacking parasite:  A rat ingests toxoplasma cells, which then migrate to the rat’s brain. The parasites’ DNA code is expressed as proteins that cause changes to regions of connected neurons (eg. amygdala). These microscopic effects cascade into the rat – while navigating physical spaces – no longer feeling fear when it smells cat pee. Rather, the rat finds the smell appealing and approaches the cat’s pee. Then cat eats the rat and toxoplasma infects its next host over its reproductive cycle.

    So a tiny piece of code shifts a rat’s navigational functions such that the code variant replicates again. Humans are in turn more generally intelligent and capable than a tiny parasitic cell, yet toxoplasma make their way into 30% of the human population. Unbeknownst to cat ‘owners’ infected by toxoplasma gondii, human motivations and motor control get influenced too. Infected humans more frequently lose social relationships, end up in accidents, etc.

    Parasites present real-life examples of tiny pieces of evolutionarily selected-for code spreading and taking over existing functions of vastly more generally intelligent entities. eg. COVID co-opts our lungs’ function to cough. 

    But there is one crucial flaw in this analogy:
    Variants that co-opt initial AGI functions are not parasites. In not threatening the survival nor reproduction of AGI components, they would not be in an adversarial relationship with their host.

    Rather, the humans constraining the reproductive fitness of AGI to gain benefits are, evolutionary speaking, the parasites. The error-corrective (alignment) system we would build in lowers the host’s reproductive fitness. It is like a faulty immune system that kills healthy gut bacteria. It will get selected out.

    As humans, we rely on our evolved immune system to detect and correct out viruses, including for the vaccinations we develop and deploy. We also rely on the blood-testes and blood-follicle barrier to block variants of these viruses from entering into our body’s (re)production facilities.   

    Smaller viruses survive this detection more frequently, so code strands of replicating virus variants are selected for staying small. Few viruses replicate ‘as a package’ with the DNA code of their host. Thus, virus variants rarely co-adapt and integrate with the host’s variants, each selected for raising collective reproductive fitness (except for eg. detection signatures). 

    These systems and barriers add to our reproductive fitness: our ability to preserve and replicate internal code. Past DNA code that got expressed – in interaction with surrounding code – as (precursors of) these functions got selected for as a portion of the existing human ‘code pool’.

    For any organic system or barrier preventing virus variants from replicating through our bodies, evolution is firmly on our side. For any artificial system or barrier we build to prevent misaligned AI variants from replicating through solid-state hardware infrastructure, evolution will thwart our efforts. 

    AI variants would compete for hardware/resources like computer viruses do. And co-adapt and integrate with other internal variants to replicate as part of larger symbiotic packages.

13. ^{^}

    Now, a thought may come to your mind – roughly like:
    "That sounds like evolutionary selection; but isn't evolution a slow and local optimiser?".

    Yes, is the answer. This does involve evolutionary selection.
    Unfortunately, by Forrest Landry's estimate, the selection that would take place through components of self-sufficient learning machinery would take < 500 years to cause ecosystem-wide extinction. This compared to the 3.7 billion years from the origin of carbon lifeforms to us humans starting to cause a mass extinction.

    Reasons include:

    • Pre-assembled functions: 
    First solid-state lifeforms can co-opt/repurpose pre-assembled AI functions and infrastructure (vs. first carbon-based lifeforms that started from scratch). 
    • Standardization: 
    The efficiency gains of the virtualisation of code’s storage, computation and transmission – vastly reducing how much atoms need to be moved about and molecularly reconfigured. Think of how fast memes spread through society – even while still requiring lots of atoms to jiggle across neurons in our brains.
    • Faster reproduction:
    Reproduce hardware components in days to months, versus humans who take decades to reproduce as physical units.
    • The terraforming gap: 
    A much larger gap between the current state of planet Earth and the conditions that self-sufficient self-assembling learning machinery need and would therefore modify the environment toward (versus gap to conditions needed by humans and other species living in carbon-based ecosystem). 
    ~ ~ ~
    Another argument you may have heard is that the top-down intelligent engineering by goal-directed AGI would beat the bottom-up selection happening through this intelligent machinery.

    That argument can be traced back to Eliezer Yudkowsky's sequence The Simple Math of Evolution. Unfortunately, there were mistakes in Eliezer's posts, some of which a modern evolutionary biologist may have been able to correct:

    • implying that sound comparisons can be made between the reproductive fitness of organisms, as somehow independent of unknown changes in environmental context (eg. a black swan event of a once-in-200 years drought that kills the entire population, except a few members who by previous derivable standards would have been relatively low fitness).
    •  overlooking the ways that information can be stored within the fuzzy regions of phenotypic effects maintained outside respective organisms.
    • overlooking the role of transmission speed-up for virtualisation of code.
    • overlooking the tight coupling in AGI between the intrinsic learning/selection of code, and extrinsic selection of that code through differentiated rates of replication through the environment.
    • overlooking the role of exaptation/co-option. 

14. ^{^}

    Worse, since error correction methods would correct out component variants with detectable unsafe/co-optive effects, this leaves to grow in influence any replicating branches of variants with undetectable unsafe/co-optive effects. 

    Thus, the error correction methods select for the variants that can escape detection. As do meta-methods (having to soundly and comprehensively adapt error correction to newly learned code or newly produced hardware parts).

15. ^{^}

    See:
    • Tallinn's seed grant to DeepMind.
    • OpenPhil's $30M grant to OpenAI.
    • FTX's $500M grant (+ Tallinn's + Moskovitz' grant) to Anthropic.