Disclaimer

Written quickly^[1]. It's better to draft my objections poorly, than to not draft them at all.

Introduction

I am sceptical that "foom"^[2] is some of not physically possible/feasible/economically viable.
[Not sure yet what level of scepticism I endorse.]

I have a few object level beliefs that bear on it. I'll try and express them succinctly below (there's a summary at the end of the post for those pressed for time).

Note that my objections to foom are more disjunctive than they are conjuctive. Each is independently a reason why foom looks less likely to me.

Beliefs

I currently believe/expect the following to a sufficient degree that they inform my position on foom.

Diminishing Marginal Returns

1.0. Marginal returns to cognitive investment (e.g. compute) decay at a superlinear rate (e.g. exponential) across some relevant cognitive domains (e.g. some of near human, human spectrum, superhuman, strongly superhuman).

1.1. Marginal returns to real world capabilities from cognitive amplification likewise decay at a superlinear rate across relevant cognitive domains.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

Broad Human Cognitive Spectrum

2. The human cognitive spectrum (1st percentile human to peak human) is broad in an absolute sense.

On many useful cognitive tasks(chess, theoretical research, invention, mathematics, etc.), beginner/dumb/unskilled humans are closer to a chimpanzee/rock than peak humans (for some fields, only a small minority of humans are able to perform the task at all, or perform the task in a useful manner^[3], for other like chess, beginners are simply closer to the lowest attainable scores than to the scores obtained by peak humans [600 - 800 is a lot closer to 0 than to 2700 - 2900]).

Median humans are probably also closer to a rock than to peak humans (on e.g. inventing general relativity pre 1920).

Peak humans may be closer to bounded superintelligences than beginner/median humans.

E.g. Magnus Carlsen is closer in ELO to Stockfish than median human.

I expect Magnus Carlsen to be closer in ELO to a bounded superintelligence than to a median human.

Narrow Optimisers Outperform General Optimisers on Narrow Domains

3.0. I believe that for similar levels of cognitive investment narrow optimisers outperform general optimisers on narrow domains.

This is because they are not constrained by the pareto frontier across many domains and are more able to pursue the optimum in their narrow domains.

I expect this to translate to many narrow domains (I wouldn't be surprised if we get superhuman language performance without "dangerously capable" systems [we got superhuman art without dangerously capable systems".].

E.g. future LLMs may be able to write very compelling ("bestseller" status) long form fiction in an hour.)

I expect a superintelligence to not win against dedicated chess/Go bots with comparable cognitive endowments (compute budgets, comparably efficient cognitive algorithms/architectures).

"Not win" is too conservative: I expect the ASI to lose unless it adopts the strategy of just running the bot (or depending on the level of superhuman, it might be able to force a tie). I simply do not think a general optimiser (no matter how capable) with comparable cognitive endowment can beat a narrow optimiser at their own game. Optimisation across more domains constrains the attainable optimum in any domain; the pareto frontier is an absolute limit.

I wouldn't be surprised if this generalises somewhat beyond Go.

Are narrow AI superhuman real world strategists viable?

The answer is not obviously "no" to me.

3.1. I believe that general intelligence is not compact.

Deployment Expectations and Strategic Conditions

4.0. I expect continuous progress in cognitive capabilities for several years/decades more.

There may be some paradigm shifts/discontinuous jumps, but I expect that the world would have already been radically transformed when superhuman agents arrive.
 

4.1. I expect it to be much more difficult for any single agent to attain decisive cognitive superiority to civilisation, or to a relevant subset of civilisation.

Especially given 3.

Superhuman agents may not be that much more capable than superhuman narrow AI amplified humans.

4.2. Specifically, I expect a multipolar world in which many actors have a suite of superhuman narrow AIs that make them "dangerously capable" relative to 2020s earth, but not relative to their current time (I expect the actors to be in some sort of equilibrium).

I'm not convinced the arrival of superhuman agents in such a world would necessarily shatter such an equilibrium.

Or be unilateral "existentially dangerous" relative to said world.

Hence, I expect failure to materialise as dystopia not extinction.

"Superintelligence" is a High Bar

5. "Superintelligence" requires a "very high" level of strongly superhuman cognitive capabilities

Reasons:

• Arguments $\#1$ through $\#4$
• Attaining decisive strategic advantage seems difficult. 
  • E.g. I doubt:
    • A +12 SD human could do so during most of human history
    • Human intelligence in a chimpanzee body easily takes over a chimpanzee tribe

My intuition is that the level of cognitive power required to achieve absolute strategic dominance is crazily high.

And it's a moving target that would rise with the extant effective level of civilisation.

Summary

Courtesy of chatGPT:

The author presents several objections to the idea of a rapid, exponential increase in AI capabilities known as an "intelligence explosion" or "foom". The objections include the belief that marginal returns to cognitive investment decay at a superlinear rate, that narrow optimizers outperform general optimizers on narrow domains, and that it will be difficult for a single agent to attain decisive cognitive superiority over civilization. The author also believes that the arrival of superhuman agents in a world with multiple actors possessing superhuman narrow AI will not necessarily shatter the existing equilibrium.

1. ^{^}

   Half an hour to touch up a stream of consciousness Twitter thread I wrote yesterday.

2. ^{^}

   An "intelligence explosion" scenario where there's a very short time period where AI systems rapidly grow in intelligence until their cognitive capabilities far exceed humanity's.

3. ^{^}

   E.g. inventing the dominant paradigm in a hard science seems beyond the ability of most humans. I'm under the impression that pre 1920 < 1,000 (and plausibly < a 100) people could have invented general relativity.

   Some have claimed that without Einstein we may not have gotten general relativity for decades.