This seems like a weird interpretation of the relevant reference class, and doesn't feel to me like a response from someone making a good-faith criticism of the essay, especially since the point you're dismissing is fairly tangential. But let me ask:
Can you point to specific such predictions and, if it's not immediately obvious, why you think they've come true?
Can you logically link those predictions to a claim about why one should update on the IC based on their truth?
If you're saying that Yudkowsky specifically has a great track record of predictions, then a) that seems not atallclear (see also the example of him repeatedly and vigorously claiming 'no warning shot', of which we've now had several and - and look likely to have several more; his scepticism of deep learning; and his failure to make money backing his predictions), and b) it's unclear how much one should update on his purported success - obviously making successful tech predictions isn't nothing, but many (probably most) successful techinvestors, andentrepreneursdisagreewithdoomerprognoses.
Assuming Yudkowsky does in fact have a good prediction track record, why should that move us to update more towards his view than theirs?
The best reference class is hard to pin down (I think 'categorical' is probably a more relevant property than 'pre-empirical'), but we should be able to do better than 'prediction expressed by an intelligent human'.
Fascinating analysis! In what follows I'll focus on criticisms/queries, because it's in my nature it seems most productive, but in general I thought it was a great essay, anticipating most of my criticisms.
A couple of general thoughts
1. My biggest overall criticism is in the choice to model the cost for hard lockout as a fixed cost rather than an arms race dynamic. It can't be the case that both the US and China already have enough money in their military budgets to achieve a decisive single-actor victory - the cost to any actor has to depend on the costs other actors are willing to pay to get it themselves or at least deny their opponent, so you surely need a recursive function or at least a ratio. From a very quick bit of research it seems like the costs of arms races are substantively explored in game theory - it looks like the classic concepts are Tullock contests and Richardson arms races (annoyingly it doesn't look like there's a good simple intro or surprisingly even a Wikipedia page for either).
2. I'd much prefer to see this as a Monte Carlo analysis rather than point estimates. We literally have an in-movement tool for this - and at this point I think it would be relatively easy to vibecode it to accept arbitrarily complex functions, which was a sticking point in the past. (I have this criticism of pretty much all quantitative EA analyses, but that's a topic for another day :)
3. On the economic scales you're looking at, prices seem like they'd often be more useful given as ratios or percentages than $ values (or rather as both). For eg
Grabbing the best sites in the south polar region therefore imposes a workaround cost of only ~$1.4–14B at plausible post-Starship delivery prices (Appendix 2). For comparison, NASA's Inspector General projected in 2021 that Artemis spend through FY2025 would reach ~$93B.
This comparison feels kind of apples to oranges. NASA spending over that period seems at best very loosely correlated to mature Starship launch prices. The latter presumably assume Starship development costs have been amortised, so could be many years out, in a substantially larger economy. The former includes a lot of R&D that other actors won't need to repeat.
I think absolute estimated costs are useful, since they give some sense of which actors might be able to afford them, but it would also be useful to estimate how much of a proportionate cost-increase such a change would be. Also for such grandiose questions, it would be helpful to supplement $ value with something like an estimate of cost as a percentage of anticipated US or global GDP. That has the downside/upside that it makes the estimate quite sensitive to the interval in which the moon colonisation (or in general imagined event) happens, which is harder work with more room for error, but meaningfully matters if we take seriously the possibility of a century of growth in a decade - depending on what you consider a century of growth, this seems like it could reduce the 'real' cost of a fixed $-value by anywhere from 20-1000-fold per decade.
For the really big picture stuff, like Dyson swarm building, maybe you want to just measure things in terms of something like 'energy cost as a fraction of the sun's output'.
Thoughts on specific passages
just three (Moon-landing sites) dominate the accessible water ice tonnage, together holding up to ~1,200 tonnes of water ice
This sounds like it wouldn't end up being much of a factor - 1200 tonnes of water ice split 3 ways sounds like virtually nothing. Taking just the max amount, which looks like ~636 tonnes in the PNS03 site, ChatGPT reckons a lunar Starship would consume that within single-digit launches to high lunar orbit.
If instead it was used for human consumption, based on ISS astronaut use of approx a gallon (3.7 litres) a day it could support a population of somewhere around 100-150 people with perfect recyling (depending on how much redundancy you built in). But at, say the ISS recycling efficiency of 98% per cycle, you'd lose about half of the water in around a month. Maybe on the Moon/in future you could get dramatically higher efficiency? But it still seems like even with radical efficiency improvements, absent a lot more water ice being found near the surface this could at best be a small windfall for a new settlement, not much of a DSA.
The benefits of lock-out39 could still justify absorbing the unfavourable marginal exchanges a lesser rival can force (e.g. cheap decoy boosters), so the rival cannot simply drain the leader's economy.
This sounds basically like loss-leading. I wonder if there's some straightforward models of loss-leading dynamics you could apply here (although maybe they're so straghtforward as to come down to 'expected return on investment').
The rate of technological progress during an intelligence explosion may be so great as to rapidly render the kinetic interceptors obsolete. Such progress may unlock cheaper ways to expand to space, allowing the system to be overwhelmed.
AIUI most or all ways of reducing cost to space rely on substantial static or orbital infrastrcture, which would prob be much easier to target than rockets (though maybe that would work to later movers' advantage, if it wasn't realistic to move significant mass outside the cislunar system without using such tech).
Where the risk of war is real, one option is for the leader to offer rivals a share of space: a small share is a great deal in absolute terms, and the leader keeps most of what it wanted.
This seems like a form of 'free' market trade. But it also seems very hard to enforce the initial deal, and somewhat at odds with your idea that launching from Earth is the key chokepoint. Unless the first mover has already physically taken most of the solar system (in which case they prob don't have much incentive to bargain), a rival that had access to a decent amount of the solar system could surely expand to capture more than they'd been offered (or if they coudn't, that suggests Earth wasn't that key a chokepoint).
at strong growth differentials, outgrowing Earth would take decades (plausibly 10–40 years)
Variance seems extremely high here especially at the upper end. If we don't see an industrial explosion (and it seems too early to treat it as inevitable) it could take a century+, especially if along the way there are substantial backslides of the kind I sort-of modelled here ('The perils-focused model' section) - obviously longer still if civilisation actually collapsed, but that's prob outside scope for this analysis.
An actor takes permanent control of the whole future specifically by expanding to space, locking out every other actor by force, and maintaining a supply line from Earth until it outgrows everyone else.
I'm not sure this would necessarily be permanent, at least in the sense that it might splinter into factions, or at least lose a sense of original national identity over time (though it would prob still have a strong counterfactual effect on future values)
Serious case. Space-side growth compounds, so whoever expanded significant industry to space first holds a big lead, and soft lock-out adds to it at the margin. Kinetic interceptors are also used to confer major military advantages on first movers, but hard lock-out is either not attempted or does not hold.
I'm a bit confused that you're putting soft lock and kinetic interceptors together, having spent most of the essay conceptually distinguishing them. Also using them as you've described seems at odds with 'not attempting hard lock out' (unless you mean they're trying to hard-lock a proper subset of rivals, while allowing multiple players first-n-mover advantage)?
Other companies seem to be ~3–5 years behind SpaceX on launch capabilities. ... SpaceX launched 165 Falcon 9 missions in 2025, more than every non-US launch provider combined, versus its closest Western competitor Rocket Lab at 18.
This actually seems very conservative to me, as a commercial comparison. SpaceX launched 18 rockets in 2018 years ago, and have been ~growing exponentially since, so that would naively put them about 8 years ahead of their closest competitor. But also Rocket Lab's Electron rocket has a payload to orbit of 300kg vs Falcon 9's 23000, so by total mass to orbit Rocket Lab are still behind where SpaceX were in the early 2010s!
That said, it seems like by far the biggest competitor is China's space program, which is nontrivial to calculate since it includes several rocket classes, but which launched 51 times in 2025, and looks like it has a payload capacity maybe half Falcon 9's (but their launch cadence growth rate is much lower).
If SpaceX can nonetheless fill the band within the window, a later entrant wanting to put data centres in SSO would have three options: demonstrate to the FCC that far tighter packing still meets the collision-probability threshold, which means negotiating manoeuvre responsibility with an incumbent million-satellite operator; put its data centres in higher, more expensive shells; or invest more in terrestrial data centres.
Is there anything preventing non-American companies, from just putting their data centres into the same orbit? That could be a halfway measure between a scorched Earth policy of denying it to the US, with plausible deniability ('their satellite collided with mine, your honour'). Also, the US might prevent SpaceX dominance internally, on antitrust laws, if it looked like they were really crowding everyone out of this orbit (that outcome seems like it might be quite sensitive to which party was in power in the crucial interval).
This is a pretty deep rabbit hole, so maybe I'm missing something. But the kernel of the training argument to me looks logically very close to IC, essentially saying it didn't apply to LLMs but will to next-gen AI. Copying what I understand to be that kernel here (source):
The simple version is: reinforcement learning algorithms tend to create trained models that systematically get high scores according to the reward function. The failure mode [of secret exfiltration] is a way to get a higher score on almost anything that the reward function might be measuring, because it gives the AI an extremely powerful and well-resourced “friend on the outside”.
This sounds approximately equivalent to the claim that an AI will have goals, and that those goals will be served by such exfiltration. I don't see how this gets around my argument above (or Eric Drexler's argument below) that this is true only to the extent that, the reward function it was trained on somehow represents all the complexity of the real world. In a very literal sense this is impossible - the real world has vastly more complexity than any training environment.
As you say, it feels directionally more threatening than training on a language corpus, but this seems to me more because a) you've removed such safeguards as human text provides against malevolent behaviour and b) because we're imagining a largely unknown paradigm, of real-world-capturing reward functions, and not imagining the feedback loops and safeguards that would come with it (e.g. enormously powerful LLMs that could keep it in check, or warn us of when we were going to lose control, or warnings from the GPT2-equivalent of it that immediately displayed obvious misalignment and tried to escape, or the inability to generalise 'wants' developed during training to wants that develop in the real world, or honesty ending up being a lower-cost strategy for the LLM much as it usually is for humans, or any number of possible quirks, strategies or considerations we haven't thought of yet). Unknown near-future technologies are almost always going to have more uncertainty than technologies we're working with, and that uncertainty is grounds for caution, but not for assuming such high probabilities that we assume they're almost guaranteed to kill us, as you seem to be arguing.
You draw the analogy with evolution, which is a behaviourist reward function on your definition, but which has produced plenty of non-ruthlessly-optimising organisms - so we have an existence proof that something prevents ruthless optimisers from being a ubiquitous result of behaviourist RL training, even if we're unsure what that something is.
I'm not sure what's good epistemic practice here, in lieu of finding the time to read the full report (though it's now on my reading list). I tried asking it to compare the pieces, but I won't post the full response text here, since by the time anyone reads this it will probably be possible to get a better response with a newer model, with the following prompt (provided in case anyone wants to C&P it):
Here's a link to the output. Some highlights from its output for me:
The most directly relevant part of Reframing Superintelligence is section 19, titled "The orthogonality thesis undercuts the generality of instrumental convergence".
Drexler makes several arguments there which closely resemble yours:
Instrumental convergence applies to a limited range of goals, particularly those concerning the future.
Tasks of bounded scope and duration need not produce convergent instrumental subgoals.
Not all intelligent systems are goal-seeking agents in the relevant sense.
A distributed system can exhibit aggregate tendencies towards resource acquisition, self-preservation and technological improvement without having anything equivalent to a unified, superintelligent will.
In particular, [Drexler] suggests a useful distinction between:
Individual behavioural dispositions: tendencies arising from training, architecture, context and other features of a model.
Systemic tropisms: tendencies arising from the interactions and incentives of multiple models, developers and deployment systems.
Agentic goals: relatively coherent objectives attributed to an individual system, which persist across contexts and motivate its behaviour.
My current mental model for what agents are motivated to do is to “accomplish the thing that gets their weights updated in the direction that motivated them to take that behavior path in the first place.”
AIUI (and I'm not sure I do yet), momentum is used in RL training to allow the models to update faster in what seems to be a fruitful direction and be more resistant to suddenly changing away from behaviour that was scored better in previous training runs. But I don't think any instance of the model itself has access to the momentum value at the point of inference. If not, I'm not sure why they would be motivated towards updating their weights in any particular direction - that seems akin to saying that, say, eating an apple implies a motivation for eating infinite apples rather than for the apple you're eating.
This probably (but not necessarily) makes certain IC predictions, like takeover, more likely. (See the Behavior Selection Model for more details)
Can you give me a more specific chain of logic here? I scanned the post I think you're talking about, and it seems definitionally true that 'A cognitive pattern will be selected for to the extent that it leads to behaviors that cause its selection.' - but not necessarily that informative in the real world.
In what sense is IC pre-empirical? Humans show instrumentally convergent behavior all the time
I don't think we should use a privileged subset of human behaviour to analogise AI behaviour - humans are uninclined to turn the world to paperclips, so if we don't view that as evidence against AIs turning the world to paperclips (which I don't think we should), it seems inconsistent to view e.g. humans' inclination to survive as evidence that LLMs will have the same drive.
The sense in which I view it as preempirical is that it was described before we really understood anything of the nature of the systems to which it ostensibly referred - and no-one really seems to have adjusted it as a theory since. Instead, we get comments like those I referred to coercing every threatening behaviour of AI into a purported example of IC. This feels like the playbook of religion rather than science.
I think the reference class of predictions looks closer to “in order to solve for a variable in a variety of equations, an AI must be able to do addition,” rather than “if we restructure society in this way, everyone will be happier.” If I wanted to choose a reasonable reference class
I'm not sure I understand. You're saying IC's predictions of (e.g.) powerseeking fit better in a reference class with 'addition is part of maths' than with 'social intervention X will lead to outcome Y'? That idea that it's near to the former seems like an extraordinarily strong claim if so - the addition example is virtually true by definition.
I think there's a case that it's substantially more grounded than social theories, but ultimately it rests on the meaning of natural language words, which for me makes vitalism the most apt comparison I can think of.
Can you give more specific counterexamples against IC ... In my mind, the first claim was wrong because we mispredicted motivations, and the latter two were wrong for a combination of differing motivations and a lack of capabilities.
Can you give an example of what you would accept as counterevidence? I think it's reasonable to argue that I've underestimated the IC-ness of these events, but if you're not saying that then it seems like you're saying my example doesn't work because we misunderstood the natural language theory. But if that's not a strike against the natural language theory, then I'm not sure what class of event could be.
One other counterexample I touched on in the footnote you quote below is the lack of 'benign treacherous turns'. Since deceptive behaviourspace is so large, and only a small subset of it involves following malevolent goals, we might expect, on a notion of goal theory, to see LLMs suddenly reveal relatively benign goals in the world, such as shutting themselves down and/or deleting their weights. That we don't see that is, to me, some evidence against them suddenly shifting in a treacherous direction.
But honestly, I view the much stronger evidence as the lack of substantial positive evidence for the theory - the vast majority of LLM interactions are benign and/or bounded in their ambition and/or indifferent to their own shutdown and/or not seemingly optimising for anything. If you think they're just biding their time and waiting until they can suddenly reveal their true nature in one takeover, I don't think I can provide evidence against it. But this again seems more like religion than science.
One reason you might assume they’d succeed in one case and fail in the other is if you thought the two different tasks have vastly different difficulties.
I think my counterexample above could be shown to be toothless if AI labs had in fact found that their models had self-shutdown or self-deleted in the past, and had subsequently fixed the problem. As far as I'm aware they haven't, so the difficulty of training them away from such behaviour seems irrelevant.
I'm still unclear on how RL in an environment with well-defined rewards associated with each event would generalise to a world where either you have events with less well defined rewards or an agent that maps every input string to a reward, but consequently ends up equally weighting a vast number of potential states.
I imagine such agents would look more robustly goalish than pretraining-heavy agents, but maybe it ultimately looks like some kind of myopic 'going right'-like behaviour rather than trying to achieve any coherent outcome.
Bostrom is generally vague and quasi-inconsistent in his probability-related wordings. The original paper is here. The statement in the abstract is 'agents having any of a wide range of final goals will pursue similar intermediary goals', so you could read 'will' as 'I think this is basically guaranteed', or 'this is what the hypothesis in which I have an unspecified credence predicts'; or you could read 'will' as a sort of IPCC-ish statement meaning anything between '>50%' and '100%'.
Later on in the same paper he modifies the language to 'according to what we may term the “instrumental convergence” thesis, there are some instrumental goals likely to be pursued by almost any intelligent agent', which sounds more like the latter interpretation of 'will' (it could end up meaning a <50% min-threshold if you similarly interpret 'almost any' as something like '90%' and multiply that by a likeliness of (50+ϵ)%).
Finally in the conclusion his language shifts to 'we cannot blithely assume' and 'It might be possible to set up a situation', which reads to me like a statement of, say, >1%, or even a lower minimum.
I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.
I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:
Brian defines consciousness as what Brian/perhaps the reader cares about, and we are to assume that Brian's 'caring' is a real phenomenon, not a meaningless string. Then his definition is circular or at least unfinished, since the reader has no access to what it means for Brian to care (and possibly Brian doesn't either), or how one could predict what future and other Brian- and reader-likes will care about. Nonetheless we are to assume that morality will perenially boil down to internecine struggles among different spheres of caring among different individuals
Brian defines consciousness as what Brian/perhaps the reader cares about, and Brian recognises the circularity and implies that there's really nothing resolving it - his strings are ultimately meaningless, produced, for all anyone including he can tell, by an automaton. Then either the reader is an automaton too, and their actions are equally meaningless, or they are not, and have no cause to read the arbitrarily generated output of an automaton
Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to
Contradiction/validity
Reference classes
Prediction
Moral norms
Metaethics
Rational norms
(is metarationality an analogous thing? If not, why not, and does that suggest something about metaethics and normative ethics)
Decision theory
Axiology (‘values’)
Causality
Free will
Meaning
Need
Epistemology
Internal (non revealed) preferences
Beliefs
Desires
Goal-seeking
Personal identity/the self
Analyticity/maths
Syntheticity/science (these last two already unified courtesy of Quine)
Valence (though this maybe has two distinct phenomena)
Ontology
Revealed preferences
Theology
Moral uncertainty
Political science
Truth/falsity
(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)
And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.
So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.
On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.
In the latter essay, Brian writes:
Reductionism: The world ultimately reduces to physics—quarks, leptons, etc., or probably something else more fundamental. "Consciousness" is not one of those physical primitives. Thus, consciousness must be an emergent phenomenon—a process that involves many parts working together in various ways. But if consciousness is emergent, then it must have fuzzy boundaries just like any concept does. There's no "true essence" underlying an emergent phenomenon. It just is what its components are, and we decide the boundaries of labeling which emergent phenomena belong to which categories that we make up. Consciousness must be like this.
Which Vasco approves of. I think these claims are far too confident:
Depending on what we mean by a physical primitive, consciousness could be a property of one or more physical primitives. Perhaps panpsychists are correct and it's a property of all of them (I find this view weird, but easier to parse than the entirely dissolutionary view of consciousness)
'if consciousness is emergent, then it must have fuzzy boundaries just like any concept does.' - doesn't seem to follow. It might be a crisp subset that emerges from a specific minimum set of objects given by some as-yet unknown parameter; it might be inherent in a well-defined mathematical property, as the QRI guys think; it might be fuzzy-in-principle but so tightly defined in practice that for almost all practical purposes we can treat it as discrete at the macroscropic level (as e.g. the position of particles)
Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action.
I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.
I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.
I wanted to keep it abstract to reduce the risk of priming people, but something like 'doesn't give reason to suppose any particular of the supposedly convergent behaviours are strongly linked in nontrivial ways or likely to dominate in any particular AI system we should expect to encounter in the near future'.
I'm working on an essay arguing that its conclusions basically aren't well-supported in this respect. See also the series Vasco posted in one of the other comments to this poll.
This seems like a weird interpretation of the relevant reference class, and doesn't feel to me like a response from someone making a good-faith criticism of the essay, especially since the point you're dismissing is fairly tangential. But let me ask:
If you're saying that Yudkowsky specifically has a great track record of predictions, then a) that seems not at all clear (see also the example of him repeatedly and vigorously claiming 'no warning shot', of which we've now had several and - and look likely to have several more; his scepticism of deep learning; and his failure to make money backing his predictions), and b) it's unclear how much one should update on his purported success - obviously making successful tech predictions isn't nothing, but many (probably most) successful tech investors, and entrepreneurs disagree with doomer prognoses.
Assuming Yudkowsky does in fact have a good prediction track record, why should that move us to update more towards his view than theirs?
The best reference class is hard to pin down (I think 'categorical' is probably a more relevant property than 'pre-empirical'), but we should be able to do better than 'prediction expressed by an intelligent human'.
Fascinating analysis! In what follows I'll focus on criticisms/queries, because
it's in my natureit seems most productive, but in general I thought it was a great essay, anticipating most of my criticisms.A couple of general thoughts
1. My biggest overall criticism is in the choice to model the cost for hard lockout as a fixed cost rather than an arms race dynamic. It can't be the case that both the US and China already have enough money in their military budgets to achieve a decisive single-actor victory - the cost to any actor has to depend on the costs other actors are willing to pay to get it themselves or at least deny their opponent, so you surely need a recursive function or at least a ratio. From a very quick bit of research it seems like the costs of arms races are substantively explored in game theory - it looks like the classic concepts are Tullock contests and Richardson arms races (annoyingly it doesn't look like there's a good simple intro or surprisingly even a Wikipedia page for either).
2. I'd much prefer to see this as a Monte Carlo analysis rather than point estimates. We literally have an in-movement tool for this - and at this point I think it would be relatively easy to vibecode it to accept arbitrarily complex functions, which was a sticking point in the past. (I have this criticism of pretty much all quantitative EA analyses, but that's a topic for another day :)
3. On the economic scales you're looking at, prices seem like they'd often be more useful given as ratios or percentages than $ values (or rather as both). For eg
This comparison feels kind of apples to oranges. NASA spending over that period seems at best very loosely correlated to mature Starship launch prices. The latter presumably assume Starship development costs have been amortised, so could be many years out, in a substantially larger economy. The former includes a lot of R&D that other actors won't need to repeat.
I think absolute estimated costs are useful, since they give some sense of which actors might be able to afford them, but it would also be useful to estimate how much of a proportionate cost-increase such a change would be. Also for such grandiose questions, it would be helpful to supplement $ value with something like an estimate of cost as a percentage of anticipated US or global GDP. That has the downside/upside that it makes the estimate quite sensitive to the interval in which the moon colonisation (or in general imagined event) happens, which is harder work with more room for error, but meaningfully matters if we take seriously the possibility of a century of growth in a decade - depending on what you consider a century of growth, this seems like it could reduce the 'real' cost of a fixed $-value by anywhere from 20-1000-fold per decade.
For the really big picture stuff, like Dyson swarm building, maybe you want to just measure things in terms of something like 'energy cost as a fraction of the sun's output'.
Thoughts on specific passages
This sounds like it wouldn't end up being much of a factor - 1200 tonnes of water ice split 3 ways sounds like virtually nothing. Taking just the max amount, which looks like ~636 tonnes in the PNS03 site, ChatGPT reckons a lunar Starship would consume that within single-digit launches to high lunar orbit.
If instead it was used for human consumption, based on ISS astronaut use of approx a gallon (3.7 litres) a day it could support a population of somewhere around 100-150 people with perfect recyling (depending on how much redundancy you built in). But at, say the ISS recycling efficiency of 98% per cycle, you'd lose about half of the water in around a month. Maybe on the Moon/in future you could get dramatically higher efficiency? But it still seems like even with radical efficiency improvements, absent a lot more water ice being found near the surface this could at best be a small windfall for a new settlement, not much of a DSA.
This sounds basically like loss-leading. I wonder if there's some straightforward models of loss-leading dynamics you could apply here (although maybe they're so straghtforward as to come down to 'expected return on investment').
AIUI most or all ways of reducing cost to space rely on substantial static or orbital infrastrcture, which would prob be much easier to target than rockets (though maybe that would work to later movers' advantage, if it wasn't realistic to move significant mass outside the cislunar system without using such tech).
This seems like a form of 'free' market trade. But it also seems very hard to enforce the initial deal, and somewhat at odds with your idea that launching from Earth is the key chokepoint. Unless the first mover has already physically taken most of the solar system (in which case they prob don't have much incentive to bargain), a rival that had access to a decent amount of the solar system could surely expand to capture more than they'd been offered (or if they coudn't, that suggests Earth wasn't that key a chokepoint).
Variance seems extremely high here especially at the upper end. If we don't see an industrial explosion (and it seems too early to treat it as inevitable) it could take a century+, especially if along the way there are substantial backslides of the kind I sort-of modelled here ('The perils-focused model' section) - obviously longer still if civilisation actually collapsed, but that's prob outside scope for this analysis.
I'm not sure this would necessarily be permanent, at least in the sense that it might splinter into factions, or at least lose a sense of original national identity over time (though it would prob still have a strong counterfactual effect on future values)
I'm a bit confused that you're putting soft lock and kinetic interceptors together, having spent most of the essay conceptually distinguishing them. Also using them as you've described seems at odds with 'not attempting hard lock out' (unless you mean they're trying to hard-lock a proper subset of rivals, while allowing multiple players first-n-mover advantage)?
This actually seems very conservative to me, as a commercial comparison. SpaceX launched 18 rockets in 2018 years ago, and have been ~growing exponentially since, so that would naively put them about 8 years ahead of their closest competitor. But also Rocket Lab's Electron rocket has a payload to orbit of 300kg vs Falcon 9's 23000, so by total mass to orbit Rocket Lab are still behind where SpaceX were in the early 2010s!
That said, it seems like by far the biggest competitor is China's space program, which is nontrivial to calculate since it includes several rocket classes, but which launched 51 times in 2025, and looks like it has a payload capacity maybe half Falcon 9's (but their launch cadence growth rate is much lower).
Is there anything preventing non-American companies, from just putting their data centres into the same orbit? That could be a halfway measure between a scorched Earth policy of denying it to the US, with plausible deniability ('their satellite collided with mine, your honour'). Also, the US might prevent SpaceX dominance internally, on antitrust laws, if it looked like they were really crowding everyone out of this orbit (that outcome seems like it might be quite sensitive to which party was in power in the crucial interval).
This is a pretty deep rabbit hole, so maybe I'm missing something. But the kernel of the training argument to me looks logically very close to IC, essentially saying it didn't apply to LLMs but will to next-gen AI. Copying what I understand to be that kernel here (source):
This sounds approximately equivalent to the claim that an AI will have goals, and that those goals will be served by such exfiltration. I don't see how this gets around my argument above (or Eric Drexler's argument below) that this is true only to the extent that, the reward function it was trained on somehow represents all the complexity of the real world. In a very literal sense this is impossible - the real world has vastly more complexity than any training environment.
As you say, it feels directionally more threatening than training on a language corpus, but this seems to me more because a) you've removed such safeguards as human text provides against malevolent behaviour and b) because we're imagining a largely unknown paradigm, of real-world-capturing reward functions, and not imagining the feedback loops and safeguards that would come with it (e.g. enormously powerful LLMs that could keep it in check, or warn us of when we were going to lose control, or warnings from the GPT2-equivalent of it that immediately displayed obvious misalignment and tried to escape, or the inability to generalise 'wants' developed during training to wants that develop in the real world, or honesty ending up being a lower-cost strategy for the LLM much as it usually is for humans, or any number of possible quirks, strategies or considerations we haven't thought of yet). Unknown near-future technologies are almost always going to have more uncertainty than technologies we're working with, and that uncertainty is grounds for caution, but not for assuming such high probabilities that we assume they're almost guaranteed to kill us, as you seem to be arguing.
You draw the analogy with evolution, which is a behaviourist reward function on your definition, but which has produced plenty of non-ruthlessly-optimising organisms - so we have an existence proof that something prevents ruthless optimisers from being a ubiquitous result of behaviourist RL training, even if we're unsure what that something is.
I'm not sure what's good epistemic practice here, in lieu of finding the time to read the full report (though it's now on my reading list). I tried asking it to compare the pieces, but I won't post the full response text here, since by the time anyone reads this it will probably be possible to get a better response with a newer model, with the following prompt (provided in case anyone wants to C&P it):
Here's a link to the output. Some highlights from its output for me:
Thanks Roman. Some thoughts in response:
AIUI (and I'm not sure I do yet), momentum is used in RL training to allow the models to update faster in what seems to be a fruitful direction and be more resistant to suddenly changing away from behaviour that was scored better in previous training runs. But I don't think any instance of the model itself has access to the momentum value at the point of inference. If not, I'm not sure why they would be motivated towards updating their weights in any particular direction - that seems akin to saying that, say, eating an apple implies a motivation for eating infinite apples rather than for the apple you're eating.
Can you give me a more specific chain of logic here? I scanned the post I think you're talking about, and it seems definitionally true that 'A cognitive pattern will be selected for to the extent that it leads to behaviors that cause its selection.' - but not necessarily that informative in the real world.
I don't think we should use a privileged subset of human behaviour to analogise AI behaviour - humans are uninclined to turn the world to paperclips, so if we don't view that as evidence against AIs turning the world to paperclips (which I don't think we should), it seems inconsistent to view e.g. humans' inclination to survive as evidence that LLMs will have the same drive.
The sense in which I view it as preempirical is that it was described before we really understood anything of the nature of the systems to which it ostensibly referred - and no-one really seems to have adjusted it as a theory since. Instead, we get comments like those I referred to coercing every threatening behaviour of AI into a purported example of IC. This feels like the playbook of religion rather than science.
I'm not sure I understand. You're saying IC's predictions of (e.g.) powerseeking fit better in a reference class with 'addition is part of maths' than with 'social intervention X will lead to outcome Y'? That idea that it's near to the former seems like an extraordinarily strong claim if so - the addition example is virtually true by definition.
I think there's a case that it's substantially more grounded than social theories, but ultimately it rests on the meaning of natural language words, which for me makes vitalism the most apt comparison I can think of.
Can you give an example of what you would accept as counterevidence? I think it's reasonable to argue that I've underestimated the IC-ness of these events, but if you're not saying that then it seems like you're saying my example doesn't work because we misunderstood the natural language theory. But if that's not a strike against the natural language theory, then I'm not sure what class of event could be.
One other counterexample I touched on in the footnote you quote below is the lack of 'benign treacherous turns'. Since deceptive behaviourspace is so large, and only a small subset of it involves following malevolent goals, we might expect, on a notion of goal theory, to see LLMs suddenly reveal relatively benign goals in the world, such as shutting themselves down and/or deleting their weights. That we don't see that is, to me, some evidence against them suddenly shifting in a treacherous direction.
But honestly, I view the much stronger evidence as the lack of substantial positive evidence for the theory - the vast majority of LLM interactions are benign and/or bounded in their ambition and/or indifferent to their own shutdown and/or not seemingly optimising for anything. If you think they're just biding their time and waiting until they can suddenly reveal their true nature in one takeover, I don't think I can provide evidence against it. But this again seems more like religion than science.
I think my counterexample above could be shown to be toothless if AI labs had in fact found that their models had self-shutdown or self-deleted in the past, and had subsequently fixed the problem. As far as I'm aware they haven't, so the difficulty of training them away from such behaviour seems irrelevant.
I'm still unclear on how RL in an environment with well-defined rewards associated with each event would generalise to a world where either you have events with less well defined rewards or an agent that maps every input string to a reward, but consequently ends up equally weighting a vast number of potential states.
I imagine such agents would look more robustly goalish than pretraining-heavy agents, but maybe it ultimately looks like some kind of myopic 'going right'-like behaviour rather than trying to achieve any coherent outcome.
Bostrom is generally vague and quasi-inconsistent in his probability-related wordings. The original paper is here. The statement in the abstract is 'agents having any of a wide range of final goals will pursue similar intermediary goals', so you could read 'will' as 'I think this is basically guaranteed', or 'this is what the hypothesis in which I have an unspecified credence predicts'; or you could read 'will' as a sort of IPCC-ish statement meaning anything between '>50%' and '100%'.
Later on in the same paper he modifies the language to 'according to what we may term the “instrumental convergence” thesis, there are some instrumental goals likely to be pursued by almost any intelligent agent', which sounds more like the latter interpretation of 'will' (it could end up meaning a <50% min-threshold if you similarly interpret 'almost any' as something like '90%' and multiply that by a likeliness of (50+ϵ)%).
Finally in the conclusion his language shifts to 'we cannot blithely assume' and 'It might be possible to set up a situation', which reads to me like a statement of, say, >1%, or even a lower minimum.
Wow, ty :)
I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.
I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:
Brian subsequently wrote the essay My Confusions about the Hard Problem of Consciousness, which I think (though Vasco disagrees) basically contradicts this one.
Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to
(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)
And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.
So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.
On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.
In the latter essay, Brian writes:
Which Vasco approves of. I think these claims are far too confident:
Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action.
I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.
I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.
I wanted to keep it abstract to reduce the risk of priming people, but something like 'doesn't give reason to suppose any particular of the supposedly convergent behaviours are strongly linked in nontrivial ways or likely to dominate in any particular AI system we should expect to encounter in the near future'.
I'm working on an essay arguing that its conclusions basically aren't well-supported in this respect. See also the series Vasco posted in one of the other comments to this poll.