1. I think that "secret intelligence explosions" are something that would be quite scary and disruptive. So it seems worth spending some attention on, like this writing.
2. I feel like there are some key assumptions here. I'd naively guess:
a. An intelligence explosion like you're describing doesn't seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point.
b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an "intelligent explosion." I'd naively expect there to be a ton of competition in this area, and I'd expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I'd naively expect that if there are any discontinuous gains to be made, they'll be made by the largest actors.
I agree that abstract arguments about the cost/benefits of privacy don't make either case certain. But I think the empirical evidence is quite positive so far, against secrecy. That said, of course I expect that the situation could change quickly as major circumstances change.
a. An intelligence explosion like you're describing doesn't seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point.
I'm not exactly sure what you mean by discontinuous jump. I expect the usefulness of AI systems to be pretty "continuous" inside AI companies and "discontinuous" outside AI companies. If you think that: 1. model release cadence will stay similar 2. but, capabilities will accelerate 3. then you should also expect external AI progress to be more "discontinuous" than it currently is.
I gave some reasons why I don't think AI companies will want to externally deploy their best models (like less benefit from user growth), so maybe you disagree with that, or do you disagree with 1,2, or 3?
b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an "intelligent explosion." I'd naively expect there to be a ton of competition in this area, and I'd expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I'd naively expect that if there are any discontinuous gains to be made, they'll be made by the largest actors.
I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I'm not sure why this is implied by my post. I think my model isn't especially sensitive to single vs multiple competing IEs, but possible you're seeing something I'm not. I don't really follow
competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do)
Do you expect competition to increase dramatically from where we are at rn? If not then I think current level of competition empirically do lead to people investing a lot in AI development so I'm not sure I quite follow your line of reasoning.
I gave some reasons why I don't think AI companies will want to externally deploy their best models (like less benefit from user growth), so maybe you disagree with that, or do you disagree with 1,2, or 3?
I understand that there are some reasons that companies might do this. One 1/2/3, I'm really unsure about the details of (2). If capabilities accelerate, but predictably and slowly, I assume this wouldn't feel very discontinuous.
Also, there's a major difference between AIs getting better and them becoming more useful. Often there are diminishing returns to intelligence.
> I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I'm not sure why this is implied by my post. I think my model isn't especially sensitive to single vs multiple competing IEs, but possible you're seeing something I'm not.
Sorry, I may have misunderstood that. But if there is only one or two potential actors, that does seem to make the situation far easier. Like, it could be fairly clear to many international actors that there are 1-2 firms that might be making major breakthroughs. In that case, we might just need to worry about policing these firms. This seems fairly possible to me (if we can be somewhat competent).
Do you expect competition to increase dramatically from where we are at rn? If not then I think current level of competition empirically do lead to people investing a lot in AI development so I'm not sure I quite follow your line of reasoning.
I'd expect that market caps of these companies would be far higher if it were clear if there would be less competition later, and I'd equivalently expect these companies to do (even more) R&D.
I'm quite sure investors are quite nervous about the monopoly abilities of LLM companies.
Right now, I don't think it's clear to anyone where OpenAI/Anthropic will really make money 5+ years from now. It seems like [slightly worse AIs] often are both cheap / open-source and good enough. I think that both companies are very promising, but just that the future market value is very unclear.
I've heard that some of the Chinese strategy is, "Don't worry too much about being on the absolute frontier, because it's far cheaper to just copy from 1-2 steps behind."
I wasn't saying that "competition would greatly decrease the value of the marginal intelligence gain" in the sense of "things will get worse from where we are now", but in the sense of "things are generally worse from where they would be without such competition"
I could see claims like “our AI can now design bioweapons” or “our AI is capable of massively speeding up research” to be brushed aside as AI hype, given the steady stream of people overhyping AI capabilities.
Which means that if an AGI lab wants to make their employees less able to successfully whistleblow, they should arrange for fake "leaks" about intelligence explosions to be happening all the time, crying wolf and poisoning the epistemic environment.
Is this what that kerfuffle was about? See also this, and the rest of that account's activity.
Those may or may not be just random shitposters, but I now see a clear motivation for OpenAI to actually run those psyops.
What are some candidates for x that would convince interested members of the public?
I think the whistleblower would need to grab some actual proof of their claims: outputs of the models' R&D/bioweapons research, or logs displaying the model's capability as it autonomously researches stuff...
A copy of internal correspondence discussing all of this might also work, if there's sufficient volume of it. Or copies of internal papers which confirm/enable the breakthroughs.
I have updated upwards a bit on whistleblowers being able to make credible claims on IE. I do think that people in positions with whistleblowing potential should probably try and think concretely about what they should do, what they'd need to see to do it, and who specifically they'd get in contact with, and what evidence might be compelling to them (and have a bunch of backup plans).
This is a crosspost from the new Animal Welfare Alignment Newsletter by Anima International. You can subscribe on Substack if you are interested in following these efforts. Audio reading also available on Substack.
The goals of this post are to:
1. Raise a question I see as crucially important to the goal of aligning AI to animal welfare...
Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job.
My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI.
...
This is a linkpost for Request for Proposals: Research and Applied Work on Digital Minds.
I'm glad to announce a request for proposals for research and applied work on digital minds at Longview Ph...
Cross-posted from Substack. This is a "takes" post and may well not meet your standard for rigour.
In the next few years, I expect AI companies to build AI systems that dramatically speed up their AI research, creating a period of extremely fast growth in capabilities, where AI systems surpass human intelligence in every interesting domain (i.e. an intelligence explosion).[1]
If this happens, I think it will be viable and potentially desirable for an AI company to keep its intelligence explosion secret, even if the project is not securitised. I’m concerned that people outside of AI companies won’t know that an intelligence explosion is happening, they’ll be cut out of high-stakes decisions, and they won’t be able to orient appropriately to a world with superintelligence.
In this post, I’ll discuss:
Why AI companies might want to do an intelligence explosion in secret.
Why AI workforces might make it much cheaper to keep large engineering projects a secret.
How AI companies could finance a secret intelligence explosion.
Whether employees will be able to effectively raise the alarm.
Overall, I think that a secret intelligence explosion is somewhat unlikely, but plausible and concerning.
Why might an AI company want to keep its intelligence explosion secret?
AI companies will be less focused on consumers: AI companies will primarily use their intelligence-explosion-enabling AI systems (e.g. automated scientists) to accelerate their AI research. They’ll be less focused on selling their most cutting-edge models to millions of people (though they may still serve less capable or narrow models to generate revenue).
Superintelligence will create a huge strategic advantage for the country that builds it first. There’ll be a lot of pressure from the host country to use the AI in the interest of national security, and by default large military projects are secretive.
Alternatively, AI companies may not want to be securitised, so may want to hide their progress.[2]
It would be particularly bad for adversary states to steal an intelligence explosion enabling AI system, as they could then quickly close the gap to the leading AI company.[3] AI companies may adopt significantly more aggressive security, which may push towards more secrecy around AI capabilities and value of their IP.
Companies want their competitors to stop racing with them, and showing that they are in an intelligence explosion may push other companies to double down due to winner-takes-all dynamics.
AI companies may be worried about their governments shutting them down or forcing them to slow down if there’s public panic around the catastrophic potential or job loss due to AI systems. Or, they might be worried about being nationalised and losing control of their model to their government, particularly as powerful AI systems could be used to seize power from their government.
Even if things stay in roughly the status quo, AI development will likely outpace external oversight, especially if oversight bodies lack access to frontier systems. Even if you think that frontier labs will let AISIs get uplifted by AIs they are testing (or models that are only marginally weaker, it seems plausible that AISIs won’t want to use a model that might be subtly sabotaging their work.
If AIs automating AI research produce extremely powerful AI systems after a few months, then it’s at least plausible that AI companies will want to do it in secret.
Common Objections to the Feasibility of Keeping an Intelligence Explosion Secret
Customer interactions might be required to train powerful models, which would necessitate serving an up-to-date model to many customers.
Foreign intelligence agencies (particularly in China) would likely discover the development anyway, so there might be less pressure from governments to run a covert project.
The risk of employees leaking information or whistleblowing increases with the magnitude of the development and the number of people who are aware of the development. Building superintelligence will require a huge workforce and is a big deal.
The massive computational and energy requirements will be difficult to hide. See the following image of a satellite image of the Stargate project. The bright green rectangle in the upper right corner is the size of a football field.
AI companies will need to raise capital continually, which is much easier if you are building in public.
One of the Stargate data centre megasites currently under construction from @nunosempere. The green thing in the top right is the size of a football field.
These are all reasonable objections and make me think a secret intelligence explosion is less likely, but I don’t think any of them are damning.
Three can keep a secret if two are AIs
I occasionally speak to people who think AI companies couldn’t do an intelligence explosion secretly, even if they wanted to. I think their model is roughly:
An AI company would need to invest substantial resources to create an intelligence explosion.
So internally, a lot of people would need to be aware that the company is massively accelerating.
AI company employees don’t take secrecy or security particularly seriously, and it’s really hard to keep a secret if hundreds of people are aware of it.
This seems wrong if you think (as I do) that an intelligence explosion will be AI-driven, i.e. making substantive use of drop-in replacements for ML engineers and researchers.
An AI workforce may be better at keeping secrets than a human one
Imagine that you are a researcher in North Korea working on an extremely secret military project as part of a large team.[4] The project’s leadership can take any actions they deem necessary to prevent the US from discovering the project.
Working there is frustrating:
You’re unable to leave the facility for the duration of the project, which could last several years.
You frequently do polygraphs and are interrogated to make sure that you don’t blow the whistle to the several other interested governments.
All of your conversations with other team members are monitored.
Anything you write down is sent to a team that tries to figure out whether you are trying to betray the interests of the project.
You suspect that the news you hear about other parts of the project is fake to stop you from knowing too much about the overall goals of the project.
…
I think many autocracies would eagerly adopt these extreme measures if they were affordable and didn’t incur large productivity losses. I expect that all of these measures and more will be relatively inexpensive with AIs because we don’t grant AI systems rights, and because their software nature provides them with many secrecy-preserving features.
For example, AI systems are currently unable to leave their workplace, can be interrogated much more invasively than humans, can be comprehensively monitored, erasing their memories is trivial, and they’ll do whatever kinds of malign or uncomfortable tasks you want.[5][6]
Suppose that an AI lab wanted to keep an intelligence explosion secret, what might they do?
Replace a large fraction of their capabilities workforce, particularly junior employees, with AI workers (and assign these workers to teams outside the intelligence explosion, to prevent a suspicious exit of many employees).[7] I think it’ll be viable to have a very small team of research scientists directing thousands of AI workers in sophisticated hierarchies with colossal compute budgets (for spinning up AI workers and model training).
Use AI systems to monitor all internal computer use to catch any whistleblowing attempts.
Use AI systems to profile employees to make sure that particularly plausible whistleblowers aren’t working on sensitive projects.
Silo teams to prevent many lab employees having situational awareness, whilst maintaining productivity by giving employees AI assistants which can cheaply and reliably elevate employee information permissions while keeping their baseline permissions and general knowledge about capability progress low.[8]
I’m sure that there are many other creative ways you could use the features of an AI workforce to increase secrecy. If you're struggling to come up with strategies, try prompting an LM for ideas.
How will AI companies keep the GPUs on?
So why aren’t AI companies building in secret right now? Some hypotheses:
Founder preferences - I think Sam Altman and Mark Zuckerberg are genuinely into building in public, moving fast, shipping cool stuff etc.
Hiring - having a public and successful product makes it much easier to hire people. The talent density at AI companies is really high; most people can get a decent salary elsewhere if they want to, but there’s a kind of status that working at a company building something important and well-known gets you. A more straightforward explanation is that people are just more likely to remember to have a look at the jobs board when they are looking to move on.
Fundraising - investors care a lot about user growth and revenue.[9] Most AI companies are hoping to raise billions of dollars. It’s hard to have 100M users if your product is a secret.
I’m not sure what will happen to founder preferences at the start of an intelligence explosion. I think it’s likely that founders change their minds, given the large pressures to be more secretive and the allure of potentially moving more quickly if they adopt a more secretive posture. Hiring will be less important if increasingly complex tasks are able to be delegated to AI systems, which leaves fundraising. I think the story here is more confusing.
Governments can clearly finance megaprojects when they are sufficiently bought in (e.g. bridges, nuclear power plants). If the US government wants to bankroll an ASI megaproject, I expect they’d be able to find the capital. If there’s sufficient justification to do this, then they’ll likely also be sufficient justification to secure the project and get some more insight into its operations, which will result in the public and some parts of government having some visibility into the AI project. That said, without significant action, I’d be surprised if all important government decision-makers were well-informed, particularly if the project is securitised.
So, how viable will it be for companies to raise billions of dollars in private capital, without releasing or publicly disclosing their best models?
I’m not sure how much an intelligence explosion will cost, but prices in the range of $70M to $2T seem quite plausible to me from holding training costs fixed and naively extrapolating. Let's take $10B-100B, which is somewhere in the middle (in log space), as our mainline scenario.
Two key questions are:
How convinced will investors be by the value proposition?
How much money can one raise from private investors for ambitious mega projects, given investor conviction?
I had a quick look at privately funded megaprojects, some of the larger infrastructure projects included Global telecommunications networks ($50+ billion each),[10] the UK-France Channel Tunnel ($21 billion adjusted to current value),[11] and the Gorgon LNG project ($54 billion).[12] Some record-breaking financing includes Saudi Aramco IPO ($30B raised) and Verizon’s 2013 mega-bond ($3B).[13]
Whilst the value proposition is less clear than traditional infrastructure projects, it’s not a mystery either. Millions of people pay for AI products, and code assistants are significantly improving the productivity of software engineers. It’s perhaps not so surprising that Open AI, which released the fastest-growing app of all time, were able to raise $40B in March this year.
They could, of course, continue to charge for less capable models, which might be much cheaper to run and require far lower levels of security than intelligence explosion enabling models. Alternatively, they could partner with industries that have an extremely high ROI on intellectual labour per person (e.g., quantitative finance) and provide them with limited access to frontier models or specialised but less generally capable models, or build products or even compete directly in those industries.
How easily can an employee raise the alarm?
It may be that staff at AI companies are so wrapped up in NDAs and social incentives (particularly if AI is taken seriously as a threat or asset to national security) that they don’t blow the whistle, but let's suppose that they do want to blow the whistle.
Suppose that an AI researcher tweets, “The intelligence explosion has started, this is crazy, I’ve just seen x”. What are some candidates for x that would convince interested members of the public? I could see claims like “our AI can now design bioweapons” or “our AI is capable of massively speeding up research” to be brushed aside as AI hype, given the steady stream of people overhyping AI capabilities.[14] I’m a little more optimistic about claims like “our AI caused harm in the real world” being taken seriously. Claims like “look at the massive data centers being built” sound compelling, but massive data centres are already being built, I’m not sure what we expect to see change - maybe someone crunches the numbers and notices that there is way too much inference compute to justify the amount of publicly served models?[15]
If the intelligence explosion only needs to last a few months, will society really notice and reorient in time?[16] Most of the data centre build-out would happen before the intelligence explosion starts. The data centres used for serving weak models to consumers and the data centres used for powerful internal AI agents look very similar. Maybe you think that evals will make it very clear the intelligence explosion has started, but AI companies might simply choose not to run them, or sandbag on them, and there likely won’t be a tonne of public outrage if the model hasn’t been publicly released - how would we even know to be outraged?
In this post I’m not arguing that there will be an intelligence explosion in the next few years, the main take is if there is an intelligence explosion in the next few years it may well be done secretly. I like this detailed illustration of intelligence explosion happening in the next few years.
Even if people are just fired I’m not sure that that intelligence explosion alarm is raised when there are easily preventable counter-narratives like “we wanted to strip our team back and increase talent density like Elon did at Twitter/X”.
One reviewer was sceptical because they thought that labs would be respectful of employee preferences, and companies have high inertia preventing them for pivoting to this agent focussed strategy. I think these are reasonably counterarguments but they are fairly small updates for me on the viability of setting up a large AI agent powered team with a substantial fraction of total compute if senior AI company leadership want to do that.
Major telecommunications companies (AT&T, Verizon, and Vodafone) built extensive global networks primarily financed through corporate debt, equity offerings, and reinvested company profits.
One of history's most ambitious privately funded infrastructure endeavors funded by equity investments and commercial bank loans. Developed without direct government funding, though government guarantees were provided to support the project.
Entirely funded by energy companies (Chevron, ExxonMobil, and Shell) via corporate resources and private debt arrangements. No direct government financial contribution.
Note that I think this will be very hard as the public would need to know how much inference compute models need to run, how much models are being run for public use. If a company is doing a secret intelligence explosion presumably they are also able to significantly push the frontier and serve small, cheap to run models that are competitive with public frontier models and use most of their inference compute for an intelligence explosion.
Note that there are a lot of simple strategies like distilling a model to save inference compute and serving that model, just saying that your model is being very reward hacky so you’ve decided not to serve it for a while until you get more of a handle on it etc.
Very quickly:
1. I think that "secret intelligence explosions" are something that would be quite scary and disruptive. So it seems worth spending some attention on, like this writing.
2. I feel like there are some key assumptions here. I'd naively guess:
a. An intelligence explosion like you're describing doesn't seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point.
b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an "intelligent explosion." I'd naively expect there to be a ton of competition in this area, and I'd expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I'd naively expect that if there are any discontinuous gains to be made, they'll be made by the largest actors.
I agree that abstract arguments about the cost/benefits of privacy don't make either case certain. But I think the empirical evidence is quite positive so far, against secrecy. That said, of course I expect that the situation could change quickly as major circumstances change.
a. An intelligence explosion like you're describing doesn't seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point.
I'm not exactly sure what you mean by discontinuous jump. I expect the usefulness of AI systems to be pretty "continuous" inside AI companies and "discontinuous" outside AI companies. If you think that:
1. model release cadence will stay similar
2. but, capabilities will accelerate
3. then you should also expect external AI progress to be more "discontinuous" than it currently is.
I gave some reasons why I don't think AI companies will want to externally deploy their best models (like less benefit from user growth), so maybe you disagree with that, or do you disagree with 1,2, or 3?
b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an "intelligent explosion." I'd naively expect there to be a ton of competition in this area, and I'd expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I'd naively expect that if there are any discontinuous gains to be made, they'll be made by the largest actors.
I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I'm not sure why this is implied by my post. I think my model isn't especially sensitive to single vs multiple competing IEs, but possible you're seeing something I'm not. I don't really follow
competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do)
Do you expect competition to increase dramatically from where we are at rn? If not then I think current level of competition empirically do lead to people investing a lot in AI development so I'm not sure I quite follow your line of reasoning.
I understand that there are some reasons that companies might do this. One 1/2/3, I'm really unsure about the details of (2). If capabilities accelerate, but predictably and slowly, I assume this wouldn't feel very discontinuous.
Also, there's a major difference between AIs getting better and them becoming more useful. Often there are diminishing returns to intelligence.
> I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I'm not sure why this is implied by my post. I think my model isn't especially sensitive to single vs multiple competing IEs, but possible you're seeing something I'm not.
Sorry, I may have misunderstood that. But if there is only one or two potential actors, that does seem to make the situation far easier. Like, it could be fairly clear to many international actors that there are 1-2 firms that might be making major breakthroughs. In that case, we might just need to worry about policing these firms. This seems fairly possible to me (if we can be somewhat competent).
I'd expect that market caps of these companies would be far higher if it were clear if there would be less competition later, and I'd equivalently expect these companies to do (even more) R&D.
I'm quite sure investors are quite nervous about the monopoly abilities of LLM companies.
Right now, I don't think it's clear to anyone where OpenAI/Anthropic will really make money 5+ years from now. It seems like [slightly worse AIs] often are both cheap / open-source and good enough. I think that both companies are very promising, but just that the future market value is very unclear.
I've heard that some of the Chinese strategy is, "Don't worry too much about being on the absolute frontier, because it's far cheaper to just copy from 1-2 steps behind."
I wasn't saying that "competition would greatly decrease the value of the marginal intelligence gain" in the sense of "things will get worse from where we are now", but in the sense of "things are generally worse from where they would be without such competition"