Daniel Kokotajlo on AI 2027: Race Dynamics, Robot Economies, and the Narrow Path to a Good Future

By Luisa Rodriguez  |    Watch on Youtube   |   Listen on Spotify   |    Read transcript

Episode summary

When Daniel Kokotajlo talks to security experts at major AI labs, they tell him something chilling: “Of course we’re probably penetrated by the CCP already, and if they really wanted something, they could take it.”

This isn’t paranoid speculation. It’s the working assumption of people whose job is to protect frontier AI models worth billions of dollars. And they’re not even trying that hard to stop it — because the security measures that might actually work would slow them down in the race against competitors.

Daniel is the founder of the AI Futures Project and author of AI 2027, a detailed scenario showing how we might get from today’s AI systems to superintelligence by the end of the decade. Over a million people read it in the first few weeks, including US Vice President JD Vance. When Daniel talks to researchers at Anthropic, OpenAI, and DeepMind, they tell him the scenario feels less wild to them than to the general public — because many of them expect something like this to happen.

Daniel’s median timeline? 2029. But he’s genuinely uncertain, putting 10–20% probability on AI progress hitting a long plateau.

When he first published AI 2027, his median forecast for when superintelligence would arrive was 2027, rather than 2029. So what shifted his timelines recently? Partly a fascinating study from METR showing that AI coding assistants might actually be making experienced programmers slower — even though the programmers themselves think they’re being sped up. The study suggests a systematic bias toward overestimating AI effectiveness — which, ironically, is good news for timelines, because it means we have more breathing room than the hype suggests.

But Daniel is also closely tracking another METR result: AI systems can now reliably complete coding tasks that take humans about an hour. That capability has been doubling every six months in a remarkably straight line. Extrapolate a couple more years and you get systems completing month-long tasks. At that point, Daniel thinks we’re probably looking at genuine AI research automation — which could cause the whole process to accelerate dramatically.

At some point, superintelligent AI will be limited by its inability to directly affect the physical world. That’s when Daniel thinks superintelligent systems will pour resources into robotics, creating a robot economy in months.

Daniel paints a vivid picture: imagine transforming all car factories (which have similar components to robots) into robot production factories — much like historical wartime efforts to redirect production of domestic goods to military goods. Then imagine the frontier robots of today hooked up to a data centre running superintelligences controlling the robots’ movements to weld, screw, and build. Or an intermediate step might even be unskilled human workers coached through construction tasks by superintelligences via their phones.

There’s no reason that an effort like this isn’t possible in principle. And there would be enormous pressure to go this direction: whoever builds a superintelligence-powered robot economy first will get unheard-of economic and military advantages.

From there, Daniel expects the default trajectory to lead to AI takeover and human extinction — not because superintelligent AI will hate humans, but because it can better pursue its goals without us.

But Daniel has a better future in mind — one he puts roughly 25–30% odds that humanity will achieve. This future involves international coordination and hardware verification systems to enforce AI development agreements, plus democratic processes for deciding what values superintelligent AIs should have — because in a world with just a handful of superintelligent AI systems, those few minds will effectively control everything: the robot armies, the information people see, the shape of civilisation itself.

Right now, nobody knows how to specify what values those minds will have. We haven’t solved alignment. And we might only have a few more years to figure it out.

Daniel and host Luisa Rodriguez dive deep into these stakes in today’s interview.

This episode was recorded on September 9, 2025.

Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: CORBIT
Coordination, transcriptions, and web: Katy Moore

The interview in a nutshell

Daniel Kokotajlo, founder of the AI Futures Project and former OpenAI employee, argues that we could see artificial general intelligence (AGI) by 2029. His narrative forecast, AI 2027, describes a plausible and rapid path to superintelligence, dangerously accelerated by US-China race dynamics that could culminate in a takeover by misaligned AI. He believes our default trajectory will have terrible consequences, unless we get extremely lucky, and that a safe outcome requires urgent international coordination and transparency.

1. AGI could arrive by 2029, driven by accelerating progress

Daniel’s median forecast for AGI is 2029, with an 80–90% chance it arrives in the next decade. This view is shaped by several key factors:

• Extrapolating current trends: Progress is quantifiable and rapid. A key metric is METR’s finding that the length of complex coding tasks AI can complete reliably doubles every six months. Extrapolating this trend suggests AIs will be capable of automating large parts of AI research within a few years.
• Progress will likely accelerate, not slow down: Daniel anticipates two major drivers of acceleration that could overcome resource bottlenecks:
  • AI automating AI research: As AIs become more capable, they will be used to speed up their own development, creating a feedback loop that leads to explosive progress.
  • Emergent “agency skills”: AIs will eventually develop the core planning and adaptation skills needed to generalize from short tasks (e.g., one hour) to extremely long ones (e.g., one year), a key step towards superhuman capabilities.
• Fundamental roadblocks are unlikely: Daniel is sceptical of arguments that current AI paradigms face insurmountable barriers (like data inefficiency or a lack of “online learning”). He points to the poor track record of experts predicting what deep learning can’t do and argues that immense financial investment will likely solve these problems as they become bottlenecks.

2. US-China race dynamics will create immense pressure and risk

A key accelerant in his forecast is the escalating competition between the US and China, which he believes will override normal safety considerations.

• Industrial espionage is a near-certainty: He considers it highly plausible that China will steal the model weights of a frontier AI model from a US company.
• AI companies are already vulnerable: This view is shared by security experts within top AI labs, who operate under the assumption that they have already been infiltrated by state actors like the Chinese Communist Party (CCP).
• Speed is prioritised over security: Labs are consciously choosing not to implement robust security measures because doing so would slow them down and create a competitive disadvantage.
• A “wartime” mentality will take hold: As the strategic importance of AGI becomes undeniable, nations will enter a state of emergency akin to a wartime mobilisation, diverting vast resources to win the race at any cost.

3. A superintelligent AI could rapidly build a robot economy for takeover

Once AGI is achieved within data centres, the primary bottleneck to real-world influence becomes physical embodiment and infrastructure. Daniel forecasts a rapid and radical economic transformation to overcome this.

• Robots are the logical next step: After automating research, superintelligent AIs will direct their efforts toward building a massive robot economy (e.g., factory, construction, and mining robots) to gain physical agency and strategic advantage.
• The scaleup will be unprecedentedly fast: Managed by superintelligence, this economic transformation could happen ~5 times faster than historical precedents like the US industrial mobilisation during World War II.
• Humans will be temporary “moist robots”: Initially, AIs would direct a massive human labour force, using technology to give step-by-step instructions to unskilled workers to build factories and infrastructure.
• Autonomous replication: Eventually, this will transition to a fully autonomous robot economy capable of exponential growth. At the extreme, one could imagine a potential doubling time of weeks or even hours, similar to biological systems like grass or algae.

4. A good outcome requires coordination and transparency, not luck

Daniel warns that the “good ending” (the slowdown ending) in AI 2027 — where developers unilaterally pause to solve alignment at the last minute — is a “terrifying path,” not one to shoot for. A truly robust strategy requires systemic solutions.

• International coordination is essential: The only realistic way to avoid a catastrophic race to the bottom is through domestic regulation and verifiable international agreements to ensure AI is developed safely and slowly.
• Two technical pillars are needed for effective deals:
  Hardware verification: We must develop technology to reliably track AI chips and monitor their computational activity. This is crucial for enabling trustless, verifiable international treaties on AI development.
  Mandatory transparency: Governments must require AI companies to be transparent about their models’ capabilities, safety risks, and concerning incidents.
• Whistleblower protections are a crucial last resort: In the absence of strong regulation, protecting employees who speak out about unmitigated risks is one of our last lines of defence.

Highlights

Could China really steal frontier model weights?

Luisa Rodriguez: How plausible is it that China would steal the model weights from a frontier US AI company?

Daniel Kokotajlo: Quite plausible. This type of industrial espionage is happening all the time. The US and China are both constantly hacking each other and infiltrating each other and so forth. This is just what the spy networks do, and it’s just a question of will they devote lots of resources to it. And the answer is yes, of course they will. Because AI will be increasingly important over the next year, so they probably already have devoted a bunch of resources to it.

And this is not just my opinion. This is also the opinion of basically all the experts I’ve talked to in the industry and outside the industry. I’ve talked to people at security at these companies who are like, “Of course we’re probably penetrated by the CCP already, and if they really wanted something, they could take it. Our job is to make it difficult for them and make it annoying and stuff like that.”

I think this might be a point to mention is that, as wild as AI 2027might read to people not working at Anthropic or OpenAI or DeepMind, it is less wild to people working at these companies — because many of the people at these companies expect something like this to happen. Not all of them, of course. There’s lots of controversy and diversity of opinion even within these companies.

But I think part of the motivation to write this is to sort of wake up the world. Sam Altman is going around talking about how they’re building superintelligence in the next few years. Dario Amodei doesn’t call it superintelligence, but he’s also talking about that. He calls it “powerful AI.” These companies are explicitly trying to build AI systems that are superhuman across the board. And according to statements of their leaders, they feel like they’re a couple years away.

…

I think eventually, before the end, governments will be woken up sufficiently that they will consider other countries doing an intelligence explosion an existential threat to their country — in a similar way to if you’re a country that doesn’t have nukes and then your neighbour, who is a rival of you, has a nuclear programme, you consider that a huge deal. But perhaps even more intense than that, because there are strong norms against using nukes in this world, which might make you hope that even though your neighbours have nukes, they’re not going to use them against you. But there’s no strong norm against using superintelligence against your neighbours, you know?

Luisa Rodriguez: Right.

Daniel Kokotajlo: In fact, it’s not even like a strong norm. It’s like, this is the plan, you know? Like, you’ve talked to people at the companies and it’s sort of like, “We’re going to build superintelligence first, before China, and then we will beat China.”

What does “beating China” look like, exactly? Well, you know, they don’t say this so much publicly, but we depict in AI 2027 what that might look like. This is, unfortunately, the world that the companies are sort of building towards and lurching towards, and we can hope that it’s not going to materialise.

Why we might get a robot economy incredibly fast

Luisa Rodriguez: People talk about robotics as this incredibly hard problem that is made extra difficult, counterintuitively, because many physical tasks for humans feel extremely intuitive and easy. But when you actually try to figure out what’s going on there, it turns out to be surprisingly hard to teach an artificial physical being to repeat the same things.

How confident are you that it is even possible to build super-capable robots on the timescales you’re talking about?

Daniel Kokotajlo: Well, it’s definitely possible in principle to build them.

Luisa Rodriguez: Yeah, we do it.

Daniel Kokotajlo: Yeah. If a human can do it, then it should be possible to design a robot that can do it as well. The laws of physics will allow that.

And I think also we’re not talking here about absolutely replicating all the functions of the human body. Just like how we have birds and planes, and the birds are able to repair themselves over time in a way that planes can’t. That’s an advantage birds still have even after 100 years. There might similarly be some niche dimensions in which humans are better than the robots, at least for a while.

But take prototypes like the Tesla Optimus robot, and just imagine that it’s hooked up to a data centre that has superintelligences running on it, and the superintelligences are steering and controlling its arms so that it can weld this part of the new thing that they’re welding, or screw in this part here or whatever — and then when they’re finished, move on to the next task and do that too.

That does not at all seem out of reach. It seems like something superintelligences should be able to do. There’s already been a decent pace of progress in robotics in the last five to 10 years. And then I’m just like, well, the progress is going to go much faster when there are superintelligences driving it.

And there’s a separate question of, what about the actual scaleup? So the superintelligence is learning how to operate the robots — and there I would be like, it’s going to be incredibly fast. By definition, they’re going to be as data efficient as humans, for example, and probably better in a bunch of ways as well. But then there’s the question of physically, how do you produce that many robots that fast? I think that’s going to be more of a bottleneck.

We talked about this a little bit in AI 2027. There’s millions of cars produced every year, and the types of components and materials that go into a robot are probably similar to the types of components and materials that go into a car. I think if you were an incredibly wealthy company that had built superintelligence, and you were in the business of expanding into the physical world, you’d probably buy up a bunch of car factories or partner with car factories and convert them to produce robots of various kinds.

And to be clear, we don’t just mean humanoid robots. That’s one kind of robot that you might build, but more generally you’d want factory robots, autonomous vehicles, mining robots, construction robots — basically some package of robots that enables you to more effectively and rapidly build more factories, which then can build more robots, and more factories, and so forth. You also would want to make lots of machine tools to be in those factories, different types of specialised manufacturing equipment, different types of ore-processing equipment. It would be sort of like the ordinary human economy, except more automated.

And also, to be clear, I think that at first you would use the human economy. So at first you would be paying millions of people to come work in your special economic zones and build stuff for you and also be in your factories. And this would go better than it does normally, because you’d have this huge superintelligent labour force to direct all of these people. So you can hire unskilled humans who don’t know anything about construction, and then you could just have a superintelligence looking at them through their phone, telling them, “This part goes there, that part goes there. No, not there, the other way” — and just actually coaching them through absolutely everything. Kind of like a “moist robot,” you might say.

…

It’s not wartime yet, but data centre construction has scaled up massively. The amount of compute AI companies are using for training has scaled up massively. How fast? Something like 3x a year or something. That’s still orders of magnitude over the course of several years. And again, that’s non-wartime, ordinary humans. So wartime economy superintelligence should be substantially faster than that, just by superintelligences directing humans to go around and restructure their factories, and take apart their car for materials, and transport the materials to this smelter or whatever.

Once you actually have robots that are doing most of the work, then things will go faster still. To put an upper bound on it, it should be possible in principle to have a fully autonomous robot economy that doubles in size every few weeks, and possibly every few hours.

The reason for this is that we already have examples in nature of macro-scale objects that double that fast. Like grass doubles every few weeks, and all it needs is sun and a little bit of water. So in principle it should be possible to design a collection of robot stuff that takes in sun and a bit of water as input and then doubles every few weeks. If grass can do it, then it’s physically possible. And algae doubles in a few hours. And maybe that’s a little different because it’s so small, and maybe it gets harder as you get bigger or something.

But the point is, it does seem like the upper bound on how fast the robot economy could be doubling is scarily high. Very fast. Very fast. And it won’t start like that immediately, but first you have the human wartime-economy thing, and then you build the robots, and then the robots get improved, and you make better robots and better robots — and then eventually you’re getting to those sorts of crazy doubling times.

Updates Daniel's made since publishing AI 2027

Daniel Kokotajlo: In some sense the thing that shifted our evidence was we just made some significant improvements to our timelines model, and the new model says a different thing than what the old model says. So I’m going with the new model.

But in terms of empirical evidence or updates that have happened in the world, I would say the biggest one is the METR horizon-length study that came out shortly before we published AI 2027.

So they have a big collection of coding tasks that are organised by how long it takes a human to complete the tasks, ranging from a second or so to eight hours. And then they have AIs attempt the tasks, and they find that for any particular AI, it can generally do the tasks below a certain length, but not do the tasks above a certain length.

And this is already kind of interesting, because it didn’t necessarily have to be that way. But they’re finding that the crossover point, the length of tasks that the AIs can usually do is lengthening year over year. The better AIs are able to do longer tasks more reliably. And also interestingly, it’s forming a pretty straight line on the graph. So they’ve got a doubling time of about every six months: the length of coding tasks that AIs can do doubles.

And that’s great. We didn’t have that before. Now that that data came out, we can extrapolate that line and say, maybe they’ll be doing one-month-long tasks in a few years, maybe they’ll be doing one-year-long tasks like two years after that. So that’s wonderful. And I think that by itself kind of shifted my timelines back a little bit.

Then another thing that came out is another METR study. They did an uplift study to see how much of a speedup programmers were getting from AI assistants. And to their surprise — and to most people’s surprise — they found that actually they were getting a speed-down: they were going slower because of AI assistants.

Now, to be fair, it was a really hard mode for the AIs, because they were really experienced programmers working on really big established codebases, and they were mostly programmers who didn’t have much experience using AI tools. So it was kind of like hard mode for AI. If AI can speed them up, then it’s really impressive. But if it can’t speed them up, well, maybe it’s still speeding up other types of coding or other types of programmers.

Anyhow, they found that it didn’t speed things up. So that is some evidence in general that the AIs are less useful. But perhaps more importantly, they found that the programmers in the study were systematically mistaken about how fast they were being sped up by the AIs. So even though they were actually being slowed down, they tended to think they were being sped up a little bit. This suggests that there’s a general bias towards overestimating the effectiveness of AI coding tools.

And that is helpful, because anecdotally, when I go talk to people at Anthropic or OpenAI or these companies, they will swear by their coding assistants and say that it’s helping them go quite a lot faster. It differs a lot. I have talked to some people who say they’re basically not speeding up at all, but then I’ve also talked to people who say they think that overall progress is going twice as fast now thanks to the AIs. So it’s helpful to have this METR study, because it suggests basically that the more bullish people are just wrong and that they’re biased.

And that’s a huge relief, because suppose that current AI assistants were speeding things up by 25%. Well, according to METR’s horizon-length study, they’re only able to do roughly one-hour tasks — depends on what level of reliability you want. But if you extrapolate the trend and they’re doing one-month tasks, presumably the speedup would be a lot more, right? By contrast, if you think that there’s basically negligible speedup right now, then that gives you a lot more breathing room to think that it’s going to be a while before there’s a significant speedup.

How to get to even better outcomes

Luisa Rodriguez: Are there any things that you think are robustly good, that you already have takes on that you think will probably stick as you keep thinking about it?

Daniel Kokotajlo: I think that international coordination is pretty robustly good if you do it right. The question is getting the details right.

In the short term, I would love to see more investment in hardware-verification technology, because that’s an important component of future deals. I think that relying on mutual trust and goodwill is unfortunately not good, because there’s probably not going to be much trust and goodwill in the future — if there’s any right now — between the US and China. So instead you need the ability for them to actually verify that the deal is being complied with. So there’s a whole packet of hardware-verification technology that I wish more research was being done into, more R&D funding, et cetera.

And then also transparency in the AI companies. I think that a big general source of problem is that information about what’s happening and what will soon happen is heavily concentrated in the companies themselves and the people they deign to tell.

And this situation is not so big of a deal right now, while the pace of progress is reasonably slow. If OpenAI is sitting on some exciting new breakthrough, probably they’re going to put it up in a product six months from now, or some other company will six months from now. And it’s not that exciting. It’s not like a big deal, right?

But if OpenAI or Anthropic or some other company has just fully automated AI research and has this giant corporation within a corporation of AIs autonomously doing stuff, it’s unacceptable for it to take six months for the public to find out that that’s happening. Who knows what could have happened in those six months inside that data centre.

…

There’s a sort of arms race dynamic, where both the US and China are worried that if they don’t rapidly allow their AIs to automate the AI research and then build a whole bunch of weapons and robots and so forth, then the other side will — and then they’ll be able to win wars, possibly even dismantle nuclear deterrence, et cetera.

So there’s often just very strong demand from the leaders of China and the US and other countries to come to some sort of arrangement about what we’re going to do and what we’re not going to do, and how fast we’re going to go, and things like that. But the core problem is that they don’t trust each other. Both sides are concerned that they could agree to some sort of deal, but then secretly cheat and have an unmonitored data centre somewhere that’s got self-improving AIs running on it. So in order for such deals to happen, there needs to be some way to verify them.

That means things like tracking the chips. You don’t have to necessarily get all the chips, but you have to get a very large majority of the chips, so that you can be reasonably confident that whatever data centre they have somewhere in a black site is not a huge threat, because it’s small in comparison to the rest.

And ideally, you don’t just want to track the locations of the chips, but you also want to track what’s going on on the chips. You want to have some sort of mechanism that’s saying, we’ve banned training this type of AI, but we’re allowing inference, for example. So there’s some device that’s ensuring that the chip is not training, but is instead just doing inference.

And I think that it’s relatively easy to get to the point where you can track the chips and know are they on or off, and where are they? But probably more research is needed to get to the point where you can also distinguish between what’s going on in the chips.

And then even more research would be needed to get to the point where you can do that in a way that’s less costly for both sides. Because if people are allowing that sort of mutual penetration, that mutual verification, then naturally they’re going to be concerned about our state secrets leaking, things like that. So one of the design considerations of these hardware devices is that they be able to enforce these types of agreements, but without also causing those problems.

So this is a technical problem, and progress is being made on it. But I would love to see it funded much more, and much more work into it.

What a good post-AGI power distribution looks like

Luisa Rodriguez: How do you think the optimal or a realistically very good world looks in terms of concentration of power? Do we have a world government? Is power no longer in the hands of AI companies? How do we distribute power?

Daniel Kokotajlo: I think that if you have coordination and regulation early, you can maybe get some sort of distributed takeoff — where rather than a couple major AI projects, there’s millions, billions of different tiny GPU clusters, individual people owning a GPU or something, and AI progress is gradually happening in this distributed way across all these different factions.

But that’s just not what’s going to happen by default. That’s not the shape of the technology. There are huge returns to scale, huge returns to doing massive training runs and having huge data centres and things like that.

So I think that unless there’s some sort of international coordination to make that distributed world happen, we will end up in a very concentrated world where there’s like one to five giant networks of data centres owned by one to five companies, possibly in coordination with their governments. And in those data centres there’ll be massive training runs happening, and then the results of those training runs will be… Basically, there’ll be many copies of AIs. Rather than a million different AIs, there’ll be three or four different AIs in a million different copies each.

And this is just a very inherently power-concentrating thing. If you’ve only got one to five companies and they each have one to three of their smartest AIs in a million copies, then that means there’s basically 10 minds that between those 10 minds get to decide almost everything, if they’re superintelligent. There’s 10 minds, such that the values and goals that those minds have determine the giant armies of robots and what humans are being told things on their cell phones. All of that is directed by the values of one to 10 minds.

And then it’s like, who gets to decide what values those minds have? Well, right now, nobody — because we haven’t solved the alignment problem, so we haven’t figured out how to actually specify the values.

But hypothetically, if we make enough progress that we can scientifically write down, “We want them to be like this” and then it will happen — the training process will work as intended and the minds will have exactly these values — then it’s like, OK, I guess the CEO gets to decide. And that’s also terrifying, because that means you have maybe one to 100 people who get to decide the values that reshape the world. And it could literally be one potentially.

So that’s terrifying. And that’s one of the things I think we need to solve with our coordination plan. We need to design some sort of domestic regulation and international regime that basically prevents that sort of concentration of power from happening.

I should add that one way to spread out the power is by having there be a governance structure for the AI mind. So even if you only have 10 AIs, if there’s a governance structure that decides what values the AIs have to have that’s based on, for example, voting, where everyone gets a vote, then that’s a way of spreading out the power. Because even though you have these 10 minds, the values that they have were decided upon by this huge population.

So the world I would like to see, that I think is easier to achieve and more realistic than the “billion different GPUs” world that I described earlier, is a world where there still is this sort of concentration in a few different AIs, but there’s this huge process for deciding what values the AIs have. And that process is a democratic process that results in things like, “All humans deserve the following rights. All humans will have this share of the profits from our endeavours.” Things like that.