EG

Erich_Grunewald 🔸

Researcher @ Institute for AI Policy and Strategy
2813 karmaJoined Working (6-15 years)Berlin, Germanywww.erichgrunewald.com

Bio

Anything I write here is written purely on my own behalf, and does not represent my employer's views (unless otherwise noted).

Comments
305

(To be clear, I do think many of these charities do some good and are run with the best of intentions, etc. But I still also stand by the statement in the parent comment.)

That is the most PR-optimized list of donations I have ever seen in my life.

Thanks for sharing this. I did an Erasmus exchange year in Italy in 2010-11 that was very important for my personal growth, although it was not particularly beneficial professionally or academically.

Nice work!

On AI chip smuggling, rather than the report you listed, which is rather outdated now, I recommend reading Countering AI Chip Smuggling Has Become a National Security Priority, which is essentially a Pareto improvement over the older one.

I also think Chris Miller's How US Export Controls Have (and Haven't) Curbed Chinese AI provides a good overview of the AI chip export controls, and it is still quite up-to-date.

On timelines, I think it's worth separating out export controls on different items:

  • Controls on AI chips themselves start having effects on AI systems within a year or so probably (say 6-12 months to procure and install the chips, and 6-18 months to develop/train/post-train a model with them), or even sooner for deployment/inference, i.e. 1-2 years or so.
  • Controls on semiconductor manufacturing equipment (SME) take longer to have an impact as you say, but I think not that long. SMIC (and therefore future Ascend GPUs) is clearly limited by the 2019 ban on EUV photolithography, and I would say this was apparent as early as 2023. So I think SME controls instituted now would start having an effect on chip production in the late 2020s already, and on AI systems 1-2 years after that.

Most other relevant products (e.g., HBM and EDA software) probably fall between those two in terms of how quickly controls affect downstream AI systems.

So that means policy changes in 2025 could start affecting Chinese AI models in 2027 (for chips) and around 2030 (for SME) already, which seems relevant to even short-timeline worlds. For example, Daniel Kokotajlo's median for superhuman coders is now 2029, and IIUC Eli Lifland's median is in the (early?) 2030s.

But I would go further to say that export controls now can substantially affect compute access well into the 2030s or even the 2040s. You write that

the technical barriers [to Chinese indigenization of leading-edge chip fabrication] are higher today, but not so high that intense Chinese investment can't dent it over the course of a decade. SMEE is investing in laser-induced discharge plasma tech, with rumored trial production as soon as the end of this year. SMIC is using DUV more efficiently for (lower-yield, but still effective) chip production. There's also work on Nanoimprint lithography, immersion lithography, packaging, etc. And that won't affect market shares, until it does.

I won't have time to go into great detail here, but I have researched this a fair amount and I think you are too bullish on Chinese leading-edge chip fabrication. To be clear, China can and will certainly produce AI chips, and these are decent AI chips. But they will likely produce those chips less cost-efficiently and at lower volumes due to having worse equipment, and they will have worse performance than TSMC-fabbed chips due to using older-generation processes. The lack of EUV machines, which will likely last at least another five years and plausibly well into the 2030s, alone is a very significant constraint.

On SMEE and SMIC in particular -- you write:

SMEE is investing in laser-induced discharge plasma tech, with rumored trial production as soon as the end of this year.

SMEE was established 23 years ago to produce indigenous lithography, and 23 years later it still has essentially no market share, and it still has not produced an immersion DUV machine, let alone an EUV machine, which is far more difficult. I would not be surprised if, when the indigenous Chinese immersion DUV machine does finally arrive, it is a SiCarrier (or subsidiary) product and not an SMEE product.

SMIC is using DUV more efficiently for (lower-yield, but still effective) chip production.

In what sense do you mean SMIC is using DUV more efficiently? It is using immersion DUV multi-patterning (with ASML machines) to compensate for its lack of EUV machines. But as you note this means worse yield and lower throughput. I don't see any sense in which SMIC is using DUV more efficiently; it's just using it more, in order to get around a constraint that TSMC doesn't have. In any case, multi-patterning with immersion DUV can only take you so far; there's likely a hard stop around what's vaguely called 2 nm or 1.4 nm process nodes, even if you do multi-patterning perfectly. (For reference, TSMC is starting mass production on its "2 nm" process this year.)

On the oil analogy, it seems from

The long-term winners were definitely not the groups that extracted or refined the oil, even though they made lots of money - it was the countries that consumed the oil and built industrial capacity leading up to WWII, and could then use the controlled supply of oil. ... And as far as I can tell, no-one is restricting Chinese companies from using compute right now - they don't own it, but can use the same LLMs I do.

that you think ownership of compute does not substantially influence who will have or control the most powerful AI systems? I disagree; I think it will impact both AI developers and also companies relying on access to AI models. First, AI developers -- export controls put the Chinese AI industry as a whole at a compute disadvantage, which we see in the fact that they train less compute-intensive models, for a few reasons:

  • It is generally unappealing for major AI developers to merely rent GPUs they don't own, as a result of which they often build their own data centers (xAI, Google) or rely on partnerships for exclusive access (OpenAI, Anthropic). I think the main reasons for this are cost, (un)certainty, and greater control over the cluster set-up.
    • Chinese companies cannot build their own data centers with export-controlled chips without smuggling, and cannot embark on these partnerships with American hyperscalers. If they want to use cutting-edge GPUs, they must either rely on smuggling (which means higher prices and smaller quantities), or renting from foreign cloud providers.
  • The US likely could, if and when it wanted to, deny access of compute via the cloud to Chinese customers, at least large-scale use and at least for the large hyperscalers. So for Chinese AI developers to rely on foreign cloud compute gives the US a lot of leverage. (There are some questions around how feasible it is to circumvent KYC checks, and especially whether the US can effectively ensure these checks are done well in third countries, but I think the US could deny China most of the world's rentable cloud compute in this way.)
  • Chinese privacy law makes it harder for Chinese AI developers to use foreign cloud compute, at least for some use cases. I'm not sure exactly how strong this effect is, but it seems non-negligible.
  • For deployment/inference, you may want to have your compute located close to your users, as that reduces latency.
  • In the event of an actual conflict over or involving AI, you can seize compute located on territory you control. I hope that doesn't happen obviously, but it's definitely a reason why as an AI developer you'd prefer to use compute located in your own country, than located in a rival country or one of the rival's allies or partners.

That's AI developers. As for the AI industry more broadly, there are barriers for Chinese companies wanting to use US models like ChatGPT or Claude, which, for example, is likely one reason why Manus moved to Singapore. So the current disparity in who owns compute and where it is located means Chinese AI developers are relatively compute-poor, and since Chinese companies rely substantially on domestic Chinese models, it seems to me like the entire Chinese AI industry is impacted by these restrictions.

Also, I disagree that oil "only mattered because it enabled economic development". In WWII especially, oil was necessary for fuel-hungry militaries to function. I think AI will also be militarily important even ignoring its effects on economic development, though maybe less so than oil.

On the other hand, I think you're wrong in saying that "the chip supply chain has unique characteristics [compared to oil,] with extreme manufacturing concentration, decades-long development cycles, and tacit knowledge that make it different" - because the same is true for crude oil extraction! What matters is who refines it, and who buys it, and what it's used for.

I think the technical barriers to developing EUV photolithography from scratch are far higher than anything needed to extract, refine, or transport oil. I also think the market concentration is far higher in the AI chip design and semiconductor industries. There's no oil equivalent to TSMC's ~90% leading-edge logic chip, NVIDIA's ~90% data center GPU, or ASML's 100% EUVL machine market shares.

Second, if we're talking about takeoff after 2035, the investments in China are going to swamp western production. (This is the command economy advantage - though I could imagine it's vulnerable to the typical failure modes where they overinvest in the wrong thing, and can't change course quickly.)

Are you sure? I would guess that the chip supply chain used by NVIDIA has more investment than the Chinese counterpart. For example, according to a SEMI report, China will spend $38bn on semiconductor manufacturing equipment in 2025, whereas the US + Taiwan + South Korea + Japan is set to spend a combined ~$70bn. I would guess it looks directionally similar for R&D investment, though the difference may be smaller there.

For moderately short, 2-6 year timelines, the timelines for chip fabs are long enough that we're mostly locked in not just to overall western dominance via chips produced in Taiwan, but because fabrication plans built today are coming online closer to 2029, and the rush to build Chinese fabrication plants is already baked in. And that's just the fabs - for the top chips, the actual chip design usually takes as long or longer than building the plant.

I was under the impression the AI chip design process is more like 1.5-2 years, and a fab is built in 2-3 years in Taiwan or 4 years for the Arizona fab. It sounds like you think differently? Whatever it is, I would guess it's roughly similar across the industry, including in China. That seems like, if my numbers are right, it leaves enough room for policy now to influence the relative compute distribution of nations 5-6 years from now.

Interesting!

While the small body size of sardines and anchovies means that many individuals must be killed to produce a given amount of food, thereby scaling up the moral weight, a meaningful moral cost calculation should extend beyond these direct first-order consequences to account for indirect higher-order consequences, especially given that all food production invariably involves some level of collateral damage, as will be discussed further on.

On the other hand, sardines really are very small, and I reckon you'd need on the order of 100x as many sardines as you'd need salmons to get the same amount of calories. I wonder how many small animals would die to produce the amount of calories of plant-based food you'd get from a sardine? I'd guess <<0.1, but I'd be interested in seeing estimates here as it seems pretty cruxy.

As for traitor, I think the only group here that can be betrayed is humanity as a whole, so as long as one believes they're doing something good for humanity I don't think it'd ever apply.

Hmm, that seems off to me? Unless you mean "severe disloyalty to some group isn't Ultimately Bad, even though it can be instrumentally bad". But to me it seems useful to have a concept of group betrayal, and to consider doing so to be generally bad, since I think group loyalty is often a useful norm that's good for humanity as a whole.

Specifically, I think group-specific trust networks are instrumentally useful for cooperating to increase human welfare. For example, scientific research can't be carried out effectively without some amount of trust among researchers, and between researchers and the public, etc. And you need some boundary for these groups that's much smaller than all humanity to enable repeated interaction, mutual monitoring, and norm enforcement. When someone is severely disloyal to one of those groups they belong to, they undermine the mutual trust that enables future cooperation, which I'd guess is ultimately often bad for the world, since humanity as a whole depends for its welfare on countless such specialised (and overlapping) communities cooperating internally.

Load more