That’s an important point of clarification, thanks. I always appreciate your comments, Mr. Denkenberger.

There’s the idea of economic stimulus. John Maynard Keynes said that it would be better to spend stimulus money on useful projects (e.g. building houses), but as an intellectual provocation to illustrate his point, he said that if there were no better option, the government should pay people to dig holes in the ground and fill them back up again. Stimulating the economy is its own goal distinct from what the money actually gets spent to directly accomplish.

AI spending is an economic stimulus. Even if the data centres sit idle and never do anything economically valuable or useful — the equivalent of holes dug in the ground that were just filled up again — it could have a temporarily favourable effect on the economy and help prevent a recession. That seems like it’s probably been true so far. The U.S. economy looks recessarionary if you subtract the AI numbers.

However, we have to consider the counterfactual. If investors didn’t put all this money into AI, what would have happened? Of course, it’s hard to say. Maybe they just would have sat on their money, in which case the stimulus wouldn’t have happened, and maybe a recession would have begun by now. That’s possible. Alternatively, investors might have found a better use for their money, could have found more productive investments.

Regardless of what happens in the future, I don’t know if we’ll ever be able to know for sure what would have happened if there hadn’t been this AI investment craze. So, who knows.

(I think there are many things to invest in that would have been better choices than AI, but the question is whether, in a counterfactual scenario without the current AI exuberance, investors actually would have gone for any of them. Would they have invested enough in other things to stimulate the economy enough to avoid a recession?)

The stronger point, in my opinion, is that I don’t think anyone would actually defend spending on data centres just as an economic stimulus, rather than as an investment with an equal or better ROI as other investments. So, the general rule we all agree we want to follow is: invest in things with a good ROI, and don’t just dig and fill up holes for the sake of stimulus. Maybe there are cases where large investment bubbles prevent recessions, but no one would ever argue: hey, we should promote investment bubbles when growth is sluggish to prevent recessions! Even if there are one-off instances where that gambit pays off, statistically, overall, over the long term, that’s going to be a losing strategy.^[1]

1. ^{^}

   Only semi-relatedly, I’m fond of rule consequentialism as an alternative to act consequentialism. Leaving aside really technical and abstract considerations about which theory is better or more correct, I think, in practice, following the procedure 'follow the rule that will overall lead to the best consequences over the set of all acts' is a better idea than the procedure 'choose the act that will lead to the best consequences in this instance'. Although, of course, life is more complicated than either of these procedures allow, and there’s a lot of discernment that needs to be used on a case-by-case basis. (E.g., just individuating acts and categories of acts and deciding which rules apply to the situation you find yourself in is complicated enough. And there are rare, exceptional circumstances in which the normal rules might not make sense anymore.)

   Whenever someone tries to justify something that seems crazy or wrong, like something deceptive, manipulative, or Machiavellian, on consequentialist grounds, which typically you only see in fiction, but you also see on rare occasions in real life (and unfortunately sometimes in mild forms in the EA community), I always see flaws in the reasoning along these lines. The choice is presented as a false binary: e.g. spend $100 billion on AI data centres as an economic stimulus or do nothing. 
   
   This type of thinking overlooks that the number of possible options is almost always immensely large, and is mostly filled up by options you can’t currently imagine. People are creative and intelligent to the point of being unpredictable by you (or by anyone), so you simply can’t anticipate the alternative options that might arise if you don’t ram through your 'for the greater good' plan. But, anyway, that’s a big philosophical digression.

I typically don’t agree with much that Dwarkesh Patel, a popular podcaster, says about AI,^[1] but his recent Substack post makes several incisive points, such as:

Somehow this automated researcher is going to figure out the algorithm for AGI - a problem humans have been banging their head against for the better part of a century - while not having the basic learning capabilities that children have? I find this super implausible.

Yes, exactly. The idea of a non-AGI AI researcher inventing AGI is a skyhook. It’s pulling yourself up by your bootstraps, a borderline supernatural idea. It’s retrocausal. It just doesn’t make sense.

There are more great points in the post besides that, such as:

Currently the labs are trying to bake in a bunch of skills into these models through “mid-training” - there’s an entire supply chain of companies building RL environments which teach the model how to navigate a web browser or use Excel to write financial models.

Either these models will soon learn on the job in a self directed way - making all this pre-baking pointless - or they won’t - which means AGI is not imminent. Humans don’t have to go through a special training phase where they need to rehearse every single piece of software they might ever need to use.

… You don’t need to pre-bake the consultant’s skills at crafting Powerpoint slides in order to automate Ilya [Sutskever, an AI researcher]. So clearly the labs’ actions hint at a world view where these models will continue to fare poorly at generalizing and on-the-job learning, thus making it necessary to build in the skills that they hope will be economically valuable.

And:

It is not possible to automate even a single job by just baking in some predefined set of skills, let alone all the jobs.

We are in an AI bubble, and AGI hype is totally misguided.

1. ^{^}

   There are some important things I disagree with in Dwarkesh's post, too. For example, he says that AI has solved "general understanding, few shot learning, [and] reasoning", but AI has absolutely not solved any of those things. 

   Models lack general understanding, and the best way to see that is they can't do much useful in complex, real world contexts — which is one of the points Dwarkesh is making in the post. Few-shot learning only works well in situations where a model has already been trained on a giant amount of similar training examples. The "reasoning" in "reasoning models" is, in Melanie Mitchell's terminology, a wishful mnemonic. In other words, just naming an AI system something doesn't mean it can actually do the thing it's named after. If Meta renamed Llama 5 to Superintelligence 1, that wouldn't make Llama 5 a superintelligence.  
   
   I also think Dwarkesh is astronomically too optimistic about how economically impactful AI will be by 2030. And he's overfocusing on continual learning as the only research problem that needs to be solved, to the neglect of others.
   
   Dwarkesh's point about the variance in the value of human labour and the O-ring theory in economics also doesn't seem to make sense, if I'm understanding his point correctly. If we had AI models that were genuinely as intelligent as the median human, the economic effects would be completely disruptive and transformative in much the way Dwarkesh describes earlier in the post. General intelligence at the level of the median human would be enough to automate a lot of knowledge work. 
   
   The idea that you need AI systems equivalent to the top percentile of humans in intelligence or skill or performance or whatever before you can start automating knowledge work doesn't make sense, since most knowledge workers aren't in the top percentile of humans. This is such an obvious point that I worry I'm just misunderstanding the point Dwarkesh was trying to make. 

Good question. I’m less familiar with the self-driving car industry in China, but my understanding is that the story there has been the same as in the United States. Lots of hype, lots of demos, lots of big promises and goals, very little success. I don’t think plans count for anything at this point, since there’s been around 6-10 years of companies making ambitious plans that never materialized.

Regulation is not the barrier. The reason why self-driving cars aren’t a solved problem and aren’t close to being a solved problem is that current AI techniques aren’t up to the task; there are open problems in fundamental AI research that would need to be solved for self-driving to be solved. If governments can accelerate progress, it’s in funding fundamental AI research, not in making the rules on the road more lenient.

Seeing the amount of private capital wasted on generative AI has been painful. (OpenAI alone has raised about $80 billion and the total, global, cumulative investment in generative AI seems like it’s into the hundreds of billions.) It’s made me wonder what could have been accomplished if that money had been spent on fundamental AI research instead. Maybe instead of being wasted and possibly even nudging the U.S. slightly toward a recession (along with tariffs and all the rest), we would have gotten the kind of fundamental research progress needed for useful AI robots like self-driving cars.

@Richard Y Chappell🔸, would you please do me the courtesy of acknowledging that you misunderstood my argument? I think this was a rather uncharitable reading on your part and would have been fairly easy to avoid. Your misreading was not explicitly forestalled by the text but not supported by the text, either, and there was much in the text to suggest I did not hold the view that you took to be the thesis or argument. I found your misreading discourteous for that reason. 

Much of the post is focused on bad intellectual practices, such as:

• Not admitting you got a prediction wrong after you got it wrong
• Repeating the same prediction multiple times in a row and repeatedly getting it wrong, and seemingly not learning anything
• Making fake graphs with false data, no data, dubious units of measurement, no units of measurements, and other problems or inaccuracies
• Psychological or social psychological biases like millennialist cognitive bias, bias resulting from the intellectual and social insularity of the EA community, and possible confirmation bias (e.g. why hasn't Toby Ord's RL scaling post gotten much more attention?)
• Acceptance or tolerance of arguments and assertions that are really weak, unsupported, or sometimes just bad

I don't interpret your comment as a defense or endorsement of any of these practices (although I could if I wanted to be combative and discourteous). I'm assuming you don't endorse these practices and your comment was not intended as a defense of them.

So, why reply to a post that is largely focused on those things as if the thesis or argument or thrust of the post is something other than that, and which was not said in the text? 

On the somewhat more narrow point of AI capabilities optimism, I think the AI bubble popping within the next 5 years or so would be strong evidence that the EA community's AI capabilities optimism has been misplaced. If the large majority of people in the EA community only thought there's a 0.1% chance or a 1% chance of AGI within a decade, then the AI bubble popping might not be that surprising from their point of view. But the actual majority view seems to be more like a 50%+ chance of AGI within a decade. My impression from discussions with various people in the EA community is that many of them would find it surprising if the AI bubble popped.

The difference between a 50%+ chance of AGI within a decade and a 0.1% chance is a lot from an epistemic perspective, even if, just for the sake of argument, it makes absolutely no difference for precautionary arguments about AI safety. So, I think misestimating the probability by that much would be worthy of discussion, even if — for the sake of argument — it doesn't change the underlying case for AI safety.

It is especially worthy of discussion if the misestimation is influenced by bad intellectual practices, such as those listed above. All the information needed to diagnose those intellectual practices as bad is available today, so the AI bubble popping isn't necessary. However, people in the EA community may be reluctant to give a hard look at them without some big external event like an AI bubble popping shaking them up. As I said in the post, I'm pessimistic that even after the AI bubble pops, people in the EA community will, even then, be willing to examine these intellectual practices and acknowledge that they're bad. But it's worth a shot for me to say something about it anyway. 

There are many practical reasons to worry about bad intellectual practices. For example, people in AI safety should worry about whether they're making existential risk from AGI better or worse, and having bad intellectual practices on a systemic or widespread level will make it more likely they'll screw this up. Or, given that, according to Denkenberger in another comment on this post, funding around existential risk from AGI has significantly taken away funding around other existential risks, overestimating existential risk from AGI based on bad intellectual practices might (counterfactually) increase total existential risk just by causing funding to be less wisely allocated. And, of course, there are many other reasons to worry about bad intellectual practices, especially if they are prevalent in a community and culturally supported by that community. 

We both could list reasons on and on why thinking badly might lead to doing badly. Just one more example I'll bring up is that, in practice, most AI safety work seems to make rather definite, specific assumptions about the underlying technical nature of AGI. If AI safety has (by and large) identified an implausible AI paradigm to underlie AGI out of at least several far more plausible and widely-known candidates (largely as a result of the bad intellectual practices listed above), then AI safety will be far less effective at achieving its goals. There might still be a strong precautionary argument for doing AI safety work on even that implausible AI paradigm, but given that AI safety, has, in practice, for the most part, bet on one specific horse and not the others, it is a problem to pick the wrong paradigm. You could maybe argue for an allocation of resources weighted to different AI paradigms based on their perceived plausibility, but that would still result in a large reallocation of resources if the paradigm AI safety is betting on is highly implausible and there are several other candidates that are much more plausible. So, I think this is a fair line of argument.

What matters is not just some unidimensional measure of the EA community’s beliefs like the median year of AGI or the probability of AGI within a certain timeframe or the probability of global catastrophe from AGI (conditional on its creation, or within a certain timeframe). If bad intellectual practices make that number go up too high, it's not necessarily just fine on precautionary grounds, it can mean existential risk is increased.