I should link the survey directly here: https://aaai.org/wp-content/uploads/2025/03/AAAI-2025-PresPanel-Report-FINAL.pdf

The relevant question is described on page 66:

The majority of respondents (76%) assert that “scaling up current AI approaches” to yield AGI is “unlikely” or “very unlikely” to succeed, suggesting doubts about whether current machine learning paradigms are sufficient for achieving general intelligence.

I frequently shorthand this to a belief that LLMs won’t scale to AGI, but the question is actually broader and encompasses all current AI approaches.

Also relevant for this discussion: pages 64 and 65 of the report describe some of the fundamental research challenges that currently exist in AI capabilities. I can’t emphasize the importance of this enough. It is easy to think a problem like AGI is closer to being solved than it really is when you haven’t explored the subproblems involved or the long history of AI researchers trying and failing to solve those subproblems.

In my observation, people in EA greatly overestimate progress on AI capabilities. For example, many people seem to believe that autonomous driving is a solved problem, when this isn’t close to being true. Natural language processing has made leaps and bounds over the last seven years, but the progress in computer vision has been quite anemic by comparison. Many fundamental research problems have seen basically no progress, or very little.

I also think many people in EA overestimate the abilities of LLMs, anthropomorphizing the LLM and interpreting its outputs as evidence of deeper cognition, while also making excuses and hand-waving away the mistakes and failures — which, when it’s possible to do so, are often manually fixed using a lot of human labour by annotators.

I think people in EA need to update on:

• Current AI capabilities being significantly less than they thought (e.g. with regard to autonomous driving and LLMs)
• Progress in AI capabilities being significantly less than they thought, especially outside of natural language processing (e.g. computer vision, reinforcement learning) and especially on fundamental research problems
• The number of fundamental research problems and how thorny they are, how much time, effort, and funding has already been spent on trying to solve them, and how little success has been achieved so far

Thank you for pointing this out, David. The situation here is asymmetric. Consider the analogy of chess. If computers can’t play chess competently, that is strong evidence against imminent AGI. If computers can play chess competently — as IBM’s Deep Blue could in 1996 — is not strong evidence for imminent AGI. It’s about 30 years later and we still don’t have anything close to AGI. 

AI investment is similar. The market isn’t pricing in AGI. I’ve looked at every analyst report I can find, and whatever other information I can get my hands on about how AI is being valued. The optimists are expecting AI to be a fairly normal, prosaic extension of computers and the Internet, enabling office workers to manipulate spreadsheets more efficiently, making it easier for consumers to shop online, they foresee social media platforms having chatbots that are somehow popular and profitable, LLMs playing some role in education, and chatbots doing customer support — which seems like one of the two areas, along with coding, where generative AI has some practical usefulness and financial value, although this is a fairly incremental step up from the pre-LLM chatbots and decision trees that were already widely used in customer support.

I haven’t seen AGI mentioned as a serious consideration in any of the stuff I’ve seen from the financial world.

What is "the outgroup"?

She, but thank you!

Huh? Why would it matter whether or not I'm part of "the outgroup"...? What does that mean?

Can you explain what you mean by "contextualizing more"? (What a curiously recursive question...)

You definitely have more popular opinions (among the EA Forum audience), and also you seem to court controversy less, i.e. a lot of your posts are about topics that aren't controversial on the EA Forum. For example, if you were to make a pseudonymous account and write posts/comments arguing that near-term AGI is highly unlikely, I think you would definitely get a much lower karma to submission ratio, even if you put just as much effort and care into them as the posts/comments you've written on the forum so far. Do you think it wouldn't turn out that way?

I've been downvoted on things that are clearly correct, e.g. the standard definitions of terms in machine learning (which anyone can Google); a methodological error that the Forecasting Research Institute later acknowledged was correct and revised their research to reflect. In other cases, the claims are controversial, but they are also claims where prominent AI experts like Andrej Karpathy, Yann LeCun, or Ilya Sutskever have said exactly the same thing as I said — and, indeed, in some cases I'm literally citing them — and it would be wild to think these sort of claims are below the quality threshold for the EA Forum. I think that should make you question whether downvotes are a reliable guide to the quality of contributions.

One-off instances of one person downvoting don't bother me that much — that literally doesn't matter, as long as it really is one-off — what bothers me is the pattern. It isn't just with my posts/comments, either, it's across the board on the forum. I see it all the time with other contributors as well. I feel uneasy dragging those people into this discussion without their permission — it's easier to talk about myself — but this is an overall pattern.

Whether reasoning is good or bad is always bound to be controversial when debating about topics that are controversial, about which there is a lot of disagreement. Just downvoting what you judge to be bad reasoning will, statistically, amount to downvoting what you disagree with. Since downvotes discourage and, in some cases, disable (through the forum's software) disagreement, you should ask: is that the desired outcome? Personally, I rarely, pretty much never, downvote based on what I perceive to be the reasoning quality for exactly this reason.

When people on the EA Forum deeply engage with the substance of what I have to say, I've actually found a really high rate of them changing their minds (not necessarily from P to ¬P, but shifting along a spectrum and rethinking some details). It's a very small sample size, only a few people, but it's something like out of five people that I've had a lengthy back-and-forth with over the last two months, three of them changed their minds in some significant way. (I'm not doing rigorous statistics here, just counting examples from memory.) And in two of the three cases, the other person’s tone started out highly confident, giving me the impression they initially thought there was basically no chance I had any good points that were going to convince them. That is the counterbalance to everything else because that's really encouraging!

I put in an effort to make my tone friendly and conciliatory, and I'm aware I probably come off as a bit testy some of the time, but I'm often responding to a much harsher delivery from the other person and underreacting in order to deescalate the tension. (For example, the person who got the ML definitions wrong started out by accusing me of "bad faith" based on their misunderstanding of the definitions. There were multiple rounds of me engaging with politeness and cordiality before I started getting a bit testy. That's just one example, but there are others — it's frequently a similar dynamic. Disagreeing with the majority opinion of the group is a thankless job because you have to be nicer to people than they are to you, and then that still isn't good enough and people say you should be even nicer.)