I have been increasingly surprised over the past year at what I perceive to be the lack of critical thinking in response to new announcements regarding AI model capabilities. I’ve written about many of these issues before, but as AI hype accelerates the problems seem to be getting worse. Here I want to highlight a few points that seem to be neglected in coverage of the growth of AI capabilities.
Advertising is treated as fact
Every new model release by a major tech company represents billions of dollars of expenditure and the potential for billions of dollars of additional investment. Model releases and their associated documentation should therefore be interpreted as advertising designed primarily to attract customers and investors. Their goal is not to provide an objective and thorough analysis of the strengths and weaknesses of each new model. Of course, this does not mean that everything reported is false, but it does mean that such releases should be treated with significant skepticism. It is therefore disappointing to me to see write-ups such as this one by Rob Wiblin, which consists almost entirely of uncritical restatement of claims made by Anthropic, with little to no additional analysis.
For example, in discussing the capabilities of Claude Mythos to identify software vulnerabilities, Wiblin says:
“Anthropic’s previous model Opus 4.6 could only successfully convert a bug it identified in the browser Firefox into an effective way to accomplish something really bad 1% of the time. Mythos could do it 72% of the time.”
This statement is selective and misleading. What Anthropic actually did was provide their models with a testing harness which mimicked Firefox 147 but without critical defence components. They then prompted the models to devise and implement a certain type of exploit. Mythos fully accomplished this 72% of the time, but nearly always did so using two specific bugs that have since been fixed. When Anthropic removed these two bugs, Mythos only fully succeeded 4.4% of the time. (It was unclear to me whether Mythos may have had any knowledge of these bugs in its training data, given they had already been fixed). While I do not doubt that Mythos has improved capabilities relative to previous models, the reporting here unjustifiably hypes the significance of the results without providing any substantive critical analysis. This is further highlighted by the fact that an independent analysis was able to find many of the same vulnerabilities using much smaller open-source models.
Wiblin also extrapolates well beyond what is even claimed by Anthropic, such as when he argues:
“Now, Anthropic doesn’t say this directly in their reports, but I think a common-sense interpretation of the above is that in any deployment where this AI has access to the kind of tools that would make it actually useful to people — the ability access some parts of the network and execute code — could probably break out of whatever software box we try to put it in, because the systems that we would be trying to restrain it are themselves made of software, and that software is going to have vulnerabilities nobody knows about that this model is superhumanly good at finding and taking advantage of”
In my view, it is reasonable to think that humans armed with improved automated techniques for identifying software vulnerabilities would be better, rather than worse, at constraining the behaviour of new models. This is in fact what Anthropic argues in their report. There may be differences of opinion about this, but this is an example of where hype seems to be substituting for genuine analysis.
Wiblin also comments on the fact that Anthropic has yet to release Mythos publicly:
“And also keep in mind that on Monday — the day before Anthropic published all of this — we learned that their annualised revenue run rate had grown from $9 billion at the end of December to $30 billion just three months later…
That exploding revenue is a pretty good proxy for how much more useful the previous release, Opus 4.6, has become for real-world tasks. If the past relationship between capability measures and usefulness continues to hold, the economic impact of Mythos once it becomes available is going to dwarf everything that came before it — which is part of why Anthropic’s decision not to release it is a serious one, and actually quite a costly one for them.
They’re sitting on something that would likely push their revenue run rate into the hundreds of billions, but they’ve decided it’s simply not worth the risk.”
Wiblin does not consider the possibility that the reason why Anthropic is publishing these claims without the corresponding model is to build hype for a model that is not actually ready for release yet, especially in the lead up to Anthropic’s upcoming IPO. Wiblin does not explain where his estimate of ‘hundreds of billions of dollars’ of revenue comes from, but it reads to me like pure marketing for potential investors. Nor does it make sense to claim that revenue is a measure of economic value when Anthropic, OpenAI, and others are massively subsidising usage. There is a discussion to be had about the implication of these issues, but it is not to be found in this (or similar) pieces I’ve seen on the subject. We need to do better than just uncritically repeating advertising talking points of billion-dollar tech companies.
Benchmarks are interpreted uncritically
Much of the claimed improvement in performance of models derives from rapidly increasing scores on various benchmarks, which are standardised tests designed to quantify model capabilities on tasks such as language, coding, reasoning, and image recognition. While these benchmark scores give the appearance of precise and objective tests, in practice they often have very limited value in assessing the rate of capability improvement in a meaningful way.
First, most benchmarks have not been validated. Validity is an important concept in research generally and especially human psychometrics. It refers to the extent to which a metric has been assessed as adequately measuring the underlying phenomenon of interest. There are many components of validity, and validity assessments require careful research to assess the relationship between test performance and the target phenomenon. However, few AI benchmarks report this sort of research. Most simply come up with tasks the researchers hope are related to the target capability. This is simply poor research practice. It cannot simply be determined by intuition whether a given set of tasks will provide reliable and valid information about the intended capability of interest. This requires carefully designed research.
Second, almost as soon as they are released the benchmark solutions begin to contaminate the training data of new models. For instance, memorisation is known to be a major problem for SWE-Bench, a widely used benchmark of software engineering tasks. A recent analysis of visual benchmarks has even found that models could outperform humans on a standard X-ray question-answering benchmark without even being provided with any images. A particularly concerning analysis found it was possible to achieve 100% on several major benchmarks without solving a single task, usually by exploiting simple vulnerabilities in the test pipeline or the way scores are computed.
Third, even when the test solutions are not publicly available, the training data often sufficiently resemble the test data that a model trained on the training data will see dramatically improved performance on the test questions as well. This would not be a problem if the train and test problems constituted a representative sample of the domain of interest, but for so many important topics (language, reasoning, coding, image recognition), the domain is so vast and hard to characterise that it is not possible to construct a representative sample in this way. Sampling also tends to favour more common and simpler problems, and even very subtle changes in the sampling method can lead to the model learning radically different representations. This means that models tend to overfit to the training data, diminishing the value of the benchmarks in assessing out-of-distribution generalisation capabilities.
The issue of benchmark contamination is granted only a few pages of the 244 in the Mythos model card. Only a few benchmarks are assessed for contamination, with Anthropic arguing that most of the improvement on these cannot be attributed to memorisation. However, their results show that model performance degrades significantly when restricted to the 20% of the benchmark questions they assess as having least probability of memorisation. This was even true for the SWE-Bench Pro benchmark, which is supposedly ‘a contamination-resistant testbed’. This highlights the importance of devoting more attention to these issues in order to better interpret the meaning of benchmark improvements.
Negative results are ignored
I rarely see discussion in EA circles of various results which indicate fundamental limitations of existing LLM-based approaches. Numerous studies have found that these models often fail to learn the appropriate task structure, but instead learn to answer questions by learning spurious correlations and superficial heuristics that work in some constrained domain or training task, but do not generalize to variations of the task. There are also significant known limitations of the chain of thought approach which underpins reasoning models, with thought chains often being unfaithful to the actual computations that generate model predictions. In my view, there is reason to believe that these problems reflect fundamental limitations of the machine learning techniques that underpin leading models.
As a further interesting example, Claude Opus 4.7 shows a significant regression in performance on long context tasks based on the MRCR benchmark, which interestingly is precisely a benchmark that uses adversarial methods to distract the model from the task. Anthropic’s response is:
“We kept MRCR in the system card for scientific honesty, but we've actually been phasing it out slowly. It's built around stacking distractors to trick the model, which isn't how people actually use long context.”
In my view, this response indicates that Anthropic is more interested in ensuring their model works in typical use cases, rather than assessing whether it actually has robust generalisable capabilities that are indicative of what we might call ‘genuine intelligence’. This is particularly relevant for arguments relying on extrapolations of improvements in model capabilities to novel tasks and in more complex settings.
Conclusions
There is no doubt that LLM-based models have shown significant improvements in recent years. However, it is important to carefully and critically assess these advances in order to make accurate inferences about their social, political, and economic impacts. One cannot infer AI 2027-like superintelligence takeover scenarios from recent trends and developments without making significant additional assumptions about the nature of generalized intelligence, the relevance of benchmark results, and the limitations of LLM-based models. Humans have a very bad track record of predicting what tasks require ‘general intelligence’ to accomplish, and I suspect that it may be possible to develop machine learning models that can automatically perform any task with known solutions without this implying any superintelligence takeoff. These issues are complex and demand a more nuanced, informed consideration than I often see in contemporary discussions.
Given that you are criticising the epistemics of EAs taking AGI very seriously, I think it's reasonable to hold this post to a higher epistemic standard than a typical EA forum post. Apologies if this comes across as combative - I spent some time trying to tone it down with Claude and struggled to get something that wasn't just hedged/weak sauce. I am excited about more discussion of the capabilities of AI systems on the EA forum and would like more people to write up their takes on the current situation.
......
I think you are applying more rigour to the bullish case than the bearish one. For example, you say:
I think this is misleading for a few reasons:
On the claim that Anthropic talks about risks from their own models primarily to create hype: I find this hard to square with the evidence. Talking about how your B2B product might be extremely dangerous, or publishing lengthy documents critically assessing your own product and admitting to errors that would be difficult to identify independently (e.g. accidentally training against the CoT), is not a common marketing tactic. It feels like your model implies that companies should only release materials optimised for short-term interests, which doesn't predict the real differences in how AI companies approach releases.
Benchmarks are interpreted uncritically
The benchmark contamination arguments are worth engaging with in principle, but I'm not sure they're doing much work in practice - I don't think many people in EA are actually updating heavily on raw benchmark scores right now. METR, arguably EA's favourite benchmarking org, has been pretty vocal about their own benchmarks being saturated, so I think the community is reasonably aware of these limitations already.
Negative results are ignored
I'm genuinely uncertain what you want Anthropic and other AI companies to do here. Do you think "genuine intelligence" is easy to measure and well-defined? The more concrete concepts being used as proxies - coding ability, economic value generated, uplift - seem defensible on their own terms rather than as misleading substitutes for something more fundamental.
On "fundamental limits of LLMs" more broadly: these arguments have been made confidently by prominent researchers since the advent of LLMs and have not had a great track record. That doesn't make them wrong, but it's worth noting.
.....
I think this post would be much stronger if it applied its standards more symmetrically. It would also help to have a more concrete conclusion. The current takeaway is essentially "further research is needed", which is a claim you can make about most areas of research (so much so that it's been banned from multiple journals), but I don't have a great sense of what research would actually convince you that the "AI hype" is reasonable.