The authors of AI 2027 repeatedly misrepresent the scientific reports they reference as evidence to support their arguments.

I’m writing a comprehensive review of AI 2027 (to be posted soon), but in the meantime I want to briefly present an example of the misrepresentations:

In a section in AI 2027 on “Iterated distillation and amplification (IDA)” (there are no page numbers so to find a passage, use control + F search), the authors write:

“Self-improvement for general intelligence had seen minor successes before. But in early 2027, it’s seeing huge returns [with] IDA”

(I’ll outline IDA below—but understanding the details is not very important for our purposes). A few sentences later, they go on:

“Early versions of IDA have been working for many years on easily verifiable tasks, like math and coding problems that have a clear answer …

Now [they’re predicting what things will be like in 2027 here], the models have become sufficiently good at verifying more subjective things (e.g. the quality of a work product), allowing the use of IDA to improve the model at many tasks.”

Here the authors take success in “math and coding problems that have a clear answer” and extrapolate to future “models [that] have become sufficiently good at verifying more subjective things”—i.e. models that are capable in many areas beyond narrow math and coding problems (such wider capacities are essential to the predicted “superintelligent” computers). What’s their evidence that any such extrapolation is warranted?

You click on the provided link to supporting evidence and you are taken to a 2017 report titled “Supervising strong learners by amplifying weak experts”.

In that report the authors outline their attempt to get AIs to imitate the operations a human carries out as the human breaks down (they call it “decomposing”) a difficult problem into smaller, easier subproblems, and then combines the answers to those subproblems to find a solution to the larger problem. (That’s more or less what is meant by “IDA”—again, don’t worry if it’s not completely clear, it’s not important here.)

The report presents little evidence of successful implementation of the model because it is more an outline of the theory than a rigorous scientific testing of the model. It is not peer reviewed.

The authors do test the model on, as they put it, “5 toy algorithmic tasks”—i.e. math tasks with very narrow goals and methods. They report some success.

One of the five test “algorithmic tasks” in the paper, for example, is: “Given a directed graph with 64 vertices and 128 edges, find the distance from s to t.”

It’s no secret that computers can do some math. The possibility of machine intelligence revolves around the question whether computers’ success at math and similarly clear-cut tasks could be extrapolated to wider less-clear-cut problems.

The authors explicitly say that their report provides no evidence that their theory (called “Iterated Amplification”) is useful outside the bounds of narrow math (or “algorithmic”) tasks. Here’s a few passages from the article making that point:

“Having successfully applied Iterated Amplification to synthetic algorithmic problems, the natural question is whether it can actually be applied to complex real-world tasks that are “beyond human scale.” We leave a convincing demonstration to future work.”

“In our experiments questions can be algorithmically decomposed into subquestions, and we replace the human with a hand-coded algorithm. These experiments don’t shed any light on whether humans can decompose interesting real world tasks [they say the model needs humans to decompose tasks], nor on whether it would be feasible to learn messy real world decompositions.”

“Removing these simplifications is a task for future work, which will ultimately test the hypothesis that Iterated Amplification can be usefully applied to complex real-world tasks for which no other training strategy is available.”

Repeatedly, the authors state that their paper does not touch on whether their theory can be applied outside of “synthetic algorithmic problems”. I might point out the article was written eight years ago, and the theory still has found little success outside that narrow domain.

This lack of real-world application (the paper explicitly does not attempt such application), along with the authors’ repeated reminder that their paper does not address “real world” tasks, does not deter the authors of AI 2027 from referencing it as alleged evidence to support their claim that computers will be superintelligent and take over the world.

The AI 2027 authors’ link to the study appears in the sentence “Self-improvement for general intelligence had seen minor successes before.” I don’t see how anyone could construe the referenced report as having to do with “general intelligence”. To repeat, the authors of the report explicitly say that they do not address wider (“general”) problems, but rather focus on narrow math tasks. Nor does the report provide very promising proof of “self-improvement” on a level approaching anything like intelligence.

(There are two other reports referenced in the sentence “Early versions of IDA have been working for many years…”. But, again, neither warrants the claims made in AI 2027. Those reports will be discussed in my upcoming, more comprehensive review.)

In passages quoted above, the AI 2027 authors imply that AI’s relative success in “math and coding problems that have a clear answer” is evidence that soon they will “become sufficiently good at verifying more subjective things (e.g. the quality of a work product), allowing … the model [to improve] at many tasks”—i.e. the model will succeed in domains beyond narrow “math and coding problems”.

Success in such wider domains is the crux of the problem of inventing intelligent computers. If you are going to predict that that gap will be bridged—as AI 2027’s authors predict—you would need to explain how it will be bridged and present evidence.

No such evidence or explanation appears in AI 2027. In fact, the authors of the paper referenced as evidence in the passage we’re reviewing explicitly state the opposite—that the report does not contain proof their model is applicable for “more subjective things”, but has only been used in narrow math tasks.

As this is just one minor example, it might seem as if I am nitpicking. I am getting too deep into the details by reviewing the specific claims in materials referenced in AI 2027. But the details are what is important. The referenced materials are supposed to be the evidentiary basis for the claims. Without evidence, the claims are empty.

And perhaps you wonder whether the authors of AI 2027 slipped up in this one case and in fact they present more substantial evidence elsewhere. They do not. In every relevant instance they make basic mistakes like the one I’ve reviewed in this article.

These sorts of misrepresentations and unfounded extrapolations are repeated throughout AI 2027. I will soon publish my full review demonstrating that the authors repeatedly reference reports that do not contain evidence to warrant their predictions.

.

.

.

Take a look at my other articles:

On Ilya Sutskever’s failure to answer the Q: How will AGI be invented?

On Eliezer Yudkowsky’s unsubstantiated arguments

On what words mean to computers

.

Twitter/X: x.com/OscarMDavies