This is part 3 of my ongoing review of the many misleading claim in AI 2027.
I am going through various claims in AI 2027 along with the materials the authors reference as supposed evidence to support the claims, and showing that, repeatedly, the referenced reports do not contain what AI 2027's authors say they do.
AI 2027 Misrepresents Scientific Reports, Part 3
Next we’ll look at whether the references support the claims in the following:
“Agent-1 had been optimized for AI R&D tasks, hoping to initiate an intelligence explosion. OpenBrain doubles down on this strategy with Agent-2. It is qualitatively almost as good as the top human experts at research engineering (designing and implementing experiments), and as good as the 25th percentile OpenBrain scientist at “research taste” (deciding what to study next, what experiments to run, or having inklings of potential new paradigms).45”
Here, AI 2027’s authors reference two articles in relation to AI’s capacity to carry out research. I’ll address both. First, https://arxiv.org/pdf/2409.04109
The referenced report does not contain evidence to support the claims made by AI 2027’s authors.
To summarise: The authors of the referenced report had expert human NLP researchers come up with research ideas for some current proposed topics, and they also used an AI to generate new research ideas for those same proposals. They then had human researchers judge the quality of the anonymised AI- and human-produced ideas. AI ideas are judged to be more novel than human-generated ideas but not as feasible.
The authors point out the many difficulties with their proposal, which I will touch on below.
Firstly, though AI 2027’s authors cite it after stating AI will soon be good at “designing and implementing experiments”, the report didn’t study whether the AI’s ideas could translate into research.
As the authors put it: “In this current study, we focused solely on evaluating the ideas themselves. Ideas that sound novel and exciting might not necessarily turn into successful projects, and our results indeed indicated some feasibility trade-offs of AI ideas.”
Perhaps most significant are the fundamental engineering problems the report’s authors point to, stating that they have no solution to them. Those are the essential problems of idea generation: (1) coming up with diverse ideas; and (2) picking out the useful ideas from the among the useless ones.
You could come up with a million ideas for a problem, but your collection of proposals is useless if you have no way to separate out the valuable ones.
The authors of the report, as they put it, “offer some empirical evidence that LLMs cannot evaluate ideas reliably yet.”
The AI system was not able to reliably pick out the best ideas from among all the proposals it generated. When humans ranked the ideas, many of their top choices were different to the AI’s choices (human ranking was the metric for value).
A point not touched on is that the topics were conceived by humans. Choosing the topics is perhaps half the difficulty.
The authors of the report also state that there was a limit to the diversity of the ideas their model produced:
“We find that they lack idea diversity when we scale up idea generation, and they cannot currently serve as reliable evaluators.”
And: “Our ideation agent is motivated by two potential strengths of LLMs: their ability to scale by generating a vast number of ideas - far more than any human could - and the possibility of filtering these ideas to extract the best ones from the large pool. In theory, this approach could lead to high-quality ideas by leveraging inference scaling. However, we present empirical evidence that this naive assumption about scaling idea generation has significant limitations.”
They found that, while the AI produced 4000 ideas, only 200 of them were not duplicates.
The above are critical issues. Researchers are attempting to address them but no one has found solutions, as many researchers admit, including the authors of the report we’re reviewing here. When they cite the article to support their claim that AI will soon be good at carrying out research, the authors of AI 2027 don’t touch on the fact that those issues are unsolved.
.
.
.
Take a look at my other articles:
On what differentiates humans from computers
On what words mean to computers
.
Follow me on Twitter, it helps: https://x.com/OscarMDavies
