This is part 2 of my ongoing review of the many misleading claim in AI 2027.
Part 1 is here: AI 2027 Misrepresents Scientific Reports
I am going through various claims in AI 2027 along with the materials the authors reference as supposed evidence to support the claims, and showing that, repeatedly, the referenced reports do not contain what AI 2027's authors say they do.
Part 2:
The AI 2027 authors say: “Using techniques that utilize AIs to train other AIs,23 the model memorizes the Spec and learns to reason carefully about its maxims.”
Here, they seem to imply, again, that there is evidence that AI can “reason”. And it will achieve “reasoning” through being trained by other AIs.
The article they reference contains no evidence that AI can reason or that what is (in my opinion misleadingly) referred to along the lines of ‘AIs training other AIs’ will enable it to reason.
Here’s a line from the article they reference (https://arxiv.org/pdf/2212.08073) that touches on the relevant point:
“In a certain sense, work on reinforcement learning from human feedback has already taken a step in the direction of scaled supervision, since the reward signal in RL actually comes from an AI preference model (PM) rather than from immediate human oversight. However, RLHF typically uses tens of thousands of human preference labels.”
“AI preference model” means a model that has been told many times by humans what those humans prefer. The system is programmed to output what it has found has been most popular among humans. Then the researchers take the data, and they use it to “train” another AI system. So that second AI system will look at a series of question like “ice cream + bananas Or ice cream + potatoes?” and it will output an answer. If it chooses the answer “potatoes” the first AI system (the one trained on human preferences) will tell it it is wrong, and if “bananas” it will tell it it is right. One is “right” and one “wrong” because “I like ice cream with bananas” has appeared more times in the dataset (the dataset might be “everything written on the internet”) than “I like ice cream with potatoes”.
This is what is meant by “the reward signal in RL actually comes from an AI preference model (PM) rather than from immediate human oversight”. There is no “immediate human oversight” in the sense that the “oversight” (telling the model it is either right or wrong in its output) is not a human sitting there and responding to the output, rather it is a machine that has taken note of what humans in its dataset have preferred in the past, and gauged which choices have been preferred a majority or supermajority of times (maybe the threshold is more than 50%). This is what is meant by “AI trains AI”.
When you look at the details, I don’t think it sounds as remarkable as what many people might imagine is meant by “AI trains AI”. It’s not really AI training AI. It is humans training AI, via a degree of removal, and moreover, the “training” in such cases depends on a sort of popularity contest or average approval across a large number of people, which I imagine would push any model towards the average, the bland, the cliché.
“AI training AI” is a sort of autopilot chain of veering towards the average, no matter how many steps removed any particular case is from the original training data.
In AI 2027’s authors’ referencing of the above article, we find what might be called the deferral of responsibility. It’s something widespread. What I mean is, if you take the time to read through the current discussions in the AI world, you will find that you repeatedly come across large claims—for example, things along the lines of “there’s evidence AI can or will be able to reason”. And then you think to yourself, This person is making a claim so what’s their proof? And you go searching for the what or whoever it is they point to to support their argument. Sometimes they reference no one. Other times they’ll mention a name of some figure supposed to be authoritative, perhaps AI inventor Geoffrey Hinton or philosopher Daniel Dennett. Other times, they’ll reference a report. Next, if you’re curious enough, you’ll review the referenced document or referenced figure’s work to see if they do provide evidence to support the claim. Doing so, you find that the referee does not say what you were told they say, and you may also find that they too, on the point in question, refer to another source supposed to prove the claim. You then move on to that next source in search of the proof. There you again find no evidence, but perhaps a reference to a further source. And so on. You’ll never get to the end of the chain because there is no end and there is no proof. In the AI world, as in many other fields of the non-hard sciences (and perhaps life in general), many people making assertions have not themselves taken the time to follow through the logic or evidence of the claim, but seem to think rather that because many other people are making the claim, it must be valid.
.
.
.
Take a look at my other articles:
On what differentiates humans from computers
On what words mean to computers
.
Follow me on Twitter, it helps: https://x.com/OscarMDavies
