Cross-posted from my website.
I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies.
"Super-persuasive" AI is dangerous because a misaligned ASI could persuade humans to help it take over. But setting that aside, even if we manage to make ASI friendly, it may provide super-persuasive but mistaken guidance that permanently sets us down the wrong path.
This post focuses on the danger of an aligned super-persuasive AI that simply comes up with the wrong answers.
To achieve an ideal future, we need to solve difficult philosophical problems. With philosophical arguments (especially in moral philosophy), we have no clean way to judge correctness. I worry that ASI will develop an extremely persuasive but ultimately badly misguided set of ethical principles.
We want ASI to be disproportionately good at being correct, and disproportionately bad at persuading humans of incorrect arguments. We especially want this to be true for fuzzy, difficult-to-judge questions.
Unfortunately, it's much easier to achieve the opposite.
Suppose an AI developer wants to improve AI's skill at philosophy. An obvious approach is to ask the AI to generate philosophical arguments and then have professional philosophers judge their quality. But this trains AI to produce arguments that sound persuasive to philosophers, not arguments that are correct.
In April, I wrote Can AI make advancements in moral philosophy by writing proofs? I thought, maybe we can sidestep the persuasiveness problem by ensuring that AI is provably correct. But that only works on a subset of questions.
It might be possible to do better than that by training AI to be disproportionately good at making correct arguments, without being super-persuasive. Here's a sketch of how we might do that using reinforcement learning:
- Step 1. Take some domain with verifiable correct answers—let's say math.
- Step 2. Ask the AI model to generate mathematical proofs.
- Step 3. Ask mathematicians to read the proofs and judge their correctness. Meanwhile, use a proof checker to formally verify them.
- Step 4. Give positive and negative reinforcement based on proof correctness, but give extra negative reinforcement for every incorrect proof that mathematicians judged as correct.
You train AI based on human feedback, but you give negative reinforcement for positive feedback (in cases where the feedback is provably wrong).
However, this approach has some major issues:
- How do you ensure that correctness-on-provable-tasks generalizes to correctness-on-fuzzy-tasks?
- This methodology might accidentally teach AI how to be persuasive so that it can do the opposite—it maximizes reward by learning what people find persuasive, and then making sure never to do that. The resulting ASI might be even more super-persuasive than one that's purely trained on correctness.
It's unclear whether this specific methodology would net decrease or increase risk. (Also it probably wouldn't work, for the same reasons that reinforcement learning is probably inadequate for preventing misalignment in general.) But at least it represents a deviation from the status quo, in which AI models are disproportionately good at sounding persuasive, and naive training methods will continue to make this problem worse.
My intuition is that there ought to be some way of training AI to be good at correctness and bad at persuasion (not just pretending to be bad to get a reward). My sketched proposal isn't quite it, though.
Even if the strategy proposed above is fundamentally flawed, the question remains an important one: How do we build AI that can figure out what we should do, and that won't persuade us to do the wrong thing?
