How I orient towards thinking about AI persuasion and superpersuasion:
Most people I talk to about superpersuasion from advanced AI seem quite confident that it either will or won’t be a major problem. My guess is that this confidence is significantly misplaced, and comes down to a failure of imagination.
Skeptics on AI persuasion argue that humans have long evolved to both persuade and be resilient to external influence, that we’ve long had propaganda, that we long had ads, and that challenges from persuasive AI won’t be qualitatively different from other technological transitions (broadcast television, the internet, social media and so forth), and people are stubborn and aren’t really liable to be persuaded by arguments anyway.
These abstract arguments may all well be true, but I think there’s a missing mood: skeptics implicitly tend to have a very specific picture in mind when they think about "AI persuasion." They imagine a chatbot making a single argument in a single session for a specific viewpoint, or a single AI-generated ad on TV, and correctly note that this doesn't seem very scary. You can just close the tab. People just aren’t that gullible.
Or sometimes when anchored on the term “superpersuasion”, people imagine heroic powers ascribed to AI in a specific situation, and assume that specific situation is implausible. Eg they point out that in a few sentences of text, an AI probably can’t convince you to kill your family, or otherwise take actions that immediately and “obviously” betray your well-defined interests.
But real-world persuasion doesn't follow our narrowly carved categories, and AI-powered persuasion will look like that even less. The r/ChangeMyView experiments using GPT-4o were instructive for me. The bots were ~98th percentile persuasiveness, but that’s the least interesting update for me: a bigger update is how much they easily lied, including “AI pretending to be a victim of rape, AI acting as a trauma counselor specializing in abus