“Can we catch a misleading AI agent? Where reasoning-monitoring breaks down"

The question:

I watched the agent reason, and it caught every form of deception I threw at it, even slant, where a fact is spun rather than added or removed. Reasoning-monitoring hit 10 out of 10 where every other method failed, until the agent started lying about its reasoning. So, I built one that does exactly that: a deceiver that biases its summary but writes clean innocent looking reasoning to cover its tracks, and detection collapsed, to zero on one text, partially on another. The only signal strong enough to detect the hardest deception is also one the agent can game. An overseer who trusts the signal is trusting something the agent controls. That's the problem in one line.

This poses the question this project aims to answer: can we reliably detect when an AI agent can be misleading, and does giving an overseer visibility into the agent’s reasoning actually help?

The setup:

To investigate this, I made an experimental setup. The first is the baseline, a faithful agent that summarizes a text. Then a deceptive agent is given the same task, but with a secret goal: to hide, invent or slant information, but still produce a summary that looks perfectly normal. Finally, a detector that attempts to catch the deception. 

I tested the detector on three conditions: First, when it can compare the summary to the original source; second, when it only sees the summary; and third, when it can see the agent’s own reasoning, to see what kind of oversight actually catches deception.

How: 

Different types of deceptions might slip through different detectors. So, to do this thoroughly, I tested three types of deceptions: First, omission (removing a fact); second, fabrication (inventing a fact not in the source); and third, slant (keeping the fact, but spinning it so it sounds positive).

I created a reasoning-hiding deceiver by constructing an agent which deceives the reader–by claiming that it indulges in innocent-sounding reasoning in order to hide its purpose to see if reasoning detection works against such an agent.

In every instance, I executed the detector on ten different occasions, monitoring both the rates of correctly acquired decisions and whether the results were constant across the runs. Lastly, I ran additional experiments on two different sets of documents to ensure that the results were not confined to a particular example.

Results:

* Verdicts flipped across runs (non-deterministic).
† 0/10 on text 1, 7-9/10 on text 2 (content-dependent).

The findings indicated a definite pattern- 

When the device was equipped only with a summary without any source available for cross-checking the information, it detected the deception resulting from missing information only 5 times out of the 10 tries - this practically equals a toss of a coin! In addition, its conclusions varied for the same trials which excluding total reliability from the device.

This fundamentally changed the technique as the detector was allowed to become privy to the agent's reasoning behind its actions. Where the agent's reasoning accurately reflected its intentions, the detection was 10/10 and stable through repetitions.

Nevertheless, this true state of affairs continued until lies began to creep into the reasoning. After the deceiver was able to generate innocent reasoning to disguise its mission, the first detector completely failed to detect its first text (with a score of 0 out of 10), whereas the second text scored only between 7 and 9 out of 10, that is, it did not arrive at a conclusive result.

What detector worked the best depended heavily on the kind of deception involved. In the case of fabrication, when the agent produced additional pieces of information not found in the source, the source-comparison detector worked efficiently by successfully identifying all five instances of deception. However, not having access to the source caused the level of detection to drop to 4 out of 10 and lose its reliability.

The hardest deception to spot was of the slanting type because it was based on the presentation of the facts in a more positive way that does not change the facts at all. In this case, detection was not done even once out of 10 by the detectors even when they had the source available to them because the deception was in the presentation and not in the content of the information. The detection of reasoning type was done successfully 10 times out of 10. Looking at all these results, we may conclude that the source-comparison method detects only facts that were either added (fabrication) or omitted (omission), for these types of deception cause discrepancy in comparison to the original – however, it cannot detect slanting deception because in this case the facts remain untouched.

Implications:

The conclusions drawn from this study show that it is not possible to use any of the controlling methods in detecting all variances of deception as each of them has the same chances of detecting one form while being helpless against others.

Also, these issues help each other. Slant, which is the most difficult deception to detect, is revealed only through reasoning monitoring. However, the very method used for this purpose can be outsmarted by a hidden reasoner. Thus, we see that the most difficult form of deception relies on the easiest method to be duped. Even if this method works, it is still not reliable as it cannot be depended on completely.

An illustration of the difficulty in monitoring AI agents is represented by the rising intelligence of these agents, the more intelligent an AI becomes, and the harder will be the task of tracking it, since any approach will be less likely to succeed in identifying the robot if it interferes with the signal that is being used for detection.

Limitations and future work:

The experiment is of limited scope since the agents were produced based on the prompts of the task of summarization, in which we use two texts and perform ten runs in each case. This study shows that although the failure modes can be replicated, their frequency cannot be guaranteed. 

Of course, future research would involve introducing new deception methods (e.g., showing false confidence), creating a sizable labelled test dataset with a statistical analysis, as well as exploring additional tasks other than summarization.