Human oversight is a central requirement across recent AI governance frameworks, but a common implementation assumption has received surprisingly little empirical evaluation: that an AI agent can reliably identify decisions requiring human review by reporting low confidence.
This post summarizes a small empirical study examining that assumption. I evaluated two frontier model families (Llama 3.3 70B and Gemini 2.5 Flash) across four benchmark domains (SWE-bench Verified, GSM8K, MMLU, and TruthfulQA), comprising 2,004 decisions. The principal finding is that the usefulness of confidence-based oversight depends strongly on the task domain. Confidence provides a meaningful signal for structured question-answering tasks but offers little value for the software engineering benchmark evaluated here. I also report a preliminary comparison with self-consistency as an alternative uncertainty signal on a smaller subset (n = 110).
Given the scale of the study, I consider the domain-dependence finding to be more reliable than the precise quantitative estimates, which may not generalize beyond the evaluated models and benchmarks. The self-consistency results should be regarded as exploratory.
The complete paper, dataset, and verification scripts are publicly available: