When Does Human Oversight Become the Bottleneck?

I've been obsessing over the latest Geneva discussions meant to classify and regulate and answer difficult questions about autonomous weapons and kill chains this past week.

If a machine ever becomes demonstrably better than a human at certain targeting decisions, insisting on human approval in that process may actually increase harm and unnecessary casualties.

Back in the Cold War, Stanislav Petrov gives us the very opposite lesson. 

 In 1983, the automated warning system he was using for the Soviets was wrong, and Petrov, trusting his instincts and human intuition, distrusted his information and ultimately opted not to catastrophically escalate the already tense situation, potentially saving us from an irradiated wasteland and WW3.

So the billion or trillion or howevermanyillion dollar question is whether human oversight is mainly valuable because humans are better on average, or because rare human overrides are disproportionately important in catastrophic edge cases.

I wrote a longer essay on this after the latest Geneva talks:

https://www.gonzocapital.net/the-human-is-becoming-latency/

Curious how people here think about the tradeoff between average accuracy, tail risk, and competitive pressure between states and defense companies to automate.