There is a quiet assumption embedded in most AI-assisted governance tools: that better decisions are more efficient decisions. Feed the system enough data, define the objective, and let it find the optimal path. The dashboard updates. The recommendation appears. The meeting moves on.
This framing has a problem. Most genuinely hard policy decisions — housing allocation, energy transition, resource distribution, public health trade-offs — are not optimisation problems. They are value conflicts. The question is not what is the most efficient outcome, but whose priorities should shape the outcome, and how should competing interests be weighed. These are political questions, not technical ones. And when AI systems are designed to resolve them rather than surface them, we do not get a better democracy. We get technocracy with a confidence interval.
The distinction I want to draw is between AI-as-optimiser and AI-as-mediator.
An AI-as-optimiser takes a stated objective and finds the best route to it. It is useful for well-defined problems such as route planning, fraud detection, and drug interactions, where the goal is clear, and trade-offs are quantifiable. But most governance contexts do not have clear goals. They have competing stakeholder interests, incommensurable values, and legitimate disagreement about what outcomes should even look like.
An AI-as-mediator does something different. Instead of converging on a recommendation, it surfaces the structure of the disagreement. It makes visible the value trade-offs that different choices imply. It asks: if we prioritise efficiency here, what happens to equity? If we weight this community's interest, what is the cost to that one? It does not resolve the question. It makes the question more legible.
This is not a minor design choice. It changes what the AI is for.
I built a small proof of concept that sits in this second category. The Community Document Analysis tool takes civic documents, initially tested against a live Section 21 eviction notice (the legal instrument landlords use to reclaim property in England, often received by tenants with little understanding of their rights or options), and grounds LLM outputs in community-defined policy inputs, with full transparency tracing. The system is designed so that the values shaping the analysis are explicit and contestable, not buried in a model's weights.
What emerged from building it was a practical version of the theoretical argument: the hardest design decisions were not about technical architecture. They were about what should count as a relevant value input, whose definition of "community concern" should be represented, and how to surface disagreement between competing policy framings without the tool collapsing them into a single answer. Those decisions cannot be delegated to the model. They have to be deliberately made by people in advance and then made visible to users.
I call this interpretive opacity. The problem with most AI governance tools is not that they are technically opaque (though many are). It is that the interpretive choices — what matters, what counts, whose interests are centred — are invisible. A system can publish its source code and still conceal the assumptions that structure its outputs. That concealment is where power concentrates.
The governance implications are direct.
First, AI systems used in public decision-making should be required to make their objective functions legible — not just their accuracy metrics. What is the system optimising for? What trade-offs does that objective embed? These are not technical disclosures. They are political ones, and they require political accountability.
Second, the distinction between tools that support deliberation and tools that substitute for it needs to become a design criterion, not an afterthought. Evaluation frameworks for public-sector AI should ask: does this system help humans reason about trade-offs, or does it pre-empt that reasoning? The answer determines whether the tool strengthens or erodes democratic accountability.
Third, the framing of "AI-assisted decision-making" often obscures the question of who is actually making the decision. When a system produces a ranked recommendation and a time-pressured official accepts it, the AI has made the decision. Designing systems that maintain meaningful human agency, not nominal sign-off, requires deliberate structural choices about how outputs are presented, what alternatives are shown, and whether disagreement is treated as a failure or a feature.
None of this requires rejecting AI in governance. It requires being precise about what kind of AI, designed for what kind of task, and accountable to whom.
The optimiser and the mediator are not different points on a capability spectrum. They are different conceptions of what AI is for. Choosing between them is a governance question. And at the moment, most governance frameworks are not asking it.
That is the gap worth working on.
