Several decades ago, researchers investigated how measured values of fundamental physical constants evolved over time. As expected, measurements gradually converged toward true values as instrumentation improved. However, this convergence revealed two unexpected patterns. First, estimated measurement errors were, on average, half the size of actual errors. Second, measurements were systematically biased toward values considered accurate at the time.

The most plausible explanation is not deliberate fraud but a natural human heuristic: when scientists obtain seemingly implausible results, they are more likely to suspect errors and adjust their calculations or experimental setup.

Bias in Policy Evaluations

In empirical sciences, this consensus-seeking behavior slows but rarely prevents progress. Since scientists have no personal stake in whether a physical constant is higher or lower, their measurements—though pulled toward the current "Overton window" of acceptable values—still tend to fall closer to truth. Over time, the window itself shifts in the correct direction.

Policy research faces different incentives:

• Unlike measurements of the Hubble constant, research on potential trade-offs of nuclear energy, AI regulations, or gain-of-function experiments is heavily influenced by personal biases.
• Researchers receive no expectation of future reward for publishing correct results that contradict current consensus.

Consequently, instead of gradually converging toward correct answers, the Overton windows of acceptable results often drift toward socially popular viewpoints.

Empirical Testing

In long-term policy debates, empirical testing of proposed solutions is generally unfeasible. Determining whether a particular policy would help or harm humanity requires implementation, and even then—except for extreme outcomes (e.g., "AGI destroys mankind")—the results are likely to remain controversial.

However, while we cannot empirically test long-term policy recommendations, we may be able to test their authors. Consider what would happen if information about experts' biases—such as political biases or tendency toward overly optimistic or pessimistic predictions—were publicly available. In any major debate where numerous experts present recommendations, we could analyze how these recommendations correlate with different biases and use this information to select an optimal course of action.

How Can We Objectively Measure Bias?

Researchers' biases can be measured through forecasting-based empirical testing. Data from forecasting platforms demonstrate that people's biases clearly manifest in their tendency to systematically overestimate the likelihood of outcomes aligning with their personal preferences. For instance, data from Manifold Markets indicate that most users exhibit significant biases along several dimensions:

• Political orientation (left vs. right)
• Tendency to under- or overestimate the pace of technological progress
• Tendency to under- or overestimate the likelihood of catastrophic events

The data also reveal that, for markets not expected to resolve in the near future, users' biases have a far stronger impact on their betting choices than their intelligence or general expertise (estimated from average winning rates).

Thus, unless policy experts differ fundamentally from physicists and Manifold users, we should expect their research to be affected by personal biases to a similarly strong degree.

Implementation Challenges

Two major obstacles prevent implementing this approach in practice:

• Organizations that preferentially hire unbiased researchers risk backlash from sponsors and peers, as their publications would frequently challenge prevailing narratives.
• Without a hiring advantage, experts have no incentive to voluntarily measure and disclose their biases.
   

If you have ideas on how to address these challenges, please share them in the comments.