Roodman's Thoughts on Biological Anchors

A new review of Ajeya Cotra's Forecasting TAI with biological anchors (see also update here), written by David Roodman in April 2020, has been added to the folder of public reviews for Cotra's report.

Roodman's summary:

I think my main critical reaction is about the draft report’s ecumenical approach. It puts non-zero weight on several different frameworks which, conditional on the various parameter choices favored in the report, contradict one another. This mixing of distributions expresses a kind of radical uncertainty: not only are we unsure about the parameter values within each framework; we’re also unsure about which framework is most right.

This set-up is pragmatic and humble, but… I think in principle the ecumenism discards useful information, by not imposing the restriction that the various frameworks agree. In principle, they are all measuring the same thing. In pure Bayesian reasoning, if one has several uncertain measurements of the same value, each represented by a probability distribution, then one combines these primary measurements by multiplying them pointwise and rescaling the result to have total integral one. This contrasts with the pointwise averaging performed in the draft report, which is the mathematical expression of ecumenism.

In Bayesian reasoning, if two distributions for the same parameter are normal, then their combination is too; its mean is the average of the two primary means, weighting by the respective precisions (inverse variances). Weirdly, if the two primary means are far apart, so that the two distributions hardly overlap, then their combination can pop up in the no-man’s-land between them. The intuition is that the combined distribution centers on the least unlikely estimate given what we know.

I make that mathematical point less to argue for a mechanical implementation of Bayesian mixing of different perspectives than to advocate for an informal didactic that aims at unification. What is the least implausible way to reconcile the large disagreements between different frameworks? Could answers to that question help us settle on a single, favored framework, perhaps one that synthesizes ideas from more than one?

That impulse ultimately led me to favor a single framework that fuses elements from several in the draft report. The idea is to model two training levels at once, of parameters and hyperparameters. Training of parameters corresponds to the training of a single neural network, or the learning a sentient organism undergoes during maturation. Hyperparameter training corresponds to the design space exploration that AI researchers engage in and, in the biological realm, to evolution. Each parameter training run may involve huge numbers of small parameter updates; each in turn serves a single hyperparameter training step…