Playing in the dark
crosspost from substack
The amount of skill behind an outcome is often impossible to discern by scrutinizing the outcome itself, and our goal of making AI go well may be an extreme case. Shaping history is like poker: high variance, small edges, low skill expression. If that is right, two things follow: 1) we cannot trust how things seem to be going, good or bad; and 2) we should stop concentrating effort on whatever looks best right now, and instead should hold a diversified portfolio of bets, including those which seem suboptimal.
Beginning with poker, where the link between skill and results is actually measured: A strong professional’s edge over a merely competent player is something like five percent of the money wagered. The edge is small because most of the game’s skill sits in basic competence. A player who folds weak hands and bets strong ones has already captured most of what reckless play gives away. Expert skill is a thin refinement on top. The variance, meanwhile, is huge relative to the edge. Below roughly a hundred thousand hands, luck swamps skill, and the weaker player wins sessions all the time. So even in a game with a perfect scoring rule, it is difficult to discriminate between good players in all but the largest sample sizes. It is difficult to determine which habits and individual choices make money. It is hard to determine whether one’s approach needs to change, and in which direction.
Our goal is harder than poker in three ways. First, we get one hand. Poker's remedy for variance is volume, and history offers none. Humanity lives through the emergence of powerful AI once. Second, the variance is worse. Philip Tetlock spent decades scoring expert predictions about world events. Forecasting skill is real at short range and decays with distance; a few years out, even the best forecasters drift toward chance. Reality is often very surprising. Third, many of the challenges on the road to our goal are not technical, but political, and thereby often involve tradeoffs which are impossible to avoid and difficult to weigh. In the short term, it is unclear whether and how we will need to transition the workforce through an age of widespread automation. In the medium term, we will have to resolve how to distribute the rapidly growing economic pie. In the very long term, we may need to make fundamental advances in our methods of government in order to create stable widespread flourishing across deep space.
It may appear at first glance that there are technical solutions to these sorts of problems, but dig one layer deeper and there is often explosive disagreement stemming from competing loyalties and belief systems. Look no further than the failure of ‘teach truckers to code,’ ‘just tax the rich,’ and ‘the United Nations.’ Political problems which evade technical solutions are common features of reality. There is little hope that even superintelligence will help resolve them. For example, the Chinese Communist Party and the Vatican both claim the authority to appoint Catholic bishops in China. I cannot conceive how a superintelligence could design a mechanism to resolve this. It is a contest over legitimacy and obligation, not a problem with a solution waiting to be found.
Another historical case is the War in Vietnam. Two observations:
First, parachuting brilliant technical people into the government is overrated. It was tried in Vietnam by the closest analogues of today's top technical talent. Kennedy brought in Robert McNamara as Secretary of Defense, who in turn installed young quantitative analysts dubbed the ‘Whiz Kids.’ David Halberstam called the wider circle the best and the brightest. The record of their Vietnam deliberations shows careful, conflicted reasoning at every step, and yet the outcome was catastrophe.
Second, there is no reliable way to actually measure our progress on our goal. The Whiz Kids, laboring under uncertainty in every direction, managed what they could read: casualty statistics, sortie rates, kill ratios. But while the metrics improved, the war was lost. When reality refuses to show you the eval bar, whatever does show a score commands your attention. Benchmarks and evals are this field's candidates for that role, but they fail to reveal where we are in relation to our ultimate goals.
My first conclusion is that short term signals are not helpful in long term assessments. How things seem to be going is a poor measure of progress towards our goal. Furthermore, apparent success is weak evidence of skill, and apparent failure is weak evidence of its absence. The researcher who feels like a prophet and the one who feels like a fraud are both measuring themselves against limited, short term metrics. In practice this means two hard things. You cannot tell when it is time to pivot, and you cannot tell what to pivot to.
The second conclusion is about the field as a whole. We need to avoid being like small children playing soccer; we can’t all be chasing the same ball. Whenever something surprising happens: a capability jump, a scary demo, a policy window, too many of us orient to the same spot, guided by the same shared model of the risk. That would be sound if anyone could verify the model, but we cannot.
The better posture is the early-stage investor's. A venture portfolio contains many bets which mostly fail, redeemed by a few that pay off at enormously, and the winners usually looked strange at the start. If outcomes inevitably surprise, then the ideas which the consensus finds unpromising are systematically underpriced, and some work deserves funding precisely because it sounds wrong. Discipline is important too. Venture funds commit for a decade and do not redeploy on noisy interim marks, which is the right posture in a domain where interim signals mean little.
This may also be a case for more investment in researching strange new long-term forecasting techniques. Critically, it would be less useful to forecast AI capabilities; instead we should try to forecast where we are on the long, dark, branchy path to our goal. Very hard.
In summary, I suggest:
1) Do not latch on to how things are going, good or bad.
2) Invest in a broad portfolio of diversified bets.
3) Invest in new methods to figure out where we are on the path to making AI go well.
I encourage you to assess just how ‘in the dark’ you believe we are playing.
