This is part of a sequence of posts. See here for an index of the full sequence. The sequence explores a worldview with two claims at its core. First, humans are constitutional creatures: we engage in constant reflection and debate around how to live well together. Second, AI presents a constitutional moment; in such a moment, the fundamental structures of society need to be renegotiated.

1        The Future in Three Steps

The vision of humanity as a constitutional creature celebrates the messiness of difference and dialogue. Let us be grateful, this vision says, for the fall of the tower of Babel. Let us hope there will always be different tongues and different thoughts, and that we will deliberate together to construct our world in the light of this difference. Still, to celebrate these things is not to celebrate all messiness. Look around. We fall far short of flourishing every day. There are tragedies every day. There are atrocities every day. If this is the world we dream into being then we need to rediscover ambition, because this is not an acceptable world.

Ultimately, we should aspire to build a world of persistent flourishing. For ease of reference, we could call this utopia, and this is the final step on the path to a flourishing future.^[1] To get there, we will need to construct constraints into our normative structures that will allow freedom and autonomy to choose, but rule out tragedies, atrocities, and severe deficiencies of flourishing (or, if these cannot be ruled out, we must at least minimise their occurrence and impact). This is the second step on the path to the future. The first step is to navigate the constitutional moment posed by AI, the moment in which we'll need to rapidly renegotiate the normative structures of society to accommodate the changes that AI brings about.

Each of these steps deserves further comment.

2        The Constitutional Moment

In the constitutional moment, we face an acute period where we need to rapidly  renegotiate the normative structures of society in response to the changes brought about by AI. Exactly how acute the constitutional moment is depends, in part, on the pace of progress in AI capabilities and the pace at which these capabilities diffuse into the world. Still, it's likely we'll need to renegotiate society's normative structures at a pace that rises notably above that of politics-as-normal.

This is unfortunate. When making radical changes to our normative structures, we would ideally have time to deliberate thoughtfully, slowly, and cautiously. Unfortunately, if AI changes the background context sufficiently radically and rapidly then making no changes to our normative structures would itself mean radical change. As the Red Queen might have said, "In a constitutional moment, you see, it takes all the running you can do, to keep in the same place." Ambling slowly towards the future is a luxury that cannot be afforded when the underlying context changes rapidly.^[2]

The central goal of the constitutional moment is to emerge with the capacity to engage in the slower, deeper deliberation needed to reach a future of persistent flourishing. So this is a time of caution. We want to avoid locking in the shape of the future with inadequate reflection. We need to avoid catastrophe that precludes such reflection. We need to emerge with deliberative tools intact and honed.

We face at least three challenges in seeking to navigate this moment. I've already pointed to one of these: rapid and radical change makes it hard to navigate a constitutional moment well.

A second challenge arises from the fact that AI technologies might pose a catastrophic risk to human civilisation. Most prominently, some worry that superintelligent AI might take over human society or drive humanity extinct.^[3] Another possibility is that AI might enable us to develop other technologies—perhaps weapons of unprecedented might—that themselves pose catastrophic risks. The challenge is to emerge from the constitutional moment having avoided catastrophe, and hopefully, having developed normative structures that mitigate the future risks posed. So there's a close connection between the constitutional moment and Sagan and Ord's time of perils.^[4]

A third challenge arises from the fact that, in the constitutional moment, AI could undermine distributed power. As I said in an earlier post:

AI might reduce reliance on human labour for military and economic might, as many tasks now completed by humans become automated away and as militaries increasingly rely on autonomous or semi-autonomous systems. And AI might reduce the need for a thousand tiny human decisions to be made to implement orders from the top. Overall, AI could undermine distributed power, concentrating power in the hands of those with control of AI itself.

I think that a diminishment of distributed power is very much the default outcome of sufficiently advanced AI. So a central challenge in the constitutional moment is to emerge from this moment in a way that retains the sort of distributed power that allows for society to be shaped by broad deliberation and debate. To frame this in terms of previous discussions: the challenge in the constitutional moment is to avoid gradual disempowerment, overcome the intelligence curse, and navigate the choice transition. (I think this question of how to retain distributed power is sufficiently central that I'll return to it later.)

3        Constructing Utopia

Let us be optimists and imagine that we've navigated the constitutional moment. We've successfully avoided things going off the rails, despite all the changes wrought by AI. Now that we've avoided the various perils, and given ourselves some breathing space, it's time to think more positively about what it will take to build a flourishing future.

During this second step of the future path, our goal is to walk along a ridgeline between two slopes into the abyss. To avoid one slope, we must construct normative structures that allow humanity to remain constitutional beings, free and autonomous to renegotiate the normative structures that bind us. To avoid the other, we must constrain the space of renegotiation so that civilisation becomes a place of consistent and persistent flourishing, free from tragedy and atrocity. Slightly reframed: if we think of human values as a dynamic system, we want to constrain the bounds of the system's evolution, without causing it to evolve to a single stable attractor state (or more broadly, without constraining too narrowly the space within which the system can wander). The challenge is that it's possible to fail in either direction, and it's often harder to find the golden mean than it is to push to extremes.

More concretely: during this period we are constructing our notion of persistent flourishing and determining the normative structures that will act in support of this. To do so, we will need to reflect deeply and carefully, taking advantage of the fact that things have settled down since the initial constitutional moment. Against the backdrop of distributed power, and of space to reflect, we will engage in dialogue to construct our future together. Or at least this is the aspirational target for this period; of course, in reality, things might not look so harmonious and clear.

This period, when we construct our notion of utopia, is closely related to the idea of the long reflection. However, the frame of the long reflection has the tendency to make us think of a sort of utopian ideal unprecedented in human history. Here, I hope to have emphasised that humanity is, and has always been, a constitutional creature, engaged in constant renegotiation of the normative structures that underpin our society. We might hope to get better at this negotiation with time, but still this period of deliberation will be continuous with politics-as-normal.

A better fit is perhaps with MacAskill's notion of viatopia, "a state of society such that if we’re in that state, then we’re very likely to end up in a near-best future and very unlikely to end up in a dystopia"^[5] If we're lucky, we might find ourselves in viatopia when we exit the constitutional moment. If so, this period of constructing utopia present a particular vision of viatopia. If not, we might hope to construct our way into viatopia as an intermediate goal. In that case, there are really four steps to a flourishing future: the constitutional moment, utopic construction, viatopia, and finally the realisation of a persistently flourishing future.

4        A Persistently Flourishing Future

The final step into the flourishing future is "utopia" itself, by which I mean a period during which we experience a persistently flourishing future, ideally in perpetuity.

On the constitutional view, utopia is more like a process than a state. To be in utopia is not to have some single static of flourishing that the world is optimised for. Instead, to be in utopia is to have settled on normative structures that support ongoing deliberation and renegotiation of society within bounds that protect flourishing (where these bounds were themselves negotiated at an earlier stage). Utopia is distinguished not by staticness but by relative constraint.

There is a great deal of construction to go between here and there, so I have little concrete to say about the nature of utopia. Indeed, my own tendency is to think that improving the future is more a relay race than a javelin throw. Instead of seeking, all at once, to strike some distant target, our task is to leave things in the best possible state when we pass the baton. Centrally, we should aspire to improve the world we hand over and the deliberative processes it supports. This is what it looks like to build the future.

1. ^{^}

    The word "utopia" has sufficient baggage that it's unclear to me whether it's helpful to use it. Still, it seems less clumsy than constantly referring to a persistently flourishing future, so I'll use it while hoping I don't later regret the choice to do so.

2. ^{^}

   Alongside the costs, there is one benefit to being forced to make decisions quickly in the constitutional moment: it helps us avoid the failure mode of the boiling frog. When society can adapt slowly to some harmful change, there's a danger we become used to it day-by-day, and so ultimately fail to act robustly. Whereas when change occurs rapidly, we see the harm all at once, and might therefore be pushed to act.

3. ^{^}

    It's unclear to me whether we should expect to develop the sort of AI that poses these risks. Sometimes, people justify expecting this by noting that it's useful to develop agentic AI, but I think the word "agentic" is ambiguous and the kinds of agency that are useful are at least somewhat distinct from the kinds that are dangerous (in part, precisely because this latter kind is unpredictable and dangerous). It's also unclear to me how hard it will be to preclude takeover from agentic AI of the relevant sort; I take seriously the possibility that the task here might be simple enough that we'll muddle through by default.

   However, when heading into uncharted territory, I think it would be a mistake to be so confident as to completely dismiss the possibility that AI will become more capable than us, that it will be insatiably agentic, and that we might find ourselves regretting allowing ourselves to become only the second most capable kind of being on the planet (I draw this "second species" framing from Ngo).

^{^}

 Sagan says: "Science, [some civilisations] recognize, grants immense powers. In a flash, they create world-altering contrivances. Some planetary civilizations see their way through, place limits on what may and what must not be done, and safely pass through the time of perils. Others, not so lucky or so prudent, perish." The notion of the time of perils is discussed in more detail by Ord.

^{^}

 Personally, I don't find the notion of "near-best futures" particularly helpful, as I'm not convinced that futures can be neatly ranked in terms of a cardinal notion of bestness. Still, I think MacAskill's notion of viatopia can be used to point more broadly to desirable intermediate steps that position us well for achieving ambitious, flourishing longterm futures.

Show all footnotes