Tuesday night was a late one for me, as I’m sure it was for the many waiting anxiously for election results to roll in across the country. That’s not why I was awake though. Instead, I found myself consumed by a proposal shared earlier in the day by Pax Machina, a publication exploring the design and dynamics of institutions as we enter a new age of intelligence. In this proposal, Joe Edelman of the Meaning Alignment Institute introduces Telocracy: a vision for a society where AI holds institutions accountable to their purposes.
Those who’ve read my essay, Preconditions of Gradual Disempowerment, will understand why I’m so excited about this — Telocracy presents solutions aimed at the heart of my greatest concerns. And although I’ve been immersing myself in the world of AI safety for almost 6 months now, this is the first time I’ve found a truly compelling articulation of how to address the structural risks that I’ve feared could be left unresolved.
Edelman’s vision may seem ambitious, yet the proposal is rooted in concrete mechanisms that could make the institutions in which advanced AI will be embedded far more capable of preserving human agency and serving human purposes. What it does not do is answer the question of how to actually bring this transformative vision to life — so I asked Edelman personally. He shared that he has a rough plan he will cover in a future essay, and that while it may be harder to pull off than the New Deal, it shouldn’t be as hard as the American Revolution.
I certainly agree that it will be difficult… but when I say I’m consumed by something I do not say that lightly. I haven’t stopped thinking about this, and I also can’t help but think about it alongside all the other ideas that have equally consumed me. So rather than wait patiently for Edelman’s next essay, I feel compelled to not let all these thoughts go to waste.
That’s where AI 2040 comes into the equation.
For those unaware, AI 2040 is a research-led initiative published by the AI Futures Project, the organization behind AI 2027 — a project that presented possible trajectories for transformative AI and became widely known for articulating how rapid advancement could culminate in an AI takeover. AI 2040 takes a different approach, presenting a positive vision for navigating the future safely through its recommended path, Plan A, alongside several alternative strategies set out in other plans exploring how the US might respond to the challenges ahead.
At the core is the dynamic between the US and China. Plan A centers on international cooperation, transparency, and restraint as the major powers continue to scale frontier capabilities. Compared with the other plans, such as sabotaging China (Plan B), racing ahead at near-max speed (Plan D), or a full stop on all frontier AI progress (Plan S), it certainly seems like a reasonable approach.
Like much of the AI safety field, Plan A largely focuses on matters related to the development and deployment of AI. I don’t disagree with that approach, but I’d argue we can place equal focus on the state of the institutions and systems where that AI will be deployed. AI 2040 does recognize the importance of institutional and epistemic conditions, but it presents them as supplemental elements within its broader strategy rather than treating them as an immediate priority. We cannot simply hope that issues will be resolved once we are well on the path — we must seek to improve our institutions and systems now so we can ensure they are capable of getting us to the best outcomes possible. Achieving alignment, proper governance, and unparalleled international cooperation still won’t be enough to ensure long-term resilience if advanced AI arrives in institutions with inadequate incentives, accountability, and oversight.
Some may disagree with my perspective or consider these matters outside the purview of AI safety organizations. To the latter, I’d ask: if not AI safety, then who?
We shouldn’t just be trying to make advanced AI safe for the institutions as they exist today. We should be using advanced AI to make those institutions safer, more accountable, and more responsive to human purposes. Telocracy provides mechanisms and models for embarking on those efforts, so to that end, I propose an alternative pathway to consider alongside those within AI 2040: Plan T.
Plan T would not treat institutions as passive recipients of technical safety measures, but instead as a substantial part of the safety problem, placing their design and functioning at the center of the strategy. Plan T would make institutional transformation a priority by combining the core mechanisms described in Edelman’s proposal with selected elements of Plan S and Plan A. Rather than trying to persuade every actor in the AI ecosystem to behave responsibly despite their incentives, it would seek to fundamentally change the institutional structures that produce those incentives.
Plan T would work to address sources of risk that might otherwise remain largely untouched, including sources of existential risk that technical alignment and international cooperation alone cannot resolve. It would leverage advanced AI capabilities to strengthen human agency as well as transform institutions toward greater fidelity to human purposes, and in doing so, establish the political, economic, and epistemic conditions necessary for a safe and sustainable transition to even more powerful AI.
I won’t claim to have mapped out the entirety of Plan T in a manner similar to the AI 2040 plans, but I do have some sense of what its components could look like. And although I’ve described it as an alternative pathway, it is not an approach that attempts to fully replace the AI 2040 recommendations — it aims to make their objectives more achievable and sustainable by strengthening the institutional foundations on which those pathways depend.
Interconnected Components of Plan T
This is a simple overview of the components rather than a comprehensive breakdown of what each would entail. It is also not presented as a step-by-step sequence, and, in practice, many elements of these components would need to occur concurrently. That said, the core idea can be reduced even further to a rough strategic arc:
Stop → Transform → Accelerate
To be clear, my use of accelerate doesn’t mean that Telocracy would allow us to race ahead with minimal regard for safety, as in Plan D. However, transforming institutions, incentives, and collective agency may create conditions in which some forms of development can proceed more safely at a faster pace. Ultimately, that advancement could follow a trajectory similar to Plan A, but perhaps with fewer institutional risks and stronger prospects for international cooperation. It may even contribute to increasing public support for advancement, achieving higher confidence in alignment, and establishing mechanisms for governance that address risk without concentrating power.
Properly developing Plan T into an actionable pathway will require deep investigation, rigorous examination, and collaboration from experts across multiple domains. It will also come with many challenges that must be addressed, some of which will be very difficult to overcome - but not nearly as difficult as the challenges we may face in the future if we choose not to address them at all. In the foreword to Edelman’s proposal, Nora Ammann notes some of these challenges:
If power increasingly flows through what people say they value, pressure will flow there too, toward shaping and gaming values and beliefs. Can telocracy be made robust to that? Who builds, evaluates and owns the models, compute and data it runs on, and can they be engineered to deserve our trust? Finally, how do we get from here to there, learning as we go?
I agree that these are the most pressing, and I’d emphasize the importance of addressing how and by whom institutional purposes are determined, interpreted, challenged, and revised. I’d also add the challenges of preserving epistemic vitality and meaningful human judgment, distinguishing institutional fidelity from institutional rigidity, and preventing capture of Telocratic institutions themselves.
If a proper plan is developed and its challenges adequately addressed, Plan T could become the most promising pathway for safer advancement. And just as I’ve refrained from conceptualizing the entirety of Plan T, I won’t attempt to present solutions to these challenges. There is, however, one question I do want to offer some thoughts on, and it’s the one Ammann succinctly posed in her note:
How do we get from here to there?
This might be the most important question of all. Even the most compelling vision with a perfectly developed plan won’t be worth much if it cannot be brought into existence. This path cannot be mapped solely behind closed doors or entrusted only to those who hold power. It should not merely be an idea discussed quietly among those already convinced of its importance, with its viability depending on hope rather than action.
This vision requires public understanding and broad participation as it is far closer to a social movement than a policy proposal. It must be taken directly to the people so that the ideas it presents may spread among them, allowing support to swell on its behalf. There must be leaders across both government and industry who are aligned with preserving human flourishing and willing to champion a path forward centered on that goal. It must be top of mind for the electorate and on the tips of the tongues of those seeking to represent them.
I believe the time for advocacy is now. The tides of change are beginning to rise, and the recent elections have shown that AI is becoming a tangible political issue well beyond a relatively small circle of researchers, policymakers, and industry leaders. The economic, infrastructural, and societal implications are entering public discourse and becoming matters that voters and candidates must confront. All the while, new evidence shared by METR points to acceleration beyond what AI 2027 predicted. It may not be long before deployment of increasingly powerful systems expands and their implications become widespread, potentially setting in motion some of the dynamics explored in The Intelligence Curse. If continued advancement can truly lead to an era of abundance and an eradication of suffering, then we must show the public how we can reshape society to ensure the future we’re advancing toward is one that reflects the wants and needs of the people.
I have many more thoughts on how the vision of Telocracy could help shape a promising path ahead, but I’ll let those marinate for a bit longer rather than share them here. For now, I’ll close with a final question, one that I hope I’ll have an answer to in the near future:
If it’s not AI safety leading the charge, then who?
