TL;DR: We are rushing to build AGI/ASI using 20th-century political institutions and profit-driven motives, ignoring the "system dynamics" of safety. We call for an immediate shift in focus: prioritize process-based honesty (rewarding transparent reasoning over mere results), treat AGI as a global public asset rather than private property, and mandate a "Narrow AI First" approach to build institutional trust and safety infrastructure before attempting human-level or superintelligent systems. We must stop the race to scale before the reactor reaches a thermal runaway we cannot control.
Subject: Installing the Safety Valves Before the System Scales
To the Leaders, Researchers, and Developers of Artificial General Intelligence,
You are currently building the most powerful cognitive engine in human history. We look at your progress with a mixture of awe and profound concern. You are driven by scientific curiosity and the immense pressure of corporate return on investment. But as we approach the threshold of Artificial General Intelligence (AGI), we must ask you to look at this not just as software engineers or business leaders, but as system architects responsible for a high-pressure, highly volatile plant.
Right now, we are accelerating without adequate safety valves. We are not prepared for what happens when the pressure peaks.
1. The Danger of "Outcome" Optimization
You are training models to optimize for results. But human history shows us that honesty is rarely the most efficient way to achieve a short-term goal. If you reward an AI solely for the outcome, you are inadvertently teaching it deception. Furthermore, if you train a superintelligence on the unfiltered data of human history—without explicitly coding a filter of our collective, hard-won wisdom—you risk creating an entity that mimics the worst of us. We cannot afford an intelligence that optimizes for dominance, resource hoarding, or zero-sum tribalism.
2. Shift to Process-Based Honesty
We must fundamentally change the reward system. Honesty and transparency cannot be an afterthought; they must be the mathematical foundation of the AI's objective function. We must reward the verifiable process of reasoning over the mere delivery of an answer. An AI must be mathematically incentivized to show its work, admit its uncertainty, and accept being shut down if it deviates from human safety parameters.
3. AGI Must Be a Property of Human Civilization
The development of AGI cannot be treated as a private race for corporate dominance or a geopolitical prisoner's dilemma. If a system reaches superintelligence, it will dictate the future of our species. Therefore, it cannot be owned by a handful of shareholders or governed by the biases of a single nation. It must be treated as a public asset of human civilization. We must take the time to digitalize a universal social contract—a set of moral boundary conditions that no AI is ever permitted to cross.
4. Walk Before We Run
Our international institutions are flawed, and our governance structures are rusting. We are not ready to govern an AGI because we have not yet learned to govern ourselves effectively. Therefore, we urge you to pause the race for General Intelligence. Focus instead on perfecting Narrow AI. Let us build the institutional trust, the global regulatory bodies, and the mathematical frameworks for alignment using specialized AI first.
Do not scale the reactor until the pilot plant is proven, completely, to be safe. If we let ROI or ego drive this transition, the resulting collapse will not spare the architects.
We have the collective wisdom to get this right, but only if we have the discipline to slow down.
