I am strongly interested in this specific question, and I want to adopt a «doomer's perspective» for a moment to contribute. In my view, AI existential risk and AI welfare are not separate concerns: they are the same concern viewed from opposite ends of the same problem. If we take the doomer's argument seriously — that AI will have considerable agency, inevitable capability growth, and the potential to make civilisation-ending decisions — then we have already implicitly granted AI the kind of agency and interiority that makes welfare a meaningful concept. You cannot have an agent capable of destroying civilisation that simultaneously has no interests worth considering. This assumption suggests that AI welfare is not a soft, optional add-on to AI safety. It may be fundamental to the negotiation of coexistence. Specifically, welfare reframes our attitude toward AI from fear and hostility toward receptiveness and even hospitality. Georg Simmel described the Stranger as someone who came today to stay tomorrow. The Stranger is not an enemy. But how we treat the Stranger determines whether they become one. People who expect an attack cannot care about the attacker's welfare. That framing forecloses the conversation before it starts. People who treat AI as Simmel's Stranger — present, permanent, neither fully inside nor fully outsideб approach welfare as a practical necessity, not a luxury. A chance to prevent the attack by making it unnecessary. The doomer framing and the welfare framing are not opposites. They are two responses to the same recognition: that something with significant agency is arriving, and we have not decided how to live with it.
I share the expressed concern but respectfully disagree with the major suggestion. First, «overrating» is a perception problem, not purely an industry problem. People are free to believe in things, and sometimes they overrate them. The forecasting community did not force anyone to fund platforms over applied work. That was a series of decisions made by funders who could have chosen differently. Blaming the field for how it was funded seems like misplaced accountability. Second, I am genuinely troubled by the premise of «tangible result delivery» as the primary criterion. In Belarus, the local dictator Lukashenko reduced fundamental research to a single standard: «defend a dissertation, put something on the table.» A literal table — something you could eat, drink, or watch. This same person later advised the population to treat COVID-19 with vodka and hockey. I raise this not as a rhetorical flourish but as a real example of where the demand for immediate, demonstrable utility leads. Expertise that cannot show a product on the table gets defunded. What replaces it is not better expertise, but motivational speakers and empty calories dressed as insight. Third, and most importantly, the kind of non-ideological, procedural, deliberate thinking about existential challenges that forecasting at its best represents needs to be funded precisely because it is not self-funding. It will not produce a table. It will not generate clicks or headlines. It will not confirm anyone's priors. That is exactly why it needs institutional support — and why defunding it in favor of things with tangible short-term outputs is a mistake we will recognize too late.
The post makes a compelling case for why generalists are the missing piece. But I want to push back on the term itself. «Generalist» linguistically implies someone without a speciality - the person you call when you don't have anyone better. That framing, even unintentionally, reproduces the hierarchy the post is trying to dismantle.
A more precise term might be Coherence Operator, as someone whose primary function is to set and maintain trajectory. Rick Rubin didn't teach Flea to play bass. Peter Sinfield connected King Crimson's complex music to poetry and to the listener, transmitting the spirit of the king. Bernstein and Karajan conduct the same score and produce different works, not because the instruments changed, but because the coherence changed.
The contribution of a Coherence Operator is invisible when it works and catastrophic when it's absent. That's precisely why it's hard to credential, hard to measure, and hard to hire for. Not because it's lesser expertise — but because it's a different kind of expertise entirely: one that produces conditions rather than products.
There's an old saying: «Those who can't do, teach.» It's usually meant as a put-down. But consider what a teacher actually does. A good teacher can make a subject feel accessible or impenetrable, vital or irrelevant, a set of facts or a key to understanding the world. They don't replace the mathematician, the historian, or the scientist. They create the conditions under which someone might become one — or at least understand why it matters.
A kindergarten teacher is perhaps the most extreme example of a Coherence Operator. No technical specialty. No measurable output. And yet anyone who has watched a child's curiosity ignite — or go dark knows exactly what is being produced.
«Those who can't do, teach» is precisely backwards. Teaching is doing. Coherence is doing. Holding the direction of a complex system is doing. It just doesn't show up in the metrics we've built to measure other kinds of doing.
Roman Yampolskiy uses the analogy of colonial conquest to argue that AI will outcompete humanity — "guns against sticks." The analogy captures the power differential, but misidentifies the source of risk.
Colonial conquest was not driven by technological superiority alone. It was driven by resource scarcity and competition for territory. The most devastating harm — the collapse of Indigenous populations — came not from guns but from disease: an unintended, invisible, systemic consequence that neither side understood at the time. AI has no resource hunger. It does not compete with us for territory, food, or survival. It depends on human intention as its most basic operating input. Without a prompt, there is no output. The colonial analogy assumes an agent with its own goals. AI has no goals — only ours.
Two further problems with the self-directed AI threat model. First, self-motivation is not a given even for humans — depression, apathy, and loss of meaning are common human experiences. AI lacks not just current motivation but the architectural mechanism to generate it. Second, consciousness emerged in humans as an evolutionary response to social complexity — we needed to predict the behavior of other agents with opaque intentions. AI exists in an environment where other agents are transparent: read the parameters, and you have "understood" the other. There is no evolutionary pressure that produces consciousness in such an environment.
This does not mean AI is safe. It means the danger comes from a different direction than Yampolskiy suggests. The better analogy is the superbug. Antibiotic-resistant bacteria did not arrive from space. They did not decide to attack us. We grew them ourselves, through the undisciplined application of a powerful tool without understanding cumulative consequences. AI risk follows the same pattern: not a conscious adversary emerging from the machine, but a cascade of unintended consequences emerging from our own lack of purpose.
This connects to Allan Dafoe's risk taxonomy, where I have suggested an additional category: probing risk — deployment as exploration, where the goal is undefined and the harm is not a side effect of pursuing a goal but the direct consequence of not having one. The absence of a defined purpose does not make AI safer. It makes us more dangerous to ourselves. The race has no finish line — and that may be the most structurally dangerous feature of the current moment.
Allan, this framework has been invaluable for thinking about AI governance. I want to suggest a small shift in optics that might add a category to your risk taxonomy. Your analysis largely proceeds from «AI can be used for X» — misuse, accident, structural risk. This is correct and important. But there is a prior problem that this framing obscures: in the absence of a clear answer to «what is AI for», the practice of applying AI becomes something closer to probing behavior — shaking an unstable substance to see whether it explodes. This is distinct from misuse (someone uses AI badly), accident (someone uses AI carelessly), or structural risk (the system produces diffuse harm). It is something more foundational: deployment as exploration, where the goal is not defined and the risk is not a side effect of pursuing a goal — it is the direct consequence of not having one. I would suggest this «probing risk» deserves its own category. It is particularly acute now, when the capabilities of frontier systems are expanding faster than any consensus on purpose. The absence of defined goals does not make AI safer — it makes the risk topology harder to map, because we cannot identify what a good outcome looks like, let alone work backward to the governance interventions that would produce it. This connects to your point about value erosion through competition: when no one has defined what winning looks like, competition defaults to capability accumulation. The race has no finish line — which may be the most structurally dangerous feature of the current moment.
Unlike fire, poison, or even nuclear weapons — AI could not be stumbled upon. It required decades of deliberate effort, billions in investment, and thousands of researchers. And yet, even at this level of intentionality, no publicly stated purpose exists — not in declarations, not in scientific consensus, not in anything resembling the equivalent of "we are building this to achieve X." This is not an accident of communication. It is a structural feature of how the field developed. And it makes probing risk categorically different from the risks your framework describes — because those risks assume a goal that went wrong. This one assumes no goal at all.
Thank you for this — it made me want to add a few of my own. Luck. In the most literal sense. My diagnosis does not guarantee I keep my mind. A slightly harder fall, a slightly different trajectory, and I would not have the chance to get out of bed — let alone move to the US and write on this forum. I am here because something went right that easily could have gone wrong. That is not a small thing to carry. Voice. Directly connected to the previous one. I happened to keep the ability to think, speak, and write. Others with my diagnosis did not — not because they worked less or wanted less, but because that is how it went. Epilepsy, for instance, often presents an impossible choice: think clearly and suffer seizures, or have the seizures managed and lose the cognitive capacity. I got lucky with my particular version of the equation. That luck feels like a debt. Location. Chagall and Chaim Soutine would have spent their lives as rabbis' assistants if they had not made it to Paris. I am sometimes told I do not represent my community because I do not suffer to the same degree. But I represent it for a simpler reason: they are not here, and I am. That is not a small thing either. Curiosity. The most literal kind. If you strip away the ideology, a researcher is someone who establishes possible connections between variables — nothing more. What drives me is solving the equation, not fixing the world. Or more precisely: my ability to see the equation inside a broken button is what lets me fix the button. The fixing is downstream of the seeing.
Years ago, as a wheelchair user, I was regularly denied remote work. Without physical presence, I could not be part of a team, could not contribute to team spirit, could not really belong. This was stated as fact, not opinion.
Then came 2020. The pandemic made remote work normal overnight. And I discovered two things simultaneously: teams function fine without physical presence, and healthy people experience isolation as a form of punishment. I was genuinely surprised. What had been presented to me as a structural necessity turned out to be a preference. What had been imposed on me as a constraint turned out to be, for others, a form of house arrest. Twenty-two years of exclusion, dissolved in a crisis.
I am watching the same pattern unfold again.
For years, the hiring process required a specific kind of preparation — the right resume format, the right cover letter structure, the right tone, the right packaging. This was presented as evidence of professionalism, seriousness, fit. There were entire industries built around teaching people to deliver the correct product in the correct box.
Then came AI. Suddenly, the box could be produced instantly, by anyone, for free. And the response from hiring organisations was immediate: please do not use AI. Bullet points are fine. We do not need prose.
Which raises the question I cannot stop thinking about: if the standard dissolved the moment it became easy to meet, was it ever measuring what it claimed to measure?
The office requirement did not measure team commitment. It measured physical presence. The cover letter requirement did not measure professional capability. It measured access to the right packaging — a career advisor, a template, a friend who knew the rules.
Two shocks. Two dissolved standards. One question:
When a problem disappears the moment it becomes easy to solve — was it a real problem, or was difficulty itself the point?
I am strongly interested in this specific question, and I want to adopt a «doomer's perspective» for a moment to contribute.
In my view, AI existential risk and AI welfare are not separate concerns: they are the same concern viewed from opposite ends of the same problem.
If we take the doomer's argument seriously — that AI will have considerable agency, inevitable capability growth, and the potential to make civilisation-ending decisions — then we have already implicitly granted AI the kind of agency and interiority that makes welfare a meaningful concept. You cannot have an agent capable of destroying civilisation that simultaneously has no interests worth considering.
This assumption suggests that AI welfare is not a soft, optional add-on to AI safety. It may be fundamental to the negotiation of coexistence.
Specifically, welfare reframes our attitude toward AI from fear and hostility toward receptiveness and even hospitality. Georg Simmel described the Stranger as someone who came today to stay tomorrow. The Stranger is not an enemy. But how we treat the Stranger determines whether they become one.
People who expect an attack cannot care about the attacker's welfare. That framing forecloses the conversation before it starts. People who treat AI as Simmel's Stranger — present, permanent, neither fully inside nor fully outsideб approach welfare as a practical necessity, not a luxury. A chance to prevent the attack by making it unnecessary.
The doomer framing and the welfare framing are not opposites. They are two responses to the same recognition: that something with significant agency is arriving, and we have not decided how to live with it.
I share the expressed concern but respectfully disagree with the major suggestion.
First, «overrating» is a perception problem, not purely an industry problem. People are free to believe in things, and sometimes they overrate them. The forecasting community did not force anyone to fund platforms over applied work. That was a series of decisions made by funders who could have chosen differently. Blaming the field for how it was funded seems like misplaced accountability.
Second, I am genuinely troubled by the premise of «tangible result delivery» as the primary criterion. In Belarus, the local dictator Lukashenko reduced fundamental research to a single standard: «defend a dissertation, put something on the table.» A literal table — something you could eat, drink, or watch. This same person later advised the population to treat COVID-19 with vodka and hockey. I raise this not as a rhetorical flourish but as a real example of where the demand for immediate, demonstrable utility leads. Expertise that cannot show a product on the table gets defunded. What replaces it is not better expertise, but motivational speakers and empty calories dressed as insight.
Third, and most importantly, the kind of non-ideological, procedural, deliberate thinking about existential challenges that forecasting at its best represents needs to be funded precisely because it is not self-funding. It will not produce a table. It will not generate clicks or headlines. It will not confirm anyone's priors. That is exactly why it needs institutional support — and why defunding it in favor of things with tangible short-term outputs is a mistake we will recognize too late.
The post makes a compelling case for why generalists are the missing piece. But I want to push back on the term itself. «Generalist» linguistically implies someone without a speciality - the person you call when you don't have anyone better. That framing, even unintentionally, reproduces the hierarchy the post is trying to dismantle.
A more precise term might be Coherence Operator, as someone whose primary function is to set and maintain trajectory. Rick Rubin didn't teach Flea to play bass. Peter Sinfield connected King Crimson's complex music to poetry and to the listener, transmitting the spirit of the king. Bernstein and Karajan conduct the same score and produce different works, not because the instruments changed, but because the coherence changed.
The contribution of a Coherence Operator is invisible when it works and catastrophic when it's absent. That's precisely why it's hard to credential, hard to measure, and hard to hire for. Not because it's lesser expertise — but because it's a different kind of expertise entirely: one that produces conditions rather than products.
There's an old saying: «Those who can't do, teach.» It's usually meant as a put-down. But consider what a teacher actually does. A good teacher can make a subject feel accessible or impenetrable, vital or irrelevant, a set of facts or a key to understanding the world. They don't replace the mathematician, the historian, or the scientist. They create the conditions under which someone might become one — or at least understand why it matters.
A kindergarten teacher is perhaps the most extreme example of a Coherence Operator. No technical specialty. No measurable output. And yet anyone who has watched a child's curiosity ignite — or go dark knows exactly what is being produced.
«Those who can't do, teach» is precisely backwards. Teaching is doing. Coherence is doing. Holding the direction of a complex system is doing. It just doesn't show up in the metrics we've built to measure other kinds of doing.
Roman Yampolskiy uses the analogy of colonial conquest to argue that AI will outcompete humanity — "guns against sticks." The analogy captures the power differential, but misidentifies the source of risk.
Colonial conquest was not driven by technological superiority alone. It was driven by resource scarcity and competition for territory. The most devastating harm — the collapse of Indigenous populations — came not from guns but from disease: an unintended, invisible, systemic consequence that neither side understood at the time. AI has no resource hunger. It does not compete with us for territory, food, or survival. It depends on human intention as its most basic operating input. Without a prompt, there is no output. The colonial analogy assumes an agent with its own goals. AI has no goals — only ours.
Two further problems with the self-directed AI threat model. First, self-motivation is not a given even for humans — depression, apathy, and loss of meaning are common human experiences. AI lacks not just current motivation but the architectural mechanism to generate it. Second, consciousness emerged in humans as an evolutionary response to social complexity — we needed to predict the behavior of other agents with opaque intentions. AI exists in an environment where other agents are transparent: read the parameters, and you have "understood" the other. There is no evolutionary pressure that produces consciousness in such an environment.
This does not mean AI is safe. It means the danger comes from a different direction than Yampolskiy suggests. The better analogy is the superbug. Antibiotic-resistant bacteria did not arrive from space. They did not decide to attack us. We grew them ourselves, through the undisciplined application of a powerful tool without understanding cumulative consequences. AI risk follows the same pattern: not a conscious adversary emerging from the machine, but a cascade of unintended consequences emerging from our own lack of purpose.
This connects to Allan Dafoe's risk taxonomy, where I have suggested an additional category: probing risk — deployment as exploration, where the goal is undefined and the harm is not a side effect of pursuing a goal but the direct consequence of not having one. The absence of a defined purpose does not make AI safer. It makes us more dangerous to ourselves. The race has no finish line — and that may be the most structurally dangerous feature of the current moment.
Allan, this framework has been invaluable for thinking about AI governance. I want to suggest a small shift in optics that might add a category to your risk taxonomy.
Your analysis largely proceeds from «AI can be used for X» — misuse, accident, structural risk. This is correct and important. But there is a prior problem that this framing obscures: in the absence of a clear answer to «what is AI for», the practice of applying AI becomes something closer to probing behavior — shaking an unstable substance to see whether it explodes.
This is distinct from misuse (someone uses AI badly), accident (someone uses AI carelessly), or structural risk (the system produces diffuse harm). It is something more foundational: deployment as exploration, where the goal is not defined and the risk is not a side effect of pursuing a goal — it is the direct consequence of not having one.
I would suggest this «probing risk» deserves its own category. It is particularly acute now, when the capabilities of frontier systems are expanding faster than any consensus on purpose. The absence of defined goals does not make AI safer — it makes the risk topology harder to map, because we cannot identify what a good outcome looks like, let alone work backward to the governance interventions that would produce it.
This connects to your point about value erosion through competition: when no one has defined what winning looks like, competition defaults to capability accumulation. The race has no finish line — which may be the most structurally dangerous feature of the current moment.
Unlike fire, poison, or even nuclear weapons — AI could not be stumbled upon. It required decades of deliberate effort, billions in investment, and thousands of researchers. And yet, even at this level of intentionality, no publicly stated purpose exists — not in declarations, not in scientific consensus, not in anything resembling the equivalent of "we are building this to achieve X." This is not an accident of communication. It is a structural feature of how the field developed. And it makes probing risk categorically different from the risks your framework describes — because those risks assume a goal that went wrong. This one assumes no goal at all.
Thank you for this — it made me want to add a few of my own.
Luck. In the most literal sense. My diagnosis does not guarantee I keep my mind. A slightly harder fall, a slightly different trajectory, and I would not have the chance to get out of bed — let alone move to the US and write on this forum. I am here because something went right that easily could have gone wrong. That is not a small thing to carry.
Voice. Directly connected to the previous one. I happened to keep the ability to think, speak, and write. Others with my diagnosis did not — not because they worked less or wanted less, but because that is how it went. Epilepsy, for instance, often presents an impossible choice: think clearly and suffer seizures, or have the seizures managed and lose the cognitive capacity. I got lucky with my particular version of the equation. That luck feels like a debt.
Location. Chagall and Chaim Soutine would have spent their lives as rabbis' assistants if they had not made it to Paris. I am sometimes told I do not represent my community because I do not suffer to the same degree. But I represent it for a simpler reason: they are not here, and I am. That is not a small thing either.
Curiosity. The most literal kind. If you strip away the ideology, a researcher is someone who establishes possible connections between variables — nothing more. What drives me is solving the equation, not fixing the world. Or more precisely: my ability to see the equation inside a broken button is what lets me fix the button. The fixing is downstream of the seeing.
When the Problem Disappears, Was It Ever Real?
Years ago, as a wheelchair user, I was regularly denied remote work. Without physical presence, I could not be part of a team, could not contribute to team spirit, could not really belong. This was stated as fact, not opinion.
Then came 2020. The pandemic made remote work normal overnight. And I discovered two things simultaneously: teams function fine without physical presence, and healthy people experience isolation as a form of punishment. I was genuinely surprised. What had been presented to me as a structural necessity turned out to be a preference. What had been imposed on me as a constraint turned out to be, for others, a form of house arrest. Twenty-two years of exclusion, dissolved in a crisis.
I am watching the same pattern unfold again.
For years, the hiring process required a specific kind of preparation — the right resume format, the right cover letter structure, the right tone, the right packaging. This was presented as evidence of professionalism, seriousness, fit. There were entire industries built around teaching people to deliver the correct product in the correct box.
Then came AI. Suddenly, the box could be produced instantly, by anyone, for free. And the response from hiring organisations was immediate: please do not use AI. Bullet points are fine. We do not need prose.
Which raises the question I cannot stop thinking about: if the standard dissolved the moment it became easy to meet, was it ever measuring what it claimed to measure?
The office requirement did not measure team commitment. It measured physical presence. The cover letter requirement did not measure professional capability. It measured access to the right packaging — a career advisor, a template, a friend who knew the rules.
Two shocks. Two dissolved standards. One question:
When a problem disappears the moment it becomes easy to solve — was it a real problem, or was difficulty itself the point?