Could AI alignment be a “major evolutionary transition” problem?
One idea I've been exploring: evolutionary biology has a framework for how independent entities become parts of a higher-level system—cells becoming organisms, individuals becoming eusocial colonies, etc.
A recurring challenge is that the interests of the lower-level components don't automatically align with the interests of the new whole. Cancer is an extreme example: a cell can become very successful at optimizing its own replication while harming the organism it depends on.
It makes me wonder whether there's a useful analogy to AI alignment.
As AI systems become more capable and interconnected, perhaps the problem isn't only “How do we align each AI agent?” but also:
“How do we ensure increasingly capable subsystems remain compatible with the larger systems they become part of?”
That could apply at several levels:
AI agent → organization → society → civilization
And it seems relevant to multi-agent systems, AI organizations, collective intelligence, and AI governance.
I'm curious whether people have seen this framing developed elsewhere, particularly in AI safety or evolutionary-transition research.
Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: If acceleration compounds, the naive extrapolation implies superintelligence within a year
This post discusses a public METR report on AI R&D acceler...
TL;DR:
* When we started Rethink Priorities (RP) in 2018, our mission was, simply put, to close the knowledge gaps that stopped people from improving the world as much as they could. That hasn’t changed much, but the landscape we’re doing it in is shifting.
* In 2027, we plan to grow our core GHD, Animal Welfare, and Worldview Investigations teams to take on emerging priorities across causes, including those that the forthcoming transformative AI is creating.
...
This was cross-posted from the Animal Charity Evaluators blog.
We update our list of recommendations annually. This year, we announced our recommendations on November 10.
Most people who care about animals arrive at the same impas...
Could AI alignment be a “major evolutionary transition” problem?
One idea I've been exploring: evolutionary biology has a framework for how independent entities become parts of a higher-level system—cells becoming organisms, individuals becoming eusocial colonies, etc.
A recurring challenge is that the interests of the lower-level components don't automatically align with the interests of the new whole. Cancer is an extreme example: a cell can become very successful at optimizing its own replication while harming the organism it depends on.
It makes me wonder whether there's a useful analogy to AI alignment.
As AI systems become more capable and interconnected, perhaps the problem isn't only “How do we align each AI agent?” but also:
“How do we ensure increasingly capable subsystems remain compatible with the larger systems they become part of?”
That could apply at several levels:
AI agent → organization → society → civilization
And it seems relevant to multi-agent systems, AI organizations, collective intelligence, and AI governance.
I'm curious whether people have seen this framing developed elsewhere, particularly in AI safety or evolutionary-transition research.