All of Raymond D's Comments + Replies

Your concern about EA's consequentialist lens warping these fields resonates with what I found when experimenting with multi-AI deliberation on ethics. I had Claude, ChatGPT, Grok, and Gemini each propose ethical frameworks independently, and each one reflected its training philosophy - Grok was absolutist about truth-seeking, Claude cautious about harm, ChatGPT moderate and consensus-seeking. The key insight: single perspectives hide their own assumptions. It's only when you compare multiple approaches that the blindspots become visible. This makes your point about EA flooding these areas with one ontology particularly concerning. If we're trying to figure out "AI character" or "gradual disempowerment" through purely consequentialist framing, we might be encoding that bias into foundational work without realizing it. Maybe the solution isn't avoiding EA involvement, but structuring the work to force engagement with different philosophical traditions from the start? Like explicitly pairing consequentialists with virtue ethicists, deontologists, care ethicists, etc. in research teams. Or requiring papers to address "what would critics from X tradition say about this framing?" Your "gradual disempowerment" example is perfect - this seems like it requires understanding emergent structures and collective identity in ways that individual-focused utilitarian thinking might miss entirely. Would you say the risk is: * EA people not recognizing non-consequentialist framings as valid? * EA organizational culture making it uncomfortable to disagree with consequentialist assumptions? * Just sheer numbers overwhelming other perspectives in discourse?

My general take on gradual disempowerment, independent of any other issues raised here, is that I think it's a coherent scenario, but that it ultimately is very unlikely to arise in practice, because it relies on an equilibrium where the sort of very imperfect alignment needed for divergence between human and AI interests to occur over the long-run being stable, even as the reasons for why the alignment problem in humans being very spotty/imperfect being stable get knocked out. In particular, I'm relatively bullish on automated AI alignment conditional on non-power seeking/non-sandbagging if we give the AIs reward but misaligned human-level AI, so I generally think it quite rapidly resolves as either the AI is power-seeking and willing to sandbag/scheme on everything, leading to the classic AI takeover, or the AI is aligned to the principal in such a way that the principal-agency cost becomes essentially 0 over time. Note I'm not claiming that most humans won't be dead/disempowered, I'm just saying that I don't think gradual disempowerment is worth spending much time/money on. Tom Davidson has a longer post on this here.

It makes sense, but it feels like a very narrow conception of what morality ought to concern itself with. In your simulation example, I think it depends on whether we can be fully confident that simulated entities cannot suffer, which seems unlikely to me.