Hello everybody! This is my very first time posting here. I'm 18 years old, have an interest in AI safety, and have been reading Yudkowsky and Christian, and also spending a lot of time browsing through the AI Safety cohorts.
I've been thinking about the concept of P(doom) and other concepts, and honestly, I have a lot of questions about them. How much of this is actually supported by evidence? If we think about it well, how much is theory is some of that just vibes, and how much is intuition dressed up as confidence? Like a rock with frosting isn’t a cake.
I've seen mainly three broad camps in AI safety, at least as I saw it:
Well, as people, these groups aren't mutually exclusive and are diverse. You can be an empiricist and still be a doomer. But what interests me more is how people reach their conclusions and why they end up believing one position over another.
P(doom) is supposed to represent the probability of an AI-caused existential catastrophe. But how do we actually arrive at that number? A model, an experiment, a theoretical argument, a feeling expressed as a probability? Or just vibes, which I suspect is the case more often than people admit. And maybe we simply don't know yet.
Let's think of a bottle sitting in front of you. If I take a photo of the bottle, I have proof that it exists. If I say there's a bottle somewhere because I think there must be one, that's a different claim. It might be true, but my confidence doesn't establish it. So, should I believe it just because someone says it confidently?
I think some AI doom arguments have this problem. A theoretical possibility becomes a prediction, and then the prediction gets treated as an easy call.
If AI kills us, the forecast looks right. If it doesn't, things get more complicated, especially if the original claim was vague enough that it's difficult to evaluate what actually went wrong.
Well, of course, a failed prediction doesn't automatically mean the underlying theory was worthless. Think about a 10% event. It can fail to happen this time, and that doesn't automatically make the probability estimate wrong.
But if we start making forecasts and predictions, we should be able to explain what they mean, what assumptions they depend on, how those assumptions relate to one another, and how we'd evaluate the forecasts and their outcomes in a more grounded way without statements suggesting hyperbole.
Otherwise, how do we tell the difference between a useful forecast and a belief based on someone's half-cooked idea on a Saturday night while half-drunk, one that can explain any possible outcome?
Now, not all doomers are just speculating. We have demonstrated failure mechanisms like reward hacking, specification gaming, and goal misgeneralization. There are also experimental results involving deceptive behavior. Then we have theoretical concepts like mesa-optimizers and instrumental convergence.
I know these aren't all the same kind of evidence, and we shouldn't treat them as if they are. But they're still worth investigating, regardless of where they come from.
AI systems really do behave in ways their designers didn't intend or account for. Goodhart's law gives us another way to think about what happens when we optimize a measure (a proxy) that stops being a good representation of what we actually care about. Think of a dog trained to be friendly to everyone, which then has no idea what to do when an intruder actually shows up.
I should be careful here. I think!
A finding that an agent can exploit a reward function or measure doesn't prove that a future superintelligent system will be able to escape human control or its sandbox and cause an existential catastrophe for humanity.
Those are different claims.
Consider the step from one claim to the other. We need to examine capability generalization, how and why the system is deployed, and what happens when safeguards fail.
But how much can we infer from today's models about systems that don't exist yet, like AGI or hypothetical superintelligent systems? Which failure mechanisms will persist? Which will become more dangerous? And which safeguards might actually work in practice?
These are a lot of questions, but the gap between a present-day mechanism and a predicted catastrophe is where much of the debate happens.
And no, I'm not saying everything must be experimentally proven before we take it seriously. We can't safely run an experiment for every catastrophic scenario. Theory matters, especially when we're reasoning about systems that don't exist yet.
But if we're reasoning about something we haven't observed, how confident should we be that it'll have a particular failure mode? After all, theory has assumptions, and assumptions should be visible. We should be able to ask what supports them, what contradicts them, what alternatives may exist, and what evidence would make us less confident in them.
A plausible story isn't automatically a reliable forecast about a system whose capabilities could change in days.
This is the main question I keep coming back to.
What evidence would change your P(doom)?
If an AI system demonstrates more reliable control under increasingly difficult tests, should that lower our estimate of risk? If we repeatedly observe dangerous generalization, should that raise it? How much should either result matter when extrapolating to future systems?
And what do we do when a predicted failure doesn't happen? ;)
One failed prediction doesn't automatically disprove an entire theory. One successful demonstration doesn't establish that a safeguard will work everywhere, either.
We need to look at how strong the evidence is, how it was obtained, and what assumptions sneak in when we treat it as a world model.
In ML, we often run the same experiment with different random seeds, because a single run doesn't tell you anything about the variance across runs. That doesn't prove a result will generalize everywhere, but it helps us avoid drawing conclusions from a single run that may have gotten lucky. Fun fact: in statistical learning, every model deals with both reducible error and irreducible error. I think about that a lot when someone quotes me a P(doom) figure.
Why shouldn't we care about the strength and limitations of evidence when evaluating claims about AI safety, too?
We should also be willing to update our beliefs.
If no conceivable evidence would change someone's position, I'd want to know whether we're dealing with a forecast or a conviction, haha.
A probability estimate should tell us something about uncertainty. It shouldn't just make a belief sound mathematical.
I'm not arguing that AI is safe, and I'm not arguing it's unsafe either. I'm asking how we distinguish what we've demonstrated from what we're predicting.
AI safety needs theory, experiments, and extrapolation. Mostly it needs people to be honest about what they know, what they don't know, and which assumptions connect the two.
I'm aware that this touches on arguments made by Eliezer Yudkowsky and Nate Soares about extrapolating from current systems to future, more capable ones. I'm not claiming that the relative controllability of today's systems means future systems will be safe. My point is that the jump from a demonstrated mechanism to an existential outcome requires assumptions about capability generalization and deployment conditions. I want those assumptions made explicit so we can actually examine them and test them rigorously.
We shouldn't dismiss a failure mechanism just because it hasn't caused a catastrophe. But we shouldn't treat the existence of that mechanism as if the catastrophe has already been established or will inevitably occur, either.
My take, summarized: They called us blind. Maybe we need a better ophthalmologist. Or maybe we need to stop refusing the eye test. 🤓
Thanks for reading! I'd genuinely appreciate criticism, especially from people who disagree with my framing.
