Good note. Also worth keeping in mind the base rate of companies going under. FTX committing massive fraud was weird; but a young, fast-growing, unprofitable company blowing up was decidedly predictable, and IMO the EA community was banking too hard on FTX money being real.

Plus the planning fallacy, i.e., if someone says they want to do something by some date, then it'll probably happen later than that.

My off-the-cuff guess is

• 30% chance Anthropic IPOs by end of 2028
• 20% chance Anthropic IPOs in 2029 or later
• 50% chance Anthropic never IPOs—because they go under for normal business-y reasons, or they build AGI first, or we're all dead, or whatever

By default, people don't talk publicly about their donation plans, especially 1–2 years in advance. It probably just hasn't occurred to them that they should.

The responsible thing to do is to go look at the balance of what experts in a field are saying, and in this case, they're fairly split

This is not a crux for me. I think if you were paying attention, it was not hard to be convinced that AI extinction risk was a big deal in 2005–2015, when the expert consensus was something like "who cares, ASI is a long way off." Most people in my college EA group were concerned about AI risk well before ML experts were concerned about it. If today's ML experts were still dismissive of AI risk, that wouldn't make me more optimistic.

"EA" isn't one single thing with a unified voice. Many EAs have indeed denounced OpenAI.

As an EA: I hereby denounce OpenAI. They have greatly increased AI extinction risk. The founding of OpenAI is a strong candidate for the worst thing to happen in history (time will tell whether this event leads to human extinction).

I love hearing that my writing helps! And Palisade is doing important work and I think they're a good place to donate.

I'm forecasting a 0.00% to 0.02% probability range for AGI by the end of 2034, and that if I were to make 100 predictions of a similar kind, more than 95 of them would have the "correct" probability range

I kinda get what you're saying but I think this is double-counting in a weird way. A 0.01% probability means that if you make 10,000 predictions of that kind, then about one of them should come true. So your 95% confidence interval sounds like something like "20 times, I make 10,000 predictions that each have a probability between 0.00% and 0.02%; and 19 out of 20 times, about one out of the 10,000 predictions comes true."

You could reduce this to a single point probability. The math is a bit complicated but I think you'd end up with a point probability on the order of 0.001% (~10x lower than the original probability). But if I understand correctly, you aren't actually claiming to have a 0.001% credence.

I think there are other meaningful statements you could make. You could say something like, "I'm 95% confident that if I spend 10x longer studying this question, then I would end up with a probability between 0.00% and 0.02%."

My confidence interval is over 95%.

What do you mean by this? What is it that you're 95% confident about?

I don't think this should be downvoted. It's a perfectly fine example of reasoning transparency. I happen to disagree, but the disagree-vote button is there for a reason.

I won't go through this whole post but I'll pick out a few representative bits to reply to.

Deutsch’s idea of explanatory universality helps clarify the mistake. Persons are universal explainers. They create new explanations that were not contained in past data. This creativity is not extrapolation from a dataset. It is invention.

LLMs do not do this. They remix what exists in their training corpus. They do not originate explanatory theories.

This statement expresses a high degree of confidence in a claim that has, as far as I can tell, zero supporting evidence. I would strongly bet against the prediction that LLMs will never be able to originate an explanatory theory.

Until we understand how humans create explanatory knowledge, we cannot program that capacity.

We still don't know how humans create language, or prove mathematical conjectures, or manipulate objects in physical space, and yet we created AIs that can do those things.

The AI 2027 paper leans heavily on forecasting. But when the subject is knowledge creation, forecasting is not just difficult. It is impossible in principle. This was one of Karl Popper’s central insights.

I am not aware of any such insight? This claim seems easily falsified by the existence of superforecasters.

And: if prediction is impossible in principle, then you can't confidently say that ASI won't kill everyone, therefore you should regard it as potentially dangerous. But you seem to be quite confident that you know what ASI will be like.

The rationalist story claims a superintelligent AI will likely be a moral monster. This conflicts with the claim that such a system will understand the world better than humans do.

https://www.lesswrong.com/w/orthogonality-thesis