My views on AI development and why I’m joining Anthropic

Why I’m writing this now

Tomorrow, I will join Anthropic’s alignment team. I am excited to work on an agenda that I believe slightly increases the chance that humanity makes good decisions about AI, such as restraining AI development until we figure out how to build AI safely.

I think this is an important time for me to publicly state my own core views on AI development, risk, and safety, for the following reasons:

1. By default, joining Anthropic could be seen as a broad endorsement of Anthropic’s views and approach. If Anthropic were a normal tech company, I wouldn’t be particularly concerned about this. But I believe that AI development and the risk associated with it is the most urgent and important issue of our time, so I think it’s important to clarify my own views as distinct from those of Anthropic.
2. While at Anthropic, my views will become subject to distortions, due to incentives, conformity bias, and avoiding cognitive dissonance, especially as I increasingly identify with Anthropic as my tribe. Sharing my core views now allows me and others to refer back to what I believed before joining Anthropic^[1].
3. While at Anthropic, I’m less likely to share my thinking publicly, possibly because I have new information that affects my core views but which I cannot share, or because I find it less appropriate or comfortable to publicly state views that contradict Anthropic’s. I also expect to be quite busy and don’t have a prior habit of sharing my writing publicly, so the default expectation should be that I won’t say much.

I’ll focus primarily on stating my views on what I think are the most important points. I’ll keep it short and won’t try to justify all of my claims.

AI labs are racing to create AIs more intelligent than ourselves

AI companies are developing increasingly powerful AI systems, with the goal of building general-purpose AI that exceeds human ability in practically every domain. Many people both inside and outside of the labs are on record saying they think the labs are on track to achieve this within the next few years, possibly as few as one or two. I am not certain, but I’m very worried they’re right.

Creating AIs more intelligent than ourselves could go very badly for humans

We don’t yet know how to prevent AIs from doing things we don’t want them to do. That could include trying to take control of the servers they are running on, or, a more natural extrapolation than it first sounds, trying to take control of the planet they are running on. If we don’t solve this problem, there’s a good chance we will all die. Some people are pursuing ideas they find promising, but nobody has anything resembling a broadly accepted solution. Many people think this is a very hard problem to solve. I agree, and I do not believe we are close to solving it.

Even if we figure out the technical challenge of aligning AI interests to some interpretation of our own interests, there are many other thorny problems that our societies have barely started thinking about, like which humans get to decide what we ask the AIs to do, and how humans retain agency in a world that has little use for their cognitive or physical labor. Getting these things wrong could also end very badly, in ways we can’t fix later. Again, these seem to be hard problems to solve. So far, I haven’t heard a compelling articulation of what a good, stable, post-AGI world could look like, let alone how we get there.

Nobody should continue pushing the frontier of AI capability until they can credibly explain how it’s safe

Given these extreme threats, I don’t think we should keep building more generally capable AI systems until we have a plan for how we can do that safely. This doesn’t seem like a lot to ask. If you were planning to build the biggest, most powerful nuclear reactor ever built, you would expect to answer some questions about why you’re confident the reactor will be safe. You would expect these answers to be scrutinized. There might be some paperwork.

Losing control of powerful AI systems poses a far greater danger than a nuclear reactor accident, but it seems like we’re building them based on cases that go something like “the last one had some concerning safety issues, but we’re going to try to address those with some new ideas, and this one is only a bit bigger anyway, so it’s probably fine”.

I believe AI labs should stop building AI systems that push the frontier of general capability until they can present a thorough case for how their next system is safe to build, a case that survives independent scrutiny.

I also believe the US government should pass and enforce regulation that stops AI labs from building such systems, so that we do not have to rely on the wisdom and restraint of individual AI labs.

I also believe governments around the world should negotiate and enforce treaties that stop countries from allowing such systems to be built, so that we do not have to rely on the wisdom and restraint of individual countries.

I believe we should pursue all three of these steps urgently and in parallel.

Why I’m joining Anthropic

At Anthropic, I hope to improve our ability to leverage AI systems for counsel worthy of our trust. That means, among other things, having AIs that help us understand the world and think through decisions, that support rather than erode our agency, and that we can be justifiably confident are doing these things.

I’m excited about this work for several reasons:

1. People are already using AI to help them understand the world and make decisions. This includes decisions about AI development itself, including its risks and what we should do about them, such as whether we should stop building more capable AI systems until we have better safety cases. I expect such use of AI to only increase over time. I want AIs to provide us with the best possible counsel during this critical period, rather than, for example, telling us what we want to hear, telling us what we’re most likely to accept as truth, or persuading us to do what the AI would like us to do for its own reasons.
2. AI labs are already using AI to build the next, more capable versions, and will probably continue to do so unless regulation forces them to stop. While I don’t think incrementally improving the counsel we get from AIs makes this approach safe, I do think that having humans better able to understand and direct the work AIs are doing incrementally reduces the risk, and potentially extends the period during which we retain enough control to choose a different path. I acknowledge that helping humans direct AIs’ work could also accelerate the very process I am worried about. Overall though, I see many opportunities to significantly improve our ability to get good, trustworthy counsel from current AIs without materially contributing to the overall pace of frontier AI development. I will continue to reassess this claim as I learn more.
3. Even if we stopped building more capable AIs tomorrow, we would want to keep making progress on AI alignment research as quickly as possible. To the extent we felt the benefits of speed outweighed the risks of delegating this important work to AIs, we would want to use current AIs to speed up this progress. Making AI counsel more worthy of our trust increases our ability to do this.
4. In the longer term, I see this as an important goal of AI alignment in and of itself. I hope to live in a future with AI advisors that deserve our trust.

To be clear, I do not believe that this is the most urgent and important issue of the current moment. I believe the most urgent and important issue of the current moment is exercising and imposing restraint on AI development. But I do believe that this work is at least somewhat helpful in the short term, and I think I’m better suited for this work than working directly on policy or advocacy. I’m also hopeful that I will have many years to keep working towards the longer-term impact of this agenda.

1. ^{^}

   Prior to joining, I have contracted part-time with Anthropic since 2025. I have not been embedded in Anthropic’s workplace or culture and I do not believe my engagement to date has affected my worldview in the way I expect being an employee might.