I've never heard this idea proposed before so it seems novel and interesting.

As you say in the post, the AI risk movement could gain much more awareness by associating itself with the climate risk advocacy movement which is much larger. Compute is arguably the main driver of AI progress, compute is correlated with energy usage, and energy use generally increases carbon emissions so limiting carbon emissions from AI is an indirect way of limiting the compute dedicated to AI and slowing down the AI capabilities race.

This approach seems viable in the near future until innovations in energy technology (e.g. nuclear fusion) weaken the link between energy production and CO2 emissions, or algorithmic progress reduces the need for massive amounts of compute for AI.

The question is whether this indirect approach would be more effective than or at least complementary to a more direct approach that advocates explicit compute limits and communicates risks from misaligned AI.

A recent survey of AI alignment researchers found that the most common opinion on the statement "Current alignment research is on track to solve alignment before we get to AGI" was "Somewhat disagree". The same survey found that most AI alignment researchers also support pausing or slowing down AI progress.

Slowing down AI progress might be net-positive if you take ideas like longtermism seriously but it seems challenging to do given the strong economic incentives to increase AI capabilities. Maybe government policies to limit AI progress will eventually enter the Overton window when AI reaches a certain level of dangerous capability.

This is a cool project! Thanks for making it. Hopefully it makes the book more accessible.

Update: the UK government has announced £8.5 million in AI safety funding for systematic AI safety.

Thanks for writing this! It's interesting to see how MATS has evolved over time. I like all the quantitative metrics in the post as well.

Thanks for the writeup. I like how it's honest and covers all aspects of your experience. I think a key takeaway is that there is no obvious fixed plan or recipe for working on AI safety and instead, you just have to try things and learn as you go along. Without these kinds of accounts, I think there's a risk of survivorship bias and positive selection effects where you see a nice paper or post published and you don't get to see experiments that have failed and other stuff that has gone wrong.

I'm sad to hear that AISC is lacking in funding and somewhat surprised given that it's one of the most visible and well-known AI safety programs. Have you tried applying for grant money from Open Philanthropy since it's the largest AI safety grant-maker?

"In brief, the book [Superintelligence] mostly assumed we will manually program a set of values into an AGI, and argued that since human values are complex, our value specification will likely be wrong, and will cause a catastrophe when optimized by a superintelligence"

Superintelligence describes exploiting hard-coded goals as one failure mode which we would probably now call specification gaming. But the book is quite comprehensive, other failure modes are described and I think the book is still relevant.

For example, the book describes what we would now call deceptive alignment:

"A treacherous turn can result from a strategic decision to play nice and build strength while weak in order to strike later"

And reward tampering:

"The proposal fails when the AI achieves a decisive strategic advantage at which point the action which maximizes reward is no longer one that pleases the trainer but one that involves seizing control of the reward mechanism."

And reward hacking:

"The perverse instantiation - manipulating facial nerves - realizes the final goal to a greater degree than the methods we would normally use."

I don't think incorrigibility due to the 'goal-content integrity' instrumental goal has been observed in current ML systems yet but it could happen given the robust theoretical argument behind it:

If an agent retains its present goals into the future, then its present goals will be more likely to be achieved by its future self. This gives the agent a present instrumental reason to prevent alternations of its final goals."