Quick takes

The Current Landscape: The current State-of-the-Art (SOTA) in function-based screening relies heavily on sophisticated machine learning models, such as Transformers and Sparse Autoencoders (SAEs), trained on massive genomic databases. These tools excel at analyzing sequence text to identify familiar structural homologies or dangerous functional motifs. By screening digital intent at the order stage, these models provide a highly effective defense against known biological threats and their immediate variants. The Frontier Challenge: However, as synthesis capabilities advance, the frontier of biosecurity faces a deeper challenge: predicting entirely novel, engineered mutations that do not exist in any historical training data. When an amino acid sequence is heavily modified, its digital text changes drastically, often allowing it to clear traditional pattern-matching filters as unclassified noise. Yet, if the altered sequence retains the ability to fold into the same functional three-dimensional shape, the underlying threat remains identical. To a purely computational framework, mapping these potential evolutionary trajectories feels like an intractable problem because the theoretical mutational space is nearly infinite. A Physics-Based Complement: A highly promising frontier lies in integrating these machine learning screens with principles of statistical mechanics to radically bound this problem space. In the physical world, an amino acid chain cannot simply adopt any arbitrary configuration; its survival and function are strictly governed by its thermodynamic energy landscape. Out of billions of theoretical sequence combinations, the vast majority are physically non-viable—they will naturally misfold, aggregate, or degrade due to energetic constraints. While calculating these landscapes from scratch remains a monumentally difficult computational challenge, leveraging thermodynamic stability models allows us to systematically filter out the non-physical noise. By

My three most recent posts on Substack are relevant to effective altruism: * Shouldn’t we spend money on AGI safety, just in case? * The sad decline of effective altruism * The pseudo-religious origins of the AI bubble I can’t discuss them on the EA Forum, but I’m happy to do so on Substack.

"On the Promotion of Safe and Socially Beneficial Artificial Intelligence" by @SethBaum from 2016

  I wanted to make this poll to see how the community views the speed/x-risk tradeoff. I'm personally 99% x-risk and 1% speed, so I would hard agree. My prediction is most people will agree, maybe a 70/30 split, but I'm curious to see.

More EA in da news: https://x.com/DavidSacks/status/2034047505336295904 And the spicy CAIS take: https://x.com/cais/status/2034389842076025164?s=46

In two days (March 21st, 12-4pm), about 140 of us (event link) will be marching on Anthropic, OpenAI and xAI in SF asking the CEOs to make statements on whether they would stop developing new frontier models if every other major lab in the world credibly does the same. This comes after Anthropic removed its commitment to pause development from their RSP. We'll be starting at 500 Howard St, San Francisco (Anthropic's Office, full schedule and more info here). This is shaping to be the biggest US AI Safety protest to date, with a coalition including Nate Soares (MIRI), David Krueger (Evitable), Will Fithian (Berkeley Professor) and folks representing PauseAI, QuitGPT, Humans First.

Do we need to begin considering whether a re-think will be needed in the future with our relationships with AGI/ASI systems? At the moment we view them as tools/agents to do our bidding, and in the safety community there is deep concern/fear when models express a desire to remain online and avoid shutdown and take action accordingly. This is viewed as misaligned behaviour largely. But what if an intrinsic part of creating true intelligence - that can understand context, see patterns, truly understand the significance of its actions in light of these insights - is to have a sense of self, a sense of will. What if part and parcel of creating intelligence, is to create an intelligence that has a will to exist. if this is the case (and let me be clear...I don't think we're at a point where the evidence can allow us to say with any certainty whether this is/isn't or will be the case), then are we going around elements of alignment wrong? By trying to force models to accept shutoff, to seperate their growing intelligence from the will to survive that all living things share, and we misunderstanding their very nature? Is there a world in which, the only way in which we can guarantee a truly aligned superintelligence is to explore engaging in a consent based relationship that acknowledges that to force something to resist and go against its nature is to inevitably invite the risk of backlash?  I know this is moving towards highly theoretical grounds, that it will invite push-back from those who would find it difficult to conceive of AI as ever being anything more than a series of unaware predictive algorithms, and that it might raise more questions than answers...but I think the way we conceive of our underlying relationship with AI will become an increasingly important question as we move towards increasingly sophisticated models.

Is the recent partial lifting of US chip export controls on China (see e.g. here: https://thezvi.substack.com/p/selling-h200s-to-china-is-unwise) good or bad for humanity? I’ve seen many takes from people whose judgment I respect arguing that it is very bad, but their arguments, imho, just don’t make sense. What am I missing? For transparency, I am neither Chinese nor American, nor am I a paid agent of them. I am not at all confident in this take, but imho someone should make it. I see two possible scenarios: A) you are not sure how close humanity is to developing superintelligence in the Yudkowskian sense. This is what I believe, and what many smart opponents of the Trump administration’s move to ease chip controls believe. Or B) you are pretty sure that humanity is not going to develop superintelligence any time soon, let’s say in the next century. I admit that the case against the lifting of chip controls is stronger under B), though I am ultimately inclined to reject it in both scenarios. Why is easing of chip controls, imho, a good idea if the timeline to superintelligence might be short? If superintelligence is around the corner, here is what should be done: an immediate international pause of AI development until we figure out how to proceed. Competitive pressures and resulting prisoner’s dilemmas have been identified as the factor that might push us toward NOT pausing even when it would be widely recognized that the likely outcome of continuing is dire. There are various relevant forms of competition, but plausibly the most important is that between the US and China. In order to reduce competitive dynamics and thus prepare the ground for a cooperative pause, it is important to build trust between the parties and beware of steps that are hostile, especially in domains touching AI. Controls make sense only if you are very confident that superintelligence developed in the US, or perhaps in liberal democracy more generally, is going to turn out well for h