DEEP VZN is a virology research project funded by USAID with the stated goal of identifying and characterising pandemic capable viruses.  In particular, they plan to make the genomes for all viruses they characterise open-source[1].  Rob Reid and Kevin Esvelt argue in a recent podcast that this poses an immense biosecurity threat as there are tens of thousands of individuals/organisations with the means + ability to use these genomes to create and proliferate said viruses, effectively turning them into WMDs.  

I mainly wanted to make this post to make the community more aware if it wasn't already - I think if you want a more thorough breakdown of the arguments for and against what DEEP VZN intends to do, Rob+Kevin's conversation is great (and decidedly lands against).  If you would like to help, they both recommend reaching out to USAID through one of the following avenues and prompting them to more carefully consider these serious downsides to funding this project:

  • Tweeting at them @usaid
  • Submitting a message at usaid.gov/contact-us
  • Calling them directly at their main line 202-712-4300

DEEP VZN's NOFO: https://indiabioscience.org/media/articles/7200AA21RFA00005-NOFO-DEEP-VZN-signed.pdf

[1] Per the bottom of page 16 / top of page 17 of their NOFO linked above.

Comments7


Sorted by Click to highlight new comments since:

Hope it isn't too on the nose, but that was my thinking behind this project proposal.

I'm a former emerging/zoonotic disease epidemiologist, EcoHealth Alliance fellow, generally was all-in on DEEP VZN/PREDICT pandemic prediction type work for the better part of a decade, turned biosecurity-pilled apostate. AMA!

What will it take for bio people to treat dangerous information as if it was dangerous? Do they have a rosy view of the world, where no one would misuse it? Do they have incentives to spread the info? Is the open data movement unwittingly to do with it?

It's going to be a tough sell. The scientists involved are saturated with cultural norms and deep beliefs that more information is always better, and academic and funding incentives are aligned with that understanding.

I don't know that the "open data movement" is, like, radically influencing the beliefs of scientists involved in this kind of work, but rather they're both products of the same (mostly great) culture of openness.

I think the actual long-term solution is to influence trainees and help them rise to positions of influence. In the meantime we need to mitigating risks from these projects in ways that don't depend on changing hearts and minds of the senior scientists most highly invested in their continuation.

To what extent are there already similarly dangerous pathogen genomes on the internet? I'm guessing that things like smallpox are less of a worry because we already have a vaccine for them, but if many novel, certified pandemic-grade pathogen genomes are already available then adding more seems significantly less harmful.

Kevin claims there are none at the moment that he’s particularly concerned about (in large part because most such viruses we have developed vaccines / antivirals for).

I think the more important answer to this question is that most of the virus genomes available online are either from viruses that are unlikely to take off as a pandemic and/or have probably limited expected harm even if they take off. This harm being rather limited might be because most of us have some immunity against the pathogen like for the 1918 Spanish flue as most ppl got the flue (other influenza infections) before or indeed because there are plenty of vaccines available that are ready to go (like for smallpox which additionally is much harder to manufacture from the genome).

I should note that I'm not saying anything new here. This is just from the interview. Esvelt addresses this exact question at 33 / 35 mins in (depending on where you listen to it). He seems to see the claim that there are already (many) pandemic-grade pathogens available online as a common harmful misperception.

First, how much has COVID-19 played into your change of heart here? What do you think of Jeffrey Sachs, chair of the Lancet's COVID origins commission coming to the conclusion that a lab leak was the likely source and true investigation is being prevented?

 https://www.currentaffairs.org/2022/08/why-the-chair-of-the-lancets-covid-19-commission-thinks-the-us-government-is-preventing-a-real-investigation-into-the-pandemic

Second, if there's a non-zero chance that virological research resulted in the current pandemic, I think that it's equivalent to that unbelievable fact about Chernobyl that the USSR kept other reactors of the same design online lest they admit that a fault in their design caused the meltdown. Except in this case, it seems to me that Deep VZN is the equivalent of rolling out thousands more reactors of the same design. If a lab leak indeed is the source of the current COVID pandemic, do you think that fact is necessary to turn the policy tide here? Do you agree with my metaphor here, couched in the fact that we're not 100% sure?

 

Third, how much should MRNA vaccine technology change our risk-benefit analysis of virology that introduces new and more dangerous viruses into imperfect human custody? Treatment or prevention seem like the only two hard arguments for such work, and it seems to me the Moderna 48-hour miracle is an argument that the upside is even more indiscernable.

Fourth and finally, how do we effectively mount this argument when those who are advocating the risky approach are leading industry researchers who seem intent on snuffing out discussion that might pour cold water on their work? Cf. Daszak's coordination of the Lancet article and the Lancet commission member who helped approve EcoHealth grants recommending Sachs's recommendations be struck from the final report?

 

This is a several months old thread, but if you do see this I appreciate your input. I'm not in the field but desperately worried about this as an X-risk.

Curated and popular this week
 ·  · 38m read
 · 
In recent months, the CEOs of leading AI companies have grown increasingly confident about rapid progress: * OpenAI's Sam Altman: Shifted from saying in November "the rate of progress continues" to declaring in January "we are now confident we know how to build AGI" * Anthropic's Dario Amodei: Stated in January "I'm more confident than I've ever been that we're close to powerful capabilities... in the next 2-3 years" * Google DeepMind's Demis Hassabis: Changed from "as soon as 10 years" in autumn to "probably three to five years away" by January. What explains the shift? Is it just hype? Or could we really have Artificial General Intelligence (AGI)[1] by 2028? In this article, I look at what's driven recent progress, estimate how far those drivers can continue, and explain why they're likely to continue for at least four more years. In particular, while in 2024 progress in LLM chatbots seemed to slow, a new approach started to work: teaching the models to reason using reinforcement learning. In just a year, this let them surpass human PhDs at answering difficult scientific reasoning questions, and achieve expert-level performance on one-hour coding tasks. We don't know how capable AGI will become, but extrapolating the recent rate of progress suggests that, by 2028, we could reach AI models with beyond-human reasoning abilities, expert-level knowledge in every domain, and that can autonomously complete multi-week projects, and progress would likely continue from there.  On this set of software engineering & computer use tasks, in 2020 AI was only able to do tasks that would typically take a human expert a couple of seconds. By 2024, that had risen to almost an hour. If the trend continues, by 2028 it'll reach several weeks.  No longer mere chatbots, these 'agent' models might soon satisfy many people's definitions of AGI — roughly, AI systems that match human performance at most knowledge work (see definition in footnote). This means that, while the compa
 ·  · 4m read
 · 
SUMMARY:  ALLFED is launching an emergency appeal on the EA Forum due to a serious funding shortfall. Without new support, ALLFED will be forced to cut half our budget in the coming months, drastically reducing our capacity to help build global food system resilience for catastrophic scenarios like nuclear winter, a severe pandemic, or infrastructure breakdown. ALLFED is seeking $800,000 over the course of 2025 to sustain its team, continue policy-relevant research, and move forward with pilot projects that could save lives in a catastrophe. As funding priorities shift toward AI safety, we believe resilient food solutions remain a highly cost-effective way to protect the future. If you’re able to support or share this appeal, please visit allfed.info/donate. Donate to ALLFED FULL ARTICLE: I (David Denkenberger) am writing alongside two of my team-mates, as ALLFED’s co-founder, to ask for your support. This is the first time in Alliance to Feed the Earth in Disaster’s (ALLFED’s) 8 year existence that we have reached out on the EA Forum with a direct funding appeal outside of Marginal Funding Week/our annual updates. I am doing so because ALLFED’s funding situation is serious, and because so much of ALLFED’s progress to date has been made possible through the support, feedback, and collaboration of the EA community.  Read our funding appeal At ALLFED, we are deeply grateful to all our supporters, including the Survival and Flourishing Fund, which has provided the majority of our funding for years. At the end of 2024, we learned we would be receiving far less support than expected due to a shift in SFF’s strategic priorities toward AI safety. Without additional funding, ALLFED will need to shrink. I believe the marginal cost effectiveness for improving the future and saving lives of resilience is competitive with AI Safety, even if timelines are short, because of potential AI-induced catastrophes. That is why we are asking people to donate to this emergency appeal
 ·  · 1m read
 · 
We’ve written a new report on the threat of AI-enabled coups.  I think this is a very serious risk – comparable in importance to AI takeover but much more neglected.  In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here’s a very basic threat model for AI takeover: 1. Humanity develops superhuman AI 2. Superhuman AI is misaligned and power-seeking 3. Superhuman AI seizes power for itself And now here’s a closely analogous threat model for AI-enabled coups: 1. Humanity develops superhuman AI 2. Superhuman AI is controlled by a small group 3. Superhuman AI seizes power for the small group While the report focuses on the risk that someone seizes power over a country, I think that similar dynamics could allow someone to take over the world. In fact, if someone wanted to take over the world, their best strategy might well be to first stage an AI-enabled coup in the United States (or whichever country leads on superhuman AI), and then go from there to world domination. A single person taking over the world would be really bad. I’ve previously argued that it might even be worse than AI takeover. [1] The concrete threat models for AI-enabled coups that we discuss largely translate like-for-like over to the risk of AI takeover.[2] Similarly, there’s a lot of overlap in the mitigations that help with AI-enabled coups and AI takeover risk — e.g. alignment audits to ensure no human has made AI secretly loyal to them, transparency about AI capabilities, monitoring AI activities for suspicious behaviour, and infosecurity to prevent insiders from tampering with training.  If the world won't slow down AI development based on AI takeover risk (e.g. because there’s isn’t strong evidence for misalignment), then advocating for a slow down based on the risk of AI-enabled coups might be more convincing and achieve many of the same goals.  I really want to encourage readers — especially those at labs or governments — to do something