Pitch:
- Publishing dual use papers is bad.
- It’s also easy, people can upload it to arxiv/bioRxiv/medRxiv, even by accident (not knowing they are publishing something potentially harmful).
- It might be useful for these platforms to reject such publications, and the platforms seem interested.
- I’m guessing they’re lacking resources: People that will vet papers, maybe a software system, maybe money.
- Let’s talk to the platforms, ask what they need, and give it to them.
Why I think the platforms are interested in doing this
A founder from bioRxiv and medRxiv, Richard Sever, says about this screening:
- "This is desirable and in fact already happens to an extent"
- "arXiv and bioRxiv/medRxiv already communicate regularly"
I can provide the reference for this.
Request for vetting
My experience in biosecurity is about 3 hours.
Please, people doing biosecurity, reply with your opinion, even if it is very short like “sounds good” or “sounds bad”.
Looking for project lead
Do you know someone who could run this? Comment on the post (or DM me, and I’ll pass it on somewhere).
Before starting this project, please review the ways it could go wrong
As a naive example, just to be concrete: Someone gets mad that their (dangerous) article wasn’t accepted, so they publish it on Twitter and it goes viral.
But more generally, before starting this project, please talk to the people who reply “sounds bad”.
It was just Petrov Day, ”Wherever you are, whatever you're doing, take a minute to not destroy the world”.
Trying to collect some comments that seem relevant:
Daniel Greene said:
ASB seems to agree:
Tessa said:
Seems promising and I would support these efforts. One danger to watch for and try to address would be attention info-hazards e.g.:
But these risks can be managed, and are outweighed by the project's benefits in my view.
I think this project seems valuable for three main reasons:
Additionally, it is clear that servers like arXiv.org are highly resource-constrained (see https://www.scientificamerican.com/article/arxiv-org-reaches-a-milestone-and-a-reckoning/).
With that said, striking the right balance between openness and security is difficult, and this project might generate significant backlash or have unintended consequences. I wrote up some additional thoughts about this project after discussing it with Yonatan, so please reach out to me if you would like to read more about what this project might look like and why it might or might not be a good idea.
Arguments for
Arguments against
For further investigation
Implementation
How would vetters, whether a regulatory agency or an independent initiative, screen papers? In the case of DNA synthesis, which does not account for all biosecurity relevant dual use research, a minimalistic approach is for vetters to utilise an encrypted database of riskier sequences of DNA , as proposed by MIT's Prof Kevin Esvelt.
However, dual use control at the publisher level would presumably not be restricted to DNA synthesis, it would include such things as studies of remote areas at the human-animal boundary.
Next steps
Esvelt is in dialogue with the Nuclear Threat Initiative who are coordinating higher level conversation in this area. If the publishers you mentioned aren't already part of that dialogue, the best next steps may be to connect Nuclear Threat Initiative folks with those academic publishers. But, I don't think that should mean this initiative shouldn't proceed in parallel. I think there is merit in taking some action now in this space because the conversation that the Nuclear Threat Initiative and co are kindling is a slow, multilateral process - screening DNA synthesis orders is not legally required by any national government at this stage.
Time sensitivity
The cost of DNA synthesis is declining and the fixed costs of filtering could grow as fraction of the cost, therefore the viability of a voluntary screening model could its highest right now.
Participation
I'm interested in being involved, but don't know that much about academic publishing or technical genomics stuff so probably not a fit to be a (solo) project lead. Do know about management, health policy, public administration, stakeholder engagement, communications etc
+1 to contacting Nuclear Threat Initiative, they seem to be active and well connected across many relevant areas.
I definitely don't think "sounds bad" (I really, really would like it to be easier for publishers to adopt dual-use screening best practices) but I do think "sounds partly duplicative of other work" (there are other groups looking into what publishers need / want, seems good to collaborate with them and use their prior work) and "should be done thoughtfully" (for example, should be done with someone who has a good appreciation for the fact that, right now, there does not exist a set of "dual use best practices" that an organization could simply adopt).
I'm going to gesture towards some related initiatives that might be of interest, from some folks who have already undertaken (at least some of the) "Let’s talk to the platforms, ask what they need, and give it to them" step:
Anyway, I feel like one way in which this project could go wrong is viewing itself as trying to lock in a new standard, rather than running an experiment in biosecurity governance that is part of the project of Consensus-finding on risks and benefits of research.
This project would be valuable if the costs outweighed the benefits.
It could be relatively expensive (in person-hours) to run (there might be a tonne of publications to vet!) and relies on us being good (low false positive, high recall) at identifying biohazards (my prior is that this is actually pretty hard and those biohazardous publications would happen anyway). We'd also need to worry about incentivising people to make it harder to tell that their work is dangerous.
Biohazards are bad but preventing biohazards might have low marginal returns when some already exist. It's not that any new biohazard is fine; it's that marginal biohazard might be pretty rare (like something that advances the possibilities) relative to "acceptable" sort of non-marginal biohazards (i.e., another bad genome for something as bad as what's already public knowledge). Other work might advance what's possible without being a biohazard persay (i.e., AlphaFold).
I think a way to verify if this is a good project might be to talk to the Spiez lab. They run a biosecurity conference every year and invite anyone doing work that could be dangerous to attend.
I'm happy to chat more about it.
Regarding the problem being expensive (because there are many publications to vet or something like that):
I think I could help optimize that part, it sounds like an engineering problem, which is a thing I'm ok with
I'd like to see a more comprehensive model for what biosecurity risk looks like that can motivate a comparison of project ideas. In the absence of that, it's really hard to say where we get the most benefit.
We don't have to drop all our plans in favor of the one top plan
Or as Dumbledore said:
Sorry I'm late to the party- as per the OP's request for short takes with no explanation, mine is that this is probably not worth doing, fwiw.