Maybe this idea is already known to AI governance people, but I think that improving the security of model weights could also actually incentivize racing to AGI. So increasing the security of model weights is not a totally safe win with no plausible downsides. I’m not claiming this is likely or that increasing the security of weights is net negative for AI safety, but just that this is one plausible effect of increasing model weight security. 

Why might increasing weight security incentivize racing? If the model weights of the winner of the AGI race can easily be stolen, the winner won’t actually have a lead for long, as the loser will just steal the model weights. This makes winning less valuable for the winner, in terms of its effect on geopolitical competition with rivals, because they won’t stay the sole state with AGI for very long. But it also makes losing less bad for the loser, since they won’t be facing a situation where their rival has AGI and they do not for very long. So increasing the security of model weights makes winning the AGI race more valuable for the state with increased security and losing the AGI race worse for any rivals of that state. Making winning better or losing worse for a state increases the incentive that state has to win the AGI race.