Really valuable work - thank you. Looking forward to the Animal Welfare Alignment Newsletter!

I wonder how likely Canada is to join the US (whether through voluntary accession or forced annexation, or some other means), during an intelligence explosion. Very rough thoughts:

- Normal political base rates make this seem very low-probability. But the world where this is seriously entertained would very likely be far from "normal"
- If the US gains a permanent lead (not necessarily even a full DSA that could disable nuclear second-strike capability) over other AI-relevant countries, Canada might become much more strategically valuable because of land, water, energy, minerals, easier cooling for data centres, etc.
- The US might want Canada inside its security perimeter for many other reasons too.
- Canada would probably have very weak bargaining power if the US is far in the lead.
- But the US could probably get most benefits through softer, less extreme forms of integration?

So I’d guess: annexation very unlikely, accession unlikely but possible, de facto integration much more likely. I'm trying to think through what this would imply for whether Canadian citizenship would matter (e.g. if somehow there's windfall clause related reason or similar benefits/opportunities).

Edit: clearly AI-2040 answered the question of what would happen to Canada! (joke)

Does anyone have good resources on where value might accrue in worlds where AI can do most (or all) cognitive labour (this Q also applies to physical labour, but is harder)?

Context is I’m interested in learning more about the question of whether “all/most value accrues to AI companies”. My current guess is that value probably just accrues to whatever remains hard to replicate, not necessarily to the model layer.

If you follow this line of reasoning, here are some  candidates for where value might end up:

• Distribution and customer trust (I'd assume there's a decent chance users stick with institutions they already trust, especially in high-stakes domains?)
• Proprietary workflows/data (i.e not just static datasets, but also embedded org context, feedback loops, tacit knowledge etc)
• Regulatory permissions (e.g. medicine, law, finance, government procurement stuff)
• Physical-world access (robots, factories, human relationships, and so on)
• Compute/energy/inference (but TBC only insofar as these remain structurally scarce, not just temporarily supply-constrained).
• Frontier model capability (only if a small number of AI companies maintain a persistent lead, or if integration costs are really high for some reason - which seems unikely due to strong commercial incentives for them to be easy)

(to be clear I think a world where value concentrates in frontier AI companies would be a worse world compared to one where it is more diffuse, due to power concentration concerns + generally more dynamism and resilience in that world).

Thanks for this! Firstly these seem like two separate Qs (“which actors am I most worried about?” vs the specific risks posed by defender-first access programmes).

My current guess (as a relatively uninformed outsider) is that the most worrying near-term category actors with some combination of pre-existing domain knowledge, operational access, and (obviously) intent. So in bio I guess this would mean: state/state-adjacent programmes, small highly motivated teams with wet-lab access, and maybe compromised actors inside institutions that give them access. I definitely don't currently think frontier models are good enough that a literal random person could cause catastrophe if they wanted to (though this could change).

Re defender-first access specifically: yeah makes sense that this could add risk. E.g. if there's stuff like insider misuse, account compromise/coercion, leakage of outputs/workflows, etc.

I don't really have more concrete thoughts beyond this. It would be interesting to see some analysis (if there isn't already one internally) on how robust OAI's Rosalind Biodefense and Anthropic's Project Glasswing are to risks from risky actors getting into the programme.

(Somewhat relatedly I remember reading somewhere that Mythos reportedly got accessed by some unauthorised users via a third-party private Discord group).

Thanks :)

Great post!

Thanks for your comment.

I’m not able to follow what you mean here, could you explain your points more simply?