Note: I'm crossposting from Miles Brundage's Substack with the author's permission. The author may not see or respond to comments on this post. I'm posting this because I thought it was helpful and relevant, and I don't necessarily agree with any specific points made.

I've not included the full post, so I recommend clicking the link above to read the full post.

The hardest part of sorting out my post-OpenAI plans – as is often the case – has been saying deciding what not to focus on.

Below are some things I’ve decided not to focus on in my own projects this year (though I may advise others on them here and there). They’re all super important, but I think my skills/knowledge etc. are better suited to working on other problems. This is far from an exhaustive list of important AI-related topics to work on, of course, but maybe one of them is the right fit for you?

Helping people “feel the AGI” more

As discussed here (and in my writing generally), I don’t think we have much time left to sort out AI safety, security, and policy. The biggest bottleneck on getting these issues solved might turn out to be the extent to which people viscerally appreciate the gravity and urgency of the situation. Judging from conversations I regularly have with people in civil society, journalism, and the deep state, I think we’re still pretty far from most people appreciating the situation we’re in. Even some people at frontier companies seem to think they’re just building cool products and not something more analogous to (though not exactly the same as) electricity, or even a new, smarter species.

There are potentially many ways to push on this. Non-exhaustively, this could include clear writing that takes the pace of progress seriously; creating and giving demos of increasingly powerful and dangerous capabilities; coming up with better ways of visualizing, interpreting, and extrapolating AI progress; producing science fiction portrayals of AI that reflect significant expert input; producing documentaries; and probably various other things.

Technical infrastructure, social norms, and legal clarity for agents

I think some companies overstate the importance of AI agents, since you don’t need AIs to be particularly agent-y in order to have huge impacts. Even without a big push on agents, things will get crazy this year “just” due to much smarter chatbots, so calling it the year of agents feels off to me. But at the same time, agents are indeed coming and we don’t have the technical systems, societal norms, or legal clarity required for governing them.

What are the AI agent equivalents of stop lights, railroad tracks, etc. – infrastructure that we need in order to keep a powerful new technology “on the rails” while reaping its benefits?

Chan et al. use the term agent infrastructure to refer to “technical systems and shared protocols external to agents that are designed to mediate and influence their interactions with and impacts on their environments.” (from this recent paper). We need to sort that out quickly. One area that I’d particularly flag as essential is personhood credentials, which will be important both for distinguishing between humans and agents without violating privacy, as well as delegating to agents when appropriate. But there are many other things that need to be built.

We also need to be talking more about what exactly the role of “humans in the loop” should be in different economic and social contexts, and who is legally liable for major incidents. There are ideas here and there and there, and there was very briefly a big discussion about some of the liability stuff during the debate over SB 1047, but then it died down… we never solved the issue, though. Many US states are trying to propose their own path forward, which is creating a mess (which could have a silver lining of accelerating federal action, as long as that action doesn’t block reasonable risk mitigations).

More people should be making sure this all lands well, both in the short-term and the long-term.

The art of the EU AI Act deal

I’ve so far written two posts on the EU AI Act, one with Dean Ball and one on my own. I’m still planning to engage on this topic to some extent, and in particular I may write another post when the third draft comes out soon, but I won’t be giving it my full attention. More people should, though.

Regardless of whether you take more of a glass half-full or glass half-empty perspective on the Act generally or the Code of Practice specifically, it’s pretty high stakes. Simplifying a bit, there are three possible outcomes: the Act gets killed or gutted by a coalition of US tech companies and the Trump administration putting pressure on the EU; it gets implemented as-is with a Code of Practice that emerges from the normal negotiation process, and — either out of genuine concerns about compliance costs or to make the US administration happy — one or more AI companies pull out of the EU; or some secret third thing. More people should be thinking about the secret third thing.

This might look like a grand bargain between the US and EU that allows the US to claim a win (avoiding the worst excesses of the Act from going into force), while avoiding the EU having wasted several years. For example, there could be an amendment that repeals one or more portions that are considered especially objectionable to US companies (e.g., on copyright) or there could be some sort of explicit interoperability introduced between the EU side and the US side (e.g., allowing compliance with some US-backed private standard to serve as a substitute for Code of Practice compliance).

These are just rough ideas for what a deal could look like. But the point is that, at least unless and until the US is on top of things, we should be very concerned about the most serious effort at AI governance either being totally abandoned without a replacement, as well as such efforts falling short of their potential to actually raise the bar on safety while being efficient to implement. Contrary to some people’s beliefs, companies do not in fact have sufficient incentives to mitigate all major risks, and competition is driving corner-cutting that needs to be reined in somehow.

AI literacy

As I have discussed before, I don’t think AI companies have done a good job of explaining how their technology works. Misconceptions abound, and both systematic over-use and systematic under-use are common as a result. That’s pretty concerning from a “wanting people to make good decisions” perspective and also a liability for companies and policymakers as the technology starts to cause serious accidents, and then people look around and realize the companies on the forefront didn’t really try particularly hard to avoid such outcomes.

There have been some piecemeal efforts by non-profits, governments, and companies, but the growing literature on the topic shows that even very educated people routinely fall for hallucinated AI outputs, so something isn’t getting through. Onboarding instructions remain non-existent or minimal, with no companies (to my knowledge) having provided any empirical evidence that their user interface design improves outcomes in terms of appropriate reliance. And there is starting to be evidence that overuse may atrophy users’ critical thinking skills. Not ideal!

This is notably different from feeling the AGI, though not unrelated. One can look at the trendlines and conclude “hmm, seems exciting/concerning” without having much of an explicit or intuitive understanding of how to actually use AI effectively day to day, and vice versa. Ideally we’d be strong on both fronts, though.