Executive Summary

  • The field that studies existential and global catastrophic risks (X/GCRs) is gradually increasing engagement with policy communities. To best allocate and prioritise these efforts, the field should have a sufficient understanding of key areas of public policy.
  • We’ve produced a longlist of questions where the field’s understanding may need to improve. We call these ‘meta policy research questions’.
    • Some of these questions might relate to policy in general, for example: ‘What are the steps in a policy process?’
    • Some of the questions might relate to risk policy, for example: ‘How is policy made under uncertainty and risk?’
    • Some questions might relate to specific X/GCRs, for example: ‘how can policy windows be useful to the X/GCR field in engaging policy?’
  • This isn’t the full set of relevant meta policy questions to investigate, but we’d expect this covers a majority of valuable questions.
  • We would like additional suggestions on questions that may be valuable to research. Please provide feedback and suggestions through comments directly in the longlist at the end of this document
  • After this, our next step will be to shortlist the priority questions - where more work is needed and will add the most value to the field’s policy engagement. Many questions may be well-answered by existing research. Others may be impossible to answer. But an effort to catalogue and evaluate the questions may reveal promising areas of further research.
  • The list was generated by the authors’ brainstorming, consultation with others in the field, and a light literature scan. Brandon Perry and Risto Uuk’s article on AI Governance and the Policymaking Process was particularly useful in this respect.
  • We believe that the final shortlist of questions will be useful for researchers and new entrants to identify valuable areas of research. Ultimately, the investigation of the questions will be useful for those who start, agree, fund, host or deliver policy engagement projects.

In case you want to get in touch with the authors to discuss our work, feel free to contact rs2068@cam.ac.uk or gabriella.overodder@gmail.com




Concrete next step: 

Produce a long list of meta-policy research themes that will be important in engaging on policy. From this, we will draw out the most promising questions to research.

** We hope to receive suggestions for additional questions we may have missed, and feedback on the questions included in this list. **

Broader goal: 

We hope to gain a meaningful understanding of what influences the success in each step of the policy process, to inform how the X/GCR field can be more successful in engaging policy.  


Definition of ‘meta policy research’

Meta policy research encompasses research designed to improve the field’s understanding of the why, how, who, when, where, and what of policy, and ultimately the policy around existential and global catastrophic risks. It relates to the factors that influence the policy process and policymakers, and the factors that make policies, and policy engagement, more and less successful. It also relates to the scale, type, and focus of the field’s current and potential policy-related efforts. These questions could be further broken down by jurisdiction, institution and policy issue.

Note the reference to ‘the field’ signifies the X/GCR research- and policy-field. 


Value of meta policy research

Meta policy research can provide the field with valuable insights into the factors that influence success, ultimately providing valuable information in navigating what policies to prioritize, as well as when, where, and how to pursue policies. Meta policy research does not seek to identify specific policy recommendations - rather it aims to support the field’s engagement with policy.


Customers of meta policy research

Some of the groups listed will be customers of the shortlist produced, others of the answers produced from answers to the questions.  

  • Researchers: Highlight gaps and potential areas of focus
  • Future/new entrants to the field: Prompt for new research proposals / efforts
  • Head of research organisations: Help prioritise policy research efforts
  • Funders: Understand where policy work is needed; resolve areas of concern or risk
  • Policy interface/advocacy: Knowing what research exists and understanding gaps
  • Policy-makers: Knowing what research exists and understanding gaps


Longlist of Meta Policy Questions:

General Policy

  • Policy types: What are the types of policy? How do they differ?
  • Politics and power: How do politics and power dynamics shape policy-making? How may this guide efforts to influence policy?
  • Policy influence: How does policy influence happen? How do outsiders shape policy? How do insiders shape policy? What strategies can be used to improve the chances of a preferred policy to be adopted? What coalitions should be built to effectively increase chances of success?
  • Policy cycle and process: How is policy functionally made? What are the steps in a policy process? What are the timeframes of these steps? ​​What frameworks and methods are used in the policy cycle? What are the pressures and limitations of policy-making (e,g, political, financial, bureaucratic, time)
  • Policy windows: What are policy windows (focusing events, key moments and periods of outsized opportunity) for influencing policymaking, in different institutions? How may they guide efforts to influence policy? How can policy makers make use of policy windows to further policy agendas? What needs to be prepared such that policy makers can make full use of po licy windows?
  • Policy agenda: How are policy agendas formed? How does this differ between actors and times? What influences policy issues being placed higher on the agenda? What framing places them higher on the agenda? How has agenda-setting been successfully influenced in the adoption of policy?
  • Decision-making: How does policy decision-making occur? Who is involved in shaping and making decisions? How may this guide efforts to influence policy?
  • Policy stakeholders: Who are the stakeholders of policy? How do they engage in the policy cycle? What are the mechanisms to engage and influence stakeholders?q
  • Science-policy interface: How should researchers engage with policy? How does science interface with policy? What does researchers’ policy engagement look like? What makes such engagements successful? What are pitfalls to be avoided and how can they be avoided?
  • Policy implementation: How is policy implemented? What determines the success of policy implementation? What factors/actions increase the chances of policies being well implemented?
  • Policy evaluation: How is policy evaluated? How can the effect of a policy related to the risks best be evaluated? What factors/actions increase the chances of policies being well evaluated? How can unintended consequences of policies be evaluated?
  • Policy institutions: What are the types of policy institutions (e.g. legislative bodies, governance levels, government agencies)?
  • Policy debate: How does policy debate occur? What is the policy debate impact on legislation/legalese/legal texts? How does it work?
  • Policy enforcement: How is policy enforced? What is the enforcement mechanisms' impact on behavior of governed entities? What are loopholes and how are they prevented? How does the enforcement actually affect the behaviour?


General Risk Policy

All the general policy questions above can be applied specifically for risk policy. The following questions could also be added:

  • Decision-making under uncertainty: How should policy be made under uncertainty and risk? How is policy made under uncertainty and risk in practice? What are best practices where policy has been made under uncertainty and risk? What are examples of poor practices? What are pitfalls to be avoided? How can they be avoided?
  • Risk policy frameworks: What are international and national risk policies and frameworks? How can these support risk policy efforts? If they have not achieved their intended effect, what are reasons why these have not been effective in reducing risks?


General X/GCR Policy

All the general policy and general risk policy questions above can be applied specifically for X/GCR policy. The following questions could also be added:

  • General vs specific: What is the value of policy engagement at the general X/GCR level vs specific level?
  • Public perception: How are the risks understood by the public? How are the risks understood by policymakers? How would we like these groups to understand and relate to the risks? How do we achieve this? What is the perspective/framing of the risks that make them most comprehensible for policymakers? How does attention to the risk, or subsections of the risk, by different communities affect policymakers’ actions?
  • Gaps and opportunities: Where are current gaps in the supply and demand of policies related to the risks? How can we best fill this gap?
  • Policy commitments: What are key documents for policy makers’ commitments? How effective are efforts to make commitments to X/GCR risks, or individual risks specifically for later policy engagement successes? How can this be done well?
  • Future leaders: Who are future policy makers? How can they best be introduced to X/GCRs?
  • Framing: What types of framing of policy issues are most beneficial? What types are most dangerous? What case studies can be learned from? How does the framing of the risk by other actors shape policymakers’ understanding of and attitude towards it? If these are negatively impacting the assessed ideal framing of the risk, what can be done to respond to this? What kind of responses to objections are most convincing?
  • Messaging: Who are the best ‘messengers’ for highlighting X/GCRs to policymakers? How can they best be supported?


Specific Policy Aspects of X/GCRs

Policy by specific risks

All questions above can be applied to individual risks. In addition, the following questions could be asked:

  • Policy engagement: What is the field’s level of policy engagement with the risk? What have been successes from policy engagement work so far? What can be learned from this to inform future work? What have been negative effects of policy engagement work so far? What can be learned from this to inform future work? How should the field’s distribution of resources between policy work on the risks change?
  • Policy ideas: What policy ideas has the field, or adjacent fields, put forward? Where are further ideas needed?
  • Policy evaluation: What are the methods and considerations to evaluating them? What are the political considerations surrounding different sets of policies, and how does that affect their ability to be implemented? Which policies are politically feasible? What policy alternatives are more likely to be adopted that improve the odds of desired risk reduction? Which policy has the highest expected value, from its direct expected impact and indirect impact on later policy engagements? Are there enough resources, will, and support for the policy for it to be successfully implemented? Are those affected by the policy expected to update behaviour as intended by the policy? Are the goals of the policy in conflict with any other policy?
  • Case studies: Which policy ideas have been implemented? Where?
    • Outcomes: Has the policy resulted in the desired outcomes? If not, why not? What can we learn from this for future policy engagement? Has the policy caused any unintentional effects? How did this happen? How can it be mitigated?
    • Learning: How was the outcome of the policy communicated amongst policy makers? Amongst those affected by it? Which metrics were used to communicate its effect? Is the overall effect of the policy expected to be net positive? What were the political costs and/or benefits of the policy?
    • Adaptation: What changes could be made to the policy? How likely is it that changes will be made, good or bad? How can we prevent undesirable changes to the policy?
    • Opportunity Cost: Given the overall effect of the policy, how cost effective was it? Should these resources have been invested elsewhere?
  • Interconnection: How do risks and risk-policy related to one risk influence other risks? Are there examples when policy work on one risk has benefited other risks? Are the examples when policy work on one risk has had negative effects on other risks? How can these insights inform how future policy engagement is analyzed and pursued?


Policy by governance level

  • Levels of governance: What are the different governance levels to govern X/GCRs (e.g. international, national, local)? What are the benefits and drawbacks of each?
  • Types of international governance: What are the different types of international governance arrangements? Do some work better than others? How? Why? How can this be used in defining strategy and approach? What are the various actions they undertake? What level of governance should the field focus its policy engagement on?
  • Establishing international governance: How are new international arrangements successfully established? How are existing ones influenced successfully?
  • Impact of national governance: Which national governments are most impactful on X/GCRs? How are the field’s efforts allocated across these national governments?
  • Institutions: In what institutions are policies related to the risks expected to be most effective? How does this differ between the risks?

Policy by cross-cutting issues

  • Cross-cutting issues: What are the cross-cutting policy issues that impact multiple risks?
    • Geopolitical: What are the geopolitical aspects to X/GCRs? How do X/GCRs impact the geopolitical system?
    • Economic: What are the economic aspects to X/GCRs? How do X/GCRs impact the economic system?
    • Food and water: What are the food and water aspects to X/GCRs? How do X/GCRs impact food and water systems?
    • Energy: What are the energy aspects to X/GCRs? How do X/GCRs impact energy systems?
    • Infrastructure: What are the infrastructure aspects to X/GCRs? How do X/GCRs impact infrastructure?
    • Knowledge: What are the knowledge and epistemological aspects to X/GCRs? How do X/GCRs impact knowledge systems?
    • Governance: What are the governance aspects to X/GCRs? How do X/GCRs impact governance systems?
    • Health: What are the health aspects to X/GCRs? How do X/GCRs impact to health systems?
  • Effort: What effort in the field is on cross-cutting issues? Which, if any, should the field focus on?

Policy by risk policy action 

  • Risk actions types: What are the types of risk policy action - e.g. govern, understand, prevent, prepare, respond, recover, collaborate, communicate
  • Effort: Of the risk policy actions, where is the field’s current effort? How important is each step for reducing X/GCRs? Where is more effort needed? What is most cost effective? What should we prioritise?


11 comments, sorted by Highlighting new comments since Today at 4:41 AM
New Comment

I think this is a good idea, but would benefit greatly from narrowing the scope greatly, and finding what answer are already known before brainstorming what to investigate. Given that, I think you'd benefit from some of the basic works on policy analysis, rather than policy engagement, to see what is already understood. I'll specifically point to Bardach's A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving as a good place to start, followed by Weimar and Vining's book.

Thanks David! Appreciate you having a look and for the resources. 

Definitely agree, the scope will end up being much narrower. We wanted to keep this initial stage really broad - hoping to capture as many interesting and useful questions as possible. Then next step we're going to whittle it down to the highest priority questions, essentially those that would be valuable for the field to have insights into but that haven't yet been addressed by existing literature or work. Hope to get your thoughts at that stage as well!

Sounds great - and my guess is that lots of the most valuable work will be in "how can we use technique X for EA" for a variety of specific tools, rather than developing new methods, and will require deep dives into specifics.

Thanks for this. I'll fire off a bunch of quickly written quick reactions in this comment and in replies. Let me know if you'd like me to elaborate on anything.

  • Some of these questions might relate to policy in general, for example: ‘What are the steps in a policy process?’
  • Some of the questions might relate to risk policy, for example: ‘How is policy made under uncertainty and risk?’
  • Some questions might relate to specific X/GCRs, for example: ‘how can policy windows be useful to the X/GCR field in engaging policy?’

I think that third point conflates two separate things: questions related to specific x-risks/GCRs (e.g., how can policy windows be useful to the AI risk field in engaging policy?) and questions related to x-risk/GCR policy in general (but more narrowly than risk policy in general - risk policy would also include policies solely relevant to smaller scale risks). So I'd suggest a four part breakdown, like a Venn diagram, narrowing in from policy in general, to risk policy, to x-risk/GCR policy in general, to policy on specific x-risks/GCRs.

I'd also suggest adding a bucket for policy that's especially relevant/similar to x-risks/GCR policy even though it's not explicitly about x-risks/GCRs or even about risk (or where the primary reason it's relevant isn't because it's about risk). I think you've correctly identified that a notable aspect of x-risk/GCR policymaking is that it must contend with substantial risk and uncertainty, and thus that we can learn useful lessons from other policymaking efforts that must contend with substantial risk and uncertainty, even if they aren't explicitly or closely related to x-risks/GCRs. But I think other notable aspects, which likewise allow for learning from other policy areas, include:

  • Huge externalities within and between borders and generations
    • GCR reduction is in most cases like a global public good, and x-risk reduction is a transgenerational global public good
    • So we can learn from e.g. climate change policymaking for reasons unrelated to climate change itself being a GCR and involving risk and uncertainty
    • Presumably we can also learn from things like policymaking to prevent ozone depletion
  • Key role of emerging technologies
  • Key role of great power relations and perhaps "race" dynamics (including but not limited to arms races)

I guess this also suggests the high-level question "What is distinctive and challenging about x-risk/GCR policymaking, and what other areas can we therefore learn from?" (Maybe you already covered this and I missed it.) One previous research effort along those lines (but focused on AI specifically) which I found interesting was this from MIRI in 2013.

I might add something about "What are the actions funders can take to influence policy outcomes? How well does each tend to work? In what situations are they most appropriate? How can they best be implemented?"

Example actions include funding advocacy campaigns, funding research, funding track 1.5/track 2 dialogues, and funding capacity building stuff (like fellowships that create the next batch of experts). 

It might be that some actions tend to be more effective overall, some are more effective for particular risks, and some are more effective during certain types of policy windows while others are the best move when no particular window is open.

Messaging: Who are the best ‘messengers’ for highlighting X/GCRs to policymakers? How can they best be supported?

I think a related idea would be "What are the best ways of assessing what policy proposals will be tractable and what framings are best? How good are they? " E.g., how much should we invest in polling, message testing, talking to experts/grantmakers/campaigners/etc., or Tetlock-style forecasting to assess what policies might get public and policymaker support and what framings might best support them? 

(We could also of course ask "What policy proposals will be tractable and what framings are best?", and use those methods to answer it, but then that's not meta policy research.)

I'd probably add some things related to forecasting and maybe foresight, scenario planning, horizon scanning, and maybe red-teaming (of ideas)

I think you could this as similar to how you highlight the science-policy interface (likewise, forecasting should in theory be an important input into policymaking) or similar to how you and I respectively highlight risk+uncertainty and externalities+emerging tech (one distinctive thing about x-risk/GCR policy is how relevant forecasting etc is). 

A related thing is "trying to act well in advance of a problem occurring or even its shape or importance being clear to most people". One case study often mentioned in this context is Leo Szilard and nuclear weapons (e.g. here). See also https://forum.effectivealtruism.org/tag/long-range-forecasting One could in theory look into how often people have attempted to influence policy in that sort of way or on that sort of issue, when and how it's gone well or poorly, etc.

You could check out this draft research agenda of mine for additional question ideas. (I now you've already seen it, but perhaps it's worth glancing at it again in light of this particular project, and I'm also putting the link here for other people's potential benefit.)

Thanks Michael, all great points and  really useful additions. I've added those in.  Your draft research agenda was definitely inspiration for this work, though I realise I hadn't looked at it in a while, so thanks for re-sharing. It also shows that each meta-policy question can be broken down into all sorts of mini-meta policy questions. I'll be keen to speak with you about how you've approached prioritising across them all.

Glad it was helpful!

I'll be keen to speak with you about how you've approached prioritising across them all.

Unfortunately the summary on that is that I haven't really done any further work developing the agenda, prioritising across its questions, or actually working on the questions. (Where "further work" means "since initially writing and sharing the agenda.) 

That said:

Thanks for starting this discussion! I have essentially the same comment as David,  just a different body of literature: policy process studies. 

We reviewed the field in the context of our Computational Policy Process Studies paper (section 1.1). From that, I recommend Paul Cairney's work, e.g. Understanding public policy (2019), and Weible & Sabatier’s Theories of the Policy Process (2018).

Section 4 of the Computational Process Studies paper contains research directions we think are promising and can be investigated with other methods, too. The paper was accepted by Complexity and is currently undergoing revisions - the reviewers liked our summary and thrust, just the maths is too basic for the audience, so we’re expanding the model. Section 1 of our Long-term Institutional Fit working paper (update in the works, too) also ends with concrete questions we’d like answered.