[Below is an unedited copy and past of some old notes I made on AI policy in 2019. This was written as a personal project as a way of testing my fit before I started working on on future generations policy advocacy. I don’t know if I ever really planned to publish it but am publishing now for Draft Amnesty Day.

It is not well explained in the text but all I was doing here was taking some "best practices" from nuclear safety policy and applying them to AI safety. Which was an interesting exercise but does not quite reach the standard I would expect policy research.

I also think most of what is here (insofar as it can be applied at all) could be applied to biotech safety.

I no longer agree with a all of what is written here, and leave a few comments where this is the case. All comments and additions are in italics and square brackets. I also redact some links to other draft documents.]

A policy professional’s perspective on preventing TAI risks

Artificial intelligence is the future, not only for Russia, but for all humankind.
It comes with colossal opportunities, but also threats that are difficult to predict.

– Russian President Vladimir Putin

Introduction

Scope and audience of this paper

Audience.

This draft of this paper is written for individuals who are already concerned about the catastrophic, existential and extreme suffering risks associated with the development of Transformative AI (TAI). The question of how serious these risks are is not discussed here.

Aim.

This paper makes the case that there are concrete policies that can be implemented now by governments that would help mitigate TAI risks, and to set out a number of such policies. This could be used if opportunities arise to engage sympathetic policy makers on AI risks.

Scope.

This paper is focused only domestic AI safety policy, those that could be implemented by a single country and do not require international agreement. The policies come from examining what can be learned about AI safety policy from best practice in civil nuclear safety policy in the UK context. The policy suggestions are therefore within a UK policy context, although are hopefully general enough to be applicable in other countries as well.

Key recommendations for government

• Support more long term thinking in policy and politics through improvements to culture, processes, and accountability mechanisms. There are numerous policies that could do this, they are mapped out here [Link redacted. Maybe see here instead].
• Improve the general processes for identifying mitigating and planning for future risks. Assessments of catastrophic risks should go out at least 20 years and pick up on emerging risks. Responsibility should be assigned to consider what mitigation, and preparation is needed for each risk. For a more detailed breakdown of suggested policies see draft CSER paper here [Redacted] (in future will be at www.gcrpolicy.com).
• Improve the ability of government to draw on technical and scientific expertise. For example, improve hiring processes and retention for staff with a technical background in government, facilitate secondments between government and academia, etc. Some of the mechanism the UK government has used to bring in scientific expertise are set out here [Redacted].
• Civil Servants should be researching policy issues around ethics and technology and AI. This could be facilitated through the development of an internal government think tank with a mix of staff from policy and technical backgrounds and the mandate to and capacity to look at high level questions connected to tech, ethics and AI, including TAI. (Similar to CDEI or NIRO)
• [2022: Very cautious and unsure about this now] Fund external AI safety and ethics research by academia or in partnership with industry. For example, research to better estimate AI risks. 
• [2022: Don’t see huge need for this] Develop a system for recording and sharing data relevant to AI safety (similar to NEWS).  Work with industry and academia to generate guidelines about and mechanisms for information sharing, for example covering when information should be shared, shared through private channels or made public. 
• Set up a regulator in the form of a well-funded independent body of technical experts who are given very high-level goals to ensure the safe and ethical behaviour of the technology industry and government (similar to the ONR or HFEA).
• [2022: Very cautious and unsure about this now] Consider if the UK government can play a useful role to safeguard AI assets in the UK, for example by requiring and providing security clearances for staff at AI research organisations (like DeepMind) and ensuring that those organisations have excellent cybersecurity protection.

Further down the line, government should: 

[2022: Not that sure about any of these now]

• Recognise the extreme risk associated with TAI (including catastrophic, existential and extreme suffering risks) and assign responsibility to a relevant individual to mitigate and prepare for these risks.
• Consider and independent expert review of the risks and potential solutions. (such as the Stern review or the Independent Commission on Banking)
• Engage with AI safety researchers to put in place additional layers defence in depth and other measures to mitigate risks and support the Resilience to shocks from the potential development of TAI.

Additionally, this paper recommends that academics concerned about AI safety should be looking to build or demonstrate academic consensus regarding the scale of AI risks.

Background 

Background to AI

Transformative AI (TAI)[1] is an AI that could lead to rapid radical changes in human welfare or power. Potentially an Artificial General Intelligence (AGI) that can do tasks of any type or even an Artificial Super Intelligence (ASI) that is smarter than humans as we are to animals. This concept is borrowed from and explored in depth in Professor Allan Dafoe’s AI Governance: A Research Agenda. Experts predicts a 50% chance that high-level AI will be developed around 2040-2050[2].

TAI risks. This paper is focused on what policies could be adopted now that could mitigate some of the tail end risks associated with TAI, in particular the risk of TAI being misused by bad actors or the risk of an ASI that is not controllable or aligned with human goals. Both of these situations could be potentially existential risks for humanity.  For the purpose of this paper this topic is referred to as “AI safety policy”. 

Background to AI policy

The growth of AI has captured the attention of corporates, the media, the public and state actors. States keen to bring AI start-ups and innovative research to their shores and reap the potential economic and societal benefits. This has led many states from China to Tunisia to adopt AI policies or national AI strategies.[i] These strategies cover topics such as supporting the AI industry, developing or capturing AI skills, improving digital infrastructure, funding R&D, data use, safeguarding public data, AI use by government and AI regulation. In each case these strategies sit alongside work on related topics such as supporting the tech industry, software regulation, privacy and data use.

Within this process states have recognised the need to understand and prepare for the sociological, economic, ethical and legal implications that may stem from the widespread adoption of AI technology. There has been minimal action at the state level to address the implications of TAI.

The UK context

So far in the UK the Government has:

• Founded an Office for AI. This office will be implementing the priorities mentioned in the UK’s Industrial Strategy: building a workforce with machine learning skills, growing the AI sector and supporting businesses to use AI[ii]. The office will also act as a secretariat to an AI Council.[iii]
• Founded the Centre for Data Ethics and Innovation (CEDI). CDEI is an internal government think tank considering issues around data, tech, ai and ethics. A centre for ethics is perhaps what the government creates when it is not yet sure what regulation is needed.
• Encouraging AI adoption in specific areas of policy including defence[iv] transport[v] (with the Centre for Connected and Autonomous Vehicles) and healthcare[vi]
• Supporting AI research through the Turing Institute.

The policy context

AI policy making sit within a wider context. It is important to remember that a government’s policies on AI will be driven largely by the more general approach a government takes to business data and technology policy. Similarly, AI safety policy will sit within the context of wider AI policy.  (It is possible AI policy will sit under defence rather than business but this does not appear to be the current trend).

It is also helpful to distinguish two kids of AI safety policy a government can adopt

• Domestic policy. The policies a concerned government could unilaterally implement to ensure that the implications of AI are well manged within their country
• International policy. Any actions a state takes to ensure that AI safety best practice is adopted by other states and international institutions. For example, negotiating international treaties.

This paper focuses on domestic AI safety policy

Domestic AI safety policy

Best practice

Drawing from nuclear safety, we can identify seven key principles that policymakers should consider in order to reduce the chances of a technological disaster, the 6 ‘R’s & ‘S’:

• Recognition: acknowledge the potential risks.
• Responsibility: change the political landscape so that someone has responsibility and an incentive to care about the risk.
• Research: fund AI safety research and encourage policy professionals to engage in AI policy research.
• Record: set up a system so that useful AI safety knowledge, particularly cases of AI systems acting unexpectedly, can be shared.
• Regulate: set up a body of experts with powers to review and oversee risky research.
• Resilience planning: government resources could go into adding layers of defence
• Safeguarding: and supporting AI companies’ cybersecurity efforts.

This chapter looks at all of these except for regulation which is dealt with in the next section.

https://www.iaea.org/publications/10883/governmental-legal-and-regulatory-framework-for-safety

Recognition and Responsibility

[Update 2022: My views on this now are a bit more nuanced. I don’t think that governments need to directly or explicitly recognise AI existential risks. What I do think is that any polices introduced such as regulations need to be robust to future worlds where various extreme AI risks become a possibility. This could be done by mapping out the short term AI societal risks and the range of possible long term AI risks and mapping across and ensuring the categorisation of and ways of responding to short term risks is not likely to pose barriers in any future more extreme risk scenarios.] 

Governments could recognise the implications that future technologies and global trends might have and recognise that some emerging technologies, such as the AI, biotech or geoengineering, may pose threat of catastrophic and potentially existential scale. Such recognition should be based as much as possible on a comprehensive scientific understanding of the subjects at hand and ideally informed public debate.

Recognition of a policy area or potential risk should be followed by assigning responsibility for mitigating and preparing for the risk to a Minister, Department, or position. That responsible political figure would be accountable to the public should doubts arise as to the policy or safety regime in place and they or their Department would be the point of contact for concerned parties.

Factors that affect how likely a country is to recognise and assign responsibility for a risk include: academic consensus, recognition of the risk by other countries, clear actions that can be taken after a risk is recognised, public and pressure group concern, political ideology and political will. It is worth noting that there is currently a lack of academic and expert consensus on the risks of TAI. From a policy perspective it is not clear that it would currently be sensible for a Government to recognise TAI risks.

Moving away from nuclear safety best practice, another government

The UK has a number of Horizon scanning processes spread across departments that look into future trends. It is likely that CDEI will be looking into the future trends associated with AI.

Risks are manged by the Cabinet Office. They collate risk into the National Risk Register of Civil Emergencies[3]. The register risks that “have at least a 1 in 20,000 chance of occurring in the UK in the next five years”[vii]. These risks are then distributed to relevant departments and the such that the Secretary of State of that department is responsible for risk mitigation and emergency preparedness policy. For example, the risk of a radiological release from a nuclear site is the responsibility of the Secretary of State of the Department for Business Energy and Industrial Strategy and staff in that department ensure that satisfactory polices are in place. There is also a separate occasional National Security Risk Assessment carried out by the MoD.[4]

Recommendations

• Support more long term thinking in policy and politics through improvements to culture, processes, and accountability mechanisms. There are numerous policies that could do this, they are mapped here [Link redacted. Maybe see here instead].
• Improve the general processes for identifying mitigating and planning for future risks. Assessments of catastrophic risks should go out at least 20 years and pick up on emerging risks. Responsibility should be assigned to consider what mitigation, and preparation is needed for each risk. For a more detailed breakdown of suggested policies see draft CSER paper here [Redacted] (in future will be at www.gcrpolicy.com).
• Improve the ability of government to draw on technical and scientific expertise. For example, improve hiring processes and retention for staff with a technical background in government, facilitate secondments between government and academia, etc. Some of the mechanism the UK government has used to bring in scientific expertise are set out here [Redacted].

Further down the line, government’s should

• Recognise the extreme risk associated with TAI (including catastrophic, existential and extreme suffering risks) and assign responsibility to a relevant individual to mitigate and prepare for these risks.
• Consider and independent expert review of the risks and potential solutions. (such as the Stern review or the Independent Commission on Banking)

The UK context and UK recommendations

Some recognition of AI risks has already taken place within the UK. The House of Lord’s “AI in the UK: Ready, Willing, and Able?”[5] endorses the Centre for the Study of Existential Risk’s testimony on cyber-security, dual use technology, and the possible AI arms race – but stops short from recognising the risks stemming from superintelligence.

We recommend the UK

• Supports long term thinking in policy and political processes. “Government after government has failed to plan for a long-term generational timescale”[6] according to the recent (May 2019) House of Lords’ (HoL) Intergenerational Fairness Committee. The UK government should implement the majority of the options mapped here [Link redacted. Maybe see here instead]. This would include taking action to addressing cultural factors that motivate short-termism, aiming to spend more on prevention[7], encouraging more use of the lower intergenerational discount rate[8],  and implementing a permanent mechanism to ensure better political representation and due consideration of future generations[9]. This could be done through a Future Generations Bill, discussed in details here [Link redacted. Maybe see here instead].
• Improves the National Risk Register process so that it picks up on emerging risks and risks that may occur more than 5 years in the future but where action now might be useful. There should also be a strategy to handle risks that are shared across national borders to avoid tragedy of the commons type conundrums whereby self-interested states behave to the detriment of the general good.[viii] If an improved processes recognises risks from TAI then this should be included on the National Risk Register and responsibility for TAI risk assigned to a Secretary of State for a relevant Department, most likely the Secretary of State for Digital Culture Media and Sport (DCMS).

Additionally, academics concerned about AI safety should be looking to build or demonstrate academic consensus regarding the scale of AI risks, as without this it is unclear the extent to which good policy decision making processes should be picking up on these risks.

Research

[Update 2022: Less keen on this one. I would still support government funding to work on AI governance and ethics and maybe also on technical safety and misuse risk, but would worry about government AI funding fueling capabilities. I think any ask for government research funding for should be very carefully thought out with this in mind, for example restricting such funding in various ways.]

Countering risks stemming from AI, whether short term (e.g. deep fakes and fake news) or long term (e.g. existential risk) requires significant research, both technical research and policy research.

Funding should be provided to academia and the third sector to ensure that technical AI safety research is carried out, especially to incentivise researchers to look at topics that may not necessarily lead to short term gains or profits. This would have an added benefit of helping to boost the UK’s AI skill base.

Government officials are rarely experts in particular areas, but they do make up a large body of the expertise in policymaking, statecraft, international relations, and so forth. For good AI policy to be developed it is imperative that civil servants are directed to work on AI policy issues.

It is also important that there are good relationships between civil servants and academic experts. In particular cross cutting research will be needed to improve estimates of the potential risks better so that resources can be allocated appropriately.

Government funded R&D may be to support and encourage industry, where there is a market failure to produce a public good, or simply exploratory and academic purposes

Currently the UK Government funds the nuclear research though

• The Government owned[10] National Nuclear Laboratory (NNL) which researches nuclear technology and has a focus on safety. NNL also hosts Nuclear Innovation and Research Office (NIRO) which provides expert support to the UK government.
• Industry partnerships, where both industry and government input funds for a research project, as set out in the nuclear sector deal. Occasionally government will directly commission research from industry consultants.[11] (
• Academia, with funding given out by the Engineering and Physical Sciences Research Council.

At present the nuclear the main focus of research funding is not currently on safety concerns, although this is researched.

Recommendations

• Civil Servants should be researching policy issues around ethics and technology and AI. This could be facilitated through the development of an internal government think tank with a mix of staff form policy and technical backgrounds and the mandate to and capacity to look at high level questions connected to tech, ethics and AI, including TAI. (Similar to CDEI or NIRO)
• Fund external AI safety and ethics research by academia or in partnership with industry. For example, research to better estimate AI risks.

The UK context and UK recommendations

Research Councils UK have an annual budget of £3bn.[12] Just 0.1% of this could fully fund existing global technical AI research[13]. The Government’s Industrial Strategy Challenge Fund’s earmarked £93m for safer AI.[14] (In compassion the UK government are investing around £460 million in nuclear research and innovation).

Civil Servants are being ask to research issues about ethics and technology and AI within CDEI.

The UK government should fund AI safety and ethics research including funding efforts seeking to better estimate AI risks. Bodies such as CDEI should also ensure there are open communication streams between policymakers and technical researchers and the third sector.

Record

[Update 2022: Seems like a nice to have but not a key ask]

Best practice & existing precedent

In nuclear safety, when a serious nuclear or radiological problem occurs, countries will rate this event on the International Nuclear and Radiological Event Scale (INES) and if it is serious enough (and does not threaten national security to do so) the country will report the event to the International Atomic Energy Agency’s (IAEA) Unified System for Information Exchange in Incidents and Emergencies (USIE). The event details are shared globally by the Nuclear Events Web-based System (NEWS). There are also strong working relationships between nuclear operators (and regulators) in different countries. So when evidence of a malfunction was discovered in a nuclear power plant in the UK it quickly became apparent that a plant in the US had dealt with similar issues and the lessons from that plan could be applied to the UK.

In cybersecurity there are various vulnerability disclosure processes in place to ensure rapid information sharing and appropriate responses from corporate actors. Governments have played a role in developing these processes and ensuring best practice even across international boundaries, for example the Dutch Coordinated Vulnerability Disclosure Guidelines.

There have been numerous reports of AI systems behaving unexpectedly and unpredictably.[15] Whilst this is not necessarily problematic for games of Go, it may lead to danger should it occur at a large scale, or in scenarios where human lives are at risk (e.g. self-driving cars) or in the development of TAI.  Technical AI safety researchers are already looking into setting up honeypots[16] to help identify unusual behaviour by advanced AIs. It could be useful to ensure that information on unexpected behaviour is shared across AI labs.

No data sharing system for AI safety currently exists. Such as system could be set up unilaterally by a state or non-state actor, although international agreement would be ideal to ensure it’s use. 

Recommendation

• Develop a system for recording and sharing data relevant to AI safety (similar to NEWS).  Work with industry and academia to generate guidelines about and mechanisms for information sharing, for example covering when information should be shared, shared through private channels or made public. 

Resilience: AI defence and safety

[Update 2022: Not at the stage where I think there is much to do here yet, except maybe general support for research and projects aimed at civilizational resilience and recovery.]

In 1961, “a bomber was flying over North Carolina, when it lost a wing, and two of its nuclear bombs fell to the ground […]. One of the bombs broke on impact after its parachute failed. The other landed unharmed, but five of its six safety devices also failed. Defence Secretary Robert McNamara had this to say: "by the slightest margin of chance, literally the failure of two wires to cross, a nuclear explosion was averted.””[17]

Best practice in risk prevention is ‘defence in depth’. This means that a system is protected by a series of independent layers of defensive mechanisms (or ‘fail-safes’) such that if one mechanism fails, another will already be in place to protect against unexpected failures. To prevent TAI safety risks, AI researchers are already looking at how they can institute fail-safes. 

Governments have capabilities and expertise in defence, security and cyber security. One way the Government can help find more fail-safes to help defend against AI risks is to increase the space in which we are looking for options: ideas such as manual internet/electricity shut-offs, or investments in general computer security to complicate the creation of botnets, should be studied further. [18]

As well as ASI safety concerns there may be a host of policy political and governance issues that could arise very rapidly from the development of TAI that, further down the line, it may be helpful for governments to prepare for. 

Recommendation

Further down the line the Government and the MoD may want engage with AI safety researchers to put in place additional layers defence in depth and other measures to mitigate risks and support the resilience to shocks from the potential development of TAI.

Safeguarding

[Update 2022: No strong views on this. I expect it is a bit over the top for now which risks escalating race dynamics, but an idea to bare in mind. Maybe some good policies on supply chains here. Not really my area of expertise.]

“As the best AI will be expensive, digital theft will pay.26 We can expect to see sophisticated cyber-attacks against companies like Google’s DeepMind, IBM or Facebook if they are perceived to have the best AI code. Defending such civil/commercial assets may become an issue of national security”

Ministry of Defence, UK[19]

In the nuclear industry “safeguarding” is the term used to refer to the variety of mechanisms put in place nationally and internationally to prevent nuclear material intended for peaceful purposes being used for the development of nuclear weapons, as in line with the Nuclear Non-Proliferation Treaty (NPT). In practice in the UK this mean security clearances for staff in sensitive positions and high fences and the armed Civil Nuclear Constabulary at nuclear sites.

There is significant consensus that the misuse of AI, especially TAI by hostile actors poses a significant threat to global security. As such government’s who have industries that are developing AI should look to protect these AI assets from hostile actors who might use them for nefarious ends. Government defence and security capabilities should be used to support the cybersecurity of researchers and academics working on AI who might lack the ability to defend the code they develop from malicious threats. Government could also provide background personnel checks on individuals to make sure they are not influenceable by outside forces (or eventually by an AGI itself).

Recommendation

The government should consider if the UK has high risk AI assets and if so look to protect those assets: for example, by requiring and providing security clearances for staff at Deepmind and by MoD or the National Cyber Security Centre (NCSC) ensuring that DeepMind has excellent cybersecurity protection.

+ supply chain 

Regulation

[Update 2022: I am still very very keen on AI regulatory policy, if we can do it well. I think we can construct a world whereby there are skilled risk aware actors (regulators) who can step in at an emergency crunch point to pull the plug but otherwise do minimal except support safety work, prevent AI misuse, have some oversight and ensure AI companies have good governance structures. I think this world would be great. I think the community of people concerned about AI should be thinking much more about this.]

Based largely on memory but mostly draws from the International Atomic Energy Agency’s (IAEA) Safety Standards, in particular Governmental, Legal and Regulatory Framework for Safety, General Safety Requirements Part 1

Regulate

There is a lot written by governments and academics on best practice in regulation. Within the UK Government best practice is set out in the Regulators Code[20] and more recently the work of the Regulatory Futures Review[21]. For the uninitiated, Annex A looks to dispel some of the preconceptions about regulation. Including dispelling the ideas that regulation means having a detailed rulebook, being a burden on industry, watching everything industry does, enforcing rules using legal powers and so on. 

In the UK the nuclear Regulator is the Office for Nuclear Regulation (ONR). The (ONR) is responsible for regulation of the UK’s nuclear industry. They are required under the Energy Act to ensure nuclear safety, conventional safety, security and safeguarding on civil nuclear sites. They are an arm’s length Government body that reports to Parliament and are 97% funded by fees charged to dutyholders (nuclear sites). They use outcome focused regulation and dutyholders need to demonstrate that they are meeting the licence conditions, but the way they meet them is up to the dutyholder. The ONR are comprised of a body of experts, known as inspectors, who tend to have considerable experience in the nuclear industry. Inspectors monitor actions of dutyholders inspections, site self-reporting and site safety cases or other independent reports. The inspectors have a good relationship with the sites and 90 of the time soft enforcement (eg. a quick constructive conversation) is sufficient to address an issue. There is a separate regulator for Defence sites, the Defence Nuclear Safety Regulator (DNSR).

Another example of best practice UK regulation is the acclaimed Human Fertilisation and Embryology Authority (HFEA).

Drawing from best practice in regulation and especially from nuclear industry we can draw a number of conclusions about what good regulation of AI should look like.

Ideally the remit of an AI regulator should include:

• Regulating the responsible use of tech by the tech industry. Regulators function well when they have a good relationship with the industry they are regulating. For example, the ONR regulates conventional health and safety as well as nuclear safety. This helps the regulator build a single point of contact relationship with industry. Even more importantly this allows inspectors to infer conclusions about the likelihood of serious safety incidents (eg. in the development of TAI) from the likelihood of minor incidents and infractions.
• Regulating the use of tech by government including by the military.
• Other responsibilities. They should ensure the safeguarding of cyber commercial assets (discussed above), ensure reporting and data sharing (discussed above) and have a responsibility to support economic growth and support the development of tech and AI skills.

The rules should be:

• Outcome focused. Companies should be able to demonstrate the safe and ethical in any way they wish. For example, a driverless car company does not need to show it can meet specific criteria (eg. can the AI recognise humans) only that it is safe (if used it will not lead to casualties). 
• Based on high level principles
• Consider corporate structures and safety culture. For example make it clear that responsibility needs to be assigned at senior levels and that Directors can be held legally responsible for problems. Make t clear that responsible indiviauls need to be suitably quliaifed and experienced. 

This tech regulator should be set up such that it is:

• Staffed by technical experts at the top of their fields. There should be a set of inspectors each of whom can generally inspect most situations but each with a unique high level specialisation. 
• Able to use powers but in practice use them rarely. Inspectors should have powers (to access site, seize evidence, issue improvement notices, take to court) to but look to avoid using them as much as possible where soft influence will suffice.
• Independent from government. They should be an arm’s length body like the Bank of England or the ONR. In the UK they should report directly to Parliament. If it is not too much of a burden on industry, they should be funded by a levy on industry rather than by government.
• Sampling and permissioning focused. Inspectors should not look at everything a company does. In most cases inspectors should look at random samples of projects and code. The regulator may want to set out that in certain specified extreme cases where the risks or ethical implications are unclear (eg introducing driverless cars on public roads) the company should ask for permission first and demonstrate that they are safe.
• Proportionate. The amount of regulation must be proportionate to the risk and not an undue burden on industry. Potentially a regulatory sandbox with lighter rules should exist to support small-scale innovation.
• Horizon scanning. The regulator should think long term and be able to adapt to an ever-changing technological environment.

Recommendation

The UK Government should set up a regulator in the form of a well-funded independent body of technical experts is given very high-level goals to ensure the safe and ethical behaviour of the technology industry (similar to HEFA or the ONR).

+ Put in place whistleblowing mechanisms

+ Look also at the corporate structures and how responsibility is assigned to individuals within the organisations 

Concluding thoughts 

Recommendations

In the UK context

The paper recognises the positive steps that the UK Government has made to date in particular:

• Civil Servants should be researching policy issues around ethics and technology and AI. (This is already happening sufficiently within CDEI).
• it Recognises emerging risks that may occur more than 5 years in the future but where action now might be useful. The relevant Ministers should be assigned Responsibility to address any additional risks

International agreements

The author of this paper is not an expert in international treaties, or international security mechanisms but it certainly seems likely that international policy and the international stage has a role to play in ensuring that if TAI is developed the transition goes well for humanity.

In the first instance regulatory expertise is often shared and developing a good regulatory system in the UK will pave the way for good AI regulation globally.

Additionally, international agreements could be reached on the ethical use of AI (an idea suggested by former UK digital and culture secretary Matt Hancock[22]), on the development of AI weapons, on international safeguarding and non-proliferation processes and on the right level of international data sharing and reporting of errors. Potentially even an international body for sharing regulator best practice could be set up similar to the IAEA. 

Moving forwards

That said, there may not be much choice. Governments are already discussing these issues, creating specialised statutory bodies, and providing finance to early stage projects. And all it needs now is for a serious event, which could be a massive cyber-attack using a machine learning algorithm, or could be the hacking of a self-driving car to murder a person, and the public may start shouting for regulation. It would be good to have the above policy suggestions at hand for when that happens.

Annex A : introduction to Regulation

For the uninitiated, Annex A looks to dispel some of the preconceptions about regulation. Including:

1. Regulators are a part of Government. There is a scale from total industry self-regulation to government run regulators. Most regulatory bodies have some degree of independence from government and the Regulatory Futures Review recommended that government regulators be funded by dutyholders (the bodies being regulated) rather than government grants.
2. Regulation does not mean having a detailed rulebook. Best practice is to have outcome focused regulations and to allow the dutyholders to work out how to achieve the outcomes. 
3. Regulation does not mean being a burden on industry. Having a body of experts check that a system is safe provides a benefit to the dutyholder. Furthermore, a regulator may have a secondary responsibility to support industry growth. More perversely big corporates may like regulation as it keeps smaller organisations out of the market. 
4. Enforcing regulation does not mean watching everything industry does. Dutyholders may self-report. Inspectors might only go to sites if they already expect there is a problem, or may use sampling methods to test the waters and look for evidence of poor process or risk taking.
5. Enforcing regulation does not mean using legal powers. Although they will have legal enforcement powers, most of the work of a regulator is to keep dutyholders within the law without having to resort to legal powers.
6. Regulation does not mean letting government off the hook. Regulators can have oversight of Government activities as well as corporate activities.
7. Different regulators specialise in different topics or type of risk. This is somewhat true, and certainly topic expertise is important. But more important is the relationship formed with dutyholders, and different regulators regulate different parts of industry / government. (Eg the nuclear regulator regulates the nuclear industry, including conventional health and safety matters)

[1] https://www.fhi.ox.ac.uk/wp-content/uploads/GovAIAgenda.pdf 

[2] Nick Bostrom. (2014). Future Progress in Artificial Intelligence: A Survey of Expert Opinion. [online] Available at: https://nickbostrom.com/papers/survey.pdf [Accessed 15 Jul. 2018].

[3] https://www.gov.uk/government/publications/national-risk-register-of-civil-emergencies-2017-edition

[4] There was a National Security Risk Assessment in 2015 that fed into the National Security Strategy 2015. See: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/478936/52309_Cm_9161_NSS_SD_Review_PRINT_only.pdf and https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/62484/Factsheet2-National-Security-Risk-Assessment.pdf

[5] House of Lords, (2018). AI in the UK: Ready, willing and able? Report of Session 2017-19. [online] Available at: https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf [Accessed 15 Jul. 2018].

[6] https://publications.parliament.uk/pa/ld201719/ldselect/ldintfair/329/329.pdf

[7] A suggestion of Lord John Bird https://www.london.edu/news-and-events/news/big-issue-founder-dismantle-poverty-1133

[8] https://www.gov.uk/government/publications/green-book-supplementary-guidance-discounting

[9] https://www.cser.ac.uk/resources/representation-future-generations/

[10] https://www.gov.uk/government/news/announcement-on-the-national-nuclear-laboratory

[11] Eg. https://www.fnc.co.uk/news-and-events/news-archive/2018/frazer-nash-secures-contract-to-support-the-government-s-nuclear-research-and-development-programme.aspx

[12] Gov.UK. (2017). Funding for nuclear innovation. [online] Available at: https://www.gov.uk/guidance/funding-for-nuclear-innovation [Accessed 15 Jul. 2018].

[13] Based on the budgets of MIRI and CSER and GCRI.

[14] https://www.ukri.org/innovation/industrial-strategy-challenge-fund/robots-for-a-safer-world/ 

[15] Metz, C. (2018). The Sadness and Beauty of Watching Google's AI Play Go. [online] WIRED. Available at: https://www.wired.com/2016/03/sadness-beauty-watching-googles-ai-play-go/ [Accessed 15 Jul. 2018]. Key citation: “It's not a human move.”

[16] https://en.wikipedia.org/wiki/Honeypot_(computing)

[17] Future of Life Institute. (2018). Accidental Nuclear War: A Timeline of Close Calls - Future of Life Institute. [online] Available at: https://futureoflife.org/background/nuclear-close-calls-a-timeline/ [Accessed 15 Jul. 2018].

[18] Duettmann, A., Miller, M. and Peterson, C. (2018). Cyber, Nano, and AGI Risks: Decentralized Approaches to Reducing Risks. [online] Available at: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46290.pdf  [Accessed 15 Jul. 2018].

[19] P19 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/709359/20180517-concepts_uk_human_machine_teaming_jcn_1_18.pdf

[20] https://www.gov.uk/government/publications/regulators-code 

[21] https://www.gov.uk/government/publications/regulatory-futures-review

[22] Government Computing Network. (2018). Hancock wants international regulatory framework on AI - Government Computing Network. [online] Available at: https://www.governmentcomputing.com/brexit-eu/news/hancock-wants-international-regulatory-framework-ai [Accessed 15 Jul. 2018].

[i] An Overview of National AI Strategies, Tim Dutton, https://medium.com/politics-ai/an-overview-of-national-ai-strategies-2a70ec6edfd

[ii] p39, Industrial Strategy, HM Government, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/664563/industrial-strategy-white-paper-web-ready-version.pdf

[iii] The AI Council was a recommendation of the Independent report on Growing the artificial intelligence industry in the UK, https://www.gov.uk/government/publications/growing-the-artificial-intelligence-industry-in-the-uk

[iv] https://www.gov.uk/guidance/emerging-technology-for-defence-programme

[v] https://www.gov.uk/government/organisations/centre-for-connected-and-autonomous-vehicles

[vi] https://www.parliament.uk/documents/lords-committees/science-technology/life-sciences-industrial-strategy/Govt-response-Life-Sciences-Industrial-Strat.pdf

[vii] https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/644968/UK_National_Risk_Register_2017.pdf 

[viii] If a risk materialises but originates from another country the responsible Minister can avoid the blame for not acting. For example, consider the inability of the EU to progress with Bank Structural Reform, regulations for financial stability, plausibly as a result of self-interested state actors refusing to work together. https://ec.europa.eu/info/business-economy-euro/banking-and-finance/financial-supervision-and-risk-management/managing-risks-banks-and-financial-institutions/structural-reform-eu-banking-sector_en https://www.bloomberg.com/news/articles/2017-10-24/eu-buries-its-plans-to-split-up-retail-and-investment-banking