The ideal MIRI researcher is someone who’s able to think about thorny philosophical problems and break off parts of them to formalize mathematically. In the case of logical uncertainty, researchers started by thinking about the initially vague problem of reasoning well about uncertain mathematical statements, turned some of these thoughts into formal desiderata and algorithms (producing intermediate possibility and impossibility results), and eventually found a way to satisfy many of these desiderata at once. We’d like to do a lot more of this kind of work in the future.

Probably the main difference between MIRI research and typical AI research is that we focus on problems of the form “if we had capability X, how would we achieve outcome Y?” rather than “how can we build a practical system achieving outcome Y?”. We focus less on computational tractability and more on the philosophical question of how we would build a system to achieve Y in principle, given e.g. unlimited computing resources or access to extremely powerful machine learning systems. I don’t think we have much special knowledge that others don’t have (or vice versa), given that most relevant AI research is public; it’s more that we have a different research focus that will lead us to ask different questions. Of course, our different research focus is motivated by our philosophy about AI, and we have significant philosophical differences with most AI researchers (which isn’t actually saying much given how much philosophical diversity there is in the field of AI).

Work in the field of AI can inform us about what approaches are most promising (e.g., the theoretical questions in the “Alignment for Advanced Machine Learning Systems” agenda are of more interest if variants of deep learning are sufficient to achieve AGI), and can directly provide useful theoretical tools (e.g., in the field of statistical learning theory). Typically, we will want to get a high-level view of what the field is doing and otherwise focus mainly on the more theoretical work relevant to our research interests.

We definitely need some way of dealing with the fact that we don’t know which AI paradigm(s) will be the foundation of the first AGI systems. One strategy is to come up with abstractions that work across AI paradigms; we can ask the question “if we had access to extremely powerful reinforcement learning systems, how would we use them to safely achieve some concrete objective in the world?” without knowing how these reinforcement learning systems work internally. A second strategy is to prioritize work related to types of AI systems that seem more promising (deep learning seems more promising than symbolic GOFAI at the moment, for example). A third strategy is to do what people sometimes do when coming up with new AI paradigms: think about how good reasoning works, formalize some of these aspects, and design algorithms performing good reasoning according to these desiderata. In thinking about AI alignment, we apply all three of these strategies.

I largely endorse Jessica’s comment. I’ll add that I think the ideal MIRI researcher has their own set of big-picture views about what’s required to design aligned AI systems, and that their vision holds up well under scrutiny. (I have a number of heuristics for what makes me more or less excited about a given roadmap.)

That is, the ideal researcher isn’t just working on whatever problems catch their eye or look interesting; they’re working toward a solution of the whole alignment problem, and that vision regularly affects their research priorities.

I don’t think of our strategy as having changed much in the last year. For example, in the last AMA I said that the plan was to work on some big open problems (I named 5 here: asymptotically good reasoning under logical uncertainty, identifying the best available decision with respect to a predictive world-model and utility function, performing induction from inside an environment, identifying the referents of goals in realistic world-models, and reasoning about the behavior of smarter reasoners), and that I’d be thrilled if we could make serious progress on any of these problems within 5 years. Scott Garrabrant then promptly developed logical induction, which represents serious progress on two (maybe three) of the big open problems. I consider this to be a good sign of progress, and that set of research priorities remains largely unchanged.

Jessica Taylor is now leading a new research program, and we're splitting our research time between this agenda and our 2014 agenda. I see this as a natural consequence of us bringing on new researchers with their own perspectives on various alignment problems, rather than as a shift in organizational strategy. Eliezer, Benya, and I drafted the agent foundations agenda when we were MIRI’s only full-time researchers; Jessica, Patrick, and Critch co-wrote a new agenda with their take once they were added to the team. The new agenda reflects a number of small changes: some updates that we’ve all made in response to evidence over the last couple of years, some writing-up of problems that we’d been thinking about for some time but which hadn’t made the cut into the previous agenda, and some legitimate differences in intuition and perspective brought to the table by Jessica, Patrick, and Critch. The overall strategy is still “do research that we think others won’t do,” and the research methods and intuitions we rely on continue to have a MIRI-ish character.

Regarding success probability, I think MIRI has a decent chance of success compared to other potential AI risk interventions, but AI risk is a hard problem. I’d guess that humanity as a whole has a fairly low probability of success, with wide error bars.

Unless I’m missing context, I think the “medium probability of success” language comes from old discussions on LessWrong about how to respond to Pascal’s mugging. (See Rob’s note about Pascalian reasoning here.) In that context, I think the main dichotomy Eliezer had in mind was “tiny” probabilities (that can be practically ignored, like gambling in the powerball) and strategically relevant probabilities like 1% or 10%. See Eliezer’s post here. I’m fine with calling the latter probabilities “medium-sized” in the context of lottery-style errors, and calling them “small” in other contexts. With respect to ensuring that the first AGI designs developed by AI scientists are easy to align, I don’t think MIRI’s odds are stellar, though I do feel comfortable saying that they’re higher than 1%. Let me know if I’ve misunderstood the question you had in mind here.

When it comes to growth, at the moment our focus is on expanding the research team. As such, our next few hires are likely to be research fellows, and assistant research fellows[1] for both our agent foundations and machine learning technical agendas. We have two new research fellows who are signed on to join the team, Abram Demski and Mihály Bárász. Abram and Mihály will both be more focused on the AF agenda, so I’m hoping our next couple hires after them will be on the ML side. We’re prioritizing people who can write well and quickly; if you or someone you know is interested and has that sort of skill, you’re encouraged to get in touch with Alex Vermeer.

As mentioned in MIRI Update and Fundraising Case, which Nate posted here a few days ago, in the medium term our current plan is to grow our research team to 13–17 people. Since we already have a pretty solid ops foundation, I don’t anticipate that we’ll need to increase our ops capacity very much to support a research team of that size, so unless our strategy changes significantly, I expect most of our upcoming hires will be researchers.

[1] At MIRI, research fellow is a full-time permanent position. A decent analogy in academia might be that research fellows are to assistant research fellows as full-time faculty are to post-docs. Assistant research fellowships are intended to be a more junior position with a fixed 1–2 year term.

In my current view, MIRI’s main contributions are (1) producing research on highly-capable aligned AI that won’t be produced by default by academia or industry; (2) helping steer academia and industry towards working on aligned AI; and (3) producing strategic knowledge of how to reduce existential risk from highly-capable AI. I think (1) and (3) are MIRI’s current strong suits. This is not easy to verify without technical background and domain knowledge, but at least for my own thinking I’m impressed enough with these points to find MIRI very worthwhile to work with.

If (1) were not strong, and (2) were no stronger than currently, I would trust (3) somewhat less, and I would give up on MIRI. If (1) became difficult or impossible because (2) was done, i.e. if academia and/or industry were already doing all the important safety research, I’d see MIRI as much less crucial, unless there was a pivot to remaining neglected tasks in reducing existential risk from AI. If (2) looked too difficult (though there is already significant success, in part due to MIRI, FHI, and FLI), and (1) were not proceeding fast enough, and my “time until game-changing AI” estimates were small enough, then I’d probably do something different.

I’ll interpret this question as “what are the most plausible ways for you to lose confidence in MIRI’s effectiveness and/or leave MIRI?” Here are a few ways that could happen for me:

1. I could be convinced that I was wrong about the type and quality of AI alignment research that the external community is able to do. There’s some inferential distance here, so I'm not expecting to explain my model in full, but in brief, I currently expect that there are a few types of important research that academia and industry won’t do by default. If I was convinced that either (a) there are no such gaps or (b) they will be filled by academia and industry as a matter of course, then I would downgrade my assessment of the importance of MIRI accordingly.
2. I could learn that our research path was doomed, for one reason or another, and simultaneously learn that repurposing our skill/experience/etc. for other purposes was not worth the opportunity cost of all our time and effort.

One horse-sized duck AI. For one thing, the duck is the ultimate (route) optimization process: you can ride it on land, sea, or air. For another, capabilities scale very nonlinearly in size; the neigh of even 1000 duck-sized horse AIs does not compare to the quack of a single horse-sized duck AI. Most importantly, if you can safely do something with 100 opposite-sized AIs, you can safely do the same thing with one opposite-sized AI.

In all seriousness though, we don't generally think in terms of "proving the friendliness" of an AI system. When doing research, we might prove that certain proposals have flaws (for example, see (1)) as a way of eliminating bad ideas in the pursuit of good ideas. And given a realistic system, one could likely prove certain high-level statistical features (such as “this component of the system has an error rate that vanishes under thus-and-such assumptions”), though it’s not yet clear how useful those proofs would be. Overall, though, the main challenges in friendly AI seem to be ones of design rather than verification. In other words, the problem is to figure out what properties an aligned system should possess, rather than to figure out how to prove them; and then to design a system that satisfies those properties. What properties would constitute friendliness, and what assumptions about the world do they rely on? I expect that answering these questions in formal detail would get us most of the way towards a design for an aligned AI, even if the only guarantees we can give about the actual system are statistical guarantees.

(1) Soares, Nate, et al. "Corrigibility." Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.

You can browse our papers and research summaries here and see if anything clicks, but failing that, I’m not sure there’s any simple heuristic I can suggest beyond “look for lots of separate lines of indirect evidence.” One question is whether we’re working on the right problems for addressing AI risk. Relevant indicators that come to mind include:

• Stuart Russell’s alignment research group is interested in value learning and “theories of (bounded) rationality,” as well as corrigibility (1, 2).
• A number of our research proposals were cited in FLI’s research priorities document, and our agent foundations agenda received one of the larger FLI grants.
• FHI and DeepMind have collaborated on corrigibility work.
• The Open Philanthropy Project’s research advisors don’t think logical uncertainty, decision theory, or Vingean reflection are likely to be safety-relevant.
• The “Concrete Problems in AI Safety” agenda has some overlap with our research interests and goals (e.g., avoiding wireheading).

A separate question is whether we’re making reasonable progress on those problems, given that they’re the right problems. Relevant indicators that come to mind:

• An OpenPhil external reviewer described our HOL-in-HOL result as “an important milestone toward formal analysis of systems with some level of self-understanding.”
• OpenPhil’s internal and external reviewers considered a set of preliminary MIRI results leading to logical induction unimpressive.
• Our reflective oracles framework was presented at a top AI conference, UAI.
• Scott Aaronson thinks “Logical Induction” is important and theoretically interesting.
• We haven’t had any significant public endorsements of our work on decision theory by leading decision theorists.
• … and so on.

If you don’t trust MIRI or yourself to assess the situation, I don’t think there’s any shortcut besides trying to gather and weigh miscellaneous pieces of evidence. (Possibly the conclusion will be that some parts of MIRI’s research are useful and others aren’t, since we work on a pretty diverse set of problems.)

"College level math" can mean a whole lot of things...

Re: 1, "what are the main points of disagreement?" is itself currently one of the points of disagreement :) A lot of our disagreements (I think) come down to diverging inchoate mathematical intuitions, which makes it hard to precisely state why we think different problems are worth prioritizing (or to resolve the disagreements).

Also, I think that different Open Phil technical advisors have different disagreements with us. As an example, Paul Christiano and I seem to have an important disagreement about how difficult it will be to align AI systems if we don’t have a correct theoretically principled understanding of how the system performs its abstract reasoning. But while the disagreement seems to me and Paul to be one of the central reasons the two of us prioritize different projects, I think some other Open Phil advisors don’t see this as a core reason to accept/reject MIRI’s research directions.

Discussions are still ongoing, but Open Phil and MIRI are both pretty time-constrained organizations, so it may take a while for us to publish details on where and why we disagree. My own attempts to gesture at possible points of divergence have been very preliminary so far, and represent my perspective rather than any kind of MIRI / Open Phil consensus summary.

Re: 4, I think we probably spent too much time this year writing up results and research proposals. The ML agenda and “Logical Induction,” for example, were both important to get right, but in retrospect I think we could have gotten away with writing less, and writing it faster. Another candidate mistake is some communication errors I made when I was trying to explain the reasoning behind MIRI’s research agenda to Open Phil. I currently attribute the problem to me overestimating how many concepts we shared, and falling prey to the illusion of transparency, in a way that burned a lot of time (though I’m not entirely confident in this analysis).

Re 2, Sam and Eliezer have been corresponding for a while now. They’ve been exploring the possibility of pursuing a couple of different projects together, including co-authoring a book or recording a dialogue of some sort and publishing it online. Sam discussed this briefly on an episode of his podcast. We’ll mention in the newsletter if things get more finalized.

Re 3, it varies a lot month-to-month and person-to-person. Looking at the data, the average and median are pretty close at somewhere between 40–50 hours a week depending on the month. During crunch times some people might be working 60–100-hour weeks.

I’ll also mention that although people at MIRI roughly track how many hours they spend working, and on what, I don’t put much weight on these numbers (especially for researchers). If a researcher comes up with a new idea in the shower, at the gym, on their walk to work, or whatever, I don’t expect them to log those hours as work time. (Fun fact: Scott came up with logical induction on his walk to work.) Many of us are thinking about work when we aren’t at our desks, so to speak. It’s also hard to compare someone who spends 80 hours working on a problem they love and find really exciting, to someone who spends 40 hours on really grueling tasks. I prefer to focus on how much people are getting done and how they are feeling.

Re 4, for me personally, I think my biggest mistake this year was not delegating enough after transitioning into the COO role. This caused a few ops projects to be blocked on me unnecessarily, which set a few ops projects back a few months. (For example, I finished our 2015-in-review document significantly later than I would have liked.)

Relevant to 1: https://agentfoundations.org/item?id=1129

I think that the ML-related topics we spend the most effort on (such as those in the ML agenda) are currently quite neglected. See my other comment for more on how our research approach is different from that of most AI researchers.

It’s still plausible that some of the ML-related topics we research would be researched anyway (perhaps significantly later). This is a legitimate consideration that is, in my view, outweighed by other considerations (such as the fact that less total safety research will be done if AGI comes soon, making such timelines more neglected; the fact that ML systems are easy to think about due to their concreteness; and the fact that it can be beneficial to “seed” the field with high-quality research that others can build on in the future).

Additionally I think that AI alignment researchers should avoid ignoring huge theoretically-relevant parts of the problem. I would have quite a lot of difficulty thinking about AI alignment without thinking about how one might train learning systems to do good things using feedback. One of my goals with the ML agenda is to build theoretical tools that make it possible to think about the rest of the problem more clearly.

The "more research" part has gone well: we added Benya and Nate in 2014, and Patrick, Jessica, Andrew, and Scott in 2015. We’re hoping to double the size of the research team over the next year or two. MIRI’s Research and All Publications pages track a lot of our output since then, and we’ve been pretty excited about recent developmens there.

For “less outreach,” the absolute amount of outreach work we're doing is probably increasing at the moment, though it's shrinking as a proportion of our total activities as the research team grows. (Eyeballing it, right now I think we spend something like 6 hours on research per hour on outreach.)

The character of our outreach is also quite different: more time spent dialoguing with AI groups and laying groundwork for research collaborations, rather than just trying to spread safety-relevant memes to various intellectuals and futurists.

The last two years have seen a big spike of interest in AI risk, and there's a lot more need for academic outreach now that it's easier to get people interested in these problems. On the other hand, there's also a lot more supply; researchers at OpenAI, Google, UC Berkeley, Oxford, and elsewhere who are interested in safety work often have a comparative advantage over us at reaching out to skeptics or researchers who are new to these topics. So the balance today is probably similar to what Luke and others at MIRI had in mind on a several-year timescale in 2013, though there was a period in 2014/2015 where we had more uncertainty about whether other groups would pop up to help meet the increased need for outreach.

“Human interests” is an unfortunate word choice; Nate talked about this last year too, and we’ve tried to avoid phrasings like that. Unfortunately, most ways of gesturing at the idea of global welfare aren’t very clear or widely understood, or they sound weird, or they borrow arguably speciesist language (“humane,” "humanitarian," “philanthropy”...).

I’m pretty sure everyone at MIRI thinks we should value all sentient life (and extremely sure at least in the case of Eliezer, Nate, and myself), including sentient non-human animals and any sentient machines we someday develop. Eliezer thinks, as an empirical hypothesis, that relatively few animal species have subjective experience. Other people at MIRI, myself included, think a larger number of animal species have subjective experience. There's no "consensus MIRI view" on this point, but I think it's important to separate the empirical question from the strictly moral one, and I'm confident that if we learn more about what "subjective experience" is and how it's implemented in brains, then people at MIRI will update. It's also important to keep in mind that a good safety approach should be robust to the fact that the designers don’t have all the answers, and that humanity as a whole hasn’t fully developed scientifically (or morally).

I am not a MIRI employee, and this comment should not be interpreted as a response from MIRI, but I wanted to throw my two cents in about this topic.

I think that creating a friendly AI to specifically advance human values would actually turn out okay for animals. Such a human-friendly AI should optimize for everything humans care about, not just the quality of humans' subjective experience. Many humans care a significant amount about the welfare of non-human animals. A human-friendly AI would thus care about animal welfare by proxy through the values of humans. As far as I am aware, there is not a significant number of humans who specifically want animals to suffer. It is extremely common for humans to want things (like food with the taste and texture of bacon) that currently can currently be produced most efficiently at significant expense to non-human animals. However, it seems unlikely that a friendly AI would not be able to find an efficient way of producing bacon that does not involve actual pigs.

I endorse Tsvi's comment above. I'll add that it’s hard to say how close we are to closing basic gaps in understanding of things like “good reasoning”, because mathematical insight is notoriously difficult to predict. All I can say is that logical induction does seem like progress to me, and we're taking various different approaches on the remaining problems. Also, yeah, one of those avenues is a follow-up to PPRHOL. (One experiment we’re running now is an attempt to implement a cellular automaton in HOL that implements a reflective reasoner with access to the source code of the world, where the reasoner uses HOL to reason about the world and itself. The idea is to see whether we can get the whole stack to work simultaneously, and to smoke out all the implementation difficulties that arise in practice when you try to use a language like HOL for reasoning about HOL.)

Scott Garrabrant’s logical induction framework feels to me like a large step forward. It provides a model of “good reasoning” about logical facts using bounded computational resources, and that model is already producing preliminary insights into decision theory. In particular, we can now write down models of agents that use logical inductors to model the world---and in some cases these agents learn to have sane beliefs about their own actions, other agents’ actions, and how those actions affect the world. This, despite the usual obstacles to self-modeling.

Further, the self-trust result from the paper can be interpreted to say that a logical inductor believes something like “If my future self is confident in the proposition A, then A is probably true”. This seems like one of the insights that the PPRHOL work was aiming at, namely, writing down a computable reasoning system that asserts a formal reflection principle of itself. Such a reflection principle must be weaker than full logical soundness; a system that proved “If my future self proves A, then A is true” would be inconsistent. But as it turns out, the reflection principle is feasible if you replace “proves” with “assigns high probability to” and replace “true” with “probably true”.

It is an active area of research to understand logical induction more deeply, and to apply it to decision-theoretic problems that require reflective properties. For example, the current framework uses “traders” that express their “beliefs” in terms of strategies for making trades against the market prices (probabilities) output by a logical inductor. Then traders are rewarded based on buying shares in sentences that turn out to be highly valued by later market prices. It would be nice to understand this process in Bayesian terms, e.g., where traders are hypotheses that output predictions about the market and have their probabilities updated by Bayesian updating.

I’ll start by stating that, while I have some intuitions about how the paper will be received, I don’t have much experience making crisp forecasts, and so I might be miscalibrated. That said:

1. In my experience it’s pretty common for ML researchers who are more interested in theory and general intelligence to find Solomonoff induction and AIXI to be useful theories. I think “Logical Induction” will be generally well-received among such people. Let’s say 70% chance that at least 40% of ML researchers who think AIXI is a useful theory, and who spend a couple hours thinking about “Logical Induction” (reading the paper / talking to people about it), will think that “Logical Induction” is at least 1/3 as interesting/useful as AIXI. I think ML researchers who don’t find Solomonoff induction relevant to their interests probably won’t find “Logical Induction” compelling either. This forecast is based on my personal experience of really liking Solomonoff induction and AIXI (long before knowing about MIRI) but finding theoretical gaps in them, many of which “Logical Induction” resolves nicely, and from various conversations with ML researchers who like Solomonoff induction and AIXI.
2. I have less-strong intuitions about mathematicians but more empirical data. “Logical Induction” has been quite well-received by Scott Aaronson, and I think the discussion at n-Category Cafe indicates that mathematicians find this paper and the overall topic interesting. I am quite uncertain about the numbers, but I expect something like 50% of mathematicians who are interested in Bayesianism and Gödel’s incompleteness theorem to think it’s quite an interesting result after thinking about it for a couple hours.

(these predictions might seem timid; I am adjusting for the low base rates of people finding things really interesting)

Over the past couple of years I’ve been excited to see the growth of the community of researchers working on technical problems related to AI alignment.

Here a quick and non-exhaustive list of people (and associated organizations) that I’m following (besides MIRI research staff and associates) in no particular order:

• Stuart Russell and the new Center for Human-Compatible AI.
• FHI’s growing technical AI safety team, which includes:
  • Stuart Armstrong, who is also a research associate at MIRI and co-author of Safely Interruptible Agents with Laurent Orseau of DeepMind);
  • Eric Drexler;
  • Owain Evans; and
  • Jan Leike, who recently collaborated with MIRI on the paper A formal solution to the grain of truth problem.
• The authors of the Concrete Problems in AI Safety paper:
  • Dario Amodei, who is now at OpenAI, and Chris Olah and Dan Mané at Google Brain;
  • Jacob Steinhardt at Stanford;
  • Paul Christiano, who is a long-time MIRI collaborator currently at OpenAI (see also his writing at medium.com/ai-control); and
  • John Schulman, also at OpenAI.
• DeepMind’s AI safety team, led by Laurent Orseau.
• Various other individual academics; for a sampling see speakers at our Colloquium Series on Robust and Beneficial AI and grant recipients from the Future of Life Institute.

First, note that we’re not looking for “proven” solutions; that seems unrealistic. (See comments from Tsvi and Nate elsewhere.) That aside, I’ll interpret this question as asking: “if your research programs succeed, how do you ensure that the results are used in practice?” This question has no simple answer, because the right strategy would likely vary significantly depending on exactly what the results looked like, our relationships with leading AGI teams at the time, and many other factors.

For example:

• What sort of results do we have? The strategy is different depending on whether MIRI researchers develop a generic set of tools for aligning arbitrary AGI systems versus whether they develop a set of tools that only work for developing a sufficiently aligned very limited task-directed AI, and so on.[1]
• How dangerous do the results seem? Designs for alignable AI systems could feasibly yield insight into how to construct misaligned AI systems; in that case, we’d have to be more careful with the tools. (Bostrom wrote about issues surrounding openness here.)[2]

While the strategy would depend quite a bit on the specifics, I can say the following things in general:

• We currently have pretty good relationships with many of the leading AI teams, and most of the leading teams are fairly safety-conscious. If we made a breakthrough in AI alignment, and an expert could easily tell that the tools were useful upon inspection, I think it is very reasonable to expect that the current leading teams would eagerly adopt those tools.
• The “pass a law that every AGI must be built a certain way” idea does not seem feasible to me in this context.
• In the ideal case, the world will coordinate around the creation of AGI (perhaps via a single collaborative project), in which case there would be more or less only one team that needed to adopt the tools.

In short, my answer here is “AI scientists tend to be reasonable people, and it currently seems reasonable to expect that if we develop alignment tools that clearly work then they’ll use them.”

[1] MIRI’s current focus is mainly on improving the odds that the kinds of advanced AI systems researchers develop down the road are alignable, i.e., they’re the kinds of system we can understand on a deep and detailed enough level to safely use them for various “general-AI-ish” objectives.

[2] On the other hand, sharing sufficiently early-stage alignment ideas may be useful for redirecting research energies toward safety research, or toward capabilities research on relatively alignable systems. What we would do depends not only on the results themselves, but on the state of the rest of the field.

Good question. The main effect is that I’ve increased my confidence in the vague MIRI mathematical intuitions being good, and the MIRI methodology for approaching big vague problems actually working. This doesn’t constitute a very large strategic shift, for a few reasons. One reason is that my strategy was already predicated on the idea that our mathematical intuitions and methodology are up to the task. As I said in last year’s AMA, visible progress on problems like logical uncertainty (and four other problems) were one of the key indicators of success that I was tracking; and as I said in February, failure to achieve results of this caliber in a 5-year timeframe would have caused me to lose confidence in our approach. (As of last year, that seemed like a real possibility.) The logical induction result increases my confidence in our current course, but it doesn't shift it much.

Another reason logical induction doesn’t affect my strategy too much is that it isn’t that big a result. It’s one step on a path, and it’s definitely mathematically exciting, and it gives answers to a bunch of longstanding philosophical problems, but it’s not a tool for aligning AI systems on the object level. We’re building towards a better understanding of “good reasoning”, and we expect this to be valuable for AI alignment, and logical induction is a step in that direction, but it's only one step. It’s not terribly useful in isolation, and so it doesn’t call for much change in course.

I think this has been changing in recent years, yes. A number of AI researchers (some of them quite prominent) have told me that they have largely agreed with AI safety concerns for some time, but have felt uncomfortable expressing those concerns until very recently. I do think that the tides are changing here, with the Concrete Problems in AI Safety paper (by Amodei, Olah, et al) perhaps marking the inflection point. I think that the 2015 FLI conference also helped quite a bit.

Thanks, Benito. With regards to the second half of this question, I suspect that either you’ve misunderstood some of the arguments I’ve made about why our work doesn’t tend to fit into standard academic journals and conferences, or (alternatively) someone has given arguments for why our work doesn’t tend to fit into standard academic venues that I personally disagree with. My view is that our work doesn’t tend to fit into standard journals etc. because (a) we deliberately focus on research that we think academia and industry are unlikely to work on for one reason or another, and (b) we approach problems from a very different angle than the research communities that are closest to those problems.

One example of (b) is that we often approach decision theory not by following the standard philosophical approach of thinking about what decision sounds intuitively reasonable in the first person, but instead by asking “how could a deterministic robot actually be programmed to reliably solve these problems”, which doesn’t fit super well into the surrounding literature on causal vs. evidential decision theory. For a few other examples, see my response to (8) in my comments on the Open Philanthropy Project’s internal and external reviews of some recent MIRI papers.

To the first part of your question, most faculty at universities have many other responsibilities beyond research which can include a mix of grant writing, teaching, supervising students, and sitting on various university councils. At MIRI most of these responsibilities simply don’t apply. We also work hard to remove as many distractions from our researchers as we can so they can spend as much of their time as possible actually making research progress. [1]

Regarding incentives, as Nate has previously discussed here on the EA Forum, our researchers aren’t subject to the same publish-or-perish incentives that most academics (especially early in their careers) are. This allows them to focus more on making progress on the most important problems, rather than trying to pump out as many papers as possible.

[1] For example, the ops team takes care of formatting and submitting all MIRI publications, we take on as much of grant application and management as is practical, we manage all the researcher conference travel booking, we provide food at the office, etc.

In short: there’s a big difference between building a system that follows the letter of the law (but not the spirit), and a system that follows the intent behind a large body of law. I agree that the legal system is a large corpus of data containing information about human values and how humans currently want their civilization organized. In order to use that corpus, we need to be able to design systems that reliably act as intended, and I’m not sure how the legal corpus helps with that technical problem (aside from providing lots of training data, which I agree is useful).

In colloquial terms, MIRI is more focused on questions like “if we had a big corpus of information about human values, how could we design a system to learn from that corpus how to act as intended”, and less focused on the lack of corpus.

The reason that we have to work on corrigibility ourselves is that we need advanced learning systems to be corrigible before they’ve finished learning how to behave correctly from a large training corpus. In other words, there are lots of different training corpuses and goal systems where, if the system is fully trained and working correctly, we get corrigibility for free; the difficult part is getting the system to behave corrigibly before it’s smart enough to be doing corrigibility for the “right reasons”.

Agree

We’re submitting “Logical Induction” for publication, yeah. Benya and Jessica (and Stuart Armstrong, a MIRI research associate based at FHI) co-authored papers in a top-10 AI conference this year, UAI, and we plan to publish in similarly high-visibility venues in the future.

We’ve thought about doing a Reddit AMA sometime. It sounds fun, though it would probably need to focus more on basic background questions; EAs have a lot of overlapping knowledge, priorities, styles of thinking, etc. with MIRI, so we can take a lot of stuff for granted here that we couldn’t on /r/science. I usually think of orgs like FHI and Leverhulme CFI and Stuart Russell’s new alignment research center as better-suited to that kind of general outreach.

Yep, we often have a number of non-MIRI folks checking the proofs, math, and citations. I’m still personally fairly involved in the writing process (because I write fast, and because I do what I can to free up the researchers’ time to do other work); this is something I’m working to reduce. Technical writing talent is one of our key bottlenecks; if you like technical writing and are interested in MIRI’s research, get in touch.

The Logical Induction paper involved multiple iterations of writing and reviewing, during which we refined the notation, terminology, proof techniques, theorem statements, etc. We also had a number of others comment on various drafts, pointing out wrong or unclear parts.

I'd mostly put OpenAI in the same category as DeepMind: primarily an AI capabilities organization, but one that's unusually interested in long-term safety issues. OpenAI is young, so it's a bit early to say much about them, but we view them as collaborators and are really happy with "Concrete Problems in AI Safety" (joint work by people at OpenAI, Google Brain, and Stanford). We helped lead a discussion about AI safety at their recent unconference, contributed to some OpenAI Gym environments, and are on good terms with a lot of people there.

Some ways OpenAI's existence adjusts our strategy (so far):

1) OpenAI is in a better position than MIRI to spread basic ideas like 'long-run AI risk is a serious issue.' So this increases our confidence in our plan to scale back outreach, especially outreach toward more skeptical audiences that OpenAI can probably better communicate with.

2) Increasing the number of leading AI research orgs introduces more opportunities for conflicts and arms races, which is a serious risk. So more of our outreach time is spent on trying to encourage collaboration between the big players.

3) On the other hand, OpenAI is a nonprofit with a strong stated interest in encouraging inter-organization collaboration. This suggests OpenAI might be a useful mediator or staging ground for future coordination between leading research groups.

4) The increased interest in long-run safety issues from ML researchers at OpenAI and Google increases the value of building bridges between the alignment and ML communities. This was one factor going into our "Alignment for Advanced ML Systems" agenda.

5) Another important factor is that more dollars going into cutting-edge AI research shortens timelines to AGI, so we put incrementally more attention into research that's more likely to be useful if AGI is developed soon.

Posts or comments on personal Twitter accounts, Facebook walls, etc. should not be assumed to represent any official or consensus MIRI position, unless noted otherwise. I'll echo Rob's comment here that "a good safety approach should be robust to the fact that the designers don’t have all the answers". If an AI project hinges on the research team being completely free from epistemic shortcomings and moral failings, then the project is doomed (and should change how it's doing alignment research).

I suspect we're on the same page about it being important to err in the direction of system designs that don't encourage arms races or other zero-sum conflicts between parties with different object-level beliefs or preferences. See also the CEV discussion above.

I discussed CEV some in this answer. I think the status is about the same: sounds like a vaguely plausible informal goal to shoot for in the very long run, but also very difficult to implement. As Eliezer notes in https://arbital.com/p/cev/, "CEV is rather complicated and meta and hence not intended as something you'd do with the first AI you ever tried to build." The first AGI systems people develop should probably have much more limited capabilities and much more modest goals, to reduce the probability of catastrophic accidents.

See also Nate's paper "The Value Learning Problem."

I share Open Phil’s view on the probability of transformative AI in the next 20 years. The relevant signposts would be answers to questions like “how are current algorithms doing on tasks requiring various capabilities”, “how much did this performance depend on task-specific tweaking on the part of programmers”, “how much is performance projected to improve due to increasing hardware”, and “do many credible AI researchers think that we are close to transformative AI”.

In designing the new ML-focused agenda, we imagined a concrete hypothetical (which isn’t stated explicitly in the paper): what research would we do if we knew we’d have sufficient technology for AGI in about 20 years, and this technology would be qualitatively similar to modern ML technology such as deep learning? So we definitely intend for this research agenda to be relevant to the scenario you describe, and the agenda document goes into more details. Much of this research deals with task-directed AGI, which can be limited (e.g. not self-improving).

Right now, we're trying to ensure that people down the road can build AGI systems that it's technologically possible to align with operators' interests at all. We expect that early systems should be punting on those moral hazards and diffusing them as much as possible, rather than trying to lock in answers to tough philosophical questions on the first go.

That said, we've thought about this some. One proposal by Eliezer years ago was coherent extrapolated volition (CEV), which (roughly) deals with this problem by basing decisions on what we'd do "if counterfactually we knew everything the AI knew; we could think as fast as the AI and consider all the arguments; [and] we knew ourselves perfectly and had better self-control or self-modification ability." We aren't shooting for a CEV-based system right now, but that sounds like a plausible guess about what we'd want researchers to eventually develop, when our institutions and technical knowledge are much more mature.

It's clear that we want to take the interests and preferences of religious extremists into account in making decisions, since they're people too and their welfare matters. (The welfare of non-human sentient beings should be taken into account too.)

You might argue that their welfare matters, but they aren't good sources of moral insight: "it's bad to torture people on a whim, even religious militants" is a moral insight you can already get without consulting a religious militant, and perhaps adding the religious militant's insights is harmful (or just unhelpful).

The idea behind CEV might help here if we can find some reasonable way to aggregate extrapolated preferences. Rather than relying on what people want in today's world, you simulate what people would want if they knew more, were more reflectively consistent, etc. A nice feature of this idea is that ISIS-ish problems might go away, as more knowledge causes more irreligion. A second nice feature of this idea is that many religious extremists' representatives at some hypothetical AI consortium could potentially endorse the same design principles as ZachWeems' representatives: both sides may think that they'll "win the argument" and convince the other side if they accumulate and share enough evidence, so this framework can reduce incentives to get into zero-sum conflicts.

Of course, ISIS won't literally be at the bargaining table. But 'take the CEV of all sentient beings' is a nice Schelling point, which reduces the risk that we'll just replace first-order conflicts about what AI systems should do with second-order conflicts about what people do or don't get to inform the process. If some people get kicked out, then AI designers may be tempted to kick other people out who are less obviously lacking in important moral insights -- or people may just worry more about the possibility that designers will do that, leading to conflict. So there are practical reasons to be egalitarian here even if you don't literally need to simulate every human (or cow, pig, etc.) in order to get good global outcomes.

As Tsvi mentioned, and as Luke has talked about before, we’re not really researching “provable AI”. (I’m not even quite sure what that term would mean.) We are trying to push towards AI systems where the way they reason is principled and understandable. We suspect that that will involve having a good understanding ourselves of how the system performs its reasoning, and when we study different types of reasoning systems we sometimes build models of systems that are trying to prove things as part of how they reason; but that’s very different from trying to make an AI that is “provably X” for some value of X. I personally doubt AGI teams be able to literally prove anything substantial about how well the system will work in practice, though I expect that they will be able to get some decent statistical guarantees.

There are some big difficulties related to the problem of choosing the right objective to optimize, but currently, that’s not where my biggest concerns are. I’m much more concerned with scenarios where AI scientists figure out how to build misaligned AGI systems well before they figure out how to build aligned AGI systems, as that would be a dangerous regime. My top priority is making it the case that the first AGI designs humanity develops are the kinds of system it’s technologically possible to align with operator intentions in practice. (I’ll write more on this subject later.)