Hello Effective Altruism Forum, I am Nate Soares, and I will be here to answer your questions tomorrow, Thursday the 11th of June, 15:00-18:00 US Pacific time. You can post questions here in the interim.
Last week Monday, I took the reins as executive director of the Machine Intelligence Research Institute. MIRI focuses on studying technical problems of long-term AI safety. I'm happy to chat about what that means, why it's important, why we think we can make a difference now, what the open technical problems are, how we approach them, and some of my plans for the future.
I'm also happy to answer questions about my personal history and how I got here, or about personal growth and mindhacking (a subject I touch upon frequently in my blog, Minding Our Way), or about whatever else piques your curiosity. This is an AMA, after all!
EDIT (15:00): All right, I'm here. Dang there are a lot of questions! Let's get this started :-)
EDIT (18:00): Ok, that's a wrap. Thanks, everyone! Those were great questions.
What are some of the most neglected sub-tasks of reducing existential risk? That is, what is no one working on which someone really, really should be?
Policy work / international coordination. Figuring out how to build an aligned AI is only part of the problem. You also need to ensure that an aligned AI is built, and that’s a lot harder to do during an international arms race. (A race to the finish would be pretty bad, I think.)
I’d like to see a lot more people figuring out how to ensure global stability & coordination as we enter a time period that may be fairly dangerous.
What is the top thing you think you'll do differently now that you're Executive Director?
What do you think is the biggest mistake MIRI has made in it's past? How have you learned from it?
What do you think has been the biggest success MIRI has had? How have you learned from that?
What metrics does MIRI use to internally measure its own success?
Congrats on the new position!
My question: what advances does MIRI hope to achieve in the next 5 years?
What question should we be asking you?
Which uncertainties about the trajectory to AI do you regard as of key strategic importance?
Working without concrete feedback, how are you planning on increasing the chance that MIRI's work will be relevant to the AI developers of the future?
That’s a good question: we don’t have a practical AGI to poke at, so why do we expect that we can do work today that’s likely to be relevant many years down the line?
I’ll answer in part with an analogy: Say you went back in time and dropped by to visit Kolmogorov back when he was trying to formalize probability theory, and you asked "working without concrete feedback, how are you planning to increase the chance that your probability theory will be relevant to people trying to reason probabilistically in the future?" It seems like the best response is for him to sort of cock his head and say "well, uh, I’m still trying to formalize what I mean by "chance" and "probability" and so on; once we’ve got those things ironed out, then we can chat."
Similarly, we’re still trying to formalize the theory of advanced agents: right now, if you handed me unlimited computing power, I wouldn’t know how to program it to reliably and "intelligently" pursue a known goal, even a very simple goal, such as "produce as much diamond as possible." There are parts of the problem of designing highly reliable advanced agents that we don’t understand e... (read more)
Asking on behalf of Daniel Satanove, former intern at MIRI (summer 2014):
What do other people who are concerned with AI safety (e.g., Elon Musk, Bill Gates, Stuart Russell, etc.) think the path to friendly AI is? Are there other people who are working directly on Friendly AI research other than MIRI?
Welcome, everybody!
Nate: on behalf of the EA community, thanks very much for showing up here. I think I speak for a lot of EAs when I say that since MIRI has such ambitious goals, it's really valuable to keep things grounded with open conversations about why you're doing what you're doing, and how it's turning out. So I think you've already won a lot of respect by making yourself available to answer questions here! Rest assured, you're not expected to answer every single question!
Everyone else: feel free to ask more questions in the next couple of hours, and to comment, and to upvote the questions you find the most interesting. We're lucky to have Nate around, so enjoy! :)
How does MIRI plan to interface with important AI researchers that disagree with key pieces in the argument for safety?
1) I see a trend in the way new EAs concerned about the far future think about where to donate money that seems dangerous, it goes:
I am an EA and care about impactfulness and neglectedness -> Existential risk dominates my considerations -> AI is the most important risk -> Donate to MIRI.
The last step frequently involves very little thought, it borders on a cached thought.
How would you be conceiving of donating your X-risk money at the moment if MIRI did not exist? Which other researchers or organizations should be being scrutinized by donors who are X-risk concerned, and AI persuaded?
It seems easy to imagine scenarios where MIRI's work is either irrelevant (e.g., mainstream AI research keeps going in a neuromorphic or heuristic trial-and-error direction and eventually "succeeds" that way) or actively harmful (e.g., publishes ideas that eventually help others to build UFAIs). I don't know how to tell whether MIRI's current strategy overall has positive expected impact. What's your approach to this problem?
In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, 5) have argued that MIRI has a medium probability of success.
What is this probability estimate based on and how is success defined?
(Note that I've asked this before, but I'm curious for more perspective.)
To what degree is MIRI now restricted by lack of funding, and is there any amount of funding beyond which you could not make effective use of it?
Among recruiting new talent and having funding for new positions, what is the greatest bottleneck?
What are your plans for taking MIRI to the next level? What is the next level?
Now that MIRI is focused on math research (a good move) and not on outreach, there is less of a role for volunteers and supporters. With the donation from Elon Musk, some of which will presumably get to MIRI, the marginal value of small donations has gone down. How do you plan to keep your supporters engaged and donating? (The alternative, which is perhaps feasible, could be for MIRI to be an independent research institution, without a lot of public engagement, funded by a few big donors.)
What's your response to Peter Hurford's arguments in his article Why I'm Skeptical Of Unproven Causes...?
That post mixes a bunch of different assertions together, let me try to distill a few of them out and answer them in turn:
One of Peter's first (implicit) points is that AI alignment is a speculative cause. I tend to disagree.
Imagine it's 1942. The Manhattan project is well under way, Leo Szilard has shown that it's possible to get a neutron chain reaction, and physicists are hard at work figuring out how to make an atom bomb. You suggest that this might be a fine time to start working on nuclear containment, so that, once humans are done bombing the everloving breath out of each other, they can harness nuclear energy for fun and profit. In this scenario, would nuclear containment be a "speculative cause"?
There are currently thousands of person-hours and billions of dollars going towards increasing AI capabilities every year. To call AI alignment a "speculative cause" in an environment such as this one seems fairly silly to me. In what sense is it speculative to work on improving the safety of the tools that other people are currently building as fast as they can? Now, I suppose you could argue that either (a) AI will never work or (b) it will be safe by defaul... (read more)
1) Your current technical agenda involves creating a math of logical uncertainty and forming world-models out of this. When (if possible) do you predict that such a math will be worked out, and will MIRI's focus move to the value learning problem then?
2) How long do you estimate that formal logic will be the arena in which MIRI's technical work takes place - that is, how long will knowing formal logic be of use to a potential researcher before the research moves to new places?
Hi Nate,
Thanks for the AMA. I’m most curious as to what MIRI’s working definition is for what has intrinsic value. The core worry of MIRI has been that it’s easy to get the AI value problem wrong, to build AIs that don’t value the correct thing. But how do we humans get the value problem right? What should we value?
Max Tegmark alludes to this in Friendly Artificial Intelligence: the Physics Challenge:
... (read more)What is your AI arrival timeline? Once we get AI, how quickly do you think it will self-improve? How likely do you think it is that there will be a singleton vs. many competing AIs?
(1) What is the probability of mankind, or a "good" successor species we turn into, surviving for the next 1000 years? (2) What is the probability of MIRI being the first organization to create an AGI smart enough to, say, be better at computer programming than any human?
1)Which are the implicit assumptions, within MIRI's research agenda, of things that "currently we have absolutely no idea of how to do that, but we are taking this assumption for the time being, and hoping that in the future either a more practical version of this idea will be feasible, or that this version will be a guiding star for practical implementations"?
I mean things like
UDT assumes it's ok for an agent to have a policy ranging over all possible environments and environment histories
The notion of agent used by MIRI assumes to some ex
Asking for a friend:
"What would it take to get hired by MIRI, if not in a capacity as a researcher? What others ways can I volunteer to help MIRI, operationally or otherwise?"
Many years ago, SIAI's outlook seemed to be one of desperation - the world was mad, and probably doomed. Only nine coders, locked in a basement, could save it. Now things seem much more optimistic, the Powers That Be are receptive to AGI risk, and MIRI's job is to help understand the issues. Is this a correct impression? If so, what caused the change?
It appears that the phrase "Friendly AI research" has been replaced by "AI alignment research". Why was that term picked?
How does existential risk affect you emotionally? If negatively, how do you cope?
What do you think of popular portrayals of AI-risk in general? Do you think there's much of a point either in trying to spread broad awareness of the issue? Do you think that any such efforts ultimately do more harm than good, and that we should try to keep AI-risk more secretive?
For example, are things like like Ex Machina, which doesn't really present the full AI arguement, but does make it obvious that AI is a risk, or Wait But Why's AI posts good?
Thanks!
What is your stance on whole brain emulation as a path to a positive singularity?
What are your biggest flaws, skill gaps, areas to grow?
Hi, I'm a software developer with good knowledge of basic algorithms and machine learning techniques. What mathematics and computer science fields should I learn to be able to make a significant impact in solving AGI problem?
Let's assume that an AGI is, indeed, created sometime in the future. Let us also assume that MIRI achieves its goal of essentialy protecting us from the existential dangers that stem from it. My question may well be quite naive, but how likely is it for a totalitarian "New World Order" to seize control of said AGI and use it for their own purposes, deciding who gets to benefit from it and to what degree?
This is something I, myself, get asked a lot and while it takes into account the current state of society which look nothing like the next ones probably will, I can't seem to properly reject as a possibilty.
Do you think a fast takeoff is more likely?
What are MIRI's plans for publication over the next few years, whether peer-reviewed or arxiv-style publications?
More specifically, what are the a) long-term intentions and b) short-term actual plans for the publication of workshop results, and what kind of priority does that have?
Hi Nate!
Daniel Dewey at FHI outlined some strategies to mitigate existential risk from a fast take-off scenario here: http://www.danieldewey.net/fast-takeoff-strategies.pdf
I expect you to agree with the exponential decay model, if not – why?
I would also like your opinion on his four strategic categories, namely:
Thanks for your attention!
Are you single? What are some strategic methods that would make one successful at seducing you? (I'm giving a very liberal interpretation to "I'm also happy to answer questions about (...) whatever else piques your curiosity" :P)
1) What was the length of time between you reading the sequences and doing research on the value alignment problem?
2) What portion of your time will now be spent on technical research? Also, what is Eliezer Yudkowsky spending most of his work-time on? Is he still writing up introductory stuff like he said in the HPMOR author notes?
3) What are any unstated pre-requisites for researching the value-alignment problem that aren't in MIRI's research guide? e.g. could include Real Analysis or particular types of programming ability
What is your best characterisation of Robin Hanson's arguments against FOOM, and what is your analysis of the strengths and weaknesses of his argument?
I remember reading that you had plans to change the world via economic/political influence, and then you realized that existential risk was more important. The same thing happened to me.
What was that experience like for you? How long did it take you to change your mind? Other thoughts?
What are some ways in which you've changed your mind? Recently, important things, things that come to mind, whatever you want.
What path did MIRI's staff take there? How many came from other charities?
Three questions:
1: As a past MIRI researcher, which one of the technical problems in the technical research agenda currently looks like the biggest pain in the ass/the one requiring the most lead time to solve?
2: When you become executive director, will that displace all of your research work, or will you still have a few thought cycles left over to contribute mathematically to workshops/do some part-time research?
3: My current life plan is "speedrun college in 3 years (mostly done), speedrun employment by living in a van and spending under 14k/year s... (read more)
Kieran Allen asks:
I know that in the past LessWrong, HPMOR, and similar community-oriented publications have been a significant source of recruitment for areas that MIRI is interested in, such as rationality, EA, awareness of the AI problem, and actual research associates (including yourself, I think). What, if anything, are you planning to do to further support community engagement of this sort? Specifically, as a LW member I'm interested to know if you have any plans to help LW in some way.
I have a friend studying a masters' degree in artificial intelligence, and he says:
How much does an internship at MIRI as a researcher pay?
Is MIRIs hope/ambition that that CEV (http://wiki.lesswrong.com/wiki/Coherent_Extrapolated_Volition) or something resemblant of CEV will be implement, or is this not something you have a stance on?
(I'm not asking whether you think CEV should be the goal-system of the first superintelligence. I know it's possible to have strategies such as first creating an oracle and then at some later point implement something CEV-like.)
As someone who's spent a significant amount of time thinking about possible rearrangements of civilization, reading On Saving The World was both tantalizing and frustrating (as well as cementing your position as one of the most impressive people I am aware of). I understand building up from the ground, covering all the pre-requisites and inferential distance, would be a huge effort and currently not worth your time, but I feel like even a terse summary without any detailed justifications for suggestions based on of all those years of thought would be highl... (read more)
What are your contrarian beliefs?
Cheeky question:
You probably believe in many strange things that most people do not. Nonetheless, I think you are very clever and trust you a lot. Can you think of any unusual beliefs you have that have implications for asset prices?
There are different inputs needed to advance AI safety: money, research talent, executive talent, and others. How do you see the tradeoff between these resources, and which seems most like a priority right now?
anonymous question from a big fan of yours on tumblr:
"Re: Nate Soares (thanks for doing this btw, it's really nice of you), two questions. First, I understand his ethical system described in his recent "should" series and other posts to be basically a kind of moral relativism; is he comfortable with that label? Second, does he only intend it for a certain subset of humans with agreeable values, or does it apply to all value systems, even ones we would find objectionable?"
(I'm passing on questions without comment from anyone without an e-a.com account or who wants anonymity here. )
What are some of your techniques for doing good research?
So as I understand it, what MIRI is doing now is to think about theoretical issues and strategies and write papers about this, in the hope that the theory you develop can be made use of by others?
Does MIRI think of ever:
Also (feel free to skip this part of the question if it is too big/demanding):
Personally, I ... (read more)
I guess I am way late to the party, but.....
What part of the MIRI research agenda do you think is the most accessible to people with the least background?
How could AI alignment research be made more accessible?
Are there any areas of the current software industry that developing expertise in might be useful to MIRI's research agenda in the future?
Do you believe a terminal value could ever be "rational"? Or is that a Wrong Question?
Hey Nate, congratulations! I think we briefly met in the office in February when I asked Luke about his plans; now it turns out I should have been quizzing you instead!
I have a huge list of questions; basically the same list I asked Seth Baum, actually. Feel free to answer as many or as few as you want. Apologies if you've already written on the subject elsewhere; feel free to just link if so.
What is your current marginal project(s)? How much will they cost, and what's the expected output (if they get funded).
What is the biggest mistake you've made?
What is... (read more)
Will you still be answering questions now, or in future?
I usually see MIRI's goal in its technical agenda is "to ensure that the development of smarter-than-human intelligence has a positive impact on humanity." Is there any chance of expanding this to include all sentient beings? If not, why not? Given that nonhuman animals vastly outnumber the human ones, I would think the most pressing question for AI is its effect on nonhuman animals rather than on human ones.