Chris Olah on working at top AI labs without an undergrad degree

80000_Hours

This is a linkpost for #108 - Chris Olah on working at top AI labs without an undergrad degree. You can listen to the episode on that page, or by subscribing to the '80,000 Hours Podcast' wherever you get podcasts.

In this interview Chris and Rob discuss Chris’ personal passions over the years, including his attempts to reduce what he calls ‘research debt’ by starting a new academic journal called Distill, focused just on explaining existing results unusually clearly. They also cover:

Why highly thoughtful cold emails can be surprisingly effective, but average cold emails do little
Strategies for growing as a researcher
Thinking about research as a market
How Chris thinks about writing outstanding explanations
The concept of ‘micromarriages’ and ‘microbestfriendships’
And much more.

…it’s actually much easier to do unusual things when you’re validated by a third party. …adults in my life totally came around once I was given $100,000 to go and work on stuff, in a way that they really were not supportive beforehand.
–Chris Olah

Key points

Should people go to university?

Chris Olah: I applied for the Thiel Fellowship, which is a program that provides financial support for people under the age of 20 to go and work on ambitious projects or do unusual things, and I got it, and I was like, “Well, I have two options. One is to go back to university, and the other is I can work on whatever I want for two years.” It turns out that wasn’t a difficult decision.
Chris Olah: I had a lot of experience in doing things cutting against pressure, from the previous stuff. But I think at the time, I framed it — and especially framed it to other people — as, well, I can do this for two years, and then I can still go back to university. That seems like an amazing opportunity. One other thing that comes into play here is it’s actually much easier to do unusual things when you’re validated by a third party. I think people, when they hear about the Thiel Fellowship they’re like, “Ah, the high-value thing is that they’re providing funding.” That’s certainly part of it, but I think that actually the higher value thing was actually like, adults in my life totally came around once I was given $100,000 to go and work on stuff, in a way that they really were not supportive in beforehand. I think there’s also just a really big effect in terms of legitimizing an untraditional path and making it easier.
Chris Olah: I get a lot of emails from people asking me if they should go to university. I think it’s maybe the single most common question I get asked. I think for almost everyone who emails me, they should go to university. The reason that I think that is I think that if you want to benefit from going and doing something else, you have to have a lot of, I think, self-discipline and willingness to go and work hard on things, and self-motivation to work hard on things without an external forcing function. I think that often people don’t have this, and then this kind of thing doesn’t work as well for them.
Chris Olah: On the other hand, I think for the people — and maybe to give some more context, — I think a lot of the people who I saw really thrive in the Thiel Fellowship, some had already before the age of 20 done undergrad degrees. So there were those ones. But I think a lot of people had done really significant personal projects involving software or science or something like this. I think that’s actually a pretty good test. If you have been able to, out of self-motivation, go and do your own large personal project — and obviously you are in a privileged enough position to be able to support yourself — then you’re likely to be able to do well in something like the Thiel Fellowship, or taking a year off, or taking a few years off. But if you aren’t, it’ll be much more challenging.
See also Chris’ essay on whether people should go to university.

Lessons from Chris’ unconventional career track

Chris Olah: I think probably the most useful thing I’ve extracted has been thinking about the Pareto frontier of skills. For example, a lot of my early contributions to machine learning were basically being able to create these really helpful illustrations of complicated ideas. What skills did I need to do that? Well, I needed both to understand machine learning, and I needed to be able to draw. I wasn’t an exceptionally good artist or scientific illustrator, and I wasn’t exceptionally knowledgeable about machine learning. But very plausibly, for a while, I was the person in the world who was the best of the intersection of machine learning and drawing. If you think of these two-dimensional plots of different skills, or three-dimensional plots of different skills, and you think about the Pareto frontier, very often society is good at producing people who are optimized for a particular skill set or set of skills that society has really validated as useful.
Chris Olah: We create entire pipelines training people. But I think that often, if you can find useful intersections of skills that aren’t these couple of standard skills, there can be a lot of value. And it’s much easier to go and have a big impact, and often have a big counterfactual impact. When I’m talking to people about their own careers, I often try to frame it in terms of, what are the skills that they’re cultivating, and what do we think the Pareto frontier with regards to these skills looks like? Do we think that there’s places where, rather than going and becoming the world’s best at one skill, they can produce a lot of value by being at an intersection of skills that other people don’t have?
Rob Wiblin: Yeah, that’s really interesting. Thinking about it theoretically, I suppose part of the reason is just that there’s so many combinations of two different things that you could throw together. So the space of possible combinations is vastly larger, and so you have a lot more to choose from. It also means that you could be the only person who’s interested in X and Y, if you choose two things that are sufficiently distant. Then you have a truly unique skill set, and you might just stumble on something that no one else has even tried to find.
Chris Olah: Exactly, and now the problem is the space is exponentially big, and you want to not just find an intersection, but the intersection has to be useful. So you have to have some taste in picking the skills that you develop. But I think that there are lots of opportunities like this, and that often it’s much less competitive than going and being good at one of the skills that society already really values as a thing to optimize for.

Developing research taste and technique

Chris Olah: I think it’s often helpful to divide being a good researcher into two parts. One is taste. So your ability to go and pick good problems and go and pick good avenues to attack those problems, and things like this. The second you might call technique, or execution. Maybe if you picture a chemist working with vials and pipettes and weird things, it’s pretty clear that there’s a whole technique to going and manipulating that laboratory equipment.
Chris Olah: I think that it’s subtler in other fields, but I think that there is something — certainly in machine learning, of the technique of training models, and even just being a good programmer, and doing very minute things of manipulating your code editor, or going and manipulating distributed systems, and stuff like this — I think that there’s a question of how do you develop both of those skills. And for taste, I think that’s probably the hardest one to develop. I tried to come up with a list of exercises that one could do. An example, and I think probably the most useful one, is just write down a list of problems that you think might be important to work on, and then have somebody else, ideally your mentor, go and just rate them one to 10.
Chris Olah: Because one of the really hard things about developing taste is that you have such a slow feedback loop on learning lessons, because you have to go and do the entire project. What you want to do is use a mentor or use somebody else as a cheap proxy for getting feedback, and then if you disagree with their feedback, you can either talk to them about it, or maybe you even want to go and do that experiment. I think that could be useful. I think there’s lots of other things. I think reading about the history of science is helpful. I think going and trying to write just about why you think things are important is helpful. In any case, I think there’s a bunch of exercises there. Then, on the technique side, I actually think the most valuable thing here is working closely with people who have good technique.
Chris Olah: I think actually, at least in machine learning, and probably other computer science disciplines, going and pair programming with people is immensely valuable. I think that there’s a lot of stuff that’s hard to communicate in other forms, but gets passed along when people are pair programming. I think for developing technique, often pair programming is the highest leverage thing to do.
See also Chris’ notes on building research taste.

Cold emails

Chris Olah: I get a lot of cold emails, and 99% of them are terrible. They’re like, “Can you do my homework for me?” or, “Can you answer this basic question that I could Google for one minute and answer?” I think people get this impression that cold emailing doesn’t work, because of course, if you send emails like that, people are overwhelmed and aren’t going to respond. Or, even if you just very generically are like… If you send a nicely written email and you’re like, “I’m trying to get into machine learning. Can you do a half-hour phone call with me to talk about how to do that?” Even that, you’re not very likely to get a response from. But I think the thing that people miss is that if you write really good cold emails, it’s actually not that hard to be the best email I received that week.
Chris Olah: And I think that if you’re willing to invest energy in understanding what a researcher or a group is working on, and you’re specifically referring to their papers, and you have thoughtful questions about things, yeah, I think that people will pay a lot of attention to that. Then I think that it will… It very often works well. I think there’s a big gap in what people mean when they talk about cold emails, and I think that if you’re willing to put in the work, and if you just genuinely really care about what somebody is doing, and have put in the work to understand it, and can talk about it really intelligently… That’s going to come through. It’s a much more compelling reason for the person to talk to you than other things.
Chris Olah: I think there’s a lot of people who are trying to look at how to get into machine learning, and what they do is they send lots of emails to people, or they email famous people. I think what you should actually be doing is trying to figure out who you would be really excited to work with, and really understand their work. Ideally pick somebody who’s a little bit less famous maybe, and then reach out to that person with an email where you’ve put a lot of work into it being clear that you’ve read their work, and connecting your interests to theirs, and things like this. There’s a number of emails that have been really important for me, where I spent a week writing them. I think that was a totally worthwhile investment. I think that’s not how people usually think about cold emails.

Research as a market

Chris Olah: The general idea is, you think of researchers as investing in different research ideas, and if the research idea pans out, and other people don’t grab it before them, then they get some reward from that maybe. Maybe more resources, or just they get a payoff from that in some way. You can see there being this competition to go and grab promising research ideas. I think there’s roughly two strategies that you can play in this market. One is you can work on things where everyone really agrees that they’re important, and that are really popular.
Chris Olah: And what you’re doing when you do that is you’re going and making that little area of the research market a tiny bit more efficient. You’re going and making it so that ideas that are important to get done, get done a little bit more quickly. And I think that is actually genuinely a valuable thing to go and do. If the thing that you’re doing is really important, and you make it happen in expectation a week earlier or a month earlier, that’s really great. But the other strategy you can do is you can try to beat the market. You can try to work on things where you just can see that something is undervalued relative to what most of the community thinks. That’s the thing that I try to do a lot of the time. And there’s lots of reasons why you might be able to beat the market.
Chris Olah: It could be that you just care about things that other people don’t. If you care about safety, or in other areas, if you care about animal welfare, or if you have weirder goals, or different goals than a lot of people, you might be able to beat the market in that way. I think another way, though, is just, if you have some insight that you really believe is true about a problem, and that’s not a widespread insight, then that could be really helpful. That can be… I feel like that’s a lot of what I’m doing, me personally. I think that you can genuinely understand neural networks if you’re willing to input enough energy into trying to figure out what’s going on. It’s a big bet that I’m making, that most other people aren’t making.

Research debt

Chris Olah: I think in many fields, achieving a research-level understanding is like climbing a mountain. There’s all of these ideas that you have to understand and build up towards before you can go into research.
Chris Olah: Mathematics, I think, is a really striking example of this, where there’s just years and years of ideas that you’re probably going to spend climbing to the point where you can do research, because there’s just so much that isn’t piled on top. Then when you get to the top, you go and you pile some more results on top, and you make the mountain higher.
Chris Olah: I think a lot of people are proud of this, because they’re like, ah, the fact that it’s this long pilgrimage to go and get to the point where you can do research, that means that it’s especially profound, and it reflects all of the work that’s been done to date. But I think that, actually, it’s often a reflection that we haven’t put enough work into explaining things, building up really good infrastructure for learning about that field. I think this is going to come in lots of forms. It can be poor expositions, just not good explanations to things. Sometimes it’s just undigested ideas. There’s an idea that’s important, but it hasn’t been really refined to the completed version of that idea. I think it’s very common for there to be bad notations, or just bad definitions of things that make things more complicated, and all these things make it harder to go and understand the topic.
Chris Olah: One analogy that I like is sometimes in software engineering, people talk about technical debt, which is you move really fast to get to that point where you can ship some feature or something like this, and in the process, you write lots of bad code, and it’s really messy and gross, and you have bad variable names, and it isn’t documented, and then it’s hard for other people to build on top of. I think something analogous, a kind of research debt, is endemic in science.

Articles, books, and other media discussed in the show

Chris’ projects and writing

Distill (From Chris: We recently posted an announcement about putting Distill on hiatus, which is germane to this episode’s discussion of publishing and Distill.)
Inceptionism: Going Deeper into Neural Networks
Research Taste Exercises
Research Debt
Do I Need to Go to University?
Micromarriages
Concrete Problems in AI Safety
Understanding LSTM Networks

Transcript

Rob’s intro [00:00:00]

Rob Wiblin: Hi listeners, this is the 80,000 Hours Podcast, where we have unusually in-depth conversations about the world’s most pressing problems, what you can do to solve them, and whether free surgery from someone without an MD is a good deal. I’m Rob Wiblin, Head of Research at 80,000 Hours.

Last week’s episode was Chris Olah’s first podcast ever, and today we’re releasing his second podcast ever — because we ended up having so much to cover it made sense to do more than one recording session and split it in two.

Last week’s episode focused on Chris’ technical work, but this one is about Chris’ life and experiences so far. We talk about things like:

How he rose to the top of the ML field without a university degree
How he got his foot in the door at Google Brain and OpenAI
The journal that he founded — called Distill — and how to explain complex things really well
How to develop good taste in research
Why very thoughtful cold emails can be surprisingly effective, but average cold emails do little
Micromarriages and microbestfriends

…and much more.

Without further ado, I bring you Chris Olah.

The interview begins [00:00:57]

Rob Wiblin: Today, I’m speaking with Chris Olah. Chris is a machine learning researcher currently focused on neural network interpretability. Until last December, he led OpenAI’s interpretability team, but he recently left with some colleagues to help start a new AI lab focused on large models and safety. Before OpenAI, he spent four years at Google Brain developing tools to visualize what’s going on in neural networks. He was hugely influential at Google Brain, being the second author on the launch of the DeepDream article back in 2015. I think the DeepDream images are something that basically just about everyone has seen at this point.

Rob Wiblin: He also helped pioneer feature visualization, activation atlases, building blocks of interpretability, TensorFlow, and he even co-authored the famous paper Concrete Problems in AI Safety. On top of all of that, in 2018, he helped found the academic journal Distill, which is dedicated to publishing clear communication of technical concepts. Chris is himself a writer who is popular among many listeners to the show, and his blog has attracted millions of readers by trying to explain cutting-edge machine learning in highly accessible ways. He’s managed to do all of this without a degree, because he dropped out of college in 2009 to defend a friend against bogus terrorism charges. In 2012, Chris took a $100,000 Thiel Fellowship, a scholarship designed to encourage gifted young people to go straight into research or entrepreneurship rather than go to university.

Rob Wiblin: What an intro. Thanks for coming on the podcast, Chris.

Chris Olah: Thank you for having me, Rob. That is an extremely flattering introduction. I should say, my role in TensorFlow was very minor.

Rob Wiblin: Okay, but all of the rest of it stands.

Chris Olah: This is my first podcast ever. If I get terrified of the medium and never do a podcast episode again, everyone will know who to blame.

Rob Wiblin: Or alternatively, if things go well, we’ll have a fantastic scoop and I get lots of new subscribers. That’s the dream.

Rob Wiblin: Alright, I hope we get to talk about your research into AI interpretability and your many unusual life experiences as just described. But, first, what are you working on at the moment and why do you think it’s important?

Chris Olah: I think one of the craziest things about machine learning is that we have all these systems that can do these amazing things — they can classify images, translate text, write essays, recognize your voice, generate videos… And yet we can’t go and produce these systems directly. No human being knows how to write a computer program directly that does those kinds of things. Instead, we go and produce systems that do these things, and we have no idea what those systems are doing. So the thing that I’ve always felt has just been the question that I’ve been obsessed with, and just feels like the burning question in machine learning to me, is: How in the wide world are these systems going and doing all of these crazy things that we don’t know how to do? I care about that for safety reasons, and honestly, I also just care about it because it seems like this incredibly crazy thing about the world that I just want to understand.

Rob Wiblin: Yeah, that makes a lot of sense. It sounds like, looking at your CV, it’s been something like an eight-year journey for you, working on this problem. Trying to pick away at it, and taking neural networks from being these black boxes to things that we can properly understand and build on.

Chris Olah: Yeah. it’s not the only thing that I’ve done for the last eight years, but it’s definitely been the biggest one, and I’ve tried lots of things. A lot of the things I tried early on didn’t work very well, but over time I think we’ve really developed. Not just me, but lots of other people and lots of collaborators that I’ve worked with have really been able to get to a point where we can actually very significantly understand neural networks, and can actually just look at their weights and read entire algorithms for doing things that we didn’t really know how to do before off of them. That’s been really cool to see.

Defending a falsely accused friend [00:04:24]

Rob Wiblin: Alright, well, we’ll return to some of those techniques later on. Let’s talk now about the very unusual and circuitous career path that you’ve ended up going on. We can see if people can learn any lessons from it, or maybe whether it’s just too weird to be generalizable. Obviously, you now do machine learning research really professionally, but unusually you don’t have a PhD or even an undergraduate degree. What’s the story of how that ended up happening?

Chris Olah: Oh gosh, well, when I was in high school, I became really involved in a community technology space called HackLab. So the G20 came to Toronto, and one of our members was doing security research, where he did things like he was recording where temporary cameras got put up. The police thought this was really suspicious, and they raided his house and they found a hobby chemistry and electronics lab, and they decided he was making bombs. One of the police officers conducting the raid had served in Afghanistan and thought he could recognize explosives, and he misrecognized chemicals as explosives.

Chris Olah: It was obviously a really awful situation for him. I didn’t know him that well, I knew him a bit through HackLab, but a lot of us really rallied in supporting him. I had a lot more flexibility than other people because I was a university student in my first year. It just seemed really important to me to try to support him, and so I started going to court whenever he had bail hearings and stuff like this, and trying to do court support-type stuff to just help him and help his family a little bit, and I took notes and things like this. Then he was under house arrest for one year, and in the second year I took a year off university so I could go and be at his trial full time. He was found innocent of all charges in the end, but that led to me initially dropping out of university.

Rob Wiblin: Yeah, I read a bit about this, there are records from 2008 on your personal website. It is an astonishing and pretty tragic story. You must have been — I’d imagine that I would be as well — just so outraged and disgusted in order to basically drop out of university. It sounded like you were focusing a lot of your time on tracking this case and trying to help him as much as you could, to make sure that this guy didn’t get some long terrorist prison sentence for something that he absolutely hadn’t done.

Chris Olah: Yeah, I don’t know that I actually helped very much. I think really at best I saved him a little bit of money by going and doing some things that saved his lawyers a bit of time here and there. I think the most impactful thing I did was I transcribed an interrogation session and put it on YouTube, and then his lawyers could go and sort of… There’s this nice feature where you can click on a line from the transcript and it’ll jump to that point in the video. I think that was probably the highest impact thing I did. But I think actually it wasn’t primarily outrage at first. There was a lot of outrage, but I think actually the first thing was fear. I was scared for this person, I was scared for myself and for all of my friends that maybe they were going to now come after all of us as, I don’t know, some kind of co-conspirators or something like this.

Chris Olah: It was just a really scary time period, and I think maybe the main thing was just that it seems really important when certain people are systematically stepping away from someone or abandoning someone, it seems really important for other people to rally and to try and support them. Or that was a strong emotional intuition I had. I think that was probably the main motivator for me.

Rob Wiblin: Yeah, I hadn’t thought about that angle as much. I suppose it actually, on some level, required bravery. Bravery or foolishness. Because by taking such a big interest in this, as someone who’s an associate now of an accused terrorist, you’re potentially painting a target on yourself to have the police come after you, or investigate you, or potentially try to engage in reprisals if they’re frustrated with what you’re doing. I imagine your parents were… Maybe they were on board, or maybe they were just horrified at the risk that you were taking?

Chris Olah: All of the adults in my life were very, very deeply worried about this, and pushed me really hard to not do it. But I think any jeopardy that I was in would have already been there just as a result of pre-existing associations. But I didn’t know anything about law at that time, I didn’t know anything about how to think about this type of thing. It just seemed very generally scary.

Rob Wiblin: Yeah, it sounds like… It’s totally understandable why you did this, but it sounds like you think you didn’t make that much of a difference. In retrospect, do you wish that you hadn’t dropped out of university or hadn’t done this stuff? Or is that just maybe an impossible hypothetical to consider? Because then you’d be a different person?

Chris Olah: Yeah, it’s pretty hard to think about it. I do think that there’s a way in which it was altruistic, but not especially effective. On the other hand, it’s hard to feel too bad about it. I think probably this person felt more supported, and his parents probably felt more supported, and that definitely felt good. And I don’t know that my time was super high leverage at that point in my career. But it’s, yeah, I don’t know, it’s hard to think about it. It also just put me on a trajectory that made me do a lot of different things, and maybe in some ways made me more effective later on by becoming more unusual. So it’s hard to think about.

Rob Wiblin: Yeah, it makes sense that when someone is accused of a crime like that, even if people all know that it’s a false charge, people are inclined to just run away from them. Then that means it’s like someone has to take the hit of being willing to stand up and associate with them in order to prevent that person just being completely ruined or having like no social support and feeling like they’re being completely abandoned. It’s a difficult situation. I suppose it’s hard to think about it through an effective altruist lens. But I extremely admire and understand why you ended up doing that.

Rob Wiblin: Maybe I’ve just become inured to stories of misconduct by police — I suppose we don’t want to go into all the details here, because this isn’t ultimately an episode about criminal justice — but this must have shaped your view of national security services somewhat negatively. What did you think at the time, and maybe how has your opinion of that evolved since you were 18?

Chris Olah: Yeah, it definitely soured my opinion of national security agencies. There was a lot of really awful stuff. They threatened his wife at one point to try to get him to confess to something. There was just a lot of stuff where in his trial I thought the prosecution was deeply disingenuous. I should say, in my role as trying to be a supporter, I don’t think that I was always the epitome of intellectual integrity. But I think that if you are a prosecutor who’s trying to put somebody in jail, going and making disingenuous arguments… One that I remember is they argued that the reason he had chemicals that could burn bright colors was he was planning to make a rainbow bomb. After they found that he didn’t have explosives, it became that he had chemicals that could be used to make explosives.

Chris Olah: They found out that he had chemicals that could burn bright colors, and yeah, this argument that it could be a rainbow bomb. I don’t think that anyone who is approaching this in an honest way would make that kind of argument. They really caused, I think, severe harm to this person’s life, and so that really deeply soured my views. And yet, on the other hand, I think over time, especially as I’ve thought more about x-risk and bio stuff and even AI, I’ve gradually had more sympathy also for national security-type concerns. I don’t know, where I wind up is something like, just being really disappointed. Because I really want to be able to trust these organizations, and they clearly are systematically falling short. But also doing something important.

Chris Olah: Also it’s just sad, even from their perspective. They’re just going and burning goodwill left, right, and center by doing all of these things. I put on one hat and I’m really outraged at just the appalling ethics of it, and I put on the other hat and I’m just really sad at the needless waste and burning of goodwill.

Rob Wiblin: Yeah, it’s extraordinary that so much time was spent on this case, where it rapidly became apparent that there was nothing actually to investigate. I know that there’s amazing people within national security, some of the people with the greatest integrity and an incredible work ethic. It’s just national security and law enforcement is like any other industry, where there’s some people who are fantastic and there’s some people who engage in misconduct all the time. It’s just an area that is so important and where there’s so much power that when people do engage in misconduct, the consequences can be extremely unjust and extremely harmful.

Chris Olah: It’s also such a large space that I think it’s probably harder to have a high bar and to really filter things. I don’t know, I’m not an expert, so I don’t want to opine too much.

Rob Wiblin: Well, the police are one of the largest employment groups, so it is a lot of people I suppose. It probably is difficult to have such an extremely high bar such that only people who have demonstrated the greatest integrity in their life can possibly become police officers, because they’re just so many.

Why Chris didn’t go back to university [00:13:21]

Rob Wiblin: Let’s maybe come back to the path that you were taking in your career then. You’d been going to university, but you left in order to do this. I suppose your natural, boring track in life had been somewhat derailed. But you could have probably gone back to university after that, but you decided not to. Why was that?

Chris Olah: Yeah, so when I was doing this court support-type work, there were lots of months where there were lulls, and there just wasn’t anything to do, and so I had lots of free time. I was interested in 3D printers for awhile, so I got really involved in working on 3D printers. First I was designing 3D printers, and then I was working on a startup with a friend for open-source software to go and design objects for 3D printers. A lot of the so-called CAD software that you use to design objects, 3D objects, and we were trying to create open-source tools because I felt that was very important.

Chris Olah: I applied for the Thiel Fellowship, which is a program that provides financial support for people under the age of 20 to go and work on ambitious projects or do unusual things, and I got it, and I was like, “Well, I have two options. One is to go back to university, and the other is I can work on whatever I want for two years.” It turns out that wasn’t a difficult decision.

Rob Wiblin: Yeah, but a lot of people who are considering not going to university and starting a business or doing something else, they feel like they face a lot of pressure, understandably, from other people, from society, from themselves. “You have to go to university.” Was it a difficult decision to make on some level, even if you preferred the path of not going?

Chris Olah: Well, I had a lot of experience in doing things cutting against pressure, from the previous stuff.

Rob Wiblin: Yeah, that’s fair enough.

Chris Olah: But I think at the time, I framed it — and especially framed it to other people — as, well, I can do this for two years, and then I can still go back to university. That seems like an amazing opportunity. One other thing that comes into play here is it’s actually much easier to do unusual things when you’re validated by a third party. I think people, when they hear about the Thiel Fellowship they’re like, “Ah, the high-value thing is that they’re providing funding.” That’s certainly part of it, but I think that actually the higher value thing was actually like, adults in my life totally came around once I was given $100,000 to go and work on stuff, in a way that they really were not supportive in beforehand. I think there’s also just a really big effect in terms of legitimizing an untraditional path and making it easier.

Rob Wiblin: Yeah, it speaks to the signaling component of going to university, where you need some credentials to show that you’re a legitimate person who’s not crazy or not irresponsible. I guess the Thiel Fellowship was providing that as a substitute.

Chris Olah: I think there’s an element of that. I think there’s also just an element of when you do a non-traditional thing, it’s scary for people who care about you. And having some signal that, in fact you’re doing something reasonable…I think can help a lot.

Rob Wiblin: You had this essay on your website where you go into your views on the general path of not going to university and going and doing something else instead. It’s probably the best resource that people can go to if they want to properly learn all of your views on that. Do you just want to summarize briefly, is this something that a lot more people should be considering?

Chris Olah: I get a lot of emails from people asking me if they should go to university. I think it’s maybe the single most common question I get asked. I think for almost everyone who emails me, they should go to university. The reason that I think that is I think that if you want to benefit from going and doing something else, you have to have a lot of, I think, self-discipline and willingness to go and work hard on things, and self-motivation to work hard on things without an external forcing function. I think that often people don’t have this, and then this kind of thing doesn’t work as well for them.

Chris Olah: On the other hand, I think for the people — and maybe to give some more context, I think a lot of the people who I saw really thrive in the Thiel Fellowship, some had already before the age of 20 done undergrad degrees. So there were those ones. But I think a lot of people had done really significant personal projects involving software or science or something like this. I think that’s actually a pretty good test. If you have been able to, out of self-motivation, go and do your own large personal project — and obviously you are in a privileged enough position to be able to support yourself — then you’re likely to be able to do well in something like the Thiel Fellowship, or taking a year off, or taking a few years off. But if you aren’t, it’ll be much more challenging.

Rob Wiblin: Yeah, to what degree is this a path just for really talented people? I think it’s fair to say that like 20-year-old Chris Olah had an awful lot of potential. Some of the other people who I’ve seen thrive not going to university, it just seems like they were on a rocketship to begin with. It makes me wonder, if you’re someone who’s very talented, do you need to have those credentials in order to get your foot in the door in the careers that you want?

Chris Olah: My guess would be that the self-motivation thing is bigger than anything else. But I also just don’t feel that qualified to opine on how useful university would be to people who are very different from me. For example, I can imagine a world where actually for many people who are not going to pursue really intellectually challenging careers, maybe actually going to trade school might be more effective, or things like that. I could imagine there being like different reasons why not going to university might make sense for people who are very different from me. But it’s harder for me to speak to that.

Rob Wiblin: Has not going to university ever created issues with being taken seriously later down the line, or possibly immigration could be a concern as well?

Chris Olah: Yeah, so I think immigration is an underrated concern with this. Almost all visas for the U.S. require you to have an undergrad degree. I’m on this really weird ‘Alien of extraordinary ability‘ visa, which just doesn’t list it. But I think that’s really the main option you have if you don’t have an undergrad degree.

Chris Olah: I think that there’s also some social challenges, where I think actually something that I undervalued when I did not go to university was it makes it so much easier to have friendships and romantic relationships as a young person. I was cut off from a lot of those opportunities, and I think that was actually a really significant cost that I didn’t understand at the time.

Switching to machine learning [00:19:33]

Rob Wiblin: Yes, so tell us a bit more about the Thiel Fellowship, and to what extent having the social group there substituted for university perhaps?

Chris Olah: Well, it might have been more so if I’d been able to live in the Bay Area. I visited at that time, periodically, but I was still living in Canada, since I didn’t have a visa and couldn’t live in the U.S. It might’ve done more if I had been able to be there. I did make some friends, and I think that was valuable. I guess more generally it was just a really incredible period of being able to go and pursue things that I was excited about. I started with 3D printers, and I also was just doing lots of random side projects. Then after a while, about a year, I switched from 3D printers to machine learning, and that was a fairly big pivot.

Rob Wiblin: Yeah, what caused you to do that?

Chris Olah: Well, there were a few things. One is just that I had the opportunity to do so because someone I knew in Toronto, Michael Nielsen, was writing a textbook on neural networks. And he ran a seminar series to practice for writing his textbook. So I got exposed to a lot of ideas and learning just as the field was really just on the verge of taking off, right after the results from Krizhevsky in 2012. So I had the opportunity, I was aware that it was this really exciting thing, and I became very excited about it. And my main collaborator on the 3D printing stuff dropped out. Another thing that contributed a little bit was that I actually happened to meet Holden Karnofsky, of GiveWell and Open Philanthropy.

Rob Wiblin: That would have been pretty early on.

Chris Olah: Yeah, this was in 2012 or 2013. The reason that I met him was I was friends with Dario Amodei, who at that time was a grad student. And Dario, somehow, despite being on a grad student’s stipend, was giving what was probably a fiscally irresponsible amount of his stipend to GiveWell, and that led to him meeting Holden. I ended up getting dinner with Holden and Dario. During the dinner I pitched Holden on how I was working on 3D printers, and how I thought if we can have open-source tools for 3D printers we’d be able to go and bring 3D printers to everyone, and maybe we’d end scarcity and stuff like this. And Holden just really shot it down. At the time I was very miffed.

Chris Olah: Actually, the thing that Holden said was, I don’t think this is valuable, but it seems like a great way for you to develop skills. At the time, I was really miffed. But in retrospect, it was actually a very useful thing for him to say. I think this is a common theme of talking to Holden. I feel miffed at him sometimes, and then very often I’m very grateful for the things that he says.

Rob Wiblin: Yeah, what was his case against working on open-source… What do you call it, like local manufacturing? What’s the term for this?

Chris Olah: I was running open-source CAD tools. I don’t remember the exact argument that he made. I think he was just skeptical of a lot of things, and I think that there were a lot of things to be skeptical of. Not the least of which was whether 3D printers would actually be able to become really widespread at low cost in the way that I was envisioning, and I think also just whether these open-source tools were actually a critical blocker. Which in retrospect, I don’t think they were.

Rob Wiblin: Yeah, people have been talking about 3D printers for a long time, and I’ve always thought, I just don’t know how many things like that I want to make. How many things can be 3D printed that I’m actually that interested in producing for myself? Maybe I’m going to be proven wrong one day.

Chris Olah: Well, so I think the really compelling story is if you can create a 3D printer that can print itself, and can also print a wide variety of objects, you can have this feedback loop of making it really easy to go and spread them. And then anything that you can print, you can provide freely to everyone. You have these printers that you can like 80% print or something like this. Then there’s the remaining ‘vitamins’ that you can’t print. But the vitamins are all the hard parts, and so it’s an illusion that you’re making progress on this, I think, to a significant extent.

Chris Olah: I don’t know, I haven’t been involved in this space for a long time. But at one point I had a lot of fantasies about this, and I designed a 3D-printable vacuum cleaner, and worked on how you can make microscopes with 3D printers. In retrospect, it wasn’t very useful.

Rob Wiblin: Yeah. What do you think of filling this role where you shoot down someone’s plans? It’s a difficult one because you come across as, potentially, kind of an asshole.

Chris Olah: Yes.

Rob Wiblin: It’s like, maybe you’re not really in a position to confidently tell people that the idea is bad. I think onlookers don’t like you when you’re saying that someone’s plans are bad, and the person themself might object to it, and you run the risk of being wrong. And yet it does seem like sometimes it’s just really important to have someone say, “No, this is a dumb idea.” It’s a balancing act.

Chris Olah: I think it is really important. It’s something that I wish I was better at. Personally I really struggle to give people negative feedback. When I’m in a management role, I really push myself to do that, but it’s hard. I really admire some people who can just be like, “Yeah, that’s nonsense.” Just really frankly say it. I think you ideally couple it with support. Holden’s message wasn’t just, “This is a dumb idea.” It was, “This doesn’t seem like a very good idea, but you’re developing lots of good skills and you’ll probably do something useful in the future.”

Rob Wiblin: Yeah, I’ve tried to play this role a handful of times. I’ve heard someone’s plan and just been like, “No, this is terrible. You’ve really got to change it because I don’t see this having any impact.” I’ve gotten a mixed reception, I think it’s fair to say. So I feel a little bit more cautious about doing that now. I don’t know, maybe I should go back, after this anecdote, I should go back—

Chris Olah: I think it probably took a few years for me to totally come around on Holden not just being a little bit annoying at the time.

Rob Wiblin: Yeah. Did he pitch you on going into machine learning? Was he like, “Machine learning is the future?”

Chris Olah: No, not at all. I was completely separately excited about machine learning for other reasons at the time.

Rob Wiblin: Okay, so Holden, among other factors, convinced you that 3D printing maybe wasn’t the most important thing to be working on. Then, independently, you were getting excited by machine learning, which was beginning to take off at that time?

Chris Olah: Yeah, or deep learning was beginning to take off at that time.

Rob Wiblin: Yeah, interesting.

Chris Olah: Yeah, the thing that I just couldn’t get over, and I was really obsessed with, was we have these systems that can do these things that no human being knows how to write a computer program to do. And no one knows what’s going on inside of them! That question just really hooked me, and I couldn’t get it out of my head. And it became a big motivator for me.

Rob Wiblin: Interesting. Yeah, I suppose I’ve just always taken it for granted that neural nets are these combinations of nodes and weights and like it’s a bunch of numbers, and it does some stuff, and we don’t understand how it works. But that is, from one point of view, that is crazy.

Chris Olah: It’s crazy. It’s completely bonkers that we have these systems. Like in any other area, you’d say that it’s harder to go and create a system that automatically builds your systems for you than to go and get the system that can do the thing that’s hard. But, in this case, we have no idea how to go and produce the system, and people have certainly tried. For essentially all of the machine learning tasks we talk about, people have tried to go and hand-build systems that do this. We can’t build systems that can go and do the things that neural networks do. Even when we try really hard to do it by hand. Yet somehow they automatically form. It’s just truly a crazy fact of the world.

Rob Wiblin: Yeah, so we’re making them indirectly by programming a computer to say, “Well, try to do this thing,” and then when it fails to do it, we’re like, “Well, you would have done a little bit better if these numbers were a bit different in this way.” So then we’ll try again, and then we’ll change the numbers again, and just keep going until it works. But, yeah, at the end we end up with a whole bunch of numbers and we’re just like well, we don’t really know how to make sense of this. We don’t even know what any… Or at least in the past, as we’ll get to, we didn’t know what all the different parts did.

Chris Olah: Yeah, somehow those numbers correspond to a computer program. What’s going on inside it? I really want to know.

Rob Wiblin: Okay, so you were particularly—

Chris Olah: …And 19-year-old Christopher really wanted to know.

Rob Wiblin: Nice, okay.

Chris Olah: …Hasn’t really changed.

How Chris got his foot in the door in machine learning [00:27:34]

Rob Wiblin: What did you do next to be able to get into this? I suppose the idea that someone now could get a job at one of these AI labs just with a high school degree… I don’t know whether it was easier then, or whether you just particularly stood out, but how did you manage to get your foot in the door? Because people would often think that’s the stuff that’s potentially really hard if you’ve taken this unconventional path.

Chris Olah: I should say I’m not the only one who doesn’t have credentials and is at these labs. I think Alec Radford doesn’t have a PhD. I’m not sure that he has an undergrad degree. Then there’s a number of other researchers who are like this, so I’m not unique in it. For me, I was really lucky that I got to work with Michael Nielsen, and I think he basically treated me like one of his grad students for six months or a year. And that was a really helpful experience, and I think just helped me mature a ton as a researcher. Then I started cold emailing different labs and asking if I could visit. Yoshua Bengio put out a call for PhD students, and I wrote to him. I spent about a week writing him an email asking if he’d consider me.

Chris Olah: He ended up considering me, and actually I ended up getting accepted and I didn’t end up going. I started visiting different research groups and giving talks on research that I’d done. I had gotten some very modestly interesting results, and… I think in a lot of ways the field was much smaller, and it was growing — I think relative to its present size, it was growing very rapidly. So that also did make it easier to get into.

Rob Wiblin: Yeah, interesting. I guess your first serious role was an internship at Google Brain, is that right?

Chris Olah: Yeah, so I gave a talk at Google and Jeff Dean very kindly offered to take me on as an intern. I ended up doing an internship at Google, and it went on and on and on, and I ended up being an intern for an entire two years.

Rob Wiblin: Is that common?

Chris Olah: Apparently I’m not the person who did the longest internship in Google’s history, but I think I was a good competitor for that position.

Rob Wiblin: Were you doing this extended internship in lieu of doing other formal training? Was this the place where you learned all the things you needed to know in order to get properly hired?

Chris Olah: Yeah, I think I learned a lot through it. I also learned a lot from Michael and from self-study and stuff like this. But, yeah, I think there’s probably a significant element of that. I was just so lucky to be surrounded by so many really wonderful people who were really generous with their time, and especially, with doing different stuff. Google Brain was about 30 people at a time. Rather than building neural networks to go and do different things — I did some of that, and then I got to be involved in lots of projects, and did some work with generative models and contributed a little bit to TensorFlow and things like this — but the main thing I was doing was just trying to figure out what was going on inside these neural networks.

Chris Olah: And really very few people were going and thinking about that at that time. Matthew Zeiler did a little bit of work, although he sort of stopped. Alex Mordvintsev, who would become a close collaborator of mine, was doing some work in this space. I just spent a lot of my time writing blog posts about machine learning and trying to visualize them and trying to understand what was going on and trying lots of things that didn’t work very well.

DeepDream [00:30:52]

Rob Wiblin: Yeah, what were some of the highlights, or the most interesting things that you did in… You were at Google Brain for six years or something like that?

Chris Olah: Yeah, five-ish, five-something years, maybe six.

Rob Wiblin: Yeah, what were some of the highlights?

Chris Olah: Probably the craziest thing was DeepDream. So the really early DeepDream results, Alex Mordvintsev discovered. And he gave a talk and I was just so excited. I dumped all the projects that I was doing and got involved. It was just electrifyingly exciting. The idea is that you just try to make the neurons in a layer of a neural network activate by going and optimizing the input image. You can have these, if you haven’t seen them, these hallucinogenic images that are full of dog slugs and psychedelic colors and things like this. Neural networks seem like magic, but this seems like really crazy magic, and I just spent hours and hours and hours trying different things. I was trying to visualize different layers, and trying to go and fiddle with what we are doing in different ways.

Chris Olah: I found out that you could apply this to going and visualizing individual classes, and we discovered things like part of how it recognizes a barbell is looking for arms attached to the barbell. But it was just this really exciting moment of really intense mystery, and that was really cool.

Rob Wiblin: Yeah, I feel like everyone has seen the DeepDream images. I imagine the audience for the technical paper was a lot smaller than the audience for the images, but it was an incredible hit. Do you want to just describe how those images were produced? It seems like you’re reversing the neural network and then getting it to spit out an image, rather than classify something.

Chris Olah: Yeah, so the idea is that you go and you fiddle with the input image to get an image that causes the neurons in a given layer to fire a lot. As neurons begin to fire, you try to go and cause those neurons to fire more and more and more. Later on we’d come up with this… One of the things that I always feel a little sad about with DeepDream is we got all this attention initially, but I feel like we didn’t really understand the results in some important way. We didn’t really understand what we had discovered.

Chris Olah: I sometimes think about it as like, at some point in the invention of the microscope, somebody must’ve found a distorted piece of glass and realized that they could go and hold it up to things and see these distorted images, and that the distorted images have these parts that are enlarged and they can see things that they couldn’t see before. I feel like DeepDream was that stage, and then later we turned it into feature visualization and developed… Of course, this wasn’t just our work, we were building on work that other people did as well… That allowed us to have these lenses that allowed us to very clearly see and really understand a very powerful tool for understanding different parts of neural networks. But that came after the attention from DeepDream.

Rob Wiblin: Was the flood of attention helpful at all for getting taken seriously, or getting more resources? I don’t know to what degree the ML community respects creating a tool to make amazing images that people can stick up on their social media.

Chris Olah: I think people were really excited for a month or two. The thing that I remember most was getting lots of emails from people in psychology who wanted to investigate this with us. I remember getting an email from some really serious-looking professor about how they wanted to investigate how these seemed similar to psychedelic images, like could we do some joint collaboration to figure it out. I never followed up on that. But I think that a lot of it wasn’t as helpful. And maybe just because we — from my perspective at least — didn’t yet really understand the results deeply, we weren’t really able to funnel it into something that was really helpful for advancing an agenda of really understanding the neural networks, I think.

Rob Wiblin: What was it like working with so many people who I imagine were a whole bunch older and more senior than you?

Chris Olah: It was an interesting experience. Yeah, so for a while I was the youngest person, because even the interns were PhD students who were quite a bit older than me. Actually, at the time, many of my… You know it was only 30 people, and several of my colleagues had children who were my age. That was also fun. But, actually, just everybody was super sweet and was just really lovely to work with. I think the thing that was slightly challenging, and it became more challenging after I became full time, was trying to interact with my peers. I’d have experiences, for instance, where I’d try to hang out with PhD students and then they’d be like, “Dr. Olah” — because everybody assumes that I have a PhD — “Can I be your intern?” Or something like this.

Chris Olah: If you’re going into an environment really hoping that you can have peer-style interactions and become friends with people, it’s actually really disappointing when that happens, and a pretty strange experience.

Rob Wiblin: Interesting. I suppose if you spend all your time hanging out with 30 and 40 year olds as a 20 year old, do you just end up talking a lot about, I don’t know, male baldness, and how to buy a house? Like, “Let’s talk about our kitchen renovations” or something like that. I feel like the things that people talk about are potentially quite different.

Chris Olah: I mostly spoke with people about research. I tried to have a few conversations about effective altruism, and I think people were just confused by me.

Founding a team focused on circuits [00:35:58]

Rob Wiblin: So you’re spending these five years at Google Brain, and you had a whole lot of very widely read articles there. You also founded this academic journal, Distill. Do you want to flesh out maybe the rest of what happened at Google Brain, and what’s happened since then?

Chris Olah: Well, I think my career trajectory probably became less unusual and therefore less interesting probably around that point. But, yeah, after a few years at Google, I started to have a much stronger vision for how interpretability research could be pursued, and a very idiosyncratic one. I really wanted to try and build up a research group doing that type of interpretability research. I also was very interested in this idea of open-notebook science. I didn’t really see any downside to being very public about interpretability research. I just thought it’d be nice if we could just share it as we’re doing it, and try to involve other people.

Chris Olah: I talked a lot with my manager and Jeff, and they were both actually really incredibly supportive. And we just all agreed that actually Google probably wasn’t the best place for me to pursue that. So I ended up going to OpenAI and founding a team, the Clarity team, going and doing this kind of interpretability research there. That was a really positive and really incredible experience for me.

Rob Wiblin: Yeah, it seems like you’ve tracked the… I suppose this clarity or interpretability field didn’t exist when you joined. You virtually founded it, or at least you were there around the founding, and now you’ve lived to see it become a meaningful fraction of research into ML. Something that’s widely accepted and people that are interested in.

Chris Olah: Well, I think right now a lot of people are doing lots of different things called interpretability research, and lots of other people contributed to creating lots of different branches and directions in that space. But I’ve had a very particular vision for what the type of work I’m most excited about in this space is. I’ve been really fortunate to be able to help build that out, and I think just really lucky to be involved early on. I got to work with a bunch of really amazing collaborators at OpenAI, and we pursued what we called the ‘circuits agenda,’ which was the attempt to go and fully understand neural networks.

Lessons from Chris’ unconventional career track [00:38:13]

Rob Wiblin: Alright, let’s look at the unconventional career track that you’ve been on in general from 2008 through 2015. What do you think we could learn from this experience? Are there any lessons that listeners can draw as to their own experience? Maybe they should go and defend one of their friends from terrorism charges? Is there anything that people could learn, or is it just too weird?

Chris Olah: Well, I think probably the most useful thing I’ve extracted has been thinking about the Pareto frontier of skills. For example, a lot of my early contributions to machine learning were basically being able to create these really helpful illustrations of complicated ideas. What skills did I need to do that? Well, I needed both to understand machine learning, and I needed to be able to draw. I wasn’t an exceptionally good artist or scientific illustrator, and I wasn’t exceptionally knowledgeable about machine learning. But very plausibly, for a while, I was the person in the world who was the best of the intersection of machine learning and drawing. If you think of these two-dimensional plots of different skills, or three-dimensional plots of different skills, and you think about the Pareto frontier, very often society is good at producing people who are optimized for a particular skill set or set of skills that society has really validated as useful.

Chris Olah: We create entire pipelines training people. But I think that often, if you can find useful intersections of skills that aren’t these couple of standard skills, there can be a lot of value. And it’s much easier to go and have a big impact, and often have a big counterfactual impact. When I’m talking to people about their own careers, I often try to frame it in terms of, what are the skills that they’re cultivating, and what do we think the Pareto frontier with regards to these skills looks like? Do we think that there’s places where, rather than going and becoming the world’s best at one skill, they can produce a lot of value by being at an intersection of skills that other people don’t have?

Rob Wiblin: Yeah, that’s really interesting. Thinking about it theoretically, I suppose part of the reason is just that there’s so many combinations of two different things that you could throw together. So the space of possible combinations is vastly larger, and so you have a lot more to choose from. It also means that you could be the only person who’s interested in X and Y, if you choose two things that are sufficiently distant. Then you have a truly unique skill set, and you might just stumble on something that no one else has even tried to find.

Chris Olah: Exactly, and now the problem is the space is exponentially big, and you want to not just find an intersection, but the intersection has to be useful. So you have to have some taste in picking the skills that you develop. But I think that there are lots of opportunities like this, and that often it’s much less competitive than going and being good at one of the skills that society already really values as a thing to optimize for.

Rob Wiblin: Yeah. I guess you could choose the wrong combination, just where there aren’t that many complementarities between the two options. You also might fall between the cracks, I suppose, of existing disciplines. Or, a common complaint that people have about doing interdisciplinary work is that if you’re doing philosophy of economics, the economics department doesn’t like you and the philosophy department doesn’t like you, and no one really feels like you’re one of them.

Chris Olah: I guess I want to distinguish this a little bit from interdisciplinary work, which I think is something slightly different. When I was doing this scientific illustration of machine learning, it was really a pure machine learning contribution. It was something that was valuable to the machine learning community and targeted at the machine learning community. I think that there is a distinction, I think, between—

Rob Wiblin: …skills, and maybe bodies of knowledge?

Chris Olah: Yeah. You could do something that was more cross-disciplinary, like you could use machine learning for scientific drawing, or something like this, but I think that’s not really what I was doing.

Rob Wiblin: Yeah. Okay. So I suppose the 80,000 Hours Podcast might be an example of that. We’re at the intersection of being knowledgeable about effective altruist ideas and research, and, I guess, doing interviews and doing media, and communicating stuff. That’s maybe the unique selling point of this podcast. I suppose, do you have examples of classic skills that you can throw onto something and then maybe produce something interesting that others haven’t found?

Chris Olah: I have a favorite example of this phenomenon, which I’ll probably both slightly mistake and I actually owe to Michael Nielsen. But my understanding is that Richard Feynman, I guess when he… I don’t know much about physics, but he had to do all this work… I guess physicists were doing all this work going and solving complex integrals. And the usual set of techniques was to go and use tricks from complex analysis, analytic extensions, and stuff like this, and try to go and solve the integrals that way. And Feynman didn’t really know these complex analysis tools very well, but he had all of these weird tools around fractional calculus and stuff like this, and he used those instead.

Chris Olah: Maybe this gets… It’s not even that weird of a skill set, but in having a different skill set than his colleagues, he was able to have more counterfactual impact, and go and solve problems that other people couldn’t. It’s not that his tools were better, it’s just that lots of people were already trying with the other tools. And he brought a different set of tools to the table.

Rob Wiblin: Yeah. I guess some options might be knowing a lot about some technical area, plus, say, knowing about operations in organizations, or knowing about how to do business, or knowing about money, or knowing how to manage people. Maybe those are combinations…

Chris Olah: Yeah. I think people management plus technical skills is a huge superpower. I think it’s something that I am trying to become good at. I think the people who I see who are really good at it, I think are… Yeah, it’s a really amazing thing.

Rob Wiblin: Yeah. They end up very sought after.

Chris Olah: Yeah. I think any communication plus technical skill. I think web development plus science is actually really underrated. I think that often being able to build interactive interfaces allows you to go and… Well, I guess the basic pitch is, I think a lot of scientists are drawn towards being very reductionist… And maybe this is more for machine learning than other fields, I’m not sure. But they tend to go and look for summary statistics, because you can easily work with summary statistics, and make line plots and things like this. I think if you instead are able to go and create interactive tools and explore things, you tend to just interact with the data in a different way. I think there’s actually just something where… At least in machine learning, I think there’s a lot of value that gets left on the table. And I suspect elsewhere as well.

Rob Wiblin: Yeah. Do you think that this might be slightly a Bay Area phenomenon? The thing of people really appreciating people who have quirky skills that are combined, and that maybe if you were in a more conservative social situation, maybe it would be more risky? Or is that wrong?

Chris Olah: Yeah. That might be true. Although I think in a lot of cases, if you can demonstrate that your intersection of skills produces value, I think that’s really the critical thing. Once an organization is getting value out of your intersection of skills, whether it’s weird or not probably isn’t going to be the critical thing

Rob Wiblin: Yeah. It’s not going to be the make-or-break issue. I guess that’s maybe one other lesson, is that if you can show people that you can do stuff directly, then often you can rout around credentials. I think that is quite often true. The thing is, it actually is potentially quite hard to figure out how to produce value and how to do a good job without having the training. It requires someone who really has either just a lot of raw ability, or a lot of focus, or a lot of discipline to do things outside of a structured environment.

Chris Olah: Absolutely. I think it also depends a lot on the discipline. Practicing law without credentials isn’t something that’s going to fly. I think, depending on the discipline that you’re working in, and I guess this is sort of cutting the other way from my previous answer, but, how flexible the discipline is… I think this was actually a little bit of a different question than the intersection of skills thing. You can get a PhD and have an odd intersection of skills, perhaps.

Rob Wiblin: Yeah. I think probably people aren’t just going to learn medicine through apprenticeships. For understandable reasons, it’s a somewhat credential-filled field.

Chris Olah: I would be nervous to go and have a doctor who did not have an MD.

Rob Wiblin: Yeah, it reminds me of how you can get a free haircut if you’re willing to be someone’s first victim as a hairstylist. Perhaps it’s more difficult in a surgical environment.

Is it important to go to grad school or work at the best lab? [00:46:01]

Rob Wiblin: We’ve talked a bunch about whether people should go and do undergraduate degrees, but how do you feel about going to grad school? Is the picture very different?

Chris Olah: So again, I don’t know how much my advice generalizes across fields, but at least for machine learning, I usually encourage people to just ask where they can go and do the best research. For context, the question that I usually hear people asking when they’re considering going and doing a PhD is they’re like, “Well, should I just go to an industry lab and do research there? Or should I go and do a PhD, and go on and develop my machine learning there?” I think the essential thing isn’t whether it’s a PhD or not. The essential thing is how much do you think you’re going to learn from the people you’re going to be around, and just how good of a research environment is it going to be?

Chris Olah: For some people, I think quite often, going and doing a PhD will get them more mentorship and get them into a research environment that’s more suited to their tastes than the opportunities they have in industry. But for other people, it’s the other way. I think it’s probably better to try and compare concrete questions. I mean, I guess a very classic piece of advice that people give about doing PhDs is to think really hard about the group that you’re going to be working with, and try to understand them. I think, just like you should compare specific research groups when you’re thinking about doing a PhD, you should also, if you’re considering industry as well, consider the particular industry group that you would be working in, and what you would get out of working there.

Rob Wiblin: Yeah. How snooty Is ML about whether you went to the right university, or whether you were working at the best lab?

Chris Olah: I think not very. I mean, it probably depends from institution to institution, but I think most institutions, if you have impressive results, that’s the main thing they’re going to care about. I actually had one university encourage me to apply for a professorship at one point. I was very surprised by this, since I don’t have an undergrad degree or a PhD. I think of universities as being very, very traditional, so I think if universities are willing to consider stuff like that when people do good research, I think that’s a pretty strong signal that a lot of institutions are willing to focus on people’s research rather than their employment history or where they did their PhD.

Rob Wiblin: Yeah. It seems to me like maybe ML as an academic discipline is pretty unusual. It seems it’s routing around a whole bunch of the typical norms. Most researchers now just present at conferences, right, and then just people—

Chris Olah: That’s not just machine learning, that’s all of computer science.

Rob Wiblin: Ah, that’s all of computer science. I see, okay.

Chris Olah: Yeah. Computer science, by and large, doesn’t use journals as much, and relies a lot more on conferences.

Rob Wiblin: Interesting. Do you know how that happened?

Chris Olah: Not sure. I do know it leads to strange situations in academia, where somebody who’s looking at the career of a computer scientist might be like, “Oh, they just publish in conferences. They’re not doing good work.” But in fact, that’s just the norm.

Rob Wiblin: Yeah. Interesting.

Chris Olah: And I think that’s some areas of physics too. Machine learning has a significant fraction of people just publishing on arXiv and not publishing in a venue at all, but I think there’s areas of physics that have taken it much further.

Rob Wiblin: Wow.

Chris Olah: My impression is there’s some areas of physics where people aren’t really using… By and large, they aren’t using conferences or journals, or any peer-reviewed venue, and are just putting things on arXiv.

Rob Wiblin: Wow. They’re living the dream.

Chris Olah: They’re living the dream!

Rob Wiblin: I’ve said I’m really not sure what value academic publications are providing, because it doesn’t seem like peer review is doing that great of a job at clearing out bad papers. I mean, presumably it does something, but from what I understand, the test/retest validity of peer review isn’t that strong. So even if you submit the same paper to a journal twice, there’s a high chance it’ll be rejected once and accepted another time, which is a bit of a red flag.

Chris Olah: Yeah. NeurIPS did this test where they peer reviewed some subset of papers twice, and they found that if one review process accepted it, the other review process had a 50% chance of accepting it. Which, because they are accepting less than half of the papers… It’s not as bad as it sounds, but it’s still a very, very noisy process.

Rob Wiblin: Yeah. It seems like, if these fields have dispensed with the traditional journal system, then we could maybe learn from them how much value the journals were providing in the first place by seeing whether this has been a disaster. Maybe this is something I should look into.

Chris Olah: Yeah.

Strategies for growing as a researcher [00:50:17]

Rob Wiblin: I know you’ve written this article about how people can develop good taste in what to research and how to go about it. I actually didn’t have time to read that one to prepare for this interview. Could you give people a summary of your views there?

Chris Olah: Sure. First of all, I don’t consider myself at all an expert on this. This is just what’s worked for me, and when I’ve been mentoring people, things that I’ve found helpful. But I think it’s often helpful to divide being a good researcher into two parts. One is taste. So your ability to go and pick good problems and go and pick good avenues to attack those problems, and things like this. The second you might call technique, or execution. Maybe if you picture a chemist working with vials and pipettes and weird things, it’s pretty clear that there’s a whole technique to going and manipulating that laboratory equipment.

Chris Olah: I think that it’s subtler in other fields, but I think that there is something — certainly in machine learning, of the technique of training models, and even just being a good programmer, and doing very minute things of manipulating your code editor, or going and manipulating distributed systems, and stuff like this — I think that there’s a question of how do you develop both of those skills. And for taste, I think that’s probably the hardest one to develop. I tried to come up with a list of exercises that one could do. An example, and I think probably the most useful one, is just write down a list of problems that you think might be important to work on, and then have somebody else, ideally your mentor, go and just rate them one to 10.

Chris Olah: Because one of the really hard things about developing taste is that you have such a slow feedback loop on learning lessons, because you have to go and do the entire project. What you want to do is use a mentor or use somebody else as a cheap proxy for getting feedback, and then if you disagree with their feedback, you can either talk to them about it, or maybe you even want to go and do that experiment. I think that could be useful. I think there’s lots of other things. I think reading about the history of science is helpful. I think going and trying to write just about why you think things are important is helpful. In any case, I think there’s a bunch of exercises there. Then, on the technique side, I actually think the most valuable thing here is working closely with people who have good technique.

Chris Olah: I think actually, at least in machine learning, and probably other computer science disciplines, going and pair programming with people is immensely valuable. I think that there’s a lot of stuff that’s hard to communicate in other forms, but gets passed along when people are pair programming. I think for developing technique, often pair programming is the highest leverage thing to do.

Rob Wiblin: That’s really interesting. I’ve noticed… I guess in fields of work, in tasks where they have a physical embodiment, actually moving things around in the physical world, people get to see one another and they get to learn from watching them, what they’re doing, and figure out how to do it better. And for work on computers, that exists much less.

Chris Olah: Exactly.

Rob Wiblin: There just seems to be a culture in general, that you don’t… When you arrive at an organization, and you’re trying to get training, for example, from your manager, you don’t literally sit behind them all day and watch what they do. That would be… Maybe that would be sensible, but you can’t just sit behind them and watch their screen, and then see, how do they move the windows? How do they reply to people? That doesn’t happen. And I guess that means that it’s possible for people to just miss really basic stuff potentially. It sounds like maybe in programming, there is this pair programming thing in part to fill this gap because it’s maybe such a severe problem there.

Chris Olah: Yeah. I think there’s an increase in culture at a lot of organizations of pair programming. I feel like I hear people talking about it a lot more. And yeah, I’ve found that it’s really helpful for passing this stuff along. I myself am constantly learning from other people when I work with them, and I hope that they’re learning from me as well. I think your point about how when it’s not physical the technique gets hidden by that is a really good one.

Rob Wiblin: Yeah. I think that culture exists in part because people are worried about… Well, I guess there are two reasons. One is sheepishness, perhaps, about people disagreeing with how they’re going about their work. It’s easy to just hide it and never have people watch your screen. Another might be that you’re worried about confidentiality. You don’t want other people looking over your shoulders and reading your emails. There’s a real norm of not looking at other people’s screens in general, but it seems like maybe we should think about ways to work around that. Like, you would have a specified time when someone literally is just going to watch you work, and then you try to not do anything that would be too sensitive where they shouldn’t be looking at the screen. I think I’d be fascinated to see how my colleagues just go about their day. Do they switch windows as often as I do? I don’t know. Sometimes some of these basic skills are so key to your productivity that it could be worthwhile.

Chris Olah: Well, I think it’s also not just teaching. I think often you can just push through things faster if you have a second person with you. Jeff and Sanjay are famous for being extremely impactful at Google, and they’re pair programming all the time.

Rob Wiblin: So they sit together with their computers next to one another and they just work together on a problem?

Chris Olah: Yeah.

Rob Wiblin: That’s really interesting. Yeah. Do you have a theory for why that’s not more common? It seems like it might be just a really sensible way to get things done.

Chris Olah: I think it is modestly common in programming and software engineering. I guess the thing that I’m trying to highlight here is that it’s a really useful… I think people sometimes feel like it’s just a nice way to work, I think. Or it’s an effective way to work. But I think… Especially if you’re trying to develop technique, I think it’s the best way to go and transmit it that I’m aware of.

Cold emails [00:55:25]

Rob Wiblin: Yeah. I’ve seen in your articles that you’re also just generally a big proponent of writing cold emails to people. You found that that’s worked pretty well for you. Do you think that generalizes to others as well?

Chris Olah: I get a lot of cold emails, and 99% of them are terrible. They’re like, “Can you do my homework for me?” or, “Can you answer this basic question that I could Google for one minute and answer?” I think people get this impression that cold emailing doesn’t work, because of course, if you send emails like that, people are overwhelmed and aren’t going to respond. Or, even if you just very generically are like… If you send a nicely written email and you’re like, “I’m trying to get into machine learning. Can you do a half-hour phone call with me to talk about how to do that?” Even that, you’re not very likely to get a response from. But I think the thing that people miss is that if you write really good cold emails, it’s actually not that hard to be the best email I received that week.

Chris Olah: And I think that if you’re willing to invest energy in understanding what a researcher or a group is working on, and you’re specifically referring to their papers, and you have thoughtful questions about things, yeah, I think that people will pay a lot of attention to that. Then I think that it will… It very often works well. I think there’s a big gap in what people mean when they talk about cold emails, and I think that if you’re willing to put in the work, and if you just genuinely really care about what somebody is doing, and have put in the work to understand it, and can talk about it really intelligently… That’s going to come through. It’s a much more compelling reason for the person to talk to you than other things.

Rob Wiblin: Right. It sounds like you don’t think that people should write tons of cold emails to all kinds of people, but if there is someone whose work you’re really into, whose work you really understand, then you should not be sheepish about emailing them. Because even if they’re getting other emails, your one is really going to potentially stand out, if you can demonstrate that you have actually read their paper.

Chris Olah: Well, and I think the other thing is, I think there’s a lot of people who are trying to look at how to get into machine learning, and what they do is they send lots of emails to people, or they email famous people. I think what you should actually be doing is trying to figure out who you would be really excited to work with, and really understand their work. Ideally pick somebody who’s a little bit less famous maybe, and then reach out to that person with an email where you’ve put a lot of work into it being clear that you’ve read their work, and connecting your interests to theirs, and things like this. There’s a number of emails that have been really important for me, where I spent a week writing them. I think that was a totally worthwhile investment. I think that’s not how people usually think about cold emails.

Rob Wiblin: That’s so interesting. How do you feel about length? Maybe we’re getting a little bit into the weeds of email technique here, but go on.

Chris Olah: I think a lot of the most impactful emails I’ve written were only a few paragraphs long, less than one page. But I read five of that person’s papers beforehand, and I think that comes through in subtle ways. And I didn’t integrate it super ham-fistedly, but I was writing to them because I genuinely was invested and cared about their work, and had shared interests with them. I think that’s very, very different.

Rob Wiblin: Yeah. I think the main lesson that I’ve learned… Well, I suppose this is a different class of cold email… This is asking people for small favors or for feedback, or offering advice. Certainly the shorter it is, the more likely people are to answer. Maybe also if you can really condense down the information that you want to convey into just a couple of sentences, people are much more likely to absorb it. Because people are flicking through their email inbox pretty quickly and often they have other things going on, and if they open an email and it’s a wall of text, then you definitely run the risk that they’re just going to close it and then never get back to it, because it’s just too much. They don’t yet know whether it’s really worth investing the time in.

Chris Olah: Yeah. I think you really want to… If you’re writing something long, you want to really optimize the introduction for that reason.

Research as a market [00:59:02]

Rob Wiblin: Do you have any thoughts on how to be a successful researcher?

Chris Olah: I guess there’s one last thing that I find pretty helpful, which is thinking about research as a market. I don’t think this is very novel, but I do find it a really powerful frame. The general idea is, you think of researchers as investing in different research ideas, and if the research idea pans out, and other people don’t grab it before them, then they get some reward from that maybe. Maybe more resources, or just they get a payoff from that in some way. You can see there being this competition to go and grab promising research ideas. I think there’s roughly two strategies that you can play in this market. One is you can work on things where everyone really agrees that they’re important, and that are really popular.

Chris Olah: And what you’re doing when you do that is you’re going and making that little area of the research market a tiny bit more efficient. You’re going and making it so that ideas that are important to get done, get done a little bit more quickly. And I think that is actually genuinely a valuable thing to go and do. If the thing that you’re doing is really important, and you make it happen in expectation a week earlier or a month earlier, that’s really great. But the other strategy you can do is you can try to beat the market. You can try to work on things where you just can see that something is undervalued relative to what most of the community thinks. That’s the thing that I try to do a lot of the time. And there’s lots of reasons why you might be able to beat the market.

Chris Olah: It could be that you just care about things that other people don’t. If you care about safety, or in other areas, if you care about animal welfare, or if you have weirder goals, or different goals than a lot of people, you might be able to beat the market in that way. I think another way, though, is just, if you have some insight that you really believe is true about a problem, and that’s not a widespread insight, then that could be really helpful. That can be… I feel like that’s a lot of what I’m doing, me personally. I think that you can genuinely understand neural networks if you’re willing to input enough energy into trying to figure out what’s going on. It’s a big bet that I’m making, that most other people aren’t making.

Rob Wiblin: Yeah. I guess to translate this into 80,000 Hours speak, one option would be to go with the thing that everyone agrees is most important, or has a particularly large impact if you can make progress on it. The problem there is that it’s probably not going to be neglected, because everyone is onto it already. Another option would be to take a punt on something that is currently really neglected, that other people aren’t working on, but of course there’s a possibility that other people who have chosen not to work in it may be right, that it’s too hard to make progress on it, or it doesn’t really matter, even if you do.

Rob Wiblin: I suppose, yeah, the ideal would be that you have some underlying insight that other people have missed. Maybe they just don’t have time to engage with it, because there’s just so many ideas out there. Lots of things get ignored. And then you can find something based on that insight that is important, and neglected, and easy to make progress on, and then if you’re lucky enough to succeed in that—

Chris Olah: Then you have it made.

Rob Wiblin: Yeah, then you have it made. Right. Exactly. Do you think you struck gold in that way with working on interpretability? It seems like it was a field ripe to develop.

Chris Olah: I think I did. I think it also played a lot to my comparative advantages. But yeah, I think I did. Other people might still disagree. I think that the jury’s still a little out on how valuable it is. But I think it is.

Explaining complex things really well [01:02:26]

Rob Wiblin: Let’s push on and talk about explaining complex things really well, which I know has been a passion of yours for many years. To start, why is it essential for a field to invest in really good explanations of things?

Chris Olah: I think in many fields, achieving a research-level understanding is like climbing a mountain. There’s all of these ideas that you have to understand and build up towards before you can go into research.

Chris Olah: Mathematics, I think, is a really striking example of this, where there’s just years and years of ideas that you’re probably going to spend climbing to the point where you can do research, because there’s just so much that is piled on top. Then when you get to the top, you go and you pile some more results on top, and you make the mountain higher.

Chris Olah: I think a lot of people are proud of this, because they’re like, ah, the fact that it’s this long pilgrimage to go and get to the point where you can do research, that means that it’s especially profound, and it reflects all of the work that’s been done to date. But I think that, actually, it’s often a reflection that we haven’t put enough work into explaining things, building up really good infrastructure for learning about that field. I think this is going to come in lots of forms. It can be poor expositions, just not good explanations to things. Sometimes it’s just undigested ideas. There’s an idea that’s important, but it hasn’t been really refined to the completed version of that idea. I think it’s very common for there to be bad notations, or just bad definitions of things that make things more complicated, and all these things make it harder to go and understand the topic.

Chris Olah: One analogy that I like is sometimes in software engineering, people talk about technical debt, which is you move really fast to get to that point where you can ship some feature or something like this, and in the process, you write lots of bad code, and it’s really messy and gross, and you have bad variable names, and it isn’t documented, and then it’s hard for other people to build on top of. I think something analogous, a kind of research debt, is endemic in science.

Rob Wiblin: Yeah. I guess this problem is probably just getting worse over history.

Chris Olah: Yeah, it’s just getting worse over history, because we’re just piling the results on. Sometimes people do write beautiful tutorials and textbooks that make it a little bit better, but I think on net it’s getting worse, and it tends to be that the older a field is, the worse this problem is.

Rob Wiblin: Yeah. So I guess, at this point, in many academic fields, if you want to really reach the frontier, you have to wait until you’re 30, and you’ve done a PhD, and then a whole bunch of extra stuff. In some fields, maybe it’s 35 or 40 before you can actually start contributing. That’s a reflection of the fact that, one, there’s just a huge knowledge base. In some cases, in some fields, I think we’re pushing up against what it’s possible for the human brain to do. So you just have to really reach your peak capability before you can add in anything new.

Rob Wiblin: And it seems like the basic concepts end up being distilled really well, because they’re taught in primary school, in high school. People figure out how to communicate those, but then the closer you get to peak, the more it’s just a mess because no one’s really figured out… There is no textbook about the most recent results. It becomes slower and slower, maybe, to reach up to the top, because everything becomes inscrutable.

Chris Olah: I think there’s often also less incentive to go and work on exposition. This is my impression. There’s a really nice article by Thurston before he passed away, where he talks about basically, in my framing of it at least, killing a field by just going and picking all the low-hanging fruit, and then dumping the field, and just all of these results that he didn’t explain, and making it totally inaccessible. I guess that’s an extreme example, but I feel like this kind of thing can genuinely hold back fields.

Rob Wiblin: Well, hold on, so you’re saying that someone could just dump a whole bunch of… Well, dump the next step, but explain it so badly that people are repelled from even comprehending it?

Chris Olah: You have to do several next steps, but you can just pile on a huge amount of progress and not communicate it at all, and now it’s a dead end where you can’t go in and get credit for going and redoing that section of the field, and all the fruits have been picked. And you can’t build on top of it, because it’s really hard to understand. Yeah. That’s not a good vision.

Rob Wiblin: That is a fascinating possibility.

Chris Olah: Returning to your earlier question for a second, I think that’s another thing to be said about why it’s important to invest in good explanations. Or, there’s something about why good explanations are valuable, which is the impact of an explanation is often non-linear. As you go and you write a better explanation, not only does it provide more value to its readers, but you also have more consumers.

Chris Olah: In a lot of things, if you make something better, there’s really sharp diminishing returns. But of course, there are diminishing returns where it becomes harder and harder to make something marginally better. But I think with explanations, there’s also a non-linear thing where as you go and you make an explanation better and better and better — and it becomes better than any existing explanation — actually its value significantly increases in this very non-linear way.

Rob Wiblin: Because that becomes the default reference that everyone’s going to read, and they’re all saving a bunch of time because it’s better? And maybe also more people—

Chris Olah: Exactly.

Rob Wiblin: —more people are like, “Oh, I could plausibly understand this, because now it’s been explained properly.” You end up expanding the interest in the topic in aggregate.

Chris Olah: Sometimes I feel a little bit sad about this, but the thing that I’ve written that’s been most read is this tutorial on LSTMs, and it’s been read several million times. If I wrote it just a little bit better, such that I saved every reader one second reading… It actually adds up to non-trivial amounts of time that I saved lots of people. When you have a 1 million multiplier on something, that actually really has a big impact. And the difference between having a 1 million multiplier on something and having a 100 or 1,000 multiplier on something can actually be a pretty sharp transition, where you manage to go from being a good explanation that is comparable to existing explanations, to an explanation that is better than the existing alternatives.

Rob Wiblin: The key idea here, to me, as an economist, is fixed costs versus variable costs. Here we have a fixed cost in making an explanation better, whereas that’s just a cost to the person who’s writing the article. They’re going to have to spend more time revising it, but then the benefits are potentially variable, depending on how many people read it. As the number of readers goes up, the amount of time that it will be plausibly justified to spend improving the explanation just balloons out. If you’re going to have a million readers on a textbook, then you really want to make sure that it’s very, very good so it saves people time.

Chris Olah: While we’re thinking of this in a sort of economics-y way, I think another thing that’s interesting to think about is, if you think about n people in the field interacting, and you increase the… As you vary the size of the field, you can ask how much effort goes into explaining things, and how much effort goes into understanding things, if you want everybody to understand all the work that everyone else is doing. And the effort to explain things grows linearly, because each person has to explain their work. But the effort to understand things grows quadratically, because each person needs to understand every other person’s work.

Chris Olah: So if you have the option to change the coefficients on both of those, where people will produce better explanations, and then the consumers go and do a little bit less work, you can change the size that a field can reach before it fragments, I think. Like, there’s some maximum size that a field can have where everybody actually understands everything that’s going on in that field, and it’s determined by how people explain things, because of this linear/quadratic cost thing.

Rob Wiblin: Yeah. Interesting. Okay, so the idea there is, if people are bad at explaining what they’re doing, then it means that a field will fission, because people will feel like they have to specialize more, because it’s just so much work to understand what those other people at the other end of the field are working on. Yeah. It’s all Greek to them.

Chris Olah: Yeah. I think that’s a natural reaction.

Rob Wiblin: So what is your philosophy of what makes a good explanation, and I guess, what’s missing from most typical explanations that makes you think that they’re not as good as they could be?

Chris Olah: I think that there are two parts to a good explanation. One is just having a really clear way of thinking about the topic, and one is executing the explanation well. And it’s hard to give any advice on how to go and have a clear and nice way of thinking about a topic. The thing that works for me is I just get really annoyed with my understanding of things until it feels nice. That’s what works for me. But I think that there is more that one can say about how to execute an explanation well. Okay, so here’s something I find mysterious. It’s often the case that you have people who are extremely knowledgeable about a topic, and they put a lot of effort into writing an explanation, and they produce an explanation that’s really hard for other people to follow.

Chris Olah: And then it gets worse. People tell them that it’s hard to follow, and they try to make it better, and the result is actually that the explanation becomes progressively worse and harder to follow. When they look at their explanation, they’re like, “Oh, it’s so easy to follow. It’s a really good explanation.” It’s very mysterious. Why is that? And I think my hypothesis would be that they have the benefit of two resources that their reader doesn’t have. So first, they don’t have to go and store things in working memory when they read their explanation. They’ve already memorized all the terms and all of the ideas. They can go and write. Oftentimes, if they try to make an explanation better, they add more and more details, and they’re trying to be really, really thorough and so on. The end result is that they’re loading up the reader’s working memory.

Chris Olah: I’m not a psychologist, but as I understand it, the reader only has seven or so slots of working memory and they fill it all up. Then it’s really hard for the reader to go and continue pulling pieces in and connecting them together, as they consume the explanation. Then similarly, they have all the motivation already. They understand why you should care about the ideas and why you should push through the hard parts, but the reader doesn’t have that. And of course it’s easy for them because they’ve already done the hard parts, and the reader doesn’t have that. I think these two deficits, the not having the same working memory benefits that the author has, and then not having the motivation that the author does, are often the reasons why people think that they’ve written a good explanation, and it’s not a very good explanation. That would be my theory of why explanations fail.

Rob Wiblin: So I guess in the first one, what’s happening is people say, “I couldn’t follow this,” and what they do is maybe add more detail. They add even more to the explanation, but then that is actually making it even harder to follow it potentially, perhaps because what they have to do is simplify it, rather than make it all exactly precise. Is that one way that things could go awry?

Chris Olah: Yeah. Although, I think it’s not exactly simplification. The thing you don’t want to do is dumb an idea down and not actually explain the important thing. But what you do want to do is think hard about how you can reduce the strain on somebody’s working memory. I think there’s lots of tricks you can do with this. I think diagrams are often really helpful, because you can spatially arrange things. Annotating equations can be helpful, and just thinking of how you structure your explanations so that there are fewer long-distance interdependencies can really help. Having little things in the margin where you remind people of important things. I think there’s lots of things like this that you can do. Just asking if you really need to introduce some terminology, or if you can get away without using some additional piece of terminology that people will need to remember. I think, yeah, all of those contribute.

Distill [01:13:12]

Rob Wiblin: Alright, so you weren’t merely annoyed by this. You decided to actually try to do something to help contribute to fixing this problem. Tell us a bit about Distill.

Chris Olah: Yeah. Something that I find especially frustrating about this is there’s lots of really great explanations that people just write as blog posts, and it isn’t treated like a real academic contribution. It doesn’t help them in getting promoted, or getting hired, or things like this usually, at least in academic jobs. It seemed to me that the production of this really valuable work — explanations, and also just interactive visualizations of things, which is another thing I really care about — just wasn’t rewarded. If we could create a way to go and reward it and support it, we’d get a lot more of this kind of work.

Chris Olah: So how can we go and create more incentives? Well, the idea behind Distill was if we could create an academic journal that could be an adaptor between weird artifacts that aren’t normal scientific papers, and the traditional academic system, that maybe could allow for people to go in and get these career rewards and support that doing more traditional academic contributions can get them. So yeah, we created a journal and yeah, that was our thesis for why it would help.

Rob Wiblin: Yeah. And how has it gone? Do you think that your thesis was broadly right? Or have you learned something from the experience of actually trying to fix the problem, as people often do?

Chris Olah: Yeah, I think a lot of parts of the thesis were wrong. I think that the two parts that I don’t believe anymore are, one, that being rewarded is the primary blocker on this type of work being done. I think it’s more the case that people don’t do this work because it’s hard, and that people who do do it, it’s a passion project where they just feel very deeply that they need to go and do this, or they’re just really excited to do it. And it’s very difficult to get to a world where the career incentives would be sufficient for people to do this work for that reason instead. Even if you get this work treated like a normal academic contribution, your tutorial is treated like a typical paper but takes five times more effort to go and produce. That’s not something that people are going to go and do.

Rob Wiblin: Okay, so if I understand it, you think you were right that there aren’t sufficiently strong academic or prestige motivations to work on doing really good distillation of ideas. That was discouraging people somewhat, but the thing that was really discouraging people was that they struggle to do it, and it takes them ages. If it were easier to do, they might be willing to do it just for the love of it, just because they want to bring water from the fountain of knowledge to everyone else.

Chris Olah: Well, I think I would say that the people who are doing it right now are doing it because they have a very strong internal motivation to do it. That, on the present margin, it’s hard to create a large enough incentive that you’ll actually really change things.

Rob Wiblin: Interesting.

Chris Olah: There’s a second reason why I think this doesn’t work. I think previously I thought institutions wouldn’t reward this kind of work if it wasn’t published in a peer-reviewed venue; wasn’t published in a legitimate academic journal or conference, or something like this. I think now it’s more that there’s some institutions that are really flexible and will reward non-traditional work, and they don’t care whether it was published in a journal. They’re just going to evaluate it on its merits. There’s more traditional institutions that just are going to look at Distill and be like, “This is too weird for us. We’re out.” It’s actually a pretty small group that is in the middle enough that this actually makes a big difference. I think both of those make the case for an institution like Distill a lot weaker than I initially thought it was.

Rob Wiblin: Right, so are you guys going to shut it down and try a different method, or do you think there’s still a niche for Distill and we should keep it around, even if it’s not going to solve the problem in total?

Chris Olah: Yeah. I do think there are other ways Distill has provided value. I think that just being an example of what you can do with explanations and with interactive visualizations and interpretability has been really valuable. It’s just useful to show what’s possible if you try really hard in this space. I think it’s also been useful as a laboratory for meta-science, where it’s given us a little bit of credibility to do things like organize. We did this weird discussion article thing where there was a somewhat controversial paper, and we just got a bunch of people to, instead of doing peer review, they just wrote papers discussing the paper, and then we can pile them together and summarize them.

Chris Olah: People spent months going and writing these, what are effectively reviews. I think that was a super cool experiment. Or we’re doing this thing that we call spreads, where we go and collect a series of articles, very incremental articles building on one topic. Those seem to have also been quite successful, I think, and really interesting. So those are things that I’m glad about with Distill, and that I think are positive. But it also has really large costs to run. I think something that I didn’t appreciate enough when I was starting it was just how political it could get, where there’s a whole component of people who are upset with us, because they think it’s corrupt that we publish in our own journal.

Chris Olah: Because it’s such a small niche community, a lot of people who are doing this work… The people who are going and investing lots of effort in creating interactive scientific papers tend to be a slightly tight-knit community, and either are involved in running Distill, or know people who are involved in running Distill. From the outside, that looks incestuous and corrupt, I think, and people are unhappy about it. Yeah, I don’t know. That’s a thing that isn’t super fun to navigate.

Rob Wiblin: Yeah. That does sound tricky. Alright. Well, it’s very interesting, and I think not uncommon, that you’ve gained a lot more insight into what is the actual nature of the problem by trying to solve it, and that perhaps you’ll try a different technique now, knowing everything that you do. But Distill looks really cool to me. I was looking around the website in preparation for this, and it’s much better than the papers that I normally have to read in terms of just visual quality and quality of explanation. So it definitely has achieved that goal.

Chris Olah: Thank you. Yeah, it’s been really wonderful to be part of. I think something I feel really proud of is, in addition to the papers that I’ve worked on, being able to support lots of people in going and producing these kinds of papers has been really cool and special.

Micromarriages [01:19:28]

Rob Wiblin: Alright, we’re almost out of time for this session. But I guess to return to some personal stuff. To get to know you better, if you just had to completely change careers and you somehow became totally indifferent to making the world a better place, what would be the most self-indulgent or most enjoyable career for you to pursue instead? If you’ve thought about this.

Chris Olah: Oh, gosh. Well, one thing I think I’d really love to do is just teach young children math. I often wish that I could interact more with young children. And I find teaching really delightful. I sometimes like to daydream about, “How would I go and teach this?” Can you turn group theory into a board game for young children? Like there’s these Cayley diagrams, and you could use them. Or sometimes I’ll babysit friends’ children, and I’ve done experiments with making knots and then trying to get them to guess whether it’s the unknot or things like this. And so I think I’d have a lot of fun with that.

Rob Wiblin: Teaching children is a real job, Chris. This is the most self-indulgent thing you can imagine doing? It’s a lot of work!

Chris Olah: Well, I don’t know. I’d want to do it in a way that’s… I guess I was maximizing just for my personal enjoyment. It would probably be a very part-time thing or something like this.

Rob Wiblin: Yeah. What’s that xkcd comic where it’s like, “When people ask me to describe my dream job, I’m never sure how literal to be.” And the person’s actual dream job is I think removing some lint from a dryer, and then using a lightsaber on a door, and then retiring to a life of luxury.

Rob Wiblin: If I was being selfish and self-indulgent about your potential alternative career, I’d really love to read Chris Olah the blogger or Substacker, or whatever it would be, Twitter maybe, although you do do Twitter. Yeah, there were just all these really amusing, delightful posts on your website that I had the luxury of being paid to read during the prep for this interview. One that stands out is… People are probably familiar with this idea of the micromort, which is a one-in-a-million chance of death.

Rob Wiblin: And then I guess, I’ve noticed recently, people have been applying this to more and more concepts. So folks might know that some people in the effective altruism community made this wonderful website called microCOVID.org, where you can specify all of these things about something that you did in what country, how many people, for how long, how crowded, was it indoors, outdoors, in order to figure out how many one-in-a-million chances of getting COVID you incurred. So if you meet someone who has COVID indoors, then you clock up 100,000 microCOVIDs, which is a 10% chance of getting COVID. But more day-to-day, you clock up these one, two, three, four, five, six microCOVIDS, and people can use that to kind of judge how much risk are they willing to take and make these judgements that are often so hard to do, these risk-reward judgements in a more coherent way.

Rob Wiblin: And you had a neat application of this kind of one-in-a-million chance thing with the ‘micromarriage,’ where you’re talking about, as you’re going about your social life or just your life in general, you can imagine that the more people you meet, you’re clocking up these one-in-a-million chances of meeting the love of your life, potentially, someone who you can have a really fulfilling relationship with. Can you lead that one up for a second for us?

Chris Olah: Yeah, I guess this trick of going and inventing units to think about things is a trick that I really like. So before we dive in, I guess some context. I wrote that post kind of as humor, but some context is that I actually often find it pretty hard to motivate myself to go to social events. When I don’t know lots of people at an event and I sort of can’t fall back on just talking about research, I often find that kind of stressful, and I often don’t want to do it.

Rob Wiblin: Yeah, totally.

Chris Olah: But I also really want to have a family someday and find a partner, and that requires you to be social. And I guess going to a dinner party that a friend’s hosting, or going to some other social event, or even going on dates or dating someone, a lot of these things, the party doesn’t lead to you meeting anyone or these other things don’t lead to where you were hoping they would go, and that can be really discouraging and really hard.

Chris Olah: And so I’ve sometimes thought that it’s useful to sort of try to be like, “Well, even though this time it didn’t work out, there was actually a chance that that could’ve led to something.” And yeah, you can use that to try to motivate yourself, and also maybe to just sort of emotionally smooth over or help yourself see that you’re making progress, even when these things can be kind of discouraging.

Rob Wiblin: Yeah, that makes a lot of sense. I suppose it’s a bit of a cliché that people who are dating seriously because they’re trying to find someone to build a life with, it can be very demoralizing, because for that, the bar is naturally pretty high. If you’re really going to commit to being with someone for decades and raising children with them, then you kind of want to do your research, and actually check that this person really is a good fit to build a life with you. But then that means that maybe the best thing is just to go on a lot of dates, even have the beginnings of many different relationships, which eats up a lot of time. And optimally, most of them are going to fail. But it’s a little bit hard to keep in mind that things are actually going well, when it seems like in the narrow sense—

Chris Olah: Exactly.

Rob Wiblin: —you’re making no progress.

Chris Olah: There’s kind of this disconnect between the thing that you know, which is, “This is perfectly normal and expected and what this should look like probably,” and the day-to-day experience of often trying things that are difficult. Breakups are really painful, and feeling like you haven’t made any progress is hard. And so I guess that’s a lot of the reason why I find it’s a useful way to think about things.

Rob Wiblin: Yeah, just taking a step back and thinking about this kind of microX concept in general, I suppose it seems like the case where it’s really useful is with low probability. Well, I guess, events that are low probability, but not so low probability that you should ignore them.

Chris Olah: I guess I’d specify I think they’re useful for low-probability, high-impact things.

Rob Wiblin: Right, yeah.

Chris Olah: And so they’re cases where you can sort of get yourself a little bit confused because the probabilities involved are so small that if you think about it in terms of probability, somehow that lens by itself can confuse you. But the impact is so large, and it’s only by thinking about the multiple of those two that you have a unit that’s sort of coherent to think about.

Rob Wiblin: Yeah. And I suppose for alternative things, you could do one in a thousand.

Chris Olah: Yeah, I sometimes use milliunits for things as well, when it seems like the probabilities are a little bit larger and the total importance of the thing is a little smaller. Sometimes that’s the natural scale.

Rob Wiblin: Yeah, so why is this helping me think about these things? I suppose, one thing is that when you’re thinking about it in terms of probability, or just intuitively, the difference between 100 micromorts and 1,000 micromorts doesn’t feel like very much, or 100 micromarriages or 1,000 micromarriages. It all just kind of blurs into, “This is very unlikely.” But actually, specifying it out in millionth chances makes it feel more measurable and real, like you’re clocking up these risks of death or these risks of something really good going on. And you can see the incremental progress, and also weigh opportunities and risks against one another in the same way that we do with more frequent events and the more normal things in daily life where we have more experience of things going well and badly.

Chris Olah: Well, I think that’s right, but I think there’s another thing that happens which is really interesting, which is, as you spend more time thinking in these units… You’ve spent so much time thinking about distance that a foot and a meter and a kilometer are… I guess for a lot of people maybe a mile… are sort of intuitive units. Similarly, I think if you think about things like this, 100 micromarriages or 1,000 micromarriages start to feel like, in some sense, meaningful things.

Chris Olah: To me now, I’m like, “Gosh, 1,000 micromarriages? That’s a lot of micromarriages. 10,000 micromarriages? That’s insane.” And these start to feel like meaningful things. Then if you think that you’ve… If you went to an event and you met a bunch of people and you, as you say, clocked up this many micromarriages, there’s a scale of what you typically get or something, where you can feel like, “Oh, yeah. That went really well,” even though there’s in some sense no tangible result.

Rob Wiblin: Yeah, and the case of the micromarriage measure, I guess that also made me update how large the benefit might be at a specific social event. Because I guess, most people end up getting married, usually to someone they like quite a bit, and they stay with them a while. So how many social events does it take for that to happen, typically? I mean, most people don’t really have time over their youth to attend more than 1,000 social events where plausibly they could meet someone. It might be a bit closer to 100 for many people. So actually, the chance of meeting someone who you might end up forming a serious relationship with at any particular social event, could be maybe somewhere between 1 in 100 and 1 and 1,000. So on a gut level, that doesn’t feel super likely in any specific instance. But actually, if you go to a bunch of events, that really clocks up pretty quickly, which is why lots of people end up paired up.

Chris Olah: Yeah, I mean I think that it assumes that going to social events is the primary mechanism by which people meet their partner, which it might not be. It could be that it’s through interaction by friends, or online dating, things like that.

Chris Olah: But yeah, I think these things can be really significant. And I think it’s just such a life-changing thing and such a potentially dramatically positive thing. Yeah, I don’t know. I’m sometimes, maybe … or not even maybe… I’m definitely a pretty weird person, so maybe this isn’t useful to other people. But I’m often surprised how much time, I don’t know, say, that I’ve spent thinking about linear algebra, and trying to really deeply think about linear algebra or other topics that are sort of these intellectual topics, compared to the amount of time that I’ve spent thinking really hard about these things that are going to shape my future in really dramatic ways. Yeah, I don’t know, it seems intuitive to me to try to think really carefully about those things.

Rob Wiblin: Okay, so we want to apply this concept to low-probability, high-consequence things. We’ve already got the micromort. So death, that seems like a big deal. I guess we got microCOVID. COVID’s not quite as bad as death, but it’s unpleasant. We’ve got micro-important relationship. What else could there be? What are the other big life events? I suppose potentially finding a really fantastic job or career?

Chris Olah: Yeah, I think that would be totally a plausible thing. Yeah, I think that totally makes sense. I sometimes found this useful for thinking about pain. So I think about the most painful thing that I’ve experienced, and then thousandths of that. And then I can use that to think about the amount of suffering that would be involved in something, so I sometimes find that useful.

Rob Wiblin: Yeah, that’s a little bit dark. But what about microfriendships? I think one of the things that’s delightful about the micromarriage is that it’s applying this thing that, so far, I’d only seen applied to negative things, to positive stuff as well.

Chris Olah: Yeah. Well, I guess a related thing is thinking hard about extreme upsides, rather than just extreme downsides. I think we often are sort of focused on avoiding extreme downsides, but there’s these really extreme upsides that matter as much or more that I think we often don’t think about very much. So yeah, finding an outstanding job, finding a partner.

Chris Olah: And I think one thing that’s a little awkward about that — especially about the friend thing — is I think we just have less of a vocabulary for describing… There’s sort of a difference between somebody who’s going to be your best friend for the rest of your life, or a very close friend and it’s going to be kind of life-changing for you, from someone who’s sort of a casual friend and you enjoy meeting them, but it’s not really life-changing in the same way.

Chris Olah: And I think marriage, hopefully, is sort of pretty clearly putting you in the category of this person who’s life-changing and hopefully really dramatically changes your life for the better. And yeah, I think the same thing with a job, where you sort of need to specify the really outstanding job. I don’t know, microbestfriend or something is actually a really interesting unit. It’s a sort of a cumbersome thing to say, but I think the closest friends that I have have really dramatically changed my life for the better and are extremely precious to me, and I’d go through a lot for a one in 1,000 chance to make another friend like that.

Rob Wiblin: Yeah. I think as a society we maybe underrate the value of friendship. There’s a lot of things about how people’s lives are structured that cause a lot of their friendships to atrophy in their 30s and 40s and 50s, and for people to end up, if they don’t make an active effort, with a pretty small friend group. But I guess that’s potentially a topic for another day.

Rob Wiblin: My guest today has been Chris Olah. Thanks so much for coming back on the 80,000 Hours Podcast, Chris.

Chris Olah: Thank you so much, Rob. It was lovely being here.

Rob’s outro [01:32:16]

Rob Wiblin: If you enjoyed this, and haven’t listened to last week’s episode 107 with Chris yet, I strongly encourage you to go back and give it a chance.

It explores topics like:

What is interpretability research, and what’s it trying to solve
How neural networks work, and how they think
‘Multimodal neurons’, and their implications for AI safety work
Digital suffering
Scaling laws
And how nice it would be if this work could succeed

If you’re interested in using your career to work on safely guiding the development of AI like Chris — or working to solve any of the problems we discuss on the show — then you can apply to speak with our team one-on-one for free. We’ve made some hires and removed our waitlist to apply for advising, so our team is keen to speak with more of you loyal podcast listeners.

They can discuss which problem to focus on, look over your plan, introduce you to mentors, and suggest roles that suit your skills. Just go to 80000hours.org/speak to learn more and apply.

The 80,000 Hours podcast is produced by Keiran Harris.

Audio mastering is by Ben Cordell.

Full transcripts are available on our website and produced by Sofia Davis-Fogel.

Thanks for joining, talk to you again soon.

Learn more

ML engineering for AI safety and robustness: a Google Brain engineer's guide to entering the field

Positively shaping the development of artificial intelligence

Research into risks from artificial intelligence

Anonymous answers: the complete collection

Effective Altruism Forum
EA Forum

Chris Olah on working at top AI labs without an undergrad degree

15

Key points

Should people go to university?

Lessons from Chris’ unconventional career track

Developing research taste and technique

Cold emails

Research as a market

Research debt

Articles, books, and other media discussed in the show

Chris’ projects and writing

Other links

Transcript

Rob’s intro [00:00:00]

The interview begins [00:00:57]

Defending a falsely accused friend [00:04:24]

Why Chris didn’t go back to university [00:13:21]

Switching to machine learning [00:19:33]

How Chris got his foot in the door in machine learning [00:27:34]

DeepDream [00:30:52]

Founding a team focused on circuits [00:35:58]

Lessons from Chris’ unconventional career track [00:38:13]

Is it important to go to grad school or work at the best lab? [00:46:01]

Strategies for growing as a researcher [00:50:17]

Cold emails [00:55:25]

Research as a market [00:59:02]

Explaining complex things really well [01:02:26]

Distill [01:13:12]

Micromarriages [01:19:28]

Rob’s outro [01:32:16]

Learn more

15

Reactions

More posts like this