This is a linkpost for #89 - Ajeya Cotra on worldview diversification and how big the future could be. You can listen to the episode on that page, or by subscribing to the '80,000 Hours Podcast' wherever you get podcasts.
In the episode, Ajeya and Rob discuss how the 'doomsday argument' should affect philanthropy, as well as:
- Which worldviews Open Phil finds most plausible, and how it balances them
- Which worldviews Ajeya doesnât embrace but almost does
- How hard it is to get to other solar systems
- The famous âsimulation argumentâ
- When transformative AI might actually arrive
- The biggest challenges involved in working on big research reports
- What itâs like working at Open Phil
- And much more
I have an infinity-to-one update that the world is just tiled with Rob Wiblins having Skype conversations with Ajeya right now. Because I would be most likely to be experiencing what Iâm experiencing in that world.
Ajeya Cotra
Key points
Worldview diversification
So, Open Phil currently splits its giving across three big buckets, or worldviews⌠The longtermism versus near-termism split is where the longtermism camp is trying to lean into the implication of total utilitarianism â that because itâs good to cause there to be more people living lives worth living than there were before, you should be focused on existential risk reduction, to preserve this large long-term future where most of the moral value is, on the total utilitarian view.
And then the near-termist perspective, I wouldnât say itâs a perspective that doesnât care about the future, or has some sort of hard-line commitment to âItâs only the people that exist today that matter, and we count it as zero if we do anything that helps the futureâ. I think itâs a little bit more like this perspective is sceptical of going down that rabbit hole that gets you to âThe only thing that matters is existential risk reductionâ, and itâs sort of regressing back to normality a little bit.
This might come from scepticism of a total view population ethicsâŚit might come from scepticism about the tractability of trying to affect existential risk, or about trying to do things that donât have great feedback loops. So thereâs this tangle of considerations that make you want to go âOkay, let me take a step back and let me try and be quantitative, and rigorous, and broadly utilitarian about pursuing a broader set of ends that are more recognized as charity or doing good for others, and that isnât super strongly privileging this one philosophical argumentâ. Thatâs how I put that split, the longtermism versus near-termism split.
And then within the near-termism camp, thereâs a very analogous question of, are we inclusive of animals or not? Where the animal-inclusive view â similar to the longtermism view â says, okay, there are many more animals in this world than there are humans, and many of them are facing conditions much worse than the conditions faced by any human, and we could potentially help them very cheaply. So even if you donât think itâs very likely that animals are morally valuable roughly comparable to humans, if you think that theyâre 1% as valuable, or 10% as valuable, or even 0.001% as valuable, then the vast majority of your efforts on this near-termism worldview should be focused on helping animals.
And so this is another instance of this dynamic where the animal-inclusive worldview does care about humans but sort of ends up focusing all of its energy on this larger population of beneficiaries. And so itâs this same thing where thereâs this claim that thereâs more at stake in the animal-inclusive worldview than in the human-centric worldview, and then thereâs a further claim that thereâs more at stake in the longtermist worldview versus the near-termist worldview.
And so, essentially, thereâs two reasonable-seeming things to do. One is to allocate according to a credence between these three worldviews, and potentially other worldviews. And then the other is try to find some way to treat the things that each of these worldviews care about as comparable, and then multiply through to find the expected amount of moral stuff at stake in each of these worldviews â and then allocate all your money to the worldview that has the most stuff at stake. Which in this case, most reasonable ways of doing this would say that would be the longtermist worldview.
Effective size of the far future
So the basic astronomical waste argument (Astronomical Waste by Nick Bostrom is the seminal paper of this longtermist worldview) essentially says that thereâs a very good chance that we could colonise space and create a society thatâs not only very large relative to what could be sustained on Earth, but also very robust, and having a very low risk of extinction once you cross that barrier.
We actually think thatâs a pretty important part of the case for longtermism. So, if we were imagining longtermism as just living in the world, where humanity will continue on Earth and things will happen, and itâll be kind of like it is now, but it might last for a long time, so there may be many future generations⌠Weâre not convinced thatâs enough to get you to reducing existential risk as your primary priority.
Because in a world where there isnât a period where weâre much more technologically mature, and much more able to defend against existential risks, the impact of reducing existential risk today is much more washed out, and doesnât necessarily echo through all of the future generations, even if there are many of them on Earth.
Why Ajeya wrote her AI timelines report
In 2016, Holden wrote a blog post saying that, based on discussions with technical advisors who are AI experts â who are also within the EA community and used to thinking about things from an EA perspective â based on discussions with those technical advisors, Holden felt that it was reasonable to expect a 10% probability of transformative AI within 20 years. That was in 2016, so that would have been 2036. And that was a kind of important plank in the case for making potential risks from advanced AI not only a focus area, but also a focus area which got a particular amount of attention from senior generalist staff.
And then in 2018/early 2019, we were in the middle of this question of weâre hoping to expand to peak giving â consistent with Cari and Dustinâs goals to give away their fortune within their lifetime â and we want to know which broad worldviews and also which focus areas within the worldviews would be seeing most of that expansion. And so then the question became more live again, and more something we wanted to really nail down, as opposed to kind of relying a bit more on deference and the earlier conversations Holden had.
And so digging into AI timelines felt like basically the most urgent question on a list of empirical questions that could impact where the budget went.
Biggest challenges with writing big reports
One thing thatâs really tough is that academic fields that have been around for a while have an intuition or an aesthetic that they pass on to new members about, whatâs a unit of publishable work? Itâs sometimes called a âpublonâ. What kind of result is big enough? What kind of argument is compelling enough and complete enough that you can package it into a paper and publish it? And I think with the work that weâre trying to do â partly because itâs new, and partly because of the nature of the work itself â itâs much less clear what a publishable unit is, or when youâre done. And you almost always find yourself in a situation where thereâs a lot more research you could do than you assumed naively, going in. And itâs not always a bad thing.
Itâs not always youâre being inefficient or youâre going down rabbit holes, if you choose to do that research and just end up doing a much bigger project than you thought you were going to do. I think this was the case with all of the timelines work that we did at Open Phil. My report and then other reports. It was always the case that we came in, we thought, I thought I would do a more simple evaluation of arguments made by our technical advisors, but then complications came up. And then it just became a much longer project. And I donât regret most of that. So itâs not as simple as saying, just really force yourself to guess at the outset how much time you want to spend on it and just spend that time. But at the same time, there definitely are rabbit holes, and there definitely are things you can do that eat up a bunch of time without giving you much epistemic value. So standards for that seemed like a big, difficult issue with this work.
What itâs like working at Open Phil
We started off, I would say, on a trajectory of being much more collaborative â and then COVID happened. The recent wave of hiring was a lot of generalist hires, and I think that now thereâs more of a critical mass of generalists at Open Phil than there was before. Before I think there were only a few, now theyâre more like 10-ish people. And itâs nice because thereâs a lot more fluidity on what those people work on. And so there are a lot more opportunities for casual one-off collaboration than there is between the program staff with each other or the generalists with the program staff.
So a lot of the feeling of collaboration and teamyness and collegiality is partly driven by like, does each part of this super siloed organisation have its own critical mass. And I feel like the answer is no for most parts of the organisation, but recently the generalist group of people â both on the longtermist and near-termist side together â have more people, more opportunities for ideas to bounce, and collaborations that make sense, than there were before. And Iâm hoping as we get bigger and as each part gets bigger, thatâll be more and more true.
Articles, books, and other media discussed in the show
Ajeyaâs work
Open Philanthropy blog posts, reports, and their general application
Nick Bostromâs work
Other papers
OpenAI blog posts
Additional links
TV and movies
Transcript
Robert Wiblin: Hi listeners, this is the 80,000 Hours Podcast, where we have unusually in-depth conversations about the worldâs most pressing problems and what you can do to solve them. Iâm Rob Wiblin, Head of Research at 80,000 Hours.
A couple of years ago, I asked Ajeya Cotra to come on the show. Unfortunately, at the time she was too busy. But Iâm very glad that last year she found time for us, because she is killing it as a grantmaker at Open Philanthropy, where she tries to figure out the highest-impact ways to make philanthropic grants.
In this conversation, we go into some of my favourite juicy topics, like the simulation argument and how likely it is that humans will actually be able to settle other planets or even other solar systems.
But we also cover some more down-to-earth topics, like how Ajeya â and Open Phil as a whole â does research to try to figure out how they can have the biggest impact with their philanthropic spending, and some of the challenges (both personal and intellectual) in doing big research projects, and trying to get them written down on paper in some form before going crazy.
Before the interview, a little bit of personal news, though. As most of you will know, late last year â in October and November â we did a user survey at 80,000 Hours where we got feedback from many of you about how our products had helped you or hurt you, or just generally what you liked about what we did and how you thought we could do better.
It turns out that this show was just incredibly popular. We were really heartened to find out just how many of you thought that it had changed your life, or was one of the most interesting and engaging things, or pieces of content that you enjoy in your life.
As a result of that feedback, and a couple of other things, weâre going to be rejigging our roles at 80,000 Hours a little bit such that Arden Koehler is going to be taking over some of my responsibilities on the written work and the website, thereby freeing me up to spend a bunch more time on these interviews and producing this show, and maybe taking it to the next level â or at the very least, hopefully just producing more content.
Arden is going to be having her hands full with lots of things this year. But hopefully sheâll still make some appearances on the show. I know a lot of you have really enjoyed some of the interviews she did last year, so you certainly havenât heard the last of her.
If youâd like to do us a solid favour, in exchange for us deciding to double down on the show and make more of it, we could really use your help getting the word out there to people who would value hearing these conversations, and might even be influenced into changing what they decide to do with their life or with their work.
We already have a pretty big audience, and Keiran and I both find that extremely gratifying. But when we do the numbers, we think thereâs got to be at least 10 or maybe 100 times as many people out there who would really enjoy listening to the show based on how many subscribers some other similar interview shows have. But we canât reach all of them and tell them that it exists on our own. So we could really use your help.
If you have a friend out there who you think would also enjoy the 80,000 Hours Podcast, maybe try to think of who they might be and think of an episode that they would particularly enjoy given their interests. Drop them a message now on WhatsApp or Signal or Facebook Messenger or whatever, letting them know that this show exists and suggesting that they check out whatever episode you think is a particularly good fit for them.
We really do appreciate your support in getting the word out there. With that little bit of ado, and hopefully good news out of the way, here is my interview with Ajeya Cotra.
The interview begins [00:03:23]
Robert Wiblin: Today Iâm speaking with Ajeya Cotra. Ajeya is a senior research analyst at Open Philanthropy, a large effective altruist-flavored foundation that expects to give away billions of dollars over the course of its existence, and which is 80,000 Hoursâ largest donor. Since joining Open Philanthropy in 2016, Ajeya has worked on a framework for estimating when transformative AI may be developed, estimates of the empirical returns to funding solutions to different problems, and how worldview diversification could be implemented in open-source budget allocations. She studied electrical engineering and computer science at UC Berkeley, where she also founded the Effective Altruists of Berkeley student group and taught a course on effective altruism. Thanks for coming on the podcast, Ajeya.
Ajeya Cotra: Thanks so much for having me. Excited to be here.
Robert Wiblin: I hope to get to talk about your work on when transformative AI might show up and humanityâs prospects for settling space, but first: What are you doing at the moment, and why do you think itâs important work?
Ajeya Cotra: Iâm a senior research analyst at Open Phil, and like you said, Open Phil is trying to give away billions of dollars. Weâre aiming to do it in the most cost-effective way possible according to effective altruist principles, and we put significant amounts of money behind a number of plausible ways of cashing out what it means to be trying to do good â whether thatâs trying to help the poorest people alive today, or trying to reduce factory farming, or trying to preserve a flourishing long-term future. We call these big-picture schools of thought âworldviewsâ, because theyâre kind of like a mash-up of philosophical commitments and heuristics about how to go about achieving things in the world, and empirical views. Iâm looking into questions that help Open Phil decide how much money should go behind each of these worldviews, and occasionally, within one worldview, what kind of big-picture strategy that worldview should pursue. We call these âworldview investigationsâ.
Ajeya Cotra: This is closely related to what 80,000 Hours calls âglobal priorities researchâ, but itâs on the applied end of that â compared with the Global Priorities Institute, which is more on the academic end of that.
Robert Wiblin: Weâll get to that in just a minute, but how did you end up doing this work at Open Phil?
Ajeya Cotra: I found out about effective altruism 10 years ago or 11 years ago now, whenever Peter Singerâs book The Life You Can Save came out. I was in high school at the time, and the book mentioned GiveWell, so I started following GiveWell. I also started following some of the blogs popping up at the time that were written by effective altruists folks â including you, Jeff Kaufman, Julia Wise, and a bunch of others. I was pretty sold on the whole deal before coming to college, so I really wanted to do something EA-oriented with my time in college and with my career. So I co-founded EA Berkeley, and was working on that for a couple years, still following all these organisations. I ended up doing an internship at GiveWell, and at the time, Open Phil was budding off of GiveWell â it was called âGiveWell Labsâ. So I was able to work on both sides of GiveWell and Open Phil. And then I got a return offer, and the next year I came back.
Ajeya Cotra: I was actually the first research employee hired specifically for Open Phil, as opposed to sort of generically GiveWell/Open Phil/everything. So I got in there right as Open Phil was starting to conceptually separate itself from GiveWell. This was in July 2016.
Robert Wiblin: Had you been studying stuff that was relevant at college, or did they choose you just because of general intelligence and a big overlap of interests?
Ajeya Cotra: I mean, I had been, in my own time, âstudyingâ all the EA material I could find. I was a big fan of LessWrong, reading various blogs. One thing I did that put me on Open Philâs/GiveWellâs radar before I joined was that I was co-running this class on effective altruism. UC Berkeley has this cool thing where undergrads can teach classes for credits â like one or two credits, normal classes are like four credits â so having to put together that class on effective altruism was a good impetus to do a deep dive into stuff, and they gave us a grant. Our class was going to give away $5,000; they were going to vote on the best charity to give it to. We got that money from GiveWell.
Ajeya Cotra: But in terms of the actual subject matter I was focused on in university, not really. It was computer science â technically like an electrical engineering and computer science degree â but I didnât really do anything practical, so it was kind of a math degree. Being quantitatively fluent I think is good for the work that Iâm doing now, but Iâm not doing any fancy math in the work that Iâm doing now. We have people from all sorts of backgrounds at Open Phil: something quant-y is pretty common, philosophy is pretty common, economics is pretty common.
Robert Wiblin: Yeah, thereâs a funny phenomenon where people study very advanced maths, and then on a day-to-day basis, it really actually does seem to make a huge contribution to their ability to think clearly, just by their willingness to multiply two numbers together on a regular basis.
Ajeya Cotra: Yeah, totally, totally.
Robert Wiblin: Thatâs the level of analysis youâre doing. But for some reason, it seems like maybe in order to be comfortable enough to do that constantly, you need to actually train up to a higher level.
Ajeya Cotra: Thatâs my line. I tell people itâs probably good to study something quantitative because it gives you these vague habits of thought. Iâm not sure exactly how much I believe it. I think philosophy does a lot of the same thing for people in a kind of different flavor. Itâs more logic and argument construction, which is also super important for this kind of work.
Worldview diversification [00:08:45]
Robert Wiblin: Weâll come back to working at Open Phil and how people end up in those roles and how listeners might be able to do that, if thatâs the kind of thing theyâre interested in. But first, letâs dive into this really interesting topic of cause prioritisation and worldview diversification, which I guess is a component of what we talk about, as you said, in global priorities research.
Robert Wiblin: At a big picture level, what is the problem of prioritisation between causes and diversifying across worldviews that Open Phil faces?
Ajeya Cotra: So, Open Phil currently splits its giving across three big buckets, or worldviews, which we wrote about in the 2017 cause prioritisation update. So, thereâs one big split, which is between longtermism and near-termism. I should caveat everything Iâm about to say by saying that this is my perspective on this stuff, and these are really fuzzy concepts to pin down and theyâre kind of in flux, so Iâm sure that somebody else coming in here whoâs done cause prioritisation work would put it slightly differently and sometimes disagree with me. But broadly speaking, there are these two big splits that produce three worldviews.
Ajeya Cotra: The first split is the longtermism versus near-termism split, and this has been discussed as the difference between the person-affecting view of population ethics versus the total view of population ethics â where, roughly speaking, the person-affecting view says that it doesnât count as good to create additional people who are living lives worth living, where the total view says that creating an additional person who is living a life worth living is in the same ballpark as saving a life, in terms of the moral good thatâs being done.
Ajeya Cotra: Thereâs something to expressing the split in those terms, but I actually think that the distinction is not a purely philosophical one, and I actually think the longtermist camp is more into philosophy than the near-termist camp. So, I think I would characterise the longtermist camp as the camp that wants to go all the way with buying into the total view â which says that creating new people is good â and then take that to its logical conclusion, which says that bigger worlds are better, bigger worlds full of people living happy lives are better â and then take that to its logical conclusion, which basically says that because the potential for really huge populations is so much greater in the future â particularly with the opportunity for space colonisation â we should focus almost all of our energies on preserving the option of having that large future. So, we should be focusing on reducing existential risks.
Robert Wiblin: Setting the scene a little bit, the main problem is that youâve got this big pile of money and you want to do as much good as possible with it, and youâve got to figure out how to divide it between the many different problems in the world. And also, I guess, try to figure out when to dispense it â whether it should be now, or whether you should save it and use it later. And basically, different attitudes or perspectives â philosophical or practical â would suggest focusing on very different problems.
Robert Wiblin: And then youâve got the question of, well, do we go all in on the one that we think is best, or do we split across a bunch of them? And then how would you split it? Thatâs the kind of big-picture problem that youâre trying to solve with this investigation. And then those are examples of some of the most plausible worldviews on which you might focus.
Ajeya Cotra: Yeah, thatâs right. In fact, the question is maybe even thornier than âdo we go all in on the perspective that we think is most plausibleâ â because we could potentially end up in a situation where we want to go all in on a perspective that we actually think gets a minority of our credence, but itâs a perspective like the longtermist view that says thereâs so much opportunity, thereâs so much effectible opportunity to do good out there, more goodness. So, if you consider a perspective thatâs mostly focused on helping people in this generation or the next couple generations â versus a perspective thatâs trying to be more ambitious and bring in this opportunity of permanently affecting the entire long-run trajectory â you might say that even if you only have like a 1% or a 10% probability on the second perspective, you should actually put all of your money there, because itâs positing that the world is bigger or thereâs more goodness in the world. So, thatâs one key question that we wrestle with.
Robert Wiblin: Okay. So, letâs dive more into this kind of normative uncertainty or moral uncertainty aspect. What are some approaches that you could take to decide how much weight to give to these different philosophical positions? I guess youâre pointing out that you might think you want to allocate them in proportion to their relative likelihood, but that runs into the problem that some of the views suggest that thereâs a whole lot more good that can be done than others, and then maybe thatâs the key issue that you have to find some way to work around.
Ajeya Cotra: Yeah. Let me quickly lay out the three worldviews I was alluding to before. So, the longtermism versus near-termism split is where the longtermism camp is trying to lean into the implication of total utilitarianism â that because itâs good to cause there to be more people living lives worth living than there were before, you should be focused on existential risk reduction, to preserve this large long-term future where most of the moral value is, on the total utilitarian view.
Ajeya Cotra: And then the near-termist perspective, I wouldnât say itâs a perspective that doesnât care about the future, or has some sort of hard-line commitment to âItâs only the people that exist today that matter, and we count it as zero if we do anything that helps the futureâ. I think itâs a little bit more like this perspective is sceptical of going down that rabbit hole that gets you to âThe only thing that matters is existential risk reductionâ, and itâs sort of regressing back to normality a little bit.
Ajeya Cotra: This might come from scepticism of a total view population ethicsâŚit might come from scepticism about the tractability of trying to affect existential risk, or about trying to do things that donât have great feedback loops. So thereâs this tangle of considerations that make you want to go âOkay, let me take a step back and let me try and be quantitative, and rigorous, and broadly utilitarian about pursuing a broader set of ends that are more recognized as charity or doing good for others, and that isnât super strongly privileging this one philosophical argumentâ. Thatâs how I put that split, the longtermism versus near-termism split.
Ajeya Cotra: And then within the near-termism camp, thereâs a very analogous question of, are we inclusive of animals or not? Where the animal-inclusive view â similar to the longtermism view â says, okay, there are many more animals in this world than there are humans, and many of them are facing conditions much worse than the conditions faced by any human, and we could potentially help them very cheaply. So even if you donât think itâs very likely that animals are morally valuable roughly comparable to humans, if you think that theyâre 1% as valuable, or 10% as valuable, or even 0.001% as valuable, then the vast majority of your efforts on this near-termism worldview should be focused on helping animals.
Ajeya Cotra: And so this is another instance of this dynamic where the animal-inclusive worldview cares about humans but sort of ends up focusing all of its energy on this larger population of beneficiaries. And so itâs this same thing where thereâs this claim that thereâs more at stake in the animal-inclusive worldview than in the human-centric worldview, and then thereâs a further claim that thereâs more at stake in the longtermist worldview versus the near-termist worldview.
Ajeya Cotra: And so, essentially, like you said, thereâs two reasonable-seeming things to do. One is to allocate according to a credence between these three worldviews, and potentially other worldviews. And then the other is try to find some way to treat the things that each of these worldviews care about as comparable, and then multiply through to find the expected amount of moral stuff at stake in each of these worldviews â and then allocate all your money to the worldview that has the most stuff at stake. Which in this case, most reasonable ways of doing this would say that would be the longtermist worldview.
Robert Wiblin: So, thatâs one way of dividing things up according to, I guess, two different potential disagreements within moral philosophy. But I saw in your notes that maybe Open Phil is leaning more towards thinking about this not just in terms of moral philosophy, but also just thinking about it in terms of dispositions, or attitudes, or these worldviews as not just representing formal positions that one may take on core philosophy questions. Can you expand on that a bit?
Ajeya Cotra: Yeah. I mean, I think that the longtermist versus near-termist split is a good illustration of this. This is a super important split, and it comes up again and again, more so than the animal-inclusive versus human-centric side of things. But itâs not the case that everyone on the near-termist team doesnât care about the long-term future or wouldnât do things that would help people that donât exist yet. So, a lot of the poverty and disease reduction work that theyâre funding ends up helping people that arenât yet born, because it reduces the incidence of malaria in a region or something like that. Target Malaria is a great example of something we funded on the near-termist side of things, thatâs this very ambitious plan of trying to eradicate malaria entirely. And itâs sort of commonsensically part of a case of that thing, that future generations who might have faced malaria wonât face malaria anymore. And thatâs sort of the way we think about it quantitatively.
Ajeya Cotra: The difference, I would say, is that the near-termist side of the organisation cares about the future in a kind of atheoretical commonsensical way, a way that broadly altruistic people tend to care about the future. So, they place value on reducing climate change, and they place value on eradicating diseases, in part because future generations will be helped too, but they donât tend to go in for this âbigger world is betterâ thesis. And then they also just kind of feel uncomfortable with basically throwing out all of the goals that seemed like good goals from a commonsensical perspective of helping others selflessly, in order to focus on this one goal that⌠You know, reducing risks of a pandemic, or reducing risks of nuclear war, that was part of a portfolio of things people cared about from a common-sense values perspective, but they werenât nearly so dominant. And it wasnât for this reason of âSpace colonisation might allow us to have such a huge population in the futureâ.
Ajeya Cotra: I would characterise the distinction as the near-termist side is less into doing this kind of philosophy and biting that particular bullet, not so much that it has a philosophical commitment that the future doesnât matter, or creating new people is always morally zero or whatever.
Robert Wiblin: Yeah. So, I guess one view would be that having these atheoretical commitments to doing stuff is just a total mistake, and that those views should be ignored because someone just simply hasnât really thought clearly about what theyâre accomplishing and what they value. But it sounds like youâre a bit more sympathetic to the atheoretical approach, and maybe you think that thereâs something to be said for it, maybe even on a rigorously philosophical point of view.
Ajeya Cotra: Yeah. I mean, I donât know that thereâs necessarily something to be said for it on a rigorously philosophical point of view, but I think thereâs something to be said for not going all in on what you believe a rigorously philosophical accounting would say to value. So, I think one way you could put it is that Open Phil is â as an institution â trying to place a big bet on this idea of doing utilitarian-ish, thoughtful, deep intellectual philanthropy, which has never been done before, and we want to give that bet its best chance. And we donât necessarily want to tie that bet â like Open Philâs value as an institution to the world â to a really hyper-specific notion of what that means.
Ajeya Cotra: So, you can think about the longtermist team as trying to be the best utilitarian philosophers they can be, and trying to philosophy their way into the best goals, and win that way. Where at least moderately good execution on these goals that were identified as good (with a lot of philosophical work) is the bet theyâre making, the way theyâre trying to win and make their mark on the world. And then the near-termist team is trying to be the best utilitarian economists they can be, trying to be rigorous, and empirical, and quantitative, and smart. And trying to moneyball regular philanthropy, sort of. And they see their competitive advantage as being the economist-y thinking as opposed to the philosopher-y thinking.
Ajeya Cotra: And so when the philosopher takes you to a very weird unintuitive place â and, furthermore, wants you to give up all of the other goals that on other ways of thinking about the world that arenât philosophical seem like theyâre worth pursuing â theyâre just like, stop⌠I sometimes think of it as a train going to crazy town, and the near-termist side is like, Iâm going to get off the train before we get to the point where all weâre focusing on is existential risk because of the astronomical waste argument. And then the longtermist side stays on the train, and there may be further stops.
Robert Wiblin: Yeah, interesting. I like the idea that rather than thinking about this as exclusively a philosophical disagreement, think about it as a disagreement on the strategy question of, whatâs our edge? Whatâs our edge over everyone else whoâs trying to do good? And one of them is, âWell, weâll be better at philosophy, and weâll reach more philosophically rigorous conclusionsâ. And the other people are like, âWeâll be better in some other way. Weâll be more empirical, or be more careful about thinking aboutâŚâ
Ajeya Cotra: More quantitative, yeah.
Robert Wiblin: More quantitative, exactly.
Ajeya Cotra: I mean, I actually think the near-termist side of the organisation empirically uses quantitative estimates way, way more than the longtermist side of the organisation does. So, on the longtermist side, weâve talked ourselves into highly prioritising causes where there are only like 10 people working on them. And so most of our effort is trying to convince potential grantees â potential people who could be helpful in this mission â that itâs reasonable to work on at all. And trying to fund people who are trying to do the basic thing that we want to do â for example, reducing global catastrophic biorisks as opposed to focusing on biorisks in general. And that is where almost all of our selection pressure has to go. But on the near-termist side of things, theyâre looking at lists of hundreds of things they could focus on, like air pollution in India, or migration from low-income countries to middle-income countries. And they have a huge list of causes and theyâre just doing the math on the number of lives that get better per dollar with each of these options.
Ajeya Cotra: So, the feel of doing near-termist work at Open Phil is definitely much more quantitative and rigorous, and in some sense it feels more like what you would have thought a cartoon EA foundation would feel like, because they have more opportunity to map things out.
Science and policy funding [00:23:10]
Robert Wiblin: So, I guess weâve listed three cluster worldviews. One is helping people now, another one is helping animals now, and the other one is helping people and animals in the longer term. Are there any others that we should have in mind that you have on the shortlist of different hats that you put on?
Ajeya Cotra: Yeah, there are a couple of smaller ones. So this always goes back to the fact that we want to be a strong foundation thatâs making the most diversified bet we can make on deeply rigorous, thoughtful philanthropy thatâs truly about helping others, rather than our particular personal values on causes. And so within that, these really feel like the big three to us, I would say. But there are also other things we would like to get experience with.
Ajeya Cotra: When we were starting out, it was important to us that we put some money in science funding and some money in policy funding. Most of that is coming through our other causes that we already identified, but we also want to get experience with those things.
Ajeya Cotra: We also want to gain experience in just funding basic science, and doing that well and having a world-class team at that. So, some of our money in science goes there as well.
Ajeya Cotra: Thatâs coming much less from a philosophy point of view and much more from a track record⌠Philanthropy has done great things in the area of science and in the area of policy. We want to have an apparatus and an infrastructure that lets us capitalise on that kind of opportunity to do good as philanthropists.
Robert Wiblin: Yeah. I guess the science funding reminds me of this kind of âprogress studiesâ perspective, which has been generating a bit of buzz on blogs and on Twitter, and I guess their thinking is something along the lines of, I donât want to just think about all of this moral philosophy and theoretical stuff, but if I look back over the last 1,000 years, what has made things better? Itâs science research, technology research, and economic growth. I donât necessarily have to have a theory of how thatâs made things better, I just want to keep pushing on this thing that seems to be the fundamental driver of the world becoming less barbaric. And so they have the whole story about how they want to speed up scientific research, improve funding to direct it to better people and better projects, and increase it, and so on.
Ajeya Cotra: Yeah. And I think thereâs really something to that. So, I feel like this isnât Open Philâs primary bet, but I could imagine in a world where there was a lot less funding going to basic science â like Howard Hughes Medical Institute didnât exist â then we would be bigger on it. Going back to the bet of trying to do deeply thoughtful intellectual philanthropy to help others, we could have looked back and seen wow, basic science has been a really big deal for humanity. And then we could have looked around and seen that basically nobodyâs acting on this, and wanted to go in much bigger on the bet.
Ajeya Cotra: And so itâs really responsive, also, to this thing you were saying about what we think our competitive advantage is. And we do think our competitive advantage is more of a top-down kind of thinking, both on the near-termist side and on the longtermist side, where the near-termist side is kind of surveying this large array of possibilities to help others in the world today and is picking the one that quantitatively seems most efficient, and the longtermist side is sort of stepping back even further and thinking about, at the root, what kinds of things even plausibly could be the most valuable thing to do on a total utilitarian perspective?
Ajeya Cotra: So, weâre mostly very top-down, but part of the reason we have the basic science program is this kind of bottom-up, very atheoretical argument that, look, this has been a huge driver of human progress and human flourishing.
Robert Wiblin: Yeah. I feel like this has been a banner week for the progress studies worldview. Because you look at politics and like, oh my God, this is just⌠But every day, thereâs some amazing scientific breakthrough coming out. I guess on Monday we had AlphaFold⌠Iâm just trying to remember all of them off the top of my head.
Ajeya Cotra: I mean, the mRNA vaccine, right?
Robert Wiblin: Oh, the mRNA vaccine, yeah. So, the vaccine stuff is coming along. It seems like the scientific community has really been killing it on COVID in the big picture.
Ajeya Cotra: Totally, totally.
Robert Wiblin: We have made massive progress now settling the protein folding issue, which has been around for many, many decades. On a gut level, I find the âletâs just improve wisdom, letâs just improve scienceâ thing to be quite appealing, then maybe on a more philosophical reflection, it seems a bit more questionable. Do you want to comment on that?
Ajeya Cotra: Oh, I was just going to say, I mean, the other thing you mentioned as kind of part of the same thing, but I think it could really be broken off into a different thing, is the economic growth worldview. So, the Tyler Cowen thing you were alluding to is very much⌠I donât know how much he leans on science so much as growth has been really good. Like growth has been so much better for human welfare in the history of the last few hundred years than redistribution has been. And, interestingly, itâs not clear exactly whether that fits in the longtermist camp or the near-termist camp, and could potentially become something that we take seriously enough and think is neglected enough that it might be another worldview that we want to put some weight behind.
Robert Wiblin: Okay. Letâs head back to the question of how you would split all of the money that you have between these different mental buckets. Youâve come up with a couple of different ways of thinking about this, and one you called âfairness agreementsâ. Whatâs that one?
Ajeya Cotra: Yeah. So, just to set the scene a little bit, the question here is: What happens when you have two worldviews, one of which is trying to help a certain set of beneficiaries, say, and then the other values those beneficiaries, but also cares even more about a much larger set of beneficiaries? And so you see this dynamic with animal-inclusive versus human-centric and with longtermism and near-termism. And I see basically three things that we could do, and I would personally want to do a mix of those things. One is to allocate all the money to the worldview that has the most at stake â that says the world is biggest and it contains the most moral value â which would be the longtermist worldview in this case. Two is to allocate according to credence, like you mentioned before.
Ajeya Cotra: And then three is the thing you were saying, which is called fairness agreements. The idea is that if you imagine these worldviews as people who were each given a third of the money before they knew any pertinent facts about the world, and then you woke up and you discovered â in an extreme case, if you woke up and discovered, actually, we seem extremely safe in terms of existential risk, the biggest risk we can think of is asteroids, and thereâs a one in 10 million chance that they kill us, and we have all these detection programsâŚand weâve really determined we donât think AI is a big risk, we donât think biorisk is a big risk, we donât think nukes are big risk â then it would feel kind of unfair to keep having a third of the money on the longtermist side, because the longtermist side probably would have made the deal before knowing anything that it would give away its money in these very, very low existential risk worlds in exchange for having more of the money in these very high existential risk worlds.
Robert Wiblin: Yeah. I guess a really stark example might be the animal-inclusive person and the human-centric near-termist person trying to negotiate ahead of time what will happen, and then they wake up in a world in which it turns out that there are no nonhuman animals. In that scenario, presumably, they would bargain ahead of time to pass the money on to the human-focused person, because thatâs just very efficient from both of their perspectives before they know what world theyâre actually going to end up in.
Ajeya Cotra: Yeah, exactly. And so I think the basic idea here is quite compelling, but the tricky bit that makes me not want to put too much of the capital into these fairness agreements is: What is that prior? So, I said something fuzzy like these three worldviews are talking âbefore they know the pertinent factsâ, but how do we approximate what they would have thought? So, if we look around at the world today, a very interesting question that is really tricky to determine is like, are there more animals suffering on factory farms than we should have expected, or fewer? And thatâs just super dependent on what you thought your prior was. So, on an intuitive level, it seems like thereâs a horrifyingly large amount of animal suffering going on. But on some sort of prior, is that the median trajectory we should have expected, say, knowing what we did at the start of the Industrial Revolution, about incentives to create these kinds of systems?
Ajeya Cotra: And then an even trickier one is, there are so many stars in the future. Is that more or fewer than we should have expected there would be? Thereâs a lot of them, but there could have been moreâŚthere could have been infinity stars. So, it depends on where you place that veil of ignorance in order to determine how to do your fairness trades. And we have a few ideas, but I donât think any of them are super knockdown.
Ajeya Cotra: One idea is just to look back on history and think, letâs take these three worldviews, maybe since philanthropy was a thing, starting in the 1800s or something. Where in time would these worldviews have wanted to allocate their money? And so you might have thought, âOkay, well, the global poverty worldview, it might have been better to transport that money back in time to the 1910s, or the 1950s, at a time when there was even more extreme global poverty, but still the means to try and address it, because there was still international communicationâ, or something. And then maybe you think that the animal-inclusive worldview and the existential risk reduction worldview would have wanted to have their money roughly now. So, if you imagine making the trade back when philanthropy started to be a real force on the scene, then you might say, âOkay, global poverty had its moment, more so than now, itâs still very much its moment now, but more so than now it was global poverty reductionâs moment in the â60sâ. So, thatâs something you might say.
Robert Wiblin: How is this different from the same issue that you would face just doing the Rawlsian veil of ignorance thing to think about what moral principles we should follow? Because this obviously is a kind of old idea. I suppose, in that case⌠So youâve got everyone debating behind this veil of ignorance, and maybe in that world itâs okay for them to see how the world is, itâs just that they donât know which person theyâre going to be. And so we just get rid of that one piece of information, and then they go away and debate it, and that feels less arbitrary. Whereas with this case, itâs like, we canât tell them that much about the world because then they would know what corner to back more. Or I guess maybe you have to think, well, they donât know what moral views they have. Was that the thing toâŚ?
Ajeya Cotra: Yeah, sort of. I think that the way to translate it into the clean Rawlsian framework is, you donât know which worldview you are, but you just see the world as it is today, and then you try to think about which worldview you would rather be on its own terms. So, with the Rawlsian veil of ignorance, you donât know what person youâre going to be, but you sort of assume these people all have the same values, in terms of like, they want to have better lives rather than worse lives. But the reason we canât do that so cleanly with the worldviews thing is you donât know how to translate a standard deviation of goodness, say, across these different metrics that youâre using, because thatâs kind of the whole question of worldview diversification.
Ajeya Cotra: The thing weâre trying to replicate is if you had a probability distribution over how much good you could do per dollar before observing the world â before observing something, like taking away some information â you want to generate these probability distributions over what is the chance of saving the world per dollar on the longtermist worldview, and how many hen lives you can save per dollar on the animal-inclusive worldview, and how many human life-years can you preserve per dollar on the human-centric worldview?
Ajeya Cotra: So, one potentially natural way to do this â that I think is actually kind of my favourite â is just think about what GiveWell and Open Phil people themselves thought as they were getting into this business in 2010, or something, and when you just actually notice yourself feeling surprised that itâs so easy to help chickens, say, put more into that worldview. And thatâs something thatâs kind of been done organically, and itâs also something we could roll forward, right? Like, we could agree now that if AI risk is either more tractable to address or bigger than we currently think it is, then the longtermist worldview gets more money relative to a world where weâre more fine than we think we are.
Robert Wiblin: Yeah. So, I guess with this, youâve got something thatâs very theoretically appealing, but then it feels like the point from which youâre starting â or the views that you had and youâre updating from â just feels kind of arbitrary. And so itâs like why should we privilege the views of Open Phil staff in 2010? That feels a bit strange. It has a less philosophically pure aspect to it. Feels very messy at that stage.
Ajeya Cotra: Yeah. I mean, one thing that I think is very trippy about this is that if you are rolling back all the way to the start of the universe or something, then the longtermist worldview should basically give away all of its influence in all of the smaller worlds that we could end up in, in exchange for getting maximum influence in the world where thereâs the biggest infinity number of stars. So, I kind of feel like the longtermist worldview wants to be like a super weird, hardcore philosopher. So, itâs kind of fair to take a slice from it on this basis, because it would have made that agreement, I think, at the beginning, because itâs a very hardcore sort of beast.
Robert Wiblin: Just maximising, yeah.
Ajeya Cotra: But I donât want to take it to zero because of that. So, I just threw out, in my notes, that maybe like a quarter of the money should be allocated according to these fairness thingies, and only some of that quarter should be the super hardcore, âtry and think about the beginning of timeâ thing, and âtry and think about if there are more stars or fewer stars than we expectedâ. And then most of it should be these kind of less principled versions.
Robert Wiblin: Yeah, interesting. Okay. So, just to explain that, youâre thinking that if we went all the way back to the beginning of the universe and we donât know how much matter and energy there is in the universe, the longtermist mindset would say, âWell, if the universe is 10 times as big, then I want to have 10 times as many resources, because everything is 10 times as importantâ. And so you kind of want to do it in proportion to the size. I guess then weâre left with like, the universe seems pretty big, but it could be a whole lot bigger. Or I suppose also, we could be earlier in the universe, when we can access more of it. But it has this issue of like⌠I have no sense of what the scale is, because it feels like itâs just unlimited how large it could be, right? So, I guess it reminds me of that paradox that all numbers are small, because you can just continue adding numbers forever, and even a million, million, billion, trillion is still like smaller than⌠Thereâs far more numbers that are bigger than that are smaller?
Ajeya Cotra: No, no, totally. I mean, itâs exactly the St. Petersburg paradox, which is⌠The probability distribution you probably had going in â of what the size of the universe is and how much matter and energy there is â is you probably had most of your probability mass on smallish amounts. And in fact, our universe as we see it now probably is within the bulk of your probability distribution, but you still assign this long tail to ever bigger numbers, and they dominate the expected value because of the shape of things. Because if youâre pretty uncertain, then youâre just not going to have a sharply decaying tail.
Robert Wiblin: I love this one. If Iâd been presenting longtermism at the pub and someone had managed to respond with this one as an objection, I would have incredibly admired that. Is there a difference between fairness agreements and the veil of ignorance approach, or are those just two terms for the same general idea?
Ajeya Cotra: Yeah, I think they are two terms for the same idea.
Robert Wiblin: Nice. Okay. So, what about the âoutlier opportunities principleâ? Whatâs going on there?
Ajeya Cotra: This is something where Holden might put it differently from me, but I conceptually think of them as the same idea as fairness or veil of ignorance. That whole cluster of considerations is like, if something is doing surprisingly well on its own terms, whatever that means, then it should get some sort of bonus. Because if you imagine theyâre sort of like business partners, or like a family, these worldviews â they care about each other. If one of them is in surprisingly great need, then the others would pitch in, is kind of how I think about it. And then the whole question is what is âsurprisingly great needâ to these different worldviews? And I threw out all these different ideas for how to think about what âsurprisinglyâ means.
Ajeya Cotra: And the outlier opportunities I think is just like a particularly easy version of that, where youâre seeing the empirical distribution of opportunities in each of these worldviews as a philanthropist, since you got into the business, and if something just looks like youâre purchasing so many points of x-risk reduction per dollar versus anything youâve seen before, then you just kind of want to seize on it. And thatâs kind of coming from this impulse that it seems like some of this money should be going to helping out the worldviews that have surprisingly great opportunities, surprisingly great need.
Robert Wiblin: I guess it seems like this would be tricky if you evaluate the mindset on its own terms, because then you could have like, what about a worldview that says, âOh, if you help just one person, then thatâs fantastically goodâ. And then it does successfully manage to help one person. And so itâs like, âOh, Iâm massively flourishing. This view is kicking assâ. And you say âWell, should we give money to do that?â But it seems like somethingâs maybe going wrong there. Itâs like you have to evaluate it on some slightly higher level.
Ajeya Cotra: I agree. So, I think thereâs pretty different types of thinking that sort of determine whether you let something into the family of worldviews that are trying to be nice to each other and cooperating with each other. Thatâs like one gate, and then the other gate is this fairness stuff that I was talking about.
Ajeya Cotra: So, there are many perspectives on how to do good that Open Phil doesnât really let in the door. The most salient one is like âcharity starts at homeâ, that you should be trying to help people you personally know, or your local community or Americans or something. And so we sort of start off with this goal of trying to be very other-centered and sensitive to scale, relative to whatever else is out there, and just really give it your best shot to be this impartial effective altruist while noticing when there are bridges where you donât want to go, you donât want to ride the train all the way to crazy town with all of your capital â although you would want to put some of your capital behind that, but trying very hard to be impartial. And then you let in some set of worldviews, given that, and then they do these more complicated fairness agreements and stuff.
Next best worldviews [00:41:05]
Robert Wiblin: Yeah, thatâs interesting. Are there any worldviews that are kind of on the borderline, like youâre not sure whether to invite them to Thanksgiving or not, whether they are part of the family? What would maybe be the next best worldview thatâs currently not in the family?
Ajeya Cotra: Yeah. So, I think itâll be very different for different people. My personal one that I struggle with⌠I mean, there are two. So the thing I mentioned about improving economic growth as a worldview, I have a lot of sympathy for. And I could pretty seriously imagine, at least for myself, wanting to let it sit at the table. The other one that I think is more borderline and probably no, is something about improving civic institutions. Thereâs something that attracts a part of me to trying to clean up your own house and be like âthe city upon the hillâ, like some dude said about America back in the day. Itâs just kind of like⌠A part of me feels pulled to improving democracy, and sort of shoring up our self governance, and all this stuff that I could maybe like pencil out through to either the near-termist human-centric worldview, or maybe the longtermist worldview, but the pull I feel isnât really coming from expecting those to pencil excellently or something.
Robert Wiblin: Yeah, thatâs interesting. I suppose one practical argument is that the location you can affect the most is the one that youâre from; that youâre already really embedded in. And then maybe some really important thing is kind of demonstration effects, like showing other places how they can be great. And so if thatâs what really matters, then you want to focus on making the place that you are the very best that it can be, so that other places can learn from it. I think that makes some intuitive sense. And then maybe youâre also adding in some kind of contractarian thing where you feel like youâre embedded in some relationship withâŚ
Ajeya Cotra: Yeah, thereâs some flavor where itâs kind of like⌠Yeah. When the George Floyd protests were happening and there were just reams and reams of videos of police brutality in the United States, I was very affected by that, and I was just kind of like⌠Thereâs something I feel when something heinous is happening in your backyard. Like, if the EA community were facing a thorny situation with a bad actor, then I would want to put a lot of my own energies that could have gone into doing my job or whatever to try to help with that, if I were in a good position to do so. And that feels kind of similar with the heinous things that are happening in the United States. And so thatâs maybe another kind of frame on it, which is kind of contractarian. Itâs kind of like, these are my people. Iâm kind of responsible for this bad stuff.
Robert Wiblin: Yeah. Itâs possible another intuition thatâs firing there, and I guess, yeah, maybe this is one way to think about it, itâs likeâŚguiding intuitions that kind of push you to want to have a conclusion. Yeah, the components of worldviews would be like looking at police just beating up peaceful protesters, and you think, âOh, wow. The military services are the armed forces that are supposed to represent this country, and theyâre kind of out of control, and theyâre no longer under civilian control, and that historically is incredibly alarming, that tends to end really badlyâ. And so youâre like, âThis is a fire. I have to put out the fire in my house before I continue improving theâŚâ
Ajeya Cotra: Yeah. I mean, I guess anti-authoritarianism is maybe an umbrella you could put on this kind of worldview, sort of like the freedom worldview. We donât have a lot behind that stuff, like human rights, and freedom of speech, and anti-censorship, and anti-police brutality. Thereâs something attractive to me from a health-of-the-nation, health-of-the-world point of view to all of those things.
Robert Wiblin: Yeah. One thing thatâs attractive about the kind of pro-science or pro-learning worldview is that it allows you to just kind of punt to people in the future who hopefully will be more informed than you. So, you say like, âLook, I donât know what utopia we want to build, and I donât really know how to get there, but one thing I can do is add my brick to the wall of just making humanity as a whole wiser and more informed, and thatâs really the best that I can hope to doâ. And I can definitely see the intuition behind that kind of worldview.
Ajeya Cotra: Yeah.
Robert Wiblin: I suppose all of these other approaches have kind of been trying to avoid the extremism or the fanaticism issue â that you could have one worldview that just dominates all of them. Is there anyone who speaks up in Open Phil for just like, being fanatical?
Ajeya Cotra: Thereâs definitely a spectrum in terms of how much⌠If each of us were doing the sort of complicated calculus of thinking about how much to put into each of these worldviews, we would definitely differ in terms of the share that would go to longtermism. I donât know if we would differ by a ton, I donât know if the sort of most pro-longtermism and least pro-longtermism have more than a factor of two or something. So, I donât think thereâs somebody thatâs like really planting their flag on super, super pro-fanaticism, maybe there is, Iâm not sure.
Ajeya Cotra: I think for me personally, I was more pro going all in on the astronomical waste argument before thinking about some of the further weird things that come up as the train keeps moving to crazy town. One of which is the thing that I mentioned about how at the beginning of time, the longtermist worldview probably would have traded off almost all of its influence in almost all of the worlds, âalmost allâ there being the mathematical definition of almost all, which is like, all but one, or something. And then another one being the various philosophical arguments that suggest that if you believe there is going to be a long-term future, you run into various confusing questions. So, one confusing question is captured in the âdoomsday argumentâ, which is like, you should be very surprised to find yourself super early in the history of a very long world, but perhaps much less surprised to find yourself super early in the history of a relatively short world. So, maybe you should think that existential risk is much larger and much more inescapable than you currently think it is.
Ajeya Cotra: And then another is the âsimulation argumentâ, which is, if thereâs a giant future world and theyâre running all sorts of computations, some smallish fraction of their computations might be simulating a world like ours, namely a world thatâs potentially on the cusp of space colonisation becoming very large. And so thereâs a lot of stuff you run into where youâre like, âWow, the world is maybe really not at all what it seems likeâ. And I think after marinating in all of that, I didnât end up with any particular conclusions that I wanted to plant my flag on or anything, but I sort of was like, âOkay, actually, this line of thinking takes me to a place weirder than I am comfortable withâ. And I sort of therefore have sympathy for people for whom the immediately previous stop was weirder than they were comfortable with, and I was more able to listen to the parts of myself that found that uncomfortable.
Pragmatic reasons to spread across different areas [00:47:39]
Robert Wiblin: Letâs come back to that in just a second. But first, what are some other non-philosophical/just purely pragmatic reasons to want to hedge your bets a bit more and spread across different areas? To me, it seems like theyâre more persuasive, maybe, than these worldview diversification considerations.
Ajeya Cotra: Yeah. So, I mean, the thing I was saying earlier about how Open Phil as an institution wants to bet on scope-sensitive, deeply thoughtful philanthropyâŚthat hasnât been done. And we donât want to have that whole bet ruined and make Open Phil look like a failure because we chose to put all of the capital that could be going to a broad array of thoughtful scope-sensitive philanthropy into one type of philanthropy that has a number of practical disadvantages â such as not being able to learn and correct as you go, or not seeing impact in your lifetime, or having causal attribution be really hard.
Robert Wiblin: Thereâs also just declining returns, right? So, thatâs one reason that you would always want to spread out across different areas, is that you might just find that youâre beginning to struggle to find really good opportunities within any one particular area.
Ajeya Cotra: I think the declining returns reason for diversification applies much more within a worldview than across worldviews.
Robert Wiblin: Explain that.
Ajeya Cotra: So, the longtermist worldview doesnât ever think that its next dollar should go to something aimed at helping humans in the present, because it thinks the future is e.g. 10â30 orders of magnitude larger than the present. So, even super lottery ticket/bank shot opportunities to help the future â including just saving to wait for something to come up, like in the Phil Trammell paper that was recently released â will pretty much always beat, from within the worldview, giving it to near-termist causes. But the declining returns thing is certainly the reason why we have more than one focus area within each worldview, like why weâre working on both AI and biosecurity. We think that AI risk is probably the bigger problem, but we think that the final dollar that we spend is probably going to be less cost effective than whatever we can do in biosecurity, so we should be doing both AI and biosecurity.
Robert Wiblin: Yeah. So, the way I think about the declining returns thing is, letâs imagine that I have my factory farming/animal-inclusive hat on, and Iâm thinking, do I want to go and recommend that Open Phil spend more time trying to do worldview diversification research and try to figure out how it should shift around the fractions that itâs giving to each of these different problem areas?
Robert Wiblin: And I guess with that hat on, I think Iâve got the best arguments. Iâm right, and so on average, if they think about it more, theyâre going to end up agreeing with me. You might think, well, in the very best case, where they just went all in on this worldview, maybe say Iâm 25% of the budget now, I could go up to 100%. But in fact, thatâs not even really that beneficial to me, because I already canât find a way to spend the money that I have now. Iâm already running out of opportunities. And so even quadrupling your budget might only accomplish 100% more. And so youâll be like, âNo, letâs just stop thinking about it. Iâm happy with the fraction that Iâve got, because thatâs actually plenty to do most of what I want to do, and I donât really want to risk the possibility of losing some of my share in order to have an expected increaseâ.
Ajeya Cotra: Yeah. I think that makes sense from the perspective of a worldview. I feel like that seems right, but it seems a little bit weird, because itâs basically the worldview being afraid that further thoughtful reflection âwhich we sort of assume will lead to an increase in better conclusions â is going to lose it money. So, from the perspective of an individual worldview thatâs kind of âselfishâ within its worldview, then I agree that declining returns means that that worldview is probably more afraid to lose money than it is happy to gain money.
Ajeya Cotra: But thatâs a different notion of declining returns than people usually mean when they say that declining returns leads to diversification, because people are usually talking about declining returns from the perspective of the decider. Where youâre sort of like, you can put $100 million into bed nets, but then the marginal bed net is going into this area that has very low malaria incidence. So, at that point, rather than buying that next bed net, youâd be better off funding deworming or youâd be better off doing cash transfers. Thatâs kind of what I think of as the ânormalâ version of diversification due to diminishing marginal returns.
Robert Wiblin: Yeah. I guess I could buy it across theories if one worldview really thought that it just had nothing that was positive â it had funded everything that generated good, and anything else would actually in fact be counterproductive. But maybe thatâs just too extreme and peculiar a view to take.
Ajeya Cotra: I think itâs very unlikely you land in that place. It seems more likely across the two near-termist worldviews, because the ratio of what seems at stake is less extreme. So, I could imagine the animal-inclusive worldview getting to a place where it spent so much money on animals and made things so much better, that â because it also cares about humans â the next dollar it would spend would actually be aiming to help humans. But I really donât see it for the longtermist versus near-termist worldview, because of the massive differences in scale posited, the massive differences in scale of the world of moral things posited, and because the longtermist worldview could always just sit and wait.
Robert Wiblin: Yeah, that makes sense. I guess Iâll just note that philosophy PhD student Hayden Wilkinson recently wrote this paper called In defence of fanaticism, which unfortunately we havenât had time to read closely because itâs a little bit technical. But weâll stick up a link to that for readers, if theyâre curious to see. I guess he claims, in the introduction to the paper, that almost nobody has ever defended fanaticism, which in moral philosophy is just going all in on one perspective and ignoring other considerations. He claims that almost no oneâs defending it, but he wants to stake out the territory of defending it and saying, âActually, this is more reasonable than hedging your betsâ. Maybe weâll be able to talk about that paper at some point in the future once Iâve actually read it.
Robert Wiblin: Has Open Phil as an organisation made any big strategic shifts in the worldviews and relative weights that it gives since⌠I guess you had a couple of different blog posts about this back in 2017, I think. People might be curious, what is the upshot of things youâve been learning since then for what Open Phil is actually going to fund?
Ajeya Cotra: Yeah. I mean, I laid out this worldview diversification question in terms of these three main worldviews and these sort of high-level philosophical considerations for how much to give to each of these worldviews â do you weight by credence, or do you give all of it to the worldview that says it has the most at stake, or do you do fairness agreements to make trades between worldviews? Since then, I think weâve moved into a bit more of an atheoretical perspective, where there may be a larger number of buckets of giving, and each of those buckets of giving will have some estimate of cost effectiveness in terms of its own kind of native units.
Ajeya Cotra: So, there may be many buckets of giving in the near-termist human-centric side, like giving to GiveWell top charities, or giving to Target Malaria, or things like that in the zone of ambitious sort of science projects to eradicate disease. Or giving to improve policy in developing countries. And we would try to math all of those out in terms of disability-adjusted life years per dollar, or maybe effective cash transfers per dollar or something. But they would each have different properties, in terms of these intangible properties that were partly driving our desire to do worldview diversification in the first place â such as like, being subjectively weirder and more speculative and less commonly considered to be a good goal, or having worse feedback loops, or having a higher risk of self delusion, or feeling more like a Pascalâs mugging. These are all kind of tags you could put on different buckets of giving. And even within one worldview, these tags might be different for different buckets.
Ajeya Cotra: So, you can imagine that improving vaccine manufacturing capability in a way that would help us prevent future COVIDs â or help us make future COVIDs better, and also, potentially, help with something much worse â could be one sort of bucket of longtermist-oriented giving that seems pretty good, in terms of seeming like a subjectively good goal to most humans, having pretty good feedback loops because you can see whatâs happening with the vaccines as youâre going, and having a pretty low risk of self delusion because youâre deferring a lot to experts in how itâs actually done and so on. Versus something like funding really unproven EAs to attempt to get a machine learning degree to try and do AI safety research. That seems like it has a higher risk of potentially self delusion because there are no experts in this area to defer to, and it has worse feedback loops, potentially. And there are things you could imagine that have even worse feedback loops, like maybe creating bunkers for people to hide out in if thereâs a bio disaster or a nuclear disaster.
Ajeya Cotra: And so basically, the rough thought here is that there are some things in each of the three big worldviews that perform really well on these subjective intangible things, that make it so that theyâre an easy sell to ourselves. And then there are others that perform worse on these intangibles. And just empirically speaking, we feel more comfortable doing the things that perform worse on intangibles when they pencil out to be better on their own terms, within their worldview bucket.
Ajeya Cotra: So, weâre kind of moving into something where we sort of talk tranche by tranche, bucket by bucket, and there might be 10 or 20 of these buckets as opposed to these three big worldviews, and we try to do our best to do the math on them in terms of the unit of value purchase per dollar, and then also think about these other intangibles and argue really hard to come to a decision about each bucket.
Effective size of the long-term future [00:57:19]
Robert Wiblin: Letâs move on to another cluster of research that you did while generally thinking about how Open Phil should allocate its money among different issues.
Robert Wiblin: Earlier we were talking about the longtermist worldview versus other worldviews. I guess one key part of figuring out how much to weight the longtermist worldview is to think, well, how big could the future be? How much benefit could you create in the long term?
Robert Wiblin: I suppose on a more simple longtermist view you could think, well, the size of the future might be that humans continue to live on Earth as they have for another billion years until the sun ultimately expands and kills everyone. But I suppose the future potentially might be a whole lot bigger than that, which is one reason that potentially you want to give the longtermist view a lot of weight. Now, do you want to go into that?
Ajeya Cotra: So the basic astronomical waste argument (Astronomical Waste by Nick Bostrom is the seminal paper of this longtermist worldview) essentially says that thereâs a very good chance that we could colonise space and create a society thatâs not only very large relative to what could be sustained on Earth, but also very robust, and having a very low risk of extinction once you cross that barrier.
Ajeya Cotra: We actually think thatâs a pretty important part of the case for longtermism. So, if we were imagining longtermism as just living in the world, where humanity will continue on Earth and things will happen, and itâll be kind of like it is now, but it might last for a long time, so there may be many future generations⌠Weâre not convinced thatâs enough to get you to reducing existential risk as your primary priority.
Ajeya Cotra: Because in a world where there isnât a period where weâre much more technologically mature, and much more able to defend against existential risks, the impact of reducing existential risk today is much more washed out, and doesnât necessarily echo through all of the future generations, even if there are many of them on Earth.
Ajeya Cotra: So I was looking into that general question of whether we will have a large and robust low x-risk future in space.
Robert Wiblin: And what are the component questions of that?
Ajeya Cotra: There are basically two parts to this project. The first part was a brief literature review/interviews about the kind of technical feasibility of space colonisation, plus much reduced existential risk worlds. And I was looking into all sorts of things, like how many stars are there? And how fast could our spaceships be? And how much mass could they carry?
Ajeya Cotra: I was trying to find the most defensible or most conservative assumptions that would still lead to this low x-risk space colonisation world, in terms of could we have biological humans colonise a small number of planets, and could that be a sort of stable low x-risk world? I ended up deciding thatâs actually fairly dicey to defend.
Ajeya Cotra: The most conservative assumptions that robustly lead to this big, safe world in space go through humans being uploaded into computers, and those computers being taken to space â as opposed to the biological bodies being taken to space and trying to alter planets by terraforming to make them habitable for biological humans. So I thought that was an interesting upshot.
Robert Wiblin: Whatâs the reason for that? Why is it that the whole âhumans go to other planets and live thereâ thing either isnât possible or wouldnât be sustainable for very long time periods?
Ajeya Cotra: I donât know that I strongly think it isnât possible. It just seems like there are a lot of questions that go away if you make the one assumption about uploaded humans.
Ajeya Cotra: The people I talked to were kind of like, âMaybe we could swing that. Maybe we could have biological humans in big spaceships traveling to these other planetsâ. But there are a lot more questions about sustaining life on the spaceship and finding planets that are suitable for terraforming then there are about preserving these computers and finding planets that are suitable for building computers on.
Ajeya Cotra: A big one is just that the spaceships need to be huge to support these human colonies, and feed them and everything. And these huge spaceships, first of all, require a lot more materials to build, and you might not be able to build as many of them. They might be much more fragile in terms of, you have this huge surface area you have to protect from stuff like space debris and things, especially if youâre going very fast.
Ajeya Cotra: And you might, with these smaller spaceships, be able to send redundantly many, many of them, so that if some of them get destroyed, itâs okay with a fraction of the material you would have spent on the big spaceship. Stuff like that.
Robert Wiblin: I mean, this isnât something Iâm an expert in, but I love to speculate about it. I guess myâŚ
Ajeya Cotra: I donât consider myself an expert either. I wrote down things that people said, I read some papers.
Robert Wiblin: Yeah. I guess, in that spirit, my impression is that I havenât heard a good reason to think that in the fullness of time, it wouldnât be possible for humans to settle Mars, and make Mars habitable. And potentially some other places within the solar system, places where actual flesh-and-blood humans could live and continue to procreate and be self-sustaining.
Robert Wiblin: But yeah, once youâre talking about going to other stars and finding planets there, it gets a lot more dicey whether itâs possible. I guess, just because humans are really not designed for space travel, that is not what we evolved to be capable of doing. We need lots of space and lots of resources.
Robert Wiblin: Yeah, so if you can get all of those materials into some big spaceship, now youâve got to go a very long way. And the amount of energy required to move something that would be such a big ship, that would be large enough to have a self-sustaining group of humans for, I guess, thousands of yearsâŚthatâs a lot of material.
Robert Wiblin: And then youâve got this trade-off between⌠You could try to go really fast. Thatâs very energy-requiring. And you also have this issue that you run into dust on the way, and dust would eventually pelt and potentially break down the ship. Itâs actually a very big issue then. Because youâd want to go as close toâŚ
Ajeya Cotra: Itâs a huge issue.
Robert Wiblin: âŚyouâd want to go as close to light speed as possible. Thatâs hard enough in the first place. And that obviously makes it a bit easier, because you wonât be there for as long, and you donât have to keep the ship turning over so many generations of humans on the trip. And I guess also time slows down when you go especially fast.
Robert Wiblin: But then the ship just gets pelted and disintegrates because of dust. Which is one reason that people thought, well, if you want to go to other stars, what you want to do is send 1,000 tiny little ships, because they have a chance of, just by good fortune, not hitting dust in the intervening space, even though theyâre going incredibly fast and dust would blow them up.
Robert Wiblin: So you not only have to have enough material that youâd have a huge ship where you can have a self-sustaining population of I guess 1,000 people â because otherwise theyâll become inbred and non-functional â you also then have to have all the resources to terraform a planet once you get there and make it viable for humans. And it could be completely different kinds of planets. So, it seems hard.
Ajeya Cotra: I mean, I think the thing that cinches it for me â in terms of why I really donât want the longtermist case resting on biological colonisation â is that Iâm really not sure the economics of it would work out. It seems like colonisation with biological humans would require much more motivation on the part of the home planet to make it happen than the smaller space probe colonisation, or colonisation with computers, that could be done with one motivated company potentially, once we have the technology.
Ajeya Cotra: So I think thatâs also part of it for me, where Iâm kind of like, not only does it seem technologically dicier, partly because of that it also seems like Iâm not sure that I can really tell someone, âHey, this is probably going to happen,â if this is the only way it can happen.
Robert Wiblin: So your point there is that settling of a star system is not like Europeans going to the Americas. No oneâs bringing back any silver or gold anytime soon, on a economically-plausible timescale. So you have to want to do it just for its own sake. And then whoâs going to fund this if itâs costing several years of global GDP to do it?
Ajeya Cotra: Yeah. I mean, we might mine asteroids and stuff. But in terms of actually diversifying off of Earth, and getting a big dose of the more robust future/permanently reduced x-risk part of the story, then I think youâd have to be much more motivated as a civilisation if it had to be biological versus if it could be computers.
Robert Wiblin: Yeah. Okay, tell us about the computer future. Can we send self-replicating computers to other star systems? Is that likely?
Ajeya Cotra: Yeah. I mean, thereâs not a lot of literature on this and not a lot of people whoâve thought about it.
Robert Wiblin: Iâm shocked to hear that.
Ajeya Cotra: I love this paper out of FHI called Eternity in Six Hours. Itâs just very fun, sci-fi almost.
Ajeya Cotra: I certainly think that people in the effective altruist community who have thought about this seem quite bullish about it â like about small ships that are traveling meaningful fractions of the speed of light, and have these onboard computers. And these computers are able to land on planets and do what they need to do to build more computers on that planet, and build more ships and send them out.
Ajeya Cotra: And itâs not like I found any particular devastating counterargument to that or anything. I think I am more uncertain than the people who are most into this, like the authors of the Eternity in Six Hours paper.
Ajeya Cotra: But it seems to me like thereâs a broad range of technological sophistication levels that (once you assume the ability to upload humans into computers) feel like the big interesting uncertainty. After that point, it feels like you donât need massively more sophisticated technology than we have now to get the lower end of the ability to colonise other planets, other stars.
Ajeya Cotra: To get the shock wave expanding over the entire observable universe, you have to assume these more intense technological capabilities. Which seems possible, but I wouldnât blame you if you were more sceptical of that. But I think the lower end doesnât involve a ton of innovation. And that was something that I learned from this project.
Robert Wiblin: Thatâs interesting. Youâre saying that weâre within firing distance of potentially being able to send computers to other stars, if we were really willing to do what it took.
Ajeya Cotra: Yeah.
Robert Wiblin: But then, would they be able to do what it takes? I guess Iâve heard that, rather than go to planets, which is kind of hard, they would probably want to go to asteroids and then grab resources from the asteroids and turn those into copies of themselves?
Ajeya Cotra: Yeah.
Robert Wiblin: Yeah, I guess thatâs the thing that Iâm most unsure about â it feels to me like the kind of technological problem that, if humanity had thousands of years, it could figure out. But it seems like itâs a slight heavy lift, given the fact that itâs a bit hard to send that much.
Robert Wiblin: Itâs hard to send a full industrial base to another star system, because thereâs so much material and so much dust in the way. So itâs like, can you squeeze through just enough that it can get off the ground at the other end?
Ajeya Cotra: One update I made from this is that â and this was a long time ago, people since have looked into space colonisation at Open Phil, and I havenât dug into their work on it â but my impression is that most of the technological uncertainty is on the software side of things, versus the spacecraft side of things or the energy side of things. For what Iâm calling the âconservative astronomical waste storyâ, where the spacecraft donât actually need to be that fast or that tiny.
Ajeya Cotra: I think that the big question is, can we create these computers on moderate amounts of computing hardware that could fit on, I donât know, a golf-ball-sized craft, or maybe a soccer-ball-sized craft? Can you basically embed artificial intelligence on there and robotics thatâs flexible enough to be able to make do with what it happens to find on another surface?
Ajeya Cotra: And that feels like more of the uncertainty to me than the question of can we make some kind of spacecraft work to colonise some stars. Not the big colonise-the-universe thing, but still enough to dramatically reduce existential risk, because youâre spread out more.
Robert Wiblin: Okay. Letâs wrap up on this empirical bit. I guess the bottom line is, having looked into this you thought, if youâre willing to buy that sending artificial intelligence or uploaded humans in some form to other star systems to do their thing would be valuable, the possibility that we canât get to other star systems doesnât reduce the value of the far future or the long-term future that much. Because itâs 50/50 likely that we can do it, and so that only halves the value or something.
Robert Wiblin: So itâs not going to be the big reduction factor that you would need to say, âOh no, actually, the futureâs not that big in expectationâ.
Ajeya Cotra: Yeah.
The doomsday argument [01:09:37]
Robert Wiblin: But thereâs other arguments that potentially could do more heavy lifting. And you mentioned them earlier, the âdoomsday argumentâ and the âsimulation argumentâ. Maybe just lay out the doomsday argument, and, I suppose, how persuasive you find that line of reasoning? How much can it do to shrink the expected size of future life?
Ajeya Cotra: Yeah. So the doomsday argument is basically that if you find yourself on apparently the cusp of the ability to colonise space according to the previous research, you should be very surprised if there seems to be a very long future ahead of you as a civilisation, and you find yourself at the very earliest bit.
Ajeya Cotra: So in other words, letâs say God flips a coin at the beginning of the universe. And He either makes 10 boxes labeled one through 10, each of which has a human in it, or He makes 10 billion boxes labeled one through 10 billion, each of which has a human in it.
Ajeya Cotra: After he does this, you wake up in a box, and you walk outside to see that your box is labeled âThreeâ. So the intuition that itâs trying to elicit is âOh, probably we landed on the 10 boxes side instead of the 10 billion boxes sideâ. Because if it were the 10 billion boxes, I should have seen e.g. 7 billion as my number, rather than three.
Ajeya Cotra: The argument is, thatâs what you should believe in the case of the boxes. And the question of whether our world has a big, bright future or will be wiped out quite early in its history is like God flipping that coin and creating either 10 boxes or 10 billion boxes.
Ajeya Cotra: So finding yourself early in history should make you think thereâs actually a lot more existential risk thatâs a lot more intractable than you thought. And humanity isnât going to have a long future, and thereâs not much you can do about it.
Robert Wiblin: Yeah. So I guess depending on how far back you think humans goâŚwhich I guess is a bit of a messy question, because it was just a continuous gradual evolution into being the humans that we are now⌠But maybe thereâs been like 100 billion humans ever at any point, and so we think weâre at about 100 billion.
Robert Wiblin: And the question is, if thereâs going to be 1 trillion humans, then itâs not that surprising that we would find ourselves in the first 100 billion. But if thereâs going to be 1,000 trillion, then itâs starting to look as though itâs a bit odd, itâs a bit like drawing the box labeled âthreeâ.
Robert Wiblin: So maybe thatâs a reason to think that there wonât be that many people in the future. Because if there are going to be so many, then what a coincidence that we should find ourselves at this incredibly early stage.
Robert Wiblin: Is this sound reasoning? What do philosophers make of this? Because it feels like itâs proving too much just by⌠I mean, you havenât looked at almost anything, and youâre managing to prove that weâll destroy ourselves on the basis of pure theory.
Ajeya Cotra: Yeah. I mean, Iâm definitely suspicious of things that have strong conclusions about what kind of world weâre living in from pure philosophy. But I actually think both sides of this debate end up having something like that.
Ajeya Cotra: So this doomsday argument relies on whatâs called the âself-sampling assumptionâ. These are extremely confusingly named. But basically, the doomsday argument gets its weight from the assumption that before you look at your box, you should have been 50/50 on whether God flipped the âsmall worldâ coin or the âbig worldâ coin.
Ajeya Cotra: And because before you looked you were 50/50, then you make a massive update towards the âsmall worldâ, because youâre sort of an early number. And that intuition is coming from, well, God flipped a coin. So itâs 50/50.
Ajeya Cotra: The other perspective you could take on this is that before you look at your number, there should be a massive update already in favor of being in the âbig worldâ, just because you exist. There are more people existing and having experiences in the world with 10 billion boxes than in the world with 10 boxes. So then when you look, you are back to 50/50.
Ajeya Cotra: Thatâs roughly the two assumptions. âSelf-samplingâ is the first one, and âself-indicationâ is the second one. But I never remember that.
Robert Wiblin: Yeah. Okay, so basically the idea is that before you get out of your box, if itâs the second world, thereâs 10 billion people?
Ajeya Cotra: 10 billion boxes. Yeah.
Robert Wiblin: Yeah, okay. So thereâs that many boxes, and so youâre like, âWell, thereâs way more people in this world. So Iâm far more likely to be thereâ. And I guess itâs doing something like not taking for granted that you would exist. Itâs imagining 10 million versus 9,999,990 empty, where there is no person.
Robert Wiblin: I guess it fits my intuitions that I would think I would be more likely to be in the world where thereâs more people. What do you make of that?
Ajeya Cotra: So, I think that is pretty reasonable, but it also leads to this kind of presumptuous⌠The thing you were saying about how you come up with this really confident theory about what kind of world youâre living in based on pure philosophy applies to this approach, but in the other direction.
Ajeya Cotra: So itâs less weird because you know about God flipping the coin. And you kind of want to end up at 50/50 after looking at your box in this thought experiment. But the thing where you massively update in favor of being in the 10 billion boxes world over the 10 boxes world, when you take it to the real world, can be applied to basically be, you donât need to look at any physics to know that our world is spatially infinite.
Ajeya Cotra: You just know that, because you exist. Since there was some chance that the world is spatially infinite, and thereâs infinity update in favor of being in the world with infinite people. And so youâre just like, âI donât care what the physicists say. And I basically donât care how much evidence they find that the universe is finiteâ.
Robert Wiblin: I see. So you can either be presumptuous in the first case, or you accept this other presumption thatâs like, I can deduce from pure theory that the universe must be enormous âindeed infinitely large â because thatâs a world with so many more people in it, and so Iâm far more likely to be in that one. And so youâve got to bite one of these bullets, or accept one of these on unpleasant conclusions.
Ajeya Cotra: You could even take it further, because this thing about being a person is kind of under-defined, right? So you could be like, âActually I know with extreme confidence that thereâs infinite Rob Wiblins experiencing this exact thing, because that was physically possibleâ.
Ajeya Cotra: âAnd I have an infinity-to-one update that the world is just tiled with Rob Wiblins having Skype conversations with Ajeya right now. Because I would be most likely to be experiencing what Iâm experiencing in that worldâ.
Robert Wiblin: Does that end up being just the same as saying that the universe is infinite? I suppose it has to be infinite and not obviously repeating just some identical pattern that doesnât include me in it.
Ajeya Cotra: Well, but thereâs infinite universes with different densities of Rob, right? Some of them have more Rob, they happen to have more Rob, some of them. Like the physics is arranged such that Rob is a really common pattern.
Robert Wiblin: So I look out at the night sky and I see all these stars, but really I should be very overwhelmingly confident that thatâs an illusion. And in fact, all of that space is actually full of me having conversations with you.
Ajeya Cotra: Yeah.
Robert Wiblin: I see. That does seem counterintuitive. I was going to say that the presumptuous philosopher who thinks that the universe must be infinite⌠I guess weâve got some semi-confirmation of that by looking out at a universe that seems like it could be infinite, or at least we donât have strong reason to think that it is finite, based on the evidence that we have. So maybe they might get a pass on that.
Robert Wiblin: But yeah, the idea that itâs absolutely tiled very densely with us having this conversation is a far from appealing conclusion. So where do you fall on this?
Ajeya Cotra: I am more on the side of the presumption to big. So Iâm more on the side where when you wake up you think that youâre in the 10 billion boxes world, and then you update back to 50/50 or whatever. But I do think thatâs because I kind of want to end up with a normal conclusion, and I donât love the thing that I just said about this solipsist conclusion, basically.
The simulation argument [01:16:58]
Ajeya Cotra: So that leads into the second weird argument about how the future might be small, or why the ratio between the future and the present might be smallish, which is the simulation argument.
Robert Wiblin: Right, so I guess we have a bunch of things that all fit together a bit here. So weâve got this idea that, oh, we could influence a whole lot of people. But then weâre like, it doesnât feel right. It feels like this has to be overconfident somehow.
Robert Wiblin: And then the doomsday argument is one reason to think that, well, the future canât be big. And then it was like, well, how would that be? And then I guess the simulation argument potentially offers an explanation for how that actually would fit with our observations.
Ajeya Cotra: Yeah. So the doomsday argument assumes that thereâs really high, unavoidable x-risk or something. And thatâs why the future is small. But the simulation argument takes it in another direction.
Ajeya Cotra: The simulation argument says: Grant that thereâs a big future, with all these computations running, and a large number of flourishing humans in space, running on computers. In that world, then, some small fraction of their resources might be spent simulating worlds like ours â namely worlds where humans are on one planet, and they seem to be maybe on the cusp of colonising space in this way, in the next several decades.
Ajeya Cotra: Then in that case, if such simulations are even a pretty small fraction of the resources of this presumed giant future, like one in 1 million of the resources, or one in 100,000, then almost all of the people having experiences like ours are in simulations rather than in âthe real worldâ.
Ajeya Cotra: And the ratio of the value of what we think of as the future and what we think of as the present is basically bounded by one over the fraction of the resources in the future spent on simulation. So if they spent 0.1% of their resources on simulations, then the ratio of the value of the future to the value of the present is at most 1,000.
Robert Wiblin: Hmm. Okay. So you could have different arguments in favor of the simulation argument. So one would just be, say, you think itâs very intuitively plausible that we are going to go out and settle space and capture all this energy and have lots of very fast computers.
Robert Wiblin: And if we did that, it would be very plausible that we would simulate a time just like this very frequently, so that most of the people in a situation like the one in which we find ourselves are in simulations rather than the original â I guess people call it the âbasement universeâ. The one thatâs originally not simulated.
Robert Wiblin: Another angle would be to say, âItâs so suspicious that we look up at the sky and thereâs so many stars, so much space and matter that is being put to no use, and it seems like we could just go and use it. That would have the extreme implication that weâre at this very special time, and we have this potential to have enormous impact over lots of other beings. That canât be right. So I want a debunking explanation that makes the world seem more sensibleâ.
Robert Wiblin: And that explanation is going to be that, well, for whatever reason, even if I didnât think it was super plausible that future people would want to run a simulation of something as boring as this podcast, Iâm going to think that anyway. Or I independently have a reason to think that they are.
Ajeya Cotra: Yeah, that seems right. I think people have gotten there through both forks. I think Iâm a little bit more on the side of seeking a debunking explanation for what seems like this enormous amount of value lying on the table.
Ajeya Cotra: And I also have some, for myself, of the Fermi paradox. âWhy arenât there aliensâ would also be something that either the doomsday argument or the simulation argument could debunk.
Robert Wiblin: I see. Yeah, that makes sense. Okay, so youâre saying that if itâs so surprising that the universe seems barren of other life, even though weâre here, the doomsday argument would say, âWell, weâre not going to be here for long, and neither was anyone elseâ. And the simulation argument would say, âOh, itâs because weâre in the Truman Show and itâs not a real sky. Itâs a make-believe sky that theyâve put up there to entertain usâ.
Robert Wiblin: Okay. I think some people get off the boat with this a little bit because people start explaining why it is that future super-civilisations â potentially harvesting the energy from suns â would want to run a simulation of what weâre doing. And that starts to sound pretty kooky. Did you spend very much time thinking about the different rationales that people provide for why we would be here?
Ajeya Cotra: Yeah. I mean, so it kind of comes back to fanaticism a little bit, and how probabilities are weird, and Pascalâs muggings are weird. Even if you think itâs really unlikely, are you going to think itâs like a one in 10 to the 40 chance, if you previously thought there were 10 to the 40 persons in the big long-term future?
Ajeya Cotra: Because if youâre not willing to go there, if youâre going to say itâs a one in 10 million chance, youâve brought the ratio down of the future to the present from 10 to the 40 to 10 million, right? So I think itâs hard.
Ajeya Cotra: I kind of share your skepticism of, why would people be simulating us? And I kind of, in this project, went down a long rabbit hole about why that might be. But I think the ultimate thing is that we came up with this argument. We can imagine ourselves simulating our past, for whatever reason. We can imagine one in 1 billion of us being crazy historical replicationists or whatever, like Civil War reenactments. It just takes like a pretty small fraction.
Ajeya Cotra: Itâs quite hard to say that my probability of non-trivial amounts of simulation is anywhere in the range of one over the size of the future. And thatâs what it takes for the argument not to bite.
Robert Wiblin: I see. Okay, so some people might just get off the boat and say, âI just donât want to engage in this sort of reasoning at all. This is too much for me and so Iâm not going to use this style of reasoningâ. But I guess weâre not those people. So we have to answer the questions, like how much does this deflate the expected size of the future? We have to do something about, well, how likely is this kind of reasoning to be right?
Robert Wiblin: Is there some other very similar-ish argument that we havenât yet thought of that would also demonstrate that weâre in a simulation for some reason, or a reason that there would be lots of people in the situation that weâre in, that we havenât yet thought of? Or maybe thereâs a reason why they wouldnât do it, and so there wonât be so many? But then it seems like your entire worldview hinges on this wild speculation about this universe that you canât see and beings that you donât know. And how many of you they would want.
Ajeya Cotra: Yeah. I mean, it dampens the ratio of what you believe to be the long-term future to what you believe to be the present by a lot. But it also implies that most of the impacts of all of your actions â longtermist and near-termist â are to do with how you change the resource intensivity of the simulation youâre in, and what they would have used the resources for otherwise and stuff.
Ajeya Cotra: So, itâs kind of like the astronomical waste argument, in the sense that it re-frames all of your random actions as only mattering in that they impact the probability of getting to the long-term future. But it just replaces the probability of getting to the long-term future with whatever effects you have on the outside universe.
Robert Wiblin: Hmm. Okay. So how does this actually affect your estimates of what we ought to do? Is there any way of summarising that? Or is this an area where you just kind of threw up your hands a bit?
Robert Wiblin: I guess my take on this has been, it should affect it somewhat, but I donât really understand exactly how. And it doesnât seem like itâs such a strong argument that Iâm going to stop doing what I was going to do before I encountered this argument.
Ajeya Cotra: Iâm basically in a very similar place. I think I was at my peak astronomical waste fanaticism before doing this project. And then after doing this project, I was like, âWell, Iâve kind of discovered my limits in terms of where I get off the train to crazy townâ.
Ajeya Cotra: I am going to be living at the stop where I take astronomical waste very seriously, I take the idea of us being in a critical period of existential risk very seriously, Iâm probably going to spend most of my energies working on that. But I realise that thereâs a lot more out there in terms of, if you really have the goal to take philosophy as far as it goes, thereâs more stops past me. And I donât know where they lead.
Ajeya Cotra: Infinite ethics is another good one here, in terms of finding a non-broken version of total utilitarianism that works in an infinitely-sized world that avoids these paradoxes. I donât know if thatâs possible.
Ajeya Cotra: So I think I was kind of humbled by it. I basically gave up on this project because I was like, I donât want to get this to a publishable state and put in all the work it would take to go down the rabbit hole enough to come to particular conclusions on its basis.
Ajeya Cotra: But I do have more empathy for people who had the intuition that the astronomical waste argument is really weird. I sort of also had that intuition. And then I was like, âIâm being silly. Itâs a really strong argument. Iâm just being scope insensitive or somethingâ.
Ajeya Cotra: But actually one lesson here is just⌠Assuming a big world allows for a lot of crazy things to happen, and allows for a lot of sort of trippy questions to be raised and stuff. And so it is, in fact, a very weird and bizarre argument that opens the door to a lot of other weird and bizarre arguments.
Robert Wiblin: Yeah. I suppose I donât think of these arguments as crazy. I donât want to be like, âThis is crazyâ. I feel like I maybe stopped thinking about it, not because itâs too weird for me, because Iâm into plenty of weird stuff. But more just, I couldnât really see how me analysing this was going to shift my behavior. Or how I was getting traction on this from a research point of view, or from a practical point of view.
Robert Wiblin: Because one difference I feel is⌠The astronomical waste argument, or the idea that, oh, the long term matters a lot because it could be very big, or itâs people having really good lives⌠I feel like that cashes out in something that I can understand about how it might affect what I want to do.
Robert Wiblin: Whereas the simulation argument â and I guess to some degree, the anthropics-related uncertainties around the doomsday argument â they just feel so slippery. And Iâm just like, âI donât know where this leads, and Iâm not sure where I would even begin walkingâ.
Robert Wiblin: And so itâs very tempting, I guess, for me â and I suppose many other people have probably done this as well â to encounter these arguments and just be like, âI kind of give up, and Iâm going to stop where the tractability ended for meâ.
Robert Wiblin: But I guess I would love it if someone else much smarter than me could figure out what this actually does imply for me in my life. And then maybe I would take that seriously.
Ajeya Cotra: I would definitely be interested in funding people who want to think about this. I think it is really deeply neglected. It might be the most neglected global prioritisation question relative to its importance. Thereâs at least two people thinking about it on a timeline. So zero people, basically. Except for Paul in his spare time, I guess.
Robert Wiblin: Yeah. Paul Christiano, that is.
Ajeya Cotra: I think it could have implications⌠Yeah, Paul Christiano.
Ajeya Cotra: I think it could end up having implications for how we think about AI, and how worried we are about misaligned AI and stuff. I donât know exactly.
Ajeya Cotra: If you imagine weâre being simulated for some purpose by an outside universe, then do we want to align artificial intelligence with our goals? Or are we mostly trying to think about why we were simulated, and use AI to help us figure out how to give the outside world whatever it is they wanted? Or should we even be cooperative with the outside world?
Ajeya Cotra: And if we should be cooperative with the outside world, then does that meaningfully change how upset we are about misaligned AI? It might be misaligned with us, but we want to think about what it means for it to be aligned with where all of the value actually lies, which is the outside basement universe, or whatever.
Ajeya Cotra: I think it could have implications. Iâm not the person to think about them, but I would be very excited for other people to think about them.
Robert Wiblin: Yeah. I guess another line of argument that people have made is, well, letâs assume that we are in a simulation for the sake of a hypothetical. That means that there must be some reason why theyâve decided to run this simulation. And it means this world must be kind of interesting in some way. People write really funny tweets, and I guess Wikipediaâs good, and Netflix produces some good showsâŚbut that seems like itâs not quite enough for them.
Robert Wiblin: So, what would be the reason? And then weâre like, âWell, maybe we should go look around. What actually would be of interest?â And then I suppose people would say, âWell, maybe weâre at the cusp of some really important moment in history. Something that really would be of interest to future generationsâ. And theyâd be like, âWell, it could be a massive war or it could be development of new technology thatâs super revolutionaryâ, of which I guess AI and some biotech stuff might be on the list.
Robert Wiblin: It seems like youâd be more likely to simulate something that was really historically epically important. And so thatâs a reason to expect, if we are in a simulation, for it to be more interesting in some way. And then maybe it can be like, âWell, what does that imply?â And I donât know.
Robert Wiblin: Alright. So I guess we both find this interesting and slightly exasperating. I would be very happy for someone else to write papers that then would get us off the hook for thinking about it anymore.
Ajeya Cotra: Yes. Iâm very excited about people⌠I think itâs rare to find someone who has both the capacity and the stamina or patience for this kind of thinking, so I think itâs quite neglected and could be really high impact if you find yourself excited about it.
Robert Wiblin: Yeah. Alright. Well, weâll stick up links to the simulation argument paper, and I guess a few blog posts that flesh out possible consequences of possible things that we might infer from it, for people who want to go and explore that.
Robert Wiblin: Letâs talk now about another whole area of research that youâve been working on over the last two years, which has been this report called Forecasting Transformative AI With Biological Anchors. What was the goal of that project, and why did it matter?
Ajeya Cotra: Yeah, so this project came about because potential risks from advanced AI is a major Open Phil focus area. And one of the important considerations feeding into how much we should be allocating to risk from AI versus other longtermist goals â and also potentially how much we should be allocating to longtermism as a whole versus near-termism â is how urgent the problem of AI risk is, and how soon is it on the horizon, and how much can we anticipate now, and do things that we can expect will affect it without being washed out by a whole bunch of stuff that happens in between. So how soon really powerful AI systems are going to be developed is an important strategic question within the longtermist worldview, and sort of indirectly like an important question about how urgent the longtermist worldview as a whole is, and therefore how much weight it should get versus the near-termist worldview.
Robert Wiblin: Okay. So the goal is to try to figure out whatâs the likelihood of us having transformative AI by different dates, I guess to figure out how urgent it is to try to make sure that that goes well? Because if itâs not going to come for another 100 years, then we can potentially punt that to another generation or another philanthropic organisation to figure out in the future?
Ajeya Cotra: Yeah.
Robert Wiblin: And what was the approach that you took, and I guess how did it evolve over time?
Ajeya Cotra: Yeah, so some quick background on this project on Open Philâs side: In 2016, Holden wrote a blog post saying that, based on discussions with technical advisors who are AI experts â who are also within the EA community and used to thinking about things from an EA perspective â based on discussions with those technical advisors, Holden felt that it was reasonable to expect a 10% probability of transformative AI within 20 years. That was in 2016, so that would have been 2036. And that was a kind of important plank in the case for making potential risks from advanced AI not only a focus area, but also a focus area which got a particular amount of attention from senior generalist staff. So that there are a number of people thinking about aspects of AI at Open Phil.
Ajeya Cotra: That was roughly around when we opened up a focus area, and decided to make it a particular focus of senior staff, and weâve been in that area for a few years now. And then in 2018/early 2019, we were in the middle of this question of weâre hoping to expand to peak giving consistent with Cari and Dustinâs goals to give away their fortune within their lifetime, and we want to know which broad worldviews and also which focus areas within the worldviews would be seeing most of that expansion. And so then the question became more live again, and more something we wanted to really nail down, as opposed to kind of relying a bit more on deference and the earlier conversations Holden had.
Ajeya Cotra: And so digging into AI timelines felt like basically the most urgent question on a list of empirical questions that could impact where the budget wentâŚ
Robert Wiblin: âŚwhen and how much.
Ajeya Cotra: Because⌠Yeah, and potentially also how we should broadly strategise about what we do with the money within the AI focus area. Because it matters what tactics we take. What tactics pay off over what time scales matters for what weâd prioritise within that area.
Robert Wiblin: Alright. If I was a better person, we would eat our greens before we had our dessert, and we would walk methodically through all of the methodologies youâve used â and the various pros and cons â before we got to any conclusions. But Iâm not a patient person, so itâs not going to take forever to get there. What were your bottom line conclusions about timelines, in brief? Did you end up thinking transformative AI may come sooner or later, or maybe even that itâs just like maybe not even possible?
Ajeya Cotra: Yeah. I think the methodology I used is a little bit more robust for medians rather than either tail. So my median â depending on how Iâm feeling on a particular day â ranges anywhere between 2050 and 2060 in terms of, âI have this model and there are some parameters Iâm particularly angsty aboutâ. So thatâs between 30 to 40 years from now. And I think thatâs a quite extreme and stressful and scary conclusion, because Iâm forecasting a date by which the world has been transformed. Iâm imagining a lot is happening between now and then, a lot of wild stuff in 10 years, a lot of wild stuff in 20 years, if the median date is 35 years for fully transformative AI.
Robert Wiblin: And I guess it could also come sooner or it could also come later. So thereâs uncertainty, which I guess also might make you nervous.
Ajeya Cotra: Yeah.
Robert Wiblin: Is that sooner or later than what you thought before you set out on this project?
Ajeya Cotra: The probability by 2036, where Holden had originally said at least 10%, Iâm bouncing between 12% and 15%. So itâs definitely consistent with the at least 10% claim. With that said, the at least 10% claim was trying to shade conservative, and I think the best guesses of a number of people at Open Phil were higher than the 12â15% that I landed on. And my own best guess was more like, âOh maybe 20%, maybe 25%â. And so I think for me, it was sort of numerically an update toward longer timelines, but it also made it seem more real to me and made it seem like there was going to be a lot of stuff in between now and transformative AI. So emotionally, Iâm maybe a little bit more freaked out.
Robert Wiblin: Interesting. Okay. So your timelines moved out, so you thought maybe itâll take a bit longer, but then it felt more concrete and less speculative, and like a real thing that could really happen, and that made you say like, âOh wow, this actually really mattersâ.
Ajeya Cotra: Yeah. And it more feels like thereâs quite high probabilities of some amount of powerful AI that could have unpredictable consequences, if not fully transformative or human-level AI. So thereâs something where I think I was previously thinking of AI as kind of like biosecurity, which is like, weâre doing preparations now for a sudden event that might or might not happen that could change the world in this very discreet way. And that was one flavor of scary. But now Iâm thinking of AI much more viscerally, as this onrushing tide. AI is getting better and better. You can certainly do some stuff with AI, maybe itâll take us all the way to transformative AI, but itâs more relentless and more changing what the world looks like on the way there. And so thatâs a different flavor of scary.
Robert Wiblin: Yeah. So I guess you said that this is in a sense an aggressive forecast, but from memory, the big survey or the forecasting survey that they did of ML experts or AI research scientists a couple of years ago, I mean, it had people all the way from 2025 to 2100, I guess, and a few outliers beyond 2100. And so this is pretty consistent with that. Itâs just like, well, sometime in the next 100 years, and maybe 30 or 40 years sounds like about the median estimate. So itâs not out of line, I guess, with what some other people have said, although you thought about it a lot more.
Ajeya Cotra: Itâs not wildly out of line with one interpretation of that survey⌠So that survey asks several different questions. In my mind, the headline results of that survey is that the researchers were quite inconsistent with themselves, in terms of.. Asking when AI can do all of the tasks a human can do leads to sooner timelines reported than asking when AI will be able to do AI research. There were many particular tasks for which the timelines were substantially longer than the timeline for all tasks. And so it depends on how you think the researchers would net out if forced to reflect. Like, I believe the median for the all tasks thing was like, maybe it was 2060 or something, like it was 50 years out or it was like a little bit longer than what Iâm saying. But youâre right that itâs not a lot longer. But then the medians for some of the other ones were more like 80 years for particular tasks that the researchersâ survey implied would take longer than just the all tasks question.
Robert Wiblin: Yeah. Although your threshold here is transformative AI, which I guess is like a substantially bigger shift than just AI being able to do the tasks that humans can do. Or am I misunderstanding, is this a higher or lower threshold?
Ajeya Cotra: Thereâs a question of what will happen to the world if we have AI that can do most of the tasks humans can do. And we believe itâll be wild, and it will lead to the world moving much, much faster than itâs currently moving, and will lead to future technologies being discovered at a much more rapid pace â in part because AIs can be made to run faster in subjective time than humans, and in part because they have other advantages, like they can be arranged to have really perfect memory, no sleep, all that stuff. And so one disconnect is that I donât think that most of the people who answered that survey are imagining the consequences of AI that can solve most human tasks to be as radical as what weâre imagining. So what we define as transformative AI is like⌠In Holdenâs blog post, he defined it as AI that has at least as profound an impact as the Industrial Revolution.
Ajeya Cotra: And then in my report, I have a more quantified operationalisation thatâs roughly, AI that is the primary driver of growth rates. The growth of the world economy is ten times faster than it is in 2020. So in 2020, the world economy is growing at 2â3% per year, and thatâs roughly 25â30 years to double the economy. And so if you imagine growing 10 times faster, thatâs like two to three years to double the economy. And the reason weâre choosing that threshold â itâs an arbitrary threshold â but the idea is that once weâre at that case, there are probably further speed-ups in the future. And basically history is compressed when growth is really fast. And so once the world is doubling every two to three years, human plans and human timescale actions become dramatically less relevant to the world, and thatâs the threshold weâre looking to forecast.
Recent AI developments [01:39:28]
Robert Wiblin: So it feels to me as a casual observer that this field is moving really fast. I guess this year weâve had GPT-3 andâŚ
Ajeya Cotra: Yeah, AlphaFold.
Robert Wiblin: People are amazed at what they can do, and now we got AlphaFold, yeah. Did it feel a bit like by the time you finished this, that things have moved on a bit, andâŚ
Ajeya Cotra: Yeah.
Robert Wiblin: Two years is so long in the AI world.
Ajeya Cotra: It definitely felt that way, and I definitely questioned whether this general approach of doing these detailed forecasts and careful investigations is sustainable or reasonable. I think I had a bit more of an exaggerated sense of that. As we were writing the report, I had preliminary conclusions in mind in early 2020, and I was working on really getting things right and really nailing things before publishing. And then in the meantime, GPT-3 was published. And I think GPT-3 really caused a major shift in a lot of academicsâ views of timelines, or like gut-level views of timelines. Where before, a lot of academics were fairly dismissive of GPT-2, but I saw very little dismissive attitudes towards GPT-3. And I saw a lot of people who just straight up said on Twitter, âI thought AGI was at least 50 years away, but now I think itâs 10 years awayâ.
Ajeya Cotra: And so a lot of my report is framed as addressing a sceptical academic audience, and convincing them that itâs reasonable to expect substantial probability of crazy AI capabilities soon. And now, at least some chunk of those people â because of GPT-3 and other developments â they feel like itâs belaboring the point, and they want to argue with me from the other side. They now think the timelines are much shorter than I think they are. And I have counter-arguments to their views too, but the whole orientation was addressing an audience that shifted in between when I was writing it and when I got to publish it.
Robert Wiblin: Yeah. I guess itâs slightly satisfying in that? You have some confirmation that maybe you were right, if theyâve already changed their mind.
Ajeya Cotra: I wish I had gotten it out and had a lot of people arguing with me that it can never happen.
Robert Wiblin: Yeah.
Ajeya Cotra: But I think itâs not so much⌠I feel kind of salty about that or something, but itâs not a big deal. And I ultimately do think itâs not fast enough yet⌠At least, if youâre trying to be smart about it and trying to be efficient, I think thereâs still plenty of room to do analysis like this.
Robert Wiblin: Yeah. How did this shift your attitude towards AI safety research? And I guess in the same vein, do you do anything differently in your life outside of work because of your expectation that AI may advance pretty quickly and really change the world while youâre still alive?
Ajeya Cotra: In terms of how it affects my vision for AI, I think coming into this I was a partisan to the âgradual take-offâ view â probably AI is going to be like other technologies in the sense that thereâs going to be less powerful AI first, itâs going to proliferate, thereâs going to be lots of different things you do with it before you get full-blown human-level AI. And so that was kind of my bias coming in, but I do think that doing this work made me feel more confident about it and feel more like I have some picture of how the gradual take-off would go. And part of it is not so much the particular research that I did in the report, but just spending a year immersed in ML and realising, âOh yeah, thereâs stuff that I think will improve in the next five years based on this technology.â
Ajeya Cotra: And so that was one thing. And then that kind of leads to a proliferation of other expectations. One is like, it makes me think thereâll probably be early failures of AI systems all over the place. You know, systems that are doing pretty important (but not that important) stuff will fail in ways that are kind of analogous to the ways we worry that even more critical, more general systems will fail. I think that we can have an AI safety issue thatâs far short of a global catastrophic risk, but gets everyoneâs attention on this problem.
Ajeya Cotra: And I think on net that will probably be a really good thing, and will probably relieve a lot of what Open Phil is feeling now, with not having people who share our vision of what the main problems are to fund. But it could also present challenges that I think maybe the EA community hasnât thought through as much, because weâve been so used to this world where weâre a very small set of people who care about this.
Ajeya Cotra: I think it might not be possible to play AI safety as an inside baseball game. We might need to think more about how we message to the public. We might need to think more about how maybe itâs actually quite inevitable that this will become a top-tier political issue that has a lot of eyes on it, like climate change.
Robert Wiblin: Yeah. I was going to say itâs going to look a bit more like climate or COVID or something like that.
Robert Wiblin: Okay. Letâs go back to learning about the methodology, and inspecting whether it makes sense. So the title is Forecasting Transformative AI With Biological Anchors. What does it look like to forecast something using biological anchors? What does that mean?
Ajeya Cotra: Thereâs a history of trying to forecast when we get artificial general intelligence by basically trying to estimate how powerful the human brain is â if you think of it as a computer â and then trying to extrapolate from hardware trends when weâll get computers that powerful. So this has a long-ish tradition in the scheme of futurism. Ray Kurtzweil and Hans Moravec in the â80s and â90s were thinking along these lines. And I think the big flaw in the earlier iterations of this thinking is basically that they werenât thinking about the effort it would take â either software effort or machine learning training effort â to find the program that you should run on the brain-sized computers once you had the brain-sized computers.
Ajeya Cotra: So I think it led them to estimate timelines that were too aggressive because of this. The Moravec timeline that he estimated in the â90s was that roughly 2020 would be when we would have human-level AI. Because he said, âWell, this is how powerful I think the human brain is as a computer, and this is when I think computers of that size will become widely available. Theyâll be like $1,000, so somebody could just buy themâ. And he was actually right about the second half of that. He was right that computers roughly as powerful as he estimated the human brain to be would be available around now. But â and people pointed this out at the time â he wasnât accounting for, well, actually it took evolution many, many lifetimes of animals to find the human brain, the human mind, the arrangement that needs to occur on this hardware.
Ajeya Cotra: And so in 2015/2016, our technical advisors â particularly Paul Christiano and Dario Amodei â were thinking about how to extend this basic framework of like, think about how powerful a computer would need to be to match the human brain in raw power. But add on top of that, well, how is the search for that going to work? Like if we were to do something like evolution, or if we were to do something like ML training, how much money, how much computation would it take to do that? And then when would that be affordable? Which pushes timelines out relative to what Moravec was thinking about â which was just when the one computer to run the one human brain might be affordable.
Four key probability distributions [01:46:54]
Robert Wiblin: Okay. So at the heart of your model⌠Estimate four different probability distributions for four different components, and then put them together to estimate the likelihood of us being able to run transformative AI at acceptable cost by any given day. Can you maybe walk us through the four key probability distributions that you need to estimate here?
Ajeya Cotra: Yeah. So one is, letâs say we wanted to train a single ML model that would constitute transformative AI, which I just call a âtransformative modelâ. Letâs say we wanted to do that given todayâs algorithms. So whatever architectures we can come up with given what we know today â and whatever gradient descent or whatever ML training techniques we could come up with today â how much computation would it take to train a transformative model? And so thatâs one piece of it. That produces this wide probability distribution which I call the â2020 training compute requirements distributionâ. Thatâs a mouthful. And then basically, conceptually, once you have that, that is a snapshot of one year. But you expect that algorithmic progress is going to cause us to get better at doing any given thing over time. So you expect that if the median computation is X in 2020, then maybe in 2030, itâs gone down 10 times. So now itâs like (0.1)(X). So thatâs the second thing, the algorithmic progress piece. Itâs just trying to translate from 2020 to a future year.
Robert Wiblin: Okay. So first off youâve got like how much computational power would we need today, given what we have now to train a transformative AI. And then youâve also got this other thing of like, over time that is actually declining, because weâre getting better at training ML algorithms, they can do more with less compute. And then so youâve got an estimate of like, how strong is that effect?
Ajeya Cotra: Yeah.
Robert Wiblin: Okay. And then whatâs the third and fourth?
Ajeya Cotra: And then the third and fourth are basically together estimating how much computation a lab, or some government project, or some project trying to train transformative AI would have available to it. So one piece of that is how hardware prices are declining over time. So you can buy more computation with $1 in the future than you can now. And then another piece of that is how investment is increasing over time. So as people sort of see more potential in AI and as the world as a whole grows richer, then the frontier project will be willing to put in more money to attempt to train transformative AI.
Robert Wiblin: I see. So these are kind of the economic ones. So the first one is how quickly do the really good computers get cheap, and then you want to divide that, I guess, by how much are we going to be willing to spend to try to train a transformative model. So is it a trillion dollars, a hundred billion dollars, yeah, how many resources will we throw at this.
Ajeya Cotra: Yeah.
Robert Wiblin: Okay. Letâs go back to the first one, which is how much compute would you need to train a transformative model now. How did you try to estimate that?
Ajeya Cotra: This is where the biological anchor part comes in. So there are a number of different hypotheses about how good our algorithms are today. So one of them is, actually, we will need to sort of replicate the process of natural selection that created humans. And we can think about how expensive it was for evolution to lead to humans, and then think about, are our algorithms better than that or worse than that? And make some adjustments there, and generate that probability distribution. And then on the other end, actually our algorithms are within striking distance of human learning. So like the learning that a baby does as it grows up into a functional adult. And so we can think about how much computation that constitutes, because we have an estimate of how powerful the brain is, and we know how long it takes to grow up to be a functional adult, and then we can think about making adjustments from there. And so those are two anchors, and then there are more complicated anchors that fill out the middle.
Robert Wiblin: Okay. So those are both kind of crazy anchors in a sense. Or theyâre both at two quite extreme levels. Because I suppose the evolutionary one assumes that even though weâre designing this process, we canât do any better than natural selection has done designing the human brain over billions of years, or I guess at least hundreds of millions of years in effect. It was like so many people. Simulating all of their brains for all that time. It seems like surely we can do better than that by quite a decent margin, or youâd hope. And on the other end, weâre just imagining well you could train a model just as quickly as you could teach a baby to do things. Well, that doesnât seem quite right, because I guess a baby is born with some kind of innate knowledge, or it tends to develop naturally with a whole bunch ofâŚ
Ajeya Cotra: From evolution, it has some sort of like⌠Yeah.
Robert Wiblin: Yeah. Itâs got all of that pre-learning that itâs inheriting. And I guess on top of that, we know that babies just learn stuff so much faster than our current ML models. Theyâre much, much better at generalising from single cases. So you want to be like, this is maybe an upper and a lower bound, and then we want to find something in the middle.
Ajeya Cotra: Yeah. So the lifetime anchor is kind of like, we basically know that exactly the lifetime doesnât work. We can afford that right now, and like you were saying, we can observe that babies learn much faster than ML models. So the slightly more sophisticated version of the lifetime anchor is that maybe thereâs just a constant factor penalty. So maybe if we observe that our best models today, they took 10,000 times as many data points to learn something as a baby does, then maybe thatâs just it. Maybe itâs just however long a human takes to learn something times 10,000. Or times 1,000. Or whatever the thing we observe is, which is between 1,000 and 10,000. But I actually believe that that is not the case, and in fact the factor by which ML is worse than a baby grows as the model gets bigger.
Ajeya Cotra: So with our tiny models that are learning really simple things, the factor is smaller, and then for the bigger models learning more complex things, the factor gets bigger. And so the middle hypotheses are based on this empirical observation that training a bigger model â where bigger models can generally do more complicated, interesting things â training a bigger model takes more data. And thereâs some scaling relationship between the size of the model and the data it takes. And thatâs where most of my mass is. Most of my mass is on hypotheses that try to cash out, âOkay. How big should the model be?â Somewhere in the zone of a human brain, maybe slightly bigger, maybe slightly smaller, and we want to think about where that lands. And then given that, how much data does it take to train?
Robert Wiblin: So Iâve been hearing about these scaling laws a lot recently, but I donât feel like I have a great grasp on what they are. Maybe you could explain for me and the audience what we need to know about scaling laws?
Ajeya Cotra: Yeah. So the fundamental thing is that if youâre trying to solve some task to some level of proficiency, like letâs say I want to get to at least grandmaster level in chess, then you need to pick a model that is big enough that itâs capable of learning to be at grandmaster level â bigger models are more capable of learning more complex strategies and stuff. So thereâs some size of model thatâs big enough to learn to be a grandmaster at chess. And then you need to train it for long enough that it realises that potential. And so a model, you can think of it as like a giant collection of numbers, and those numbers store knowledge it has about the task at hand.
Ajeya Cotra: At first, these numbers are all randomly initialised. These are called parameters. And then training sets the numbers to be reasonable values, and then they can be interpreted or they represent for the model its knowledge of how to play chess. So bigger models are capable of representing more sophisticated strategies, theyâre capable of remembering more openings, etc. But they also need to see more experience to fill up all of those numbers. So roughly speaking, the amount of experience they need is going to be proportional to the number of numbers they need to fill up. Machine learning theory says that you should expect that to be roughly linear. So if you need to fill a knowledge bank of 10 million numbers, then you need to see like 10 million examples or maybe a hundred million examples. But recent empirical work suggests that itâs actually sub-linear. So itâs more like it scales to the 0.75 or something.
Robert Wiblin: Okay. So people are talking about this in part because recent discoveries have suggested that if you want to make a model thatâs twice as sophisticated â in terms of its memory, or how many strategies it thinks about â you only need 1âŚ
Ajeya Cotra: 1.7 or 1.5 times as much data.
Robert Wiblin: Okay. So itâs not quite as data-intensive as we previously thought?
Ajeya Cotra: Yeah.
Robert Wiblin: Okay. Going back to the question of how much compute it would take to train a human-level model with todayâs algorithms, is there any intuitive⌠I guess itâs an answer of a certain number of flops⌠10 to the power of something? Is there any intuitive way of communicating what probability distribution you gave it?
Ajeya Cotra: Yeah. I mean, first of all, my colleague Joe Carlsmith recently put out a great detailed report on what evidence we can gather from looking at the brain about how powerful of a computer youâd need to replicate the tasks of the brain, if you somehow found the right software. And so his estimate is that something in the range of 10 to the 13 floating point operations per second to 10 to the 17 floating point operations per second was probably sufficient based on evidence from looking at the brain, with a central estimate of 10 to the 15 flops. So Iâm leaning a lot on that. And he goes through several types of lines of evidence you could look at. But my question was kind of like, well, Iâm thinking about 2020. This estimate of brain computation would have been the same whether we estimated it in 1960 or like 2050, right? But thereâs only one period in time where weâre exactly dead on competing with biological counterparts. Before that will be worse than the biological counterparts, and after that, weâll probably be better than the biological counterparts.
Robert Wiblin: And by we, you mean the machines we make?
Ajeya Cotra: Yeah. Our design versus evolutionâs design. So weâre not thinking about how hard it was for us or evolution, but just likeâŚonce we design a product, how much better or worse is it than the product evolution designed thatâs analogous? So there, I was just kind of trying to subjectively gauge this by looking at machine learning models we have today that are attempting to do something analogous to what creatures do. The one that we have the richest amount of data for is vision. And then thereâs also motor control and how sophisticated are their movements. And then thereâs a whole other line of evidence that is actually easier to think about, which is just, forget about machine learning and think about other technology and how it compares to other natural counterparts.
Ajeya Cotra: So think about cameras and how they compare to eyes, or think about leaves and how they compare to solar panels, stuff like that. So looking at all that stuff, it seems like roughly, in the current moment, the artifacts we design are somewhat worse than the artifacts that evolution designs. And so I estimated that it would be about ten times larger than Joeâs estimate for the human brain.
Robert Wiblin: Wouldnât that stuff just be all over the place, because youâve got satellites that can see people from space and my eye canât do that. And I guess if you were like, âHow good is Robâs arm compared to this truck that can pick up an insane amount of weight?â I mean, from one point of view, maybe the truck isnât as nimble as I am, but from another point of view, it can pick up much more than I can, so it just seems itâs going to vary a lot depending on the measure.
Ajeya Cotra: Yeah I mean and on the other side, we have these cells that are like crazy nano-machines and we have nothing like that.
Robert Wiblin: Yeah. Right. I think, what is it⌠Photovoltaic cells are actually more efficient than leaves, but they donât reproduce themselves from a seed. And so from one point of view, theyâre much more impressive.
Ajeya Cotra: Yeah. Right. So the basic answer is that you⌠It is definitely really fuzzy, and youâll have people who will make the argument you made, that human technology is way cooler than nature. And youâll have people who make the argument I made, that weâre not anywhere in the ballpark. The way I try to think about it is like, you want to look in those places where humans and nature have similar cost functions. So it actually mattered to them to get the benefit, and it actually mattered to them to avoid the cost. So often in this case the cost is energy to run the thing. Because that is something humans care about because it costs money, and itâs also something nature cares about because you need the animal to be better at finding food in order to sustain a more energy-intensive thing.
Ajeya Cotra: So long-term transportation and super heavy lifting is not something that conferred much of a fitness benefit on anything. And so itâs unfair to grade nature on that basis. I think itâs actually a little more fair to grade us for not getting nanotech, but itâs still potentially kind of like, we werenât in the game, so to speak. So like thereâs something where itâs like technology that humans are in the game of making, that nature had a strong incentive to make because there was a strong fitness incentive to be good at it. Which I donât think is the case for long-distance travel and trucking and stuff.
Robert Wiblin: Okay. So youâre saying us as engineers, we take a big hit because nature has managed to produce self-replicating tiny machines and we havenât done that yet. It doesnât even seem like weâre that close. And so at that weâre quite a bit worse.
Ajeya Cotra: Yeah. We are better at these other things, like weaponry, right? I think it would confer a fitness advantage to be as good at killing things as human machines are. But I mostly try to focus on the regime where⌠Because I feel like the machine learning models are in the regime where theyâre trying to do the same things and not totally failing at it, Iâm mostly looking at other technology that also seems to be in that regime.
Robert Wiblin: I see.
Ajeya Cotra: So not thinking that much about nanotech.
Robert Wiblin: Yeah, I see. Okay. So itâs cameras and eyes and photovoltaic cells and leaves and things like that.
Ajeya Cotra: Yeah.
Robert Wiblin: Okay, letâs push on to the second component, which is how quickly our algorithms are getting better at learning and running with less compute over time. How do you estimate that, very broadly speaking? What was the conclusion?
Ajeya Cotra: Yeah, so broadly speaking, there are basically two papers Iâve seen on this. Really pretty small literature. One is Katja Graceâs paper from 2013 called Algorithmic Progress in Six Domains, and then the other is a recent blog post from OpenAI called AI and Efficiency. Both of those are basically doing a pretty similar thing for different tasks, which is theyâre asking, over successive years, when thereâs some benchmark, how much less computation did it take in future years to beat the previous state of the art?
Ajeya Cotra: And so from that, they found these trendlines. Where I believe like the trendline from AI inefficiency was halving every 16 months â or maybe 13 months â the amount of computation it takes to achieve the performance on ImageNet that was achieved in 2012. So in 2012 there was this big breakthrough year where image recognition models got a lot better, and from there we can ask how much more efficiently we can get that same level of performance.
Ajeya Cotra: And then Katja Graceâs paper looked at a lot more different tasks, but few of them were machine learning tasks. She looked at factoring numbers, and there were a bunch of other things. In that paper, it was kind of between 12 months and two years or something, two and a half years.
Robert Wiblin: âŚthe halving time?
Ajeya Cotra: The halving time, yeah. And so I was just kind of like okay, letâs take that as a starting point. And then I shaded upward a little bit because with transformative models, for all these tasks, you achieve a benchmark and then you work on making it better. You have really strong feedback loops. But with transformative AI thereâs something that is currently at an unachievable level of cost, but slowly coming down until we can meet it. Halving time for that would be slightly longer, because youâre working on these proxy tasks. And what you can actually work on and improve are benchmarks that youâve already seen. But you expect thereâs some translation between that and the ability to train a transformative model.
Robert Wiblin: Yeah. Okay, so I guess this one shouldnât be massively uncertain, at least in the past, because itâs kind of measurable in theory. Maybe some of the uncertainty comes in because these trends keep continuing, until they donât. So, projecting it forward is⌠At some point presumably weâll level off, and weâll have the best algorithm possible, and you canât get any better than that, so itâd be stuck, but we have no idea where that point is going to be, or how fast weâll get there.
Ajeya Cotra: And I do assume a leveling off, but itâs pretty arbitrary where I put it.
Robert Wiblin: Yeah. Okay, that makes sense. Letâs move onto the next one, which I guess is probably going to be pretty familiar to everyone, because itâs kind of projecting Mooreâs Law, or I guess we used to have Mooreâs Law, where itâs chips got ⌠What was it? They halved in price every year or two or something? And recently things havenât been going as fast as that, but basically youâre just trying to project that issue of the cost of an equivalently good computer chip over time.
Ajeya Cotra: Yeah, and there I think the biggest uncertainty â which I havenât looked into as much as Iâd like â is not so much the speed of the halving, but where it cuts off. I think thereâs a lot of work one could do in terms of thinking about the physical limits of different types of computing, and what is actually the best physically realisable energy efficiency of computation. I instead did an outside view thing where I was like, âWell, over the last 100 years⌠Or over the last 70 years, we had 12 orders of magnitude of progress. Maybe over the next 80 years weâll have half of that in log spaceâ. I just assumed we would have six orders of magnitude of progress. Itâs not very thoroughly done.
Robert Wiblin: Okay. Remind me, we had Mooreâs Law-ish, roughly being followed for something like 40 years, and then the last 10 years things have been noticeably slower than that? Is that because we were kind of approaching the limit of what we can do, given the current paradigm, given the current materials that weâre using?
Ajeya Cotra: Yeah, my understanding is that weâre getting to the point where itâs hard to make the transistors in the chips smaller without leading to issues with overheating, and potentially even issues with⌠Theyâre only a few atoms big at that point. There might be quantum effects that I donât understand very well.
Robert Wiblin: I see, okay. What did you project forward? Did you project forward this kind of slow progress that weâve had now, or do you think at some point it will speed up because weâll come up with a different method?
Ajeya Cotra: Yeah, so the most recent thing, like the jump from the previous state-of-the-art machine learning chip to the most recent state-of-the-art machine learning chip â the previous was the V100 and the current is the A100 â seemed to be bigger than the slightly less recent 10 year trend you were talking about. And then I also think thereâs room to change substrates, like thereâs a lot of activity in the startup world, researching optical computing â where you basically use light and a bunch of tiny mirrors instead of silicon chips to transfer information. So, itâs not going over a wire, itâs just like being bounced off a mirror so it can go faster potentially.
Robert Wiblin: Wow, thatâs going to be huge for the tiny mirror industry.
Ajeya Cotra: Yeah.
Robert Wiblin: Presently not a very large industry, I think. Sorry, go on.
Ajeya Cotra: I just split the difference and I was like, âOh well, itâll be kind of slower. Mooreâs Law was one to two years, the recent trend is three to four yearsâ⌠I just said it would be two and a half years halving time.
Robert Wiblin: Alright. And the last one, which probably I havenât thought about that much, and most people wouldnât have thought about that much, is how much is society willing to pay to train this amazing breakthrough AI model? Which I guess⌠Currently I donât know, how much is Google or the government or whatever the best research projects are⌠How much are they paying for those models? And I guess presumably over time itâs been going up, as they seem to have more applications.
Ajeya Cotra: Yeah. So, in terms of publicly calculable information, the most expensive two models, which are in a similar range to each other, are GPT-3 and AlphaStar, which is the StarCraft model. AlphaFold might be more, I havenât looked at AlphaFold, but both GPT-3 and AlphaStar are in the 1â10 million range of computation to train. And then thereâs more stuff that you need to do, like training runs you do to tinker with stuff before the final training run, and the cost of data and the cost of labor, and stuff Iâm not counting here. But just purely from computation, if you sort of math out how much computation it probably took to train them based on how big they are, itâs in the 1â10 million range.
Robert Wiblin: Itâs so little!
Ajeya Cotra: Yeah, itâs really small.
Robert Wiblin: Breakthroughs in biomedicine, like you were saying earlier, can cost billions. I guess when you consider personnel and materials and so on. And this is so⌠I guess maybe most of the cost isnât in the compute, itâs in the people, or something. But even so, itâs surprisingly small. We could spend much more than that.
Ajeya Cotra: Yeah. Right now itâs not the case that compute is most of the cost of these things. But I think probably eventually it will be the case that compute is the dominant cost, or at least like half. You might as well go to half, because youâre not even increasing the cost of your project that much. So, my basic forecast is that⌠So, OpenAI has another blog post called AI and Compute, in which they show that the super recent trend of scaling up the computation of state-of-the-art results is doubling every six months or something, or every nine months. Donât quote me on that. I think itâs six months.
Ajeya Cotra: So I basically assumed that that trend would hold until models were big enough that people would start to demand real economic value from them, before scaling up more. And then from there, I expected a slower trend to hold, that kind of converged to some fraction of gross domestic product. I was kind of like⌠Big mega projects in the past tend to be projects that last anywhere from three to five years, and then over the course of that project they spend a couple percent of GDP.
Ajeya Cotra: So, I was assuming that a project to train a transformative model would be similar in those economics in the long run, and maybe compute would be half or a third of the total cost of the project. I assumed 1% of GDP eventually. But thereâs like, basically Iâm imagining a super fast initial ramp-up that lasts just a couple more years, to get up to a billion, 10 billion, and then like much slower ramp-upâŚor somewhat slower ramp-up, thatâs more like doubling every two years, from there, to get up to 1% of GDP, and then following GDP from there.
Robert Wiblin: Okay. So, things will kind of follow what seems to be the current trend, is your prediction, roughly. Until, say, we get to something like⌠Now weâre talking about the Manhattan Project, or the Apollo program, or the Three Gorges Dam, or something thatâs really actually material â at which point itâs got to demonstrate economic value, or thereâs just no one willing to fund this thing. And at that point maybe the growth levels off, at least until the thing actually works enough to increase GDP, such that it can pay for itself.
Ajeya Cotra: Right, yeah.
Robert Wiblin: Alright. So weâve got the four different pieces. Then I guess youâre multiplying and dividing them together to get some overall thing.
Ajeya Cotra: Doing some math.
Robert Wiblin: Doing some math. I guess you do a Monte Carlo thing? To sample from each of these distributions on each one, and then produce a final distribution?
Ajeya Cotra: Yeah, so I actually, I think that would be the right thing to do, but the only thing that Iâm actually modeling out a full distribution of uncertainty for is the first one, which is the biggest piece, the 2020 training computation requirements. And then the other ones I have point estimates or like point forecasts, and then I just do an aggressive, and a conservative, and a best guess for those. But the sort of proper thing to do would be to have, to model out the uncertainty there, too.
Robert Wiblin: Okay. It seems like given that youâre reasonably unsure about all of these, or thereâs reasonable uncertainty balance, that the conclusion would be massively uncertain â because youâre compounding the uncertainty at each stage. And yet, interestingly, the actual range of time estimate⌠Itâs not from now until 10,000 years. The amount of time isnât that wide, and I guess thatâs because some things are increasing exponentially, so eventually it catches up within a reasonable time period, even in the pessimistic case? Is that whatâs going on?
Ajeya Cotra: Yeah. I mean, so the computation requirements range is⌠Itâs very wide in one sense. It spans 20 orders of magnitude. But itâs also not wide in another sense, in the sense that we have spanned more than that range, or at least almost that range, over the history of computing so far. The real work is done by⌠Most of my probability mass is on something within the biological anchors, as opposed to something astronomically larger than that. Thatâs where the work is coming from.
Ajeya Cotra: Itâs like, 20 orders of magnitude is a huge range, but between exponentially improving algorithms, and exponentially increasing spending, and exponentially decreasing hardware costs, you can shoot through that range over a matter of decades, as opposed to centuries.
Robert Wiblin: That makes sense. I guess I saw, I think at one point you added in a kludgy solution where youâre like, âAnd thereâs a 10% chance that Iâm completely wrong about this, and in fact we donât get anywhere near nowâ.
Ajeya Cotra: Yeah, I do have that. Yeah.
Robert Wiblin: I guess it makes sense. Maybe weâre just totally misunderstanding how the brain works, and weâve got this all wrong, andâŚ
Ajeya Cotra: Yeah. Maybe itâs like nanotech, right? Where like all these other technologies it seems like you can talk quantitatively about how much worse human technology is than natural technology, but thereâs some stuff like nanotech where weâre just like⌠It feels kind of silly to talk quantitatively about that, it just feels like weâre not there.
Most likely ways to be wrong [02:11:43]
Robert Wiblin: Yeah. So, imagining that youâll look back in the future and think that this report got things pretty wrong, whatâs the most likely way that you would have gotten it pretty wrong in either direction?
Ajeya Cotra: Yeah, so I mean I think thereâs more room to get it wrong, in some sense, in the direction that AI is like nowhere near where I think it is. I assigned substantial probability pretty soon, so I could be off by a factor of two in that direction and that would be scary and a big deal. But sort of like, why do I think itâs at all in this range, and could I be wrong there⌠I think a classic response is just, the move where you ask the question, given 2020 algorithms, how much compute would it take⌠Even if youâre allowing for it to be way more than we can afford, thatâs just not a fruitful intellectual move to make, and the answer is just astronomical, and thereâs no reason to think itâs near the biological anchors. Thereâs no reason to think our algorithms are as efficient as evolution.
Ajeya Cotra: And I donât think that it makes sense to assign more than 50% probability to that claim, the claim that itâs just nowhere in the range. Mainly because the range is large enough that I donât really trust most peopleâs intuitions â even expertsâ intuitions â about what would definitely not be possible if we had 15 orders of magnitude more computation. Thatâs not the kind of question that AI experts are trained to have expertise on. I have an intellectual style where I want to lean much more on an outside view, like the biological anchors view, or some of the other views that Open Phil researchers have been looking into, like just ignorance priors, once weâre talking about that kind of range.
Ajeya Cotra: One thing that I didnât get into when I was explaining the center of the probability distribution, which is, like I said before, itâs sort of assuming thereâs a model of a certain size, itâs somewhat bigger than the brain, that would be enough to be transformative. And you need some amount of data to train it. And then we have these scaling laws that say the number of samples or data points scales almost linearly but sort of sub-linearly with model size.
Ajeya Cotra: From there, thereâs a big unfixed variable, which is, what counts as one data point, or one sample? Like, are we talking about⌠Is one sample just seeing a single word, like GPT-3 is trying to predict one word and it gets feedback about whether it predicted that word correctly? Or like one image⌠the ImageNet model tries to predict an image, then gets the right answer and updates on that? Or is it more like a game of StarCraft, where the StarCraft model plays this whole game, and then it finds out whether it won, and then that propagates backwards and updates it?
Ajeya Cotra: You could imagine games that youâre training the model on that take much longer than StarCraft to resolve whether youâve won or lost. That general concept is what Iâm calling the âeffective horizon lengthâ, where currently you have models, I mentioned earlier, that AlphaStar and GPT-3 both cost about the same amount to train. But AlphaStar is much, much smaller than GPT-3. AlphaStar is 3,000 times smaller than GPT-3. That was because, in part it was because each data point for AlphaStar is a game, rather than a word.
Ajeya Cotra: So thereâs a big question mark about how dense can we get the feedback to train a transformative model? Will we be able to get away with giving it feedback once every minute? And thatâs actually rich, useful feedback⌠Or are there some things that just need to play out over a longer period of time before we can tell whether a certain direction of change is good or not?
Robert Wiblin: Unfortunately weâve got to move on, but I guess this report is online for people who are interested to learn more. Weâll stick up a link to it, and maybe also a presentation that explains what we just went through in a bit more detail, and has nice graphs. I guess the bottom line for me is that having looked into all of this, youâre pretty unsure when transformative AI might come, but you think it could be soon, could be a medium amount of time away, and I guess you havenât found any reason to think that there wonât be a transformative AI within the time period over which we could plausibly plan for thinking about how that might affect us and how it might go better or worse.
Ajeya Cotra: Yeah. I mean, I think I would frame it as like 12â15% by 2036, which is kind of the original question, a median of 2055, and then 70â80% chance this century. Thatâs how I would put the bottom line.
Biggest challenges with writing big reports [02:17:09]
Robert Wiblin: Alright. So working on big open-ended reports like this can be a bit of a mess, and I think difficult for people intellectually â and I guess also psychologically.
Ajeya Cotra: Yeah.
Robert Wiblin: What are the biggest challenges with this work? How do you think that you almost got tripped up, or that other people tend to get tripped up?
Ajeya Cotra: One thing thatâs really tough is that academic fields that have been around for a while have an intuition or an aesthetic that they pass on to new members about, whatâs a unit of publishable work? Itâs sometimes called a âpublonâ. What kind of result is big enough? What kind of argument is compelling enough and complete enough that you can package it into a paper and publish it? And I think with the work that weâre trying to do â partly because itâs new, and partly because of the nature of the work itself â itâs much less clear what a publishable unit is, or when youâre done. And you almost always find yourself in a situation where thereâs a lot more research you could do than you assumed naively, going in. And itâs not always a bad thing.
Ajeya Cotra: Itâs not always youâre being inefficient or youâre going down rabbit holes, if you choose to do that research and just end up doing a much bigger project than you thought you were going to do. I think this was the case with all of the timelines work that we did at Open Phil. My report and then other reports. It was always the case that we came in, we thought, I thought I would do a more simple evaluation of arguments made by our technical advisors, but then complications came up. And then it just became a much longer project. And I donât regret most of that. So itâs not as simple as saying, just really force yourself to guess at the outset how much time you want to spend on it and just spend that time. But at the same time, there definitely are rabbit holes, and there definitely are things you can do that eat up a bunch of time without giving you much epistemic value. So standards for that seemed like a big, difficult issue with this work.
Robert Wiblin: Okay. So yes. So this question of whatâs the publishable unit and what rabbit holes should you go down? Are there any other ways things can go wrong that stand out, or mistakes that you potentially made at some point?
Ajeya Cotra: Yeah. Looking back, I think I did a lot of what I think of as defensive writing, where basically there were a bunch of things I knew about the subject that were definitely true, and I could explain them nicely, and they lean on math and stuff, but those things were only peripherally relevant to the central point I wanted to make. And then there were a bunch of other things that were hard and messy, and mostly intuitions I had, and I didnât know how to formalise them, but they were doing most of the real work. One big example is that of the four things we talked about, the most important one by far is the 2020 computation requirements. How much computation would it take to train a transformative model if we had to do it today. But it was also the most nebulous and least defensible.
Ajeya Cotra: So I found myself wanting to spend more time on hardware forecasting, where I could say stuff that didnât sound stupid. And so as I sat down to write the big report, after I had an internal draft⌠I had an internal draft all the way back in November 2019. And then I sat down to write the publishable draft and I was like, okay, Iâll clean up this internal draft. But I just found myself being pulled to writing certain things, knowing that fancy ML people would read this. I found myself being pulled to just demonstrating that I knew stuff. And so I would just be like⌠Iâd write ten pages on machine learning theory that were perfectly reasonable intros to machine learning theory, but actually this horizon length question was the real crux, and it was messy and not found in any textbook. And so I had to do a lot to curb my instinct to defensive writing, and my instinct to put stuff in there just because I wanted to dilute the crazy speculative stuff with a lot of facts, and show people that I knew what I was talking about.
Robert Wiblin: Yeah. Thatâs understandable. How did the work affect you personally, from a happiness or job satisfaction or mental health point of view? Because I think sometimes people throw themselves against the problems like this and I think it causes them to feel very anxious, because they donât know whether theyâre doing a good job or a bad job, or they donât feel they are making progress, or they feel depressed because they worry that they havenât figured it out yet and they feel bad about that.
Ajeya Cotra: Yeah. I had a lot of those emotions. I think the most fun part of the project was the beginning parts, where my audience was mostly myself and Holden. And I was reading these arguments that our technical advisors made and basically just finding issues with them, and explaining what I learned. And thatâs just a very fun way to be⌠You have something you can bite onto, and react to, and then youâre pulling stuff out of it and restating it and finding issues with it. Itâs much more rewarding for me than looking at a blank page and no longer writing something in response to somebody else. You have to just lay it all out for somebody who has no idea what youâre talking about. And so I was starting writing this final draft â the draft that eventually became the thing posted on LessWrong â in January of 2020.
Ajeya Cotra: And I gave myself a deadline of March 9th to write it all. And in fact, I spent most of January and half of February really stressed out about how I would even frame the model. And a lot of the stuff we were talking about, about these four parts, and then the first part is if we had to do it today, how much computation would it take to train⌠All of that came out of this angsty phase, where before I was just like, how much computation does it take to train TAI, and when will we get that? But that had this important conceptual flaw that I ended up spending a lot of time on, which is like, no, that number is different in different years, because of algorithmic progress.
Ajeya Cotra: And so I was trying to force myself to just write down what I thought I knew, but I had a long period of being like this is bad. People will look at this, and if theyâre exacting, rigorous people, theyâll be like this doesnât make sense, thereâs no such thing as the amount of computation to train a transformative model. And I was very hung up on that stuff. And I think sometimes itâs great to be hung up on that stuff, and in particular, I think my report is stronger because I was hung up on that particular thing. But sometimes youâre killing yourself over something where you should just say, âThis is a vague, fuzzy notion, but you know what I meanâ. And itâs just so hard to figure out when to do one versus the other.
Robert Wiblin: Yeah. I think knowing this problem â where often the most important things canât be rigorously justified, and you just have to state your honest opinion, all things considered, given everything you know about the world and your general intuitions, thatâs the best you can do. And trying to do something else is just a fake science thing where youâre going through the motions of defending yourself against critics.
Ajeya Cotra: Yeah. Like physics envy.
Robert Wiblin: Yeah. Right. I thinkâŚ
Ajeya Cotra: I had a lot of physics envy.
Robert Wiblin: Yeah. Iâm just more indignant about that now. Iâm just like, look, I think this, you donât necessarily have to agree with me, but Iâm just going to give you my number, and Iâm not going to feel bad about it at all. And I wonât feel bad if you donât agree, because this unfortunately is the state-of-the-art process that we have for estimating, is just to say what we think. Sometimes you can do better, but sometimes you really are pretty stuck.
Ajeya Cotra: Yeah. And I think just learning the difference is really hard. Because I do think this report, I believe has made some progress toward justifying things that were previously just intuitions we stated. But then there were many things where I hoped to do that, but I had to give up. I think also, doing a report that is trying to get to a number on an important decision-relevant question is a ton of pressure, because you can be really good at laying out the arguments and finding all the considerations and stuff, but your brain might not be weighing them right. And how you weigh them, the alchemy going on in your head when you assign weights to lifetime versus evolution versus things in between make a huge difference to the final number.
Ajeya Cotra: And if you feel like your job is to get the right number, that can be really, really scary and stressful. So Iâve tried to reframe it as my job is to lay out the arguments and make a model that makes sense. How the inputs get turned into outputs makes sense and is clear to people. And so the next person who wants to come up with their views on timelines doesnât have to do all the work I did, but they still need to put in their numbers. My job is not to get the ultimate right numbers. I think reframing it that way was really important for my mental health.
Robert Wiblin: Yeah. Because thatâs something you actually have a decent shot at having control over, whether you succeed at that. Whereas being able to produce the right number is to a much greater degree out of your hands.
Robert Wiblin: Alright. Another part of this⌠I guess this is a big mood shift here, but another part of the project Iâm trying to figure out, how Open Phil should disburse its money, is trying to think well, should it be giving away more now, or should we be holding onto our money to give it away at some future time when perhaps weâll have other opportunities that could be better or worse? I guess you guys call this the âlast dollarâ project. Can you tell us a bit about that?
Ajeya Cotra: Yeah. Basically the idea is that if thereâs diminishing returns to the money weâre giving away, then the last dollar that we give away should be, in expectation, the least valuable dollar. And furthermore, if we give away one dollar today, the thing itâs trading off against, the opportunity cost, is whatever we would have spent the last dollar on. The theoretically clean answer is that if we know the value of our last dollar, then we should be giving to everything we find thatâs more cost effective than that.
Ajeya Cotra: Like, every year we should look around and fund everything that seems better than our last dollar, and hold onto all the rest. Thatâs the conceptual answer to both giving now versus giving later, and what should we spend our money on. So, the goal⌠Thereâs two sides to this. One is trying to think about what might we spend our last dollar on. How good is that? Then the other is trying to think about what should we expect about how many opportunities there are that are better than the last dollar in each future year?
Ajeya Cotra: Weâre trying to quantify it. Instead of asking âIs giving now better or worse than giving laterâ, weâre trying to get a rough sense of the allocation across time that we should expect will be reasonable. So, on the last dollar question, Iâve done some work on that on the longtermist side, in terms of what do we spend the last dollar on. For the near-termist side, my colleague Peter is working on putting together a model of allocation over time for the near-termist side. They have a bit better sense of what their last dollar is than we do, because they have⌠GiveWell has been working for more than a decade on mapping out these global health interventions that have massive room for more funding, and really high cost effectiveness. The near-termist side is mostly trying to beat the benchmark set by GiveWell, and assume that the things like GiveWell top charities will be able to absorb marginal money. Peter has been working on this model that is basically a more complex variant of Phil Trammellâs model. Phil Trammell recently put out this paper called Patient Philanthropy or something I think.
Ajeya Cotra: Peterâs working on a more complex variant of this model thatâs going to give a rough guide to how the near-termist side should spend down its money. Basically the way the model works is on the one side, your money is growing because youâre putting it in the market and itâs getting some percent return per year. And then on the other hand, opportunities to do good we assume are declining over time, because other funders are coming in and funding those things, or the world is generally getting better and problems are getting solved on their own. The baseline rate of disease is going down, stuff like that.
Ajeya Cotra: These are the two main forces and then there are a bunch of other forces you could also model. Like, maybe giving now helps you learn how to give better, and each year has diminishing returns because you canât give an organisation $100 million in one year just because you could have given it $10 million over each of ten years. Stuff like that. So, this model is trying to net all of this out and then come up with an expected allocation across time. And basically because all of the individual parameters are constant growth or decay â so thereâs a percent that your money is growing every year, and then thereâs like a percent that youâre assuming opportunities are declining every year â this spits out basically that you should give a constant fraction of your money every year, you should expect to give out a constant fraction. And that constant fraction could be zero, because you should save indefinitely if the parameters shake out that way, or it could be very high, such that youâre essentially trying to spend down as fast as possible. Or it could be somewhere in between, where youâre not drawing down your principal, but youâre giving away some of the interest. Those are the three regimes.
Robert Wiblin: Okay, so just to check that Iâve got that straight: Because GiveWell â or those who are focused on human charity today â have a better sense of what their future giving opportunities will be and how effective theyâll be, youâre using that as a baseline to compare to for the other programs, saying, âWill we have better opportunities within this program than the near-term human thing?â I guess, what was the time frame you were thinking about?
Ajeya Cotra: Our main funders, Cari and Dustin, want to spend down most of their fortune within their lifetime. So weâre usually thinking of a several-decade time span, like 50 years or 100 years, experimenting with different⌠Like what happens when you set the deadline in different places.
Robert Wiblin: I see. Then you add in a bunch of other things, like discount rate, and return on investment, and I suppose learning effects and things like that, that happen every year. And then build this into a model that says how much you should give away. How much of the decision comes down to just this really difficult, empirical question, where like the factory farming program and the biosecurity folks have to say, âWell, how good will the opportunities in this area be in 40 years time?â Which seems really hard to answer.
Ajeya Cotra: Yeah. I donât know as much about the animal-inclusive near-termist worldview, but roughly speaking I would say the human-centric near-termist worldview gets more juice out of doing a model like this, that recommends some percent you should expect to give away each year. And they still have to do a bunch of empirical work to find the opportunities that meet that bar, but theyâre able to look ahead and be like, âOkay, we need to prepare to give X% because our model says that should be roughly optimal based on what we believe.â
Ajeya Cotra: We still need to empirically find those opportunities, and we might end up finding more or less than that, but itâs kind of a guideline. On the longtermist side, itâs a lot trickier, and we donât expect it to look like the optimum is some constant fraction given away every year. We kind of expect it to more track the shape of existential risk. So, for example, on AI risk, we expect there to be more opportunities roughly like five or 10 years before transformative AI than either before or after. Before, you less know the shape of how things are going to go, and there are fewer people who want to work on this that you could fund, and impacts could be washed out by things that happened in the future. And then after that point, it might be that thereâs a ton of money in the space, and so your dollars are less leveraged. So weâre imagining thereâs some window before transformative AI where weâll be spending a lot more than weâd be spending before or after that window.
Robert Wiblin: Yeah. It seems like you build this big model, but I suppose Cari and Dustin, they want to spend all the money before they die, and I guess assuming they live a normal human lifespan, weâve got maybe 50 years to play with or something like that. In a sense thereâs not that much flexibility, and you also have to think about how quickly could you plausibly scale your ability to find really good grant opportunities? Thatâs also slightly a bottleneck at the moment. In fact, how much does this influence your decisions? Are you in practice bound by other constraints that are doing most of the work?
Ajeya Cotra: Yeah, so I think again, itâs a little bit different on the near-termist versus the longtermist side. On the longtermist side, weâre not building out much of an allocation-over-time model. Weâre mostly just focused on the question of how good is the last dollar. Weâre in a regime where we are finding substantially fewer giving opportunities than we would like to be, because these fields are small. So, we want to be funding basically everything thatâs better than that last dollar.
Ajeya Cotra: And then on the near-termist side, I think like I said before, thereâs more guidance. The model that Peter is working on, unlike Phil Trammellâs model, allows for you to set deadlines, say if you have a separate constraint where you really want to have given away the money by date X. That changes recommendations somewhat, like you would imagine. But thereâs still, depending on how the other parameters are set, besides the deadline, thereâs a range you could end up with in terms of how much you should be roughly aiming to give away per year.
Robert Wiblin: Yeah. I guess this is so important that itâs something Open Philâs going to keep tinkering with probably as long as it exists. But is there any bottom line of what fraction of the total principal that you have now you would want to be giving away each year, all things considered?
Ajeya Cotra: I donât know because Iâm not working on the near-termist side of the project. Iâd rather not speak for them, but hopefully you can grab someone from that team on your podcast pretty soon. I can talk about the last dollar thinking on the longtermist side, which is just like how good is this last dollar, not so much allocation over time, exactly.
Ajeya Cotra: There are basically two projects: one that I worked on a while back, and one that Iâm just starting to work on now. The one that I worked on a while back, basically we wanted to seek something that could be an intervention where we could robustly spend a lot of money. On the near-termist side, the analogy here is GiveDirectly. GiveDirectly is very hugely scalable â almost unboundedly scalable, in the regime of the money Open Phil has access to. We expect there to be roughly linear returns, because as youâre giving cash transfers to extremely poor people, thereâs such a large number of extremely poor people at roughly the same level of income, that youâre not really hitting diminishing returns until youâre giving away substantially more than Open Phil could afford to give away anyway.
Ajeya Cotra: Thatâs the lower bound of the near-termist last dollar. Because itâs one thing that we could put all of the money into, and we would expect all of it to get roughly 100x return, because the individuals weâre giving to are roughly 100 times poorer than the average American. So, we were seeking something that we thought that would be like the GiveDirectly of longtermism. One big intervention that we can spend a lot of money on, that has roughly linear returns in that regime.
Ajeya Cotra: The goal here was just to demonstrate that if we really wanted to, the longtermist worldview could find things to spend money on â because that is a big question, in terms of the longtermist worldview â and that it would be a reasonable return on investment. And that we think we could do better than this, but this âGiveDirectly of longtermismâ is at least reasonable. That was the goal. We turned to biosecurity for this, because basically bioscience is expensive and biotech is expensive, and thereâs a pretty big field of people we could potentially co-opt to do things that we think are valuable on longtermist grounds.
Ajeya Cotra: Thereâs not so much in AI, and other causes are less expensive to just do things in because theyâre more thinking about stuff. And so what we ended up landing on was this notion of funding, like meta R&D, to help make responses to new pathogens like COVID faster. Currently it takes several months to a year â once we learn of a new pathogen â to develop either a vaccine or an antiviral and then distribute it. There are a bunch of things we could potentially do to reduce that.
Robert Wiblin: One other suggestion Iâve heard as a baseline longtermist intervention is reducing carbon emissions through carbon offsetting, or subsidising the scale of solar energy, or something like that. Did you consider that?
Ajeya Cotra: We looked into climate change from a longtermist perspective a while back. We didnât do a super deep investigation, but like Toby Ord says in his recent book The Precipice, we felt that it was substantially less good on a longtermist perspective than the big two that we focus on, which is AI and biosecurity. We wanted a lower bound here but we also wanted to⌠We basically wanted to find the biggest thing that was still low uncertainty in terms of our ability to spend the money. Because the big questions on AI are like, do we even find anything to give the money to, given that the field is so small?
Robert Wiblin: Yeah, okay. Carry on.
Ajeya Cotra: So, we were looking at meta R&D to make responses to new pathogens faster. So currently it takes several months. We thought spending a lot of money on a lot of different fronts could bring it down to a month or a few weeks, in terms of the calendar time from ânew virus comes on the sceneâ to âwe have a vaccine or an antiviral, and weâre starting to roll it outâ. So there are a bunch of things we could potentially do. One thing is just like funding people to develop and stockpile broad-spectrum vaccines, which are vaccines that are trying to target a biological mechanism thatâs common to a big family of viruses.
Ajeya Cotra: Potentially if we found a broad-spectrum flu vaccine, say, that vaccine could protect people against a much more dangerous engineered version of the flu, too. Because itâs targeting mechanisms that are fundamental to all the different flus. And you could imagine funding tools for rapid onsite detection, like as soon as one person gets the virus, you can just go onsite, you can sequence it really quickly, you can map out its protein structure, maybe with something like AlphaFold, and then you come up with guesses about what molecules bind well to the molecules on the pathogen, so you cut out some of the trial and error in drug development.
Ajeya Cotra: So it was stuff like this that we were imagining funding, and thereâs a lot we could potentially fund, especially with the manufacture and stockpile of vaccines. This is why we were hoping it could be kind of like a âGiveDirectly of longtermismâ type thing. With that, the basic structure of the cost-effectiveness estimate was like, we expect that eventually â as civilisation becomes more technologically mature â we will have this ability to rapidly detect and prevent diseases. But weâre moving that forward in time by funding the field and beefing it up earlier. We can move forward in time by several months or a year the point at which we have the mature technology, as opposed to the current technology.
Ajeya Cotra: And then we would basically cut into some chunk of the x-risk in that window that we moved it forward. If in 2041 we would have this ability, and we caused it so that it happened in 2040 instead, then thereâs some fraction of like one yearâs worth of bio x-risk thatâs reduced.
Robert Wiblin: Okay. That makes sense. So what kind of conclusions has this led to, if anything, about whether you should spend down the resources faster or slower? Making this comparison I guess to funding this meta science decades in the future.
Ajeya Cotra: Yeah, so this estimate is roughly $200 trillion per world saved, in expectation. So, itâs actually like billions of dollars for some small fraction of the world saved, and dividing that out gets you to $200 trillion per world saved. This is quite good in the scheme of things, because itâs like less than two yearsâ worth of gross world product. Itâs like everyone in the world working together on this one problem for like 18 months, to save the world. Thatâs quite good in some sort of cosmic sense, right? Because it would be worth decades of gross world product to save the world, potentially.
Ajeya Cotra: But we were aiming for this to be conservative, because itâs likely we would spread across multiple longtermist focus areas, instead of just biosecurity, and AI risk is something that we think has a currently higher cost effectiveness. So, it didnât necessarily cause us to change how weâre expecting to spend down in the immediate term, just because weâre still in the regime where weâre trying to find grantees that are on target with what we want to fund, and are focusing on existential risk as opposed to other problems. Thatâs a huge bottleneck to getting money out the door. It wasnât like we were in a position where we were spending a lot and we realised, âOh, actually, the last dollarâs good, so we should cut back or saveâ. We werenât in that regime, and we knew we werenât.
Ajeya Cotra: But the goal of this project was just to reduce uncertainty on whether we could. Like, say the longtermist bucket had all of the money, could it actually spend that? We felt much more confident that if we gave all the money to the near-termist side, they could spend it on stuff that broadly seemed quite good, and not like a Pascalâs mugging. We wanted to see what would happen if all the money had gone to the longtermist side. Itâs like you were saying earlier, if a worldview is just getting zero marginal return, on its own perspective, from getting twice as much more money, then it just seems intuitively a lot less appealing to give that worldview more money. That was the main goal of this project.
Robert Wiblin: I guess you guys are going to keep researching this, and I suppose eventually at some point it will be like a publication that will lay this out?
Ajeya Cotra: Yeah. I mean, it might not be this particular thing. The last dollar question â and to a lesser extent, the allocation over time question â is just one thatâs always on our minds. So, the more recent work that Iâve been doing ties into the AI timelines work that I just recently completed. Itâs trying to do a last dollar cost-effectiveness estimate, but itâs less trying to look for the âGiveDirectly of longtermismâ â like one big expensive intervention â and more trying to think about 10 interventions that could each take a tenth of the money, and trying to be more like a best guess for what we actually spend the bulk of it on. And focused on AI as opposed to biosecurity, in this case.
Robert Wiblin: Does Open Phil keep track of the most important disagreements that different staff members have with one another? Iâm just imagining presumably people have views that are all over the shop on this issue, and potentially on other ones as well. I guess I could imagine you guys being the sort of people who would have a huge spreadsheetâŚtrack all of these things and then take the median, or the harmonic mean, I donât know.
Ajeya Cotra: Yeah. The harmonic mean.
Robert Wiblin: I donât even know what that is. Sorry, go on.
Ajeya Cotra: I wish we had more capacity to do this. I think GiveWell does this a lot. GiveWell, every year they have their charity recommendations, and there are these thorny questions of values and how to interpret ambiguous research. There are 10 researchers in a room arguing about them, and then they put out a spreadsheet that has columns for all 10 individuals and their disagreements. GiveWell usually reports the median.
Ajeya Cotra: We donât really have that kind of system for most things, just because Open Phil is significantly more siloed, and weâre spread out over more quite diverse topics. There are only a couple of people who have their head in each of these topics at a time. With AI timelines, there are three or four people that have had their head at least somewhat in that. And only two of those people, say, are like really deeply in it.
Ajeya Cotra: And similarly with biosecurity, there are only two or three people who think about it, and only one person whoâs really deeply in it. Thereâs a lot more deference going on across major areas than would be ideal if we had more staff and more ability to give the GiveWell treatment to each thing. Within a particular area, when itâs important and thereâs a largish number of people with some amount of expertise, we try to get polls and get estimates from a lot of different people. One area where weâre able to do this more â because we have more people who think about it â is with respect to the EA community.
Ajeya Cotra: We are experimenting with having more voices in grant-making decisions within the EA community. But most areas donât really have that, and weâre not sure that experiment actually leads to better, more efficient decisions. Itâs still up in the air.
Robert Wiblin: Yep. I might wrap this up because I guess listeners who are interested in this timing-of-giving and âpatient philanthropyâ stuff can go and listen to the interview with Phil Trammell from earlier in the year, where we go through a lot of these considerations very forensically; consider them very patiently. I guess thereâll probably be some blog posts on this topic from Open Phil in coming years because it seems like itâs going to be an important topic for you guys to figure out over the very long term.
What itâs like working at Open Phil [02:45:18]
Robert Wiblin: Before we finish, I would like to get in some discussion of what itâs like working at Open Phil, and I guess what the opportunities are at the moment. I think last time we talked about this on the show with somebody who works at Open Phil was two years ago. And since then I know the organisation has grown pretty substantially, so maybe that has shifted the culture and what itâs like to be there. So yeah. How have things changed over the last couple of years? I guess youâve been there for four years now?
Ajeya Cotra: We started off, I would say, on a trajectory of being much more collaborative â and then COVID happened. The recent wave of hiring was a lot of generalist hires, and I think that now thereâs more of a critical mass of generalists at Open Phil than there was before. Before I think there were only a few, now theyâre more like 10-ish people. And itâs nice because thereâs a lot more fluidity on what those people work on. And so there are a lot more opportunities for casual one-off collaboration than there is between the program staff with each other or the generalists with the program staff.
Ajeya Cotra: So a lot of the feeling of collaboration and teamyness and collegiality is partly driven by like, does each part of this super siloed organisation have its own critical mass. And I feel like the answer is no for most parts of the organisation, but recently the generalist group of people â both on the longtermist and near-termist side together â have more people, more opportunities for ideas to bounce, and collaborations that make sense, than there were before. And Iâm hoping as we get bigger and as each part gets bigger, thatâll be more and more true.
Robert Wiblin: I guess, as organisations become bigger, things tend to become a bit more organised and standardised and bureaucratised, which has its good sides and also has its bad sides. Has that been the case with Open Phil as well? Or are there a sufficient number of small cells so that it actually still feels like a small organisation?
Ajeya Cotra: Yeah. So I think a lot of my day-to-day feels like a pretty small organisation still, but even in a pretty siloed organisation, there are some things that itâs important to hammer out as we get to the scale weâre at, which is 45-ish people and beyond. So weâre working actively on making Open Phil more professionalised, in the sense of like, especially clearer standards for performance and promotion, and fairer compensation across the different areas. So like, what does it mean to be a program associate in farm animal welfare versus effective altruism versus science or criminal justice reform. These focus areas have different needs, and different ways they operate within their fields, but we still want it to be fair that if you are a senior program associate, and you look around and youâre wondering, why is this other person a program officer instead of a senior program associate? Or why is this other person a program associate instead of senior program associate⌠You donât want it to be the case that people can look to the left and look to the right and see people doing what they feel like is their same job, but are compensated differently for that. So thinking carefully about that is one of the things weâre aiming to do over the next year or two.
Robert Wiblin: What do you like and dislike most about your job?
Ajeya Cotra: LikesâŚobviously the mission, and I think my colleagues are just incredibly thoughtful and kind people that I feel super value-aligned with. And thatâs awesome. And then dislikes, it comes back to the thing I was saying about how itâs a pretty siloed organisation. So each particular team is quite small, and then within each team, people are spread thin. So thereâs one person thinking about timelines and thereâs one person thinking about biosecurity, and it means the collaboration you can get from your colleagues â and even the feeling of team and the encouragement you can get from your colleagues â is more limited. Because they donât have their head in what youâre up to. And itâs very hard for them to get their head in what youâre up to. And so people often find that people donât read their reports that they worked really hard on as much as they would like, except for their manager or a small set of decision makers who are looking to read that thing.
Ajeya Cotra: And so I think that can be disheartening. And then in terms of my particular job, all this stuff I was saying⌠Itâs very stressful putting together this report, in a lot of the ways that we were talking about earlier. And just feeling responsible for coming to a bottom-line number without a lot of feedback or a lot of diffusion of responsibility that comes from a bunch of people putting in the numbers. And likeâŚ
Robert Wiblin: That seems particularly hard.
Ajeya Cotra: Thatâs quite stressful. And itâs quite stressful to basically be doing work where you are just inevitably going to miss your deadlines a bunch. Youâre inevitably going to think, I know what Iâm talking about, Iâm going to write it down, but actually you didnât and you arenât, and youâre going to have to push and youâre going to have to push many times over. That can be disheartening, but I think just being aware of the dynamic has been helpful for me.
Robert Wiblin: Yeah. From memory, when Open Phil was hiring a couple of years ago, I think a thousand people applied for a bunch of jobs and then 10 people got trials, and something like five people actually got hired? So those are harsh odds. Is there anything you can say to people who I guess either donât think itâs possible theyâll get hired by Open Phil and maybe were a bit disappointed by that, or have applied and maybe didnât manage to get a trial?
Ajeya Cotra: Yeah. I guess my first thought is that Open Phil is not peopleâs only opportunity to do good. Even doing generalist research of the kind that I think Open Phil does a lot of, especially for that kind of research, I think itâs a blessing and a curse, but you just need a desk and a computer to do it. I would love to see people giving it a shot more, and I think itâs a great way to get noticed. So when we write reports, all the reports we put out recently have long lists of open questions that I think people could work on. And I know of people doing work on them and thatâs really exciting to me. So thatâs one way to just get your foot in the door, both in terms of potentially being noticed at a place like Open Phil or a place like FHI or GPI, and also just get a sense of what does it feel like to do this? And do you like it? Or are the cons outweighing the pros for you?
Ajeya Cotra: In terms of generalist roles, thatâs one thought I have. And then on a more procedural note, Open Phil is trying to be more forward-looking and long-term and patient with our recruiting pipeline. So we have a general application up, where even if youâre happy with where youâre at now and you want to stay at your current job for a couple of years, if youâre interested in eventually making a transition into this type of work, feel free to drop your name on the general application and then say what types of roles you might be interested in. And thatâs a good way to just stay in touch and stay on our radar.
Robert Wiblin: Yeah. One thing I very often say to people who are disappointed when theyâve applied for a job and they havenât gotten it is just that itâs very natural to take that as a personal insult, that you werenât good enough, but very often the most important thing is the fit between the person, the role, the organisation, the people theyâll be working with, and what they know. And that stuff could just be extremely specific. There are brilliant, incredibly smart people out there who just arenât a good fit for working at Open Phil. And thatâs not a dump on them, itâs just that they should be doing something else where theyâre just more likely to flourish, because Open Phil has this very particular culture, which I guess â as weâve just heard â is challenging in some ways. Itâs like, itâs not all a bed of roses. Thereâs also ways in which itâs challenging work intellectually and emotionally.
Ajeya Cotra: Yeah. I think the emotional element is really big there. I think itâs a certain disposition ofâŚthe cocktail of being arrogant enough and weird enough to think that you could answer these big questions, but also being finicky enough and particular enough about dotting Is and crossing Ts that you can make those weird areas just one notch more rigorous than they were before. But not 10 notches, because otherwise youâre going to be working on it for 20 years. Itâs like some kind of epistemic culture thatâs very contingent that we think some people fit into that helps thread that needle. But there are other places that want to be on a more bold, innovative, weird âarrogantâ side of the spectrum, and places that want to be on the more careful, rigorous, complete side of the spectrum too. And that just changes what you can work on and what frontier youâre operating at, basically.
Robert Wiblin: Yeah. So I think you were just about to say this earlier, but I guess for listeners who do think that this sounds like something that theyâd be interested in, where they do have the right level of attention to detail and persistence â but not too much persistence â how can they get onâŚ
Ajeya Cotra: Not too much persistence!
Robert Wiblin: Yeah, laziness can be a virtue. In fact, very often it is a virtue. How can they get on Open Philâs radar or stay abreast of opportunities to potentially meet people and escalate their involvement?
Ajeya Cotra: Yeah. So please do drop your name on our general application, and we can put up a link to that on the podcast page. And research is definitely not all Open Phil does, thereâs grantmaking stuff too, but in terms of the things Iâm working on and know best, I do think itâs possible to try it out with these open questions that we list on our other reports. And also just reading stuff written by FHI or at GPI, and thinking about, is there a piece of this I can break off? Something that seems intrinsically interesting to me, where I could make a unit of progress and put it up on the EA Forum, put it up on LessWrong, put it up on the Alignment Forum? I think thatâs a great way to just straight up add value and also get noticed by this ecosystem of organisations that are doing this work.
Robert Wiblin: Yeah. Weâve recently been trialing someone, and I think maybe the reason that they stood out was just the incredible compilation of work that they had on their personal website across a whole bunch of different⌠Like writing that theyâd done, artistic stuff that theyâd done, audio stuff that theyâd done as well. Theyâd just shown a persistent interest and ability to produce interesting stuff. So thatâs definitely one way to stand out from the crowd, because for some reason, most people donât have that.
Ajeya Cotra: A great thing to do actually is just explaining stuff that other people have said and didnât explain very well. That can be great for learning and for teaching and for demonstrating the thinking that would allow you to do more original research down the line. And this can be in any format, like writing up explainers. There was a great explainer on LessWrong about the scaling laws you were talking about actually, that was really helpful. And you could make YouTube videos explaining things, like the Robert Miles AI videos are great. So that stuff, it doesnât have to be pushing forward the frontier. You can still both add value and really make yourself stand out with explainers.
Robert Wiblin: Yeah. One further question before you go, Iâve got the weekend coming up and not a whole lot planned. Are there any good movies or TV shows youâve seen recently that you can recommend to me? The weatherâs looking pretty grim, so I guess Iâm going to be indoors.
Ajeya Cotra: So I have started⌠This is very redundant with everything everyone else has told you Iâm sure, but Iâve started The Queenâs Gambit. Iâm two episodes in, and I quite like it so far. Itâs basically like a sports movie wrapped in a prestige TV wrapper about this girl thatâs on the rise to superstardom in chess. And I also like⌠This is much less prestige, but I also like another Netflix show called Girlfriendsâ Guide to Divorce, which Iâm finding very entertaining. Itâs about snobby L.A. housewives going through divorce.
Robert Wiblin: Nice. Yeah. Netflix has been pushing Queenâs Gambit so hard. Every time I open it, itâs like theyâre insisting that I watch this thing. I feel like theyâre going to cancel my subscription if I opt out of it.
Ajeya Cotra: Also every single person in my life too. My friends and my partners, aunts, boyfriend, theyâre all in it. So I caved. Itâs good.
Robert Wiblin: Itâs good. Okay. Listeners, if youâre in the same situation as me this weekend⌠I watched Knives Out this week, which is a murder mystery with various different twistsâŚ
Ajeya Cotra: Did you know theyâre making a sequel? Itâs going to be a whole Benoit Blanc series. Itâs going to be like Hercule Poirot. Itâs going to be so big.
Robert Wiblin: Oh, wow. Yeah, no, itâs a really good character. Daniel Craig doing a Southern American accent rubbed me the wrong way for the first 10 minutes, but then I just rolled with it.
Ajeya Cotra: Itâs got a great all-star cast too. Yeah. That movie really launched me on the search for the perfect murder mystery movies. I think itâs really slim pickings in terms of good murder mysteries that are just about the mystery, instead of a character study or something else.
Robert Wiblin: Yeah. The story is impressive. I can see why so many top actors signed on, because they would have read that script and been like wow, this is really cool.
Ajeya Cotra: Yeah, totally.
Robert Wiblin: Alright. Well with that out of the way, weâve covered some fun stuff here and some pretty dense stuff, but I think I understand all of these topics a bunch better now. My guest today has been Ajeya Cotra. Thanks so much for coming on the 80,000 Hours Podcast, Ajeya.
Ajeya Cotra: Yeah. Thanks so much for having me.
As I mentioned in the middle of the interview, if youâre someone who could see themselves going into a career path like the one that Ajeya has had, or potentially, like some of the careers that other guests have had over the years, and youâd like a bit of help figuring out exactly where you should maybe aspire to end up and how you might get there, you should check out our 1-1 advising service.
Thereâs more information about that at 80000hours.org/advising. There are some free slots available at the moment, which is why weâre putting out this advertisement. And at that address, you can find out about the kinds of problems and people who we can do the most to help and the kinds of questions and people who we sometimes struggle to help and so weâre less likely to be able to advise.
But if youâve found this interview engaging and are interested to hear more things like this, then thatâs a good sign that you might be a good fit for our 1-1 advising service. So go take a look at 80000hours.org/advising and donât be shy about applying if you think it would be helpful to you.
The 80,000 Hours Podcast is produced by Keiran Harris.
Audio mastering by Ben Cordell.
Full transcripts are available on our site and made by Sofia Davis-Fogel.
Thanks for joining, talk to you again soon.
Learn more
Career review: Foundation grantmaker
Future generations and their moral significance
The case for reducing existential risks
Global priorities research
For any readers who stumble upon this later: There was an earlier linkpost for this same podcast with a lot of interesting discussion in the comments.
(This post I'm comment on now shows an earlier date of publication, but I think it was recently published as part of efforts to create an "EA Forum archive" of key content, was given an earlier date to match when the podcast was released, and didn't appear on the frontpage, hence no comments.)