## Effective Altruism ForumEA Forum

Epistemic status: Fairly uncertain. May contain errors, probabilities might not be calibrated.

## Introduction

This document reviews a number of organizations in the longtermist ecosystem, and poses and answers a number of questions which would have to be answered to arrive at a numerical estimate of their impact. My aim was to see how useful a "quantified evaluation" format in the longtermist domain would be.

In the end, I did not arrive at GiveWell-style numerical estimates of the impact of each organization, which could be used to compare and rank them. To do this, one would have to resolve and quantify the remaining uncertainties for each organization, and then convert each organization's impact to a common unit [1, 2].

In the absence of fully quantified evaluations, messier kinds of reasoning have to be used and are being used to prioritize among those organizations, and among other opportunities in the longtermist space. But the hope is that reasoning and reflection built on top of quantified predictions might prove more reliable than reasoning and reflection alone.

In practice, the evaluations below are at a fairly early stage, and I would caution against taking them too seriously and using them in real-world decisions as they are. By my own estimation, of two similar past posts, 2018-2019 Long Term Future Fund Grantees: How did they do? had 2 significant mistakes, as well as half a dozen minor mistakes, out of 24 grants, whereas Relative Impact of the First 10 EA Forum Prize Winners had significant errors in at least 3 of the 10 posts it evaluated.

To make the scope of this post more manageable, I mostly did not evaluate organizations included in Lark's yearly AI Alignment Literature Review and Charity Comparison posts, nor meta-organizations [3].

# Evaluated organizations

## Alliance to Feed the Earth in Disasters

Epistemic status for this section: Fairly sure about the points related to ALLFED's model of its own impact. Unsure about the points related to the quality of ALLFED's work, given that I'm relying on impressions from others.

### Questions

With respect to the principled case for an organization to be working on the area:

1. What is the probability of a (non-AI) catastrophe which makes ALLFED's work relevant (i.e., which kills 10% or more of humanity, but not all of humanity) over the next 50 to 100 years?
2. How much does the value of the future diminish in such a catastrophe?
3. How does this compare to work in other areas?

With respect to the execution details:

1. Is ALLFED making progress in its "feeding everyone no matter what" agenda?
2. Is that progress on the lobbying front, or on the research front?
3. Is ALLFED producing high-quality research? On a Likert scale of 1-5, how strong are their papers and public writing?
4. Is ALLFED cost-effective?
5. Given that ALLFED has a large team, is it a positive influence on its team members? How would we expect employees and volunteers to rate their experience with the organization?

Execution details about ALLFED in particular

Starting from a quick review as a non-expert, I was inclined to defer to ALLFED's own expertise in this area, i.e., to trust their own evaluation that their own work was of high value, at least compared to other possible directions which could be pursued within their cause area. Per their ALLFED 2020 Highlights, they are researching ways to quickly scale alternative food production, at the lowest cost, in the case of large catastrophes, i.e., foods which could be produced for several years if there was a nuclear war which blotted out the sun.

However, when talking with colleagues and collaborators, some had the impression that ALLFED was not particularly competent, nor its work high quality. I would thus be curious to see an assessment by independent experts about how valuable their work seems in comparison to other work in their area, or to potential work which could be done.

In 2020, ALLFED also did some work related to the COVID-19 pandemic. While there is a case to be made that the pandemic is a kind of test run for a global catastrophe, I feel that this was a bit of a distraction from their core work.

It's unclear to me whether their research process is particularly cost-efficient; I've made inquiries as to the number of full-time employees (FTEs) for 2020 and its budget for that year, but haven't been answered. The data about ALLFED's budget was not available on their webpage. Because they are not a 503 registered charity, a Form 990 isn't anywhere to be found. It is also not clear to me how many FTEs ALLFED is employing, and how many of those are dedicated to research (vs logistical support, bureaucracy, etc.)

The principled case for an organization working in the area

With regards to the chance of catastrophic risks which would make this work valuable, one guide here is Michael Aird's database of existential risk estimates, another one is Luisa Rodríguez's work on estimates of the probability of nuclear wars of varying severity. Interestingly, my intuitive estimates vary depending on whether I ask about estimates per year, or estimates in the next 100 years [4].

ALLFED has used this guesstimate model (taken from the post Cost-Effectiveness of Foods for Global Catastrophes: Even Better than Before?) to estimate its own (future) impact. For instance, the ALLFED 2020 Highlights post mentions the following while linking to the model:

I continue to believe that ALLFED's work offers the highest expected value at the margin for improving the long-term future and saving expected lives in the present generation

The model itself gives:

~60% confidence of greater cost-effectiveness than AI for the 100 millionth dollar, and ~95% confidence of greater cost-effectiveness at the margin now than AI. Anders Sandberg's version of the model produced ~80% and ~100% confidence, respectively.

The model presents some structure to estimate ALLFED's impact, namely:

• The chance of a "full-scale nuclear war" and the impact that ALLFED would have in that scenario.
• The chance of a catastrophe which kills 10% of the population, and the impact which ALLFED would have in that scenario

It seems a little bit confusing at first, but it becomes more clear once you go through it cell by cell. In any case, I disagree pretty heavily with some of the estimates in that model, though I appreciate that it's a quantified model that gives something to disagree about.

With those inputs, I arrive, per this guesstimate model at a roughly 50% probability that "marginal money now on alternate foods is more cost effective than on AI risk mitigation". This is in stark contrast with the original 95%, and at a 15% probability that $100M to alternate foods is "more cost-effective than to AI risk mitigation". I endorse the 50%, but not the 15%; I'd probably be higher on the latter. I feel that that 50% is still pretty good, but the contrast between it and the model's initial 95% is pretty noticeable to me, and makes me feel that the 95% is uncalibrated/untrustworthy. On the other hand, my probabilities above can also be seen as a sort of sensitivity analysis, which shows that the case for an organization working on ALLFED's cause area is somewhat more robust than one might have thought. ### Concluding Thoughts In conclusion, I disagree strongly with ALLFED's estimates (probability of cost overruns, impact of ALLFED's work if deployed, etc.), however, I feel that the case for an organization working in this area is relatively solid. My remaining uncertainty is about ALLFED's ability to execute competently and cost-effectively; independent expert evaluation might resolve most of it. ### Sources ## All-Party Parliamentary Group for Future Generations (APPGFG) Epistemic status for this section: Very sure that APPGFG is a very inexpensive opportunity, less sure about other considerations. ### Questions: • Is the APPGFG successfully bringing about substantial change? • Is the APPGFG successfully building capacity to bring about actual change? • Does the APPGFG have enough proposals or actionable advice for ministers to act on? • What are the possible downsides of the APPGFG? • To what extent is the APPGFG constrained by insufficient funding? • How strong is the APPGFG's network of advisors? • Is the APPGFG being cost-effective? • Does the APPGFG have room for growth? ### Tentative answers General considerations Per this writeup, the APPGFG 1. Has been figuring out how best to influence policy in the UK parliament to care more about future generations. 2. Campaigned for an "UK Future Generations Bill to embed a Commissioner for Future Generations into the structures of UK policy making", and successfully lobbied the House of Lords to establish a "Special Inquiry Committee on Risk Assessment and Risk Management," on how the UK prepares for future risks (beyond pandemics) and works internationally to prepare for global risks, which will work for one year. 3. Has been building relationships with parliamentarians. They grew a parliamentary group to include 75 parliamentarians, which can be found here. APPGFG also organized various activities for that group. 4. Has been researching possible policy suggestions: diving into policy areas, and "general research into how the effective altruism community should approach policy, risks and measuring the impact of policy interventions." Their overall theory of impact (referred to here) seems straightforward and plausible. I would further add a step where successful policy change in the UK could spur further change in other countries, particularly in the European sphere. I'm not really sure what their network of external advisors looks like; APPGFG's post mentions receiving positive feedback from the Future of Humanity Institute (FHI), the Center for the Study of Existential Risk (CSER), the UK civil service, and unspecified others. I would be comparatively more excited if the APPGFG's external advisors mostly come from FHI, rather than CSER, about which I have some reservations (more on which below, in CSER's own section). The APPGFG spent roughly$40k for one full-time employee during 2020. This seems very inexpensive. If the APP wanted to expand and thought they had someone they wanted to hire, it would be at the top of my list. It also seems likely that APPGFG's two existing employees could be paid better.

This APPGFG's writeup emphasizes that they have "not yet caused any actual changes to UK government policy", but insofar as what they're doing is capacity building, I find their capacity building work promising.

My understanding is that right now, there aren't that many longtermist related proposals which the APPGFG is able to bring forward, and that the longtermist community itself is uncertain about what kinds of policy proposals to push for. To clarify, my understanding is that policy-wise there is some work the APPGFG can do, such as pushing for the aforementioned Future Generations Bill, nudging legislation in a more longtermist direction, or maybe help shape the UK's attempt at reducing the likelihood of future COVID-19-like catastrophes. However, these proposals seem relatively small in comparison to what a "longtermist policy agenda" could be, and in fact there isn't an ambitious "longtermist policy agenda" that the APPGFG can just push for.

With that in mind, the APPGFG's strategy of embedding itself into Britain's parliamentary processes, while thinking about which more ambitious policy proposals could be brought forward in the future, seems sensible.

Possible downsides

With regards to possible downsides to the APPGFG, the main one in the common EA consciousness seems to be "poisoning the well". This refers to a possible path whether early suboptimal exposure to longtermist ideas could make the audiences more reticent to later consider similar ideas.

Two other downsides are 1) the APPGFG's current leadership getting promoted to incompetence in case the APPGFG grows substantially, and 2) the APPGFG's existence impeding the creation and growth of a more capable organization.

In the first case, maybe the APPGFG's current leadership are good lobbyists and good researchers, but would be unsuitable to lead e.g., a 20 person lobbying apparatus (and would fail to grow into the position.) But by the time the APPGFG was considering growing that much, it would be awkward to replace its leadership. In the second case, maybe there is a more promising person out there who would have done something similar to the APPGFG, but better, and who didn't because the APPGFG already existed.

My impression is that this "promotion to incompetence" dynamic may have happened in some EA research organizations, and that the Iodine Global Network may have been both too weak to establish strong, well-funded national groups, and so large that the creation of another organization to do that would be extremely awkward.

In the counterfactual world where the APPGFG didn't exist, one would still have to develop a policy agenda, and then in addition one would also have to gain familiarity with the British policy-making apparatus, and a presence within it. Whereas in the world where the APPGG does exist, one can develop a longtermist policy agenda informed by political realities, and one has a >2 year head start in establishing a presence in the British political apparatus.

Earlier capacity building seems to me to be worth some poisoning the well, and the overall probability of poisoning the well seems to me to be low. Promotion to incompetence would only be a worry if the APPGFG substantially expanded. Displacing other potentially better organizations seems (to me) to be more of a concern. But overall I think we live in a world where there are not enough people with policy expertise doing EA work, not in the world where there are many and the best are displaced.

### Conclusion

In conclusion, I feel that their logic model is solid, and that the APPGFG's capacity-building work is promising. I'm hesitant about its closeness to CSER. It's current budget seems particularly small. I'm uncertain about how they compare with other organizations in similar or adjacent spheres, and in particular with GovAI. Downsides exist, but accelerated capacity building seems to me to be worth these downsides.

I feel fairly positive about the APPGFG's chances of success:

## CSER

Epistemic status for this section: Unmitigated inside view.

### Questions

• How much of CSER's work is of high value from a long-termist perspective?

A colleague mentioned that there was something "weird" with CSER going on, and I was surprised to find out that this was actually the case.

I skimmed the past research of the members mentioned on their webpage, and I classified their researchers in terms of alignment. I came away with the impression that they had around 5 aligned researchers, around 4 researchers I'm uncertain about, and around 14 whom I'd classify as unaligned or unproductive. CSER also has 6 additional support staff.

Readers are welcome to browse CSER's team page and calculate what percentage of researchers are working on valuable directions according to one's values.

Personally, although I feel like there is a small group of strong researchers working at CSER, the proportion of researchers working on stuff I don't particularly care about or which I don't expect to be particularly valuable according to my values is too high. Commenters pointed out that this assessment is "almost unfairly subjective."

## Center for Security and Emerging Technology (CSET)

Epistemic status for this section: After doing a shallow dive and reading a portion of CSET's work , I have some models about their impact, but they are fuzzy and I don't feel particularly sure about them.

### Questions

• What is a good way to think about CSET's impact?
• How net-positive can we expect CSET's work to be? How likely is CSET to do harm? In particular, how much will CSET's work draw attention to good aspects of AI Safety and fight arms races, as opposed to drawing attention in ways that might amplify arms races or dangerous AI development?
• Is CSET acquiring influence within the US policy community and/or the current administration?
• How does Jason Matheny leaving for the Biden administration affect CSET's impact? How much power and influence does Matheny have in the new Biden administration?
• How much influence would CSET have in a future Republican administration? Might CSET become partisan?
• Does CSET 's influence translate into actual policy?
• Are CSET's researchers well-positioned to join a future US administration?
• How valuable is CSET-foretell? I.e., are the predictions eventually used to make real-world decisions?
• What is the influence of longtermism at CSET? Can we expect this to grow or shrink in the future?
• To what extent should one defer to OpenPhilanthropy's evaluation of CSET? This might be more than normal, as there may be a fair amount of private knowledge, and as models around policy change (and the reasons for believing in those models) might be particularly hard to communicate.

CSET's work can be categorized as:

• Testimonials to the US Congress
• Research
• Media appearances
• Translations
• Forecasting

Analyzing each of them in turn, I looked at past testimonies given by CSET team members to the US Senate and House of Representatives:

• Testimony Before House Homeland Security Subcommittee. This testimony briefly outlines the impact of artificial intelligence on cybersecurity. In the first place, AI systems themselves may be hacked. Secondly, AI systems can augment the capabilities of cyber attacks. Thirdly, AI might help with defense capabilities.
• Testimony Before Senate Banking Committee. The testimony considers export controls on artificial intelligence, and in particular, for data, algorithms, and computing power. It argues that export controls are the most adequate tool for the first two, but that export controls on the hardware that manufactures specialized computer chips for AI might make a difference.
• Testimony Before House Science Committee. The witness describes himself as working for OpenAI rather than for CSET, so I'm not clear to what extent I should count this towards CSET's impact. The testimony argues that we have entered the era of "good enough" AI. However, AI systems frequently exhibit biases, and they may fail, e.g., when encountering outside the training distribution, because of specification gaming. AI systems can also fail as a combination of human error and technical problems, as when recommendation engines optimize for engagement and companies are indifferent to the harms of that. Government should invest in its own capabilities to measure, assess, and forecast aspects; the testimony gives concrete suggestions. Academia should also carry out more targeted research to deal with possible AI failures. Further, industry, government and academia should engage more frequently. Testimony Before House Homeland Security Committee. The author considers how AI could be used for moderating social media platforms, and whether AI contributes to radicalization.
• Testimony Before U.S.-China Economic and Security Review Commission. The author states his affiliation as Center for the Governance of AI, FHI, and makes the case that "China is not poised to overtake the U.S. in the technology domain of AI; rather, the U.S. maintains structural advantages in the quality of S&T inputs and outputs, the fundamental layers of the AI value chain, and key subdomains of AI." It then suggests some policy recommendations to maintain the status quo of US dominance on AI.
• Testimony Before U.S.-China Economic and Security Review Commission. This testimony considers the state of AI, particularly in relationship with China, and argues in general for continued openness.
• Testimony Before Senate Foreign Relations Committee. To maintain competitiveness, the US should focus on its current asymmetric advantages: its network of allies, and its ability to attract the world's best and brightest. The US should also institute export controls on chip manufacturing equipment to ensure that democracies lead in advanced chips. The US should also invest in AI, but deploying AI in critical systems without verifying their trustworthiness poses grave risks.

Personally, I find the testimonies thoughtful and interesting. They distill complex topics into things which US Members of Congress might understand. However, it is unclear to me to what extent these testimonies actually had an impact on policy.

I thought that testimonies were particularly important because one worry outlined in Open Philanthropy's grant to found CSET was:

We worry that heavy government involvement in, and especially regulation of, AI could be premature and might be harmful at this time. We think it's possible that by drawing attention to the nexus of security and emerging technologies (including AI), CSET could lead to premature regulatory attention and thus to harm. However, we believe CSET shares our interest in caution on this front and is well-positioned to communicate carefully.

CSET indeed communicated carefully and with nuance most of the time, at least according to my reading of its testimonials to the US Congress. In particular, it seemed likely that the late Trump administration was going to take punitive actions against China, and providing expert considerations on CSET's area of expertise seemed unlikely to have done harm. There could be some scenarios in which any testimony at all increases political tensions, but this seems unlikely. However, some of the positions which CSET advocated for, e.g., openness and taking in top foreign talent from China, do map clearly across partisan lines, and if that proportion exceeds some threshold, or if CSET never gives support to uniquely Republican stances, CSET and the positions it defends might eventually come to be perceived as partisan.

With regards to research, CSET appears to be extremely prolific, per CSET's list of publications. Some publications which appeared particularly relevant for evaluation purposes are:

Interestingly, CSET's model of working within the prestigious mainstream seems to be particularly scalable, in a way which other organizations in the longtermist sphere are not. That is, because CSET doesn't specifically look for EAs when hiring, CSET's team has been able to quickly grow. This is in comparison with, for example, an organization like Rethink Priorities. The downside of this is that hires might not be aligned with longtermist interests.

Besides testimonials and research, CSET also has a large number of media appearances (cset.georgetown.edu/article/cset-experts-in-the-news through cset.georgetown.edu/article/cset-experts-in-the-news-10). I'm inclined to think that these appearances also have some kind of positive impact, though I am again uncertain of their magnitude.

CSET also carries out a large number of translations of Chinese policy and strategy documents. Lastly, I also occasionally encounter CSET's research "in the wild", e.g., these two blog posts by Bruce Schneier, a respected security expert, mentios a CSET report. This is at least some evidence that relevant experts read these.

Overall, the work that I have read appears to be lucid. But my knowledge of US policy work impact pathways is particularly fuzzy, and the pathways to influence policy are themselves fuzzy and uncertain. Further, unlike with some other organizations, there isn't an annual review I can bootstrap an evaluation from.

For this reason, it is particularly tempting for me to defer to an outside view, like OpenPhilanthropy's grant rationale for the creation of CSET, and its willingness to donate an initial $55 million in 2019, and an additional$8 million at the beginning of 2021. If OpenPhil hadn't been willing to continue to fund CSET, I'd still guess that CSET's work was valuable, but I would be fairly uncertain as to whether it was a comparatively good bet.

In conclusion, CSET's work seems within what I would expect a competent think tank would produce. Given that OpenPhilanthropy is still funding them, I expect them to still be valuable. In particular, its think-tank model seems particularly scalable.

## Future of Life Institute (FLI)

Epistemic status for this section: Uncertain about object-level facts regarding FLI.

### Questions

• What is a good breakdown of FLI's current and future activities?
• How well can FLI ensure quality with part-time employees covering sensitive topics?
• How net-positive has FLI's previous work been? Has anything been particularly negative, or have they incurred significant PR risks or similar?

FLI was also briefly covered by Larks. I think Wikipedia does a better job summarizing FLI than the FLI website:

The Future of Life Institute (FLI) is a nonprofit research institute and outreach organization in the Boston area that works to mitigate existential risks facing humanity, particularly existential risk from advanced artificial intelligence (AI). Its founders include MIT cosmologist Max Tegmark and Skype co-founder Jaan Tallinn, and its board of advisors includes entrepreneur Elon Musk.

Some notable past activities include organizing conferences---such as the Asilomar Conference, which produced the Asilomar Principles on beneficial AI---work on Lethal Autonomous Weapons Systems, giving out the future of life award, and general policy work (open letters, initiatives, pledges, video content, podcasts, etc.) FLI is also a giving vehicle, and recently announced a 25M grant program financed by Vitalik Buterin. The Centre for the Governance of AI thanks FLI on its annual report. To pick an example, for their work on Lethal Autonomous Weapons Systems, their model of impact seems to be that by raising awareness of the topic through various activities, and by pushing governments, NGOs and supranational organizations, they could institute a ban on Lethal Autonomous Weapons. This attempt would also act as a test-ground for "AI Arms Race Avoidance & Value Alignment." So far, while they have raised awareness of the topic, a ban doesn't seem to be forthcoming. Their video on slaughterbots reached a million views on youtube, but, per Seth Baum's talk in EA Global 2018, "the video was fairly poorly received by a lot of important people in international security policy communities, and because of that it has made it more difficult for the people behind the video to get their message out there to these very important audiences." The core team mentioned in their webpage had just seven members, but increased to nine as I was writing this piece. Of these nine, five mention other current affiliations, and it's unclear how many full-time equivalents FLI currently employs. In particular, I'd expect that to make inroads on their five core issues mentioned in their website (x-risk, artificial intelligence, nuclear weapons, biotechnology and climate change), a larger team would be needed. In short, I'm uncertain about how valuable policy work is, about how valuable the specific policy work which FLI has done is, and about whether FLI intends to continue doing policy work. Colleagues have mentioned that FLI isn't so much an organization as "a hat which sometimes people wear," which seems plausible. ## LessWrong Epistemic status: The graphs serve as a sanity check on my intuitions, rather than being their main drivers. ### Questions • Is LessWrong catalyzing useful research? • Is LessWrong successfully cultivating a community of people capable of grappling with important real world problems? • How does LessWrong's research output compare to that of other research institutions? • How many FTEs worth of research is LessWrong responsible for? ### Tentative answers As I understand it, LessWrong's benefits are • to catalyze concrete research • to create and maintain a community of people who are able to capably engage with real world problems See here and here for other people using different wording. With regards to concrete research outputs produced or catalyzed, some recent examples in the last three months from the list of curated posts are related to AI alignment are: With regards to community building, some interaction happens in the comments. Further, the LessWrong team organizes activities, like Solstice celebrations, Petrov Day games, talks, etc. One rough measure of the community building aspect could be the number of new users with more than 500 or 1000 karma in the last couple of years. If we search for these, we find the following: Note that this is, in a sense, unfair to recent years, because newly active users haven't had time to accumulate as much karma as old users. Nonetheless, the conclusion that the LW community recovered from its previous decline holds. It's unclear to me exactly how valuable the production of around 10 highly engaged users with the rationality community is, but the intellectual output of those new 10 users seems probably comparable to that of a small or medium-sized research institute. And the combined output of LW seems much greater. Also note that this would be 10 new highly active users per year. To the extent that these new users belong to already established organizations and just share the output of their work on LessWrong, LessWrong also seems valuable as a locus of discussion. But this doesn't seem to be the main driver of growth in highly engaged users; of the 14 users who joined since the beginning of 2019 and have accumulated more than 500 karma, only around 3 belong to EA-aligned organizations. We can also analyze the number of posts above 100 votes per year, or the total number of votes given to posts in each year. I'm using number of votes (number of people who vote) instead of karma (which includes a multiplier) because the LW API makes that easier to get. In any case, we find If, as a rough approximation, we take 100 votes (for posts) as equivalent to two researcher/weeks, 40,000 votes in 2020 would equal 200 researcher months, or 17 researcher/years. A more qualitative approach would involve, e.g., looking at the LessWrong Review for 2018, and asking how much one would be willing to pay for the creation and curation of the collected posts, or comparing their value to the value of FHI's publications for the same year. One would have to adjust for the fact that around 1/4th of the most highly upvoted posts are written by MIRI employees. In conclusion, LW seems to catalyze or facilitate a relatively large amount of research, and that it does so relatively efficiently, with around 6 FTEs (per the team page). Concretely, LessWrong appears to produce substantially more than one FTE worth of research per FTE. One key question is whether many of the LessWrong posts would have just been written elsewhere. In addition, the LessWrong codebase is also used by the EA Forum and by the AI Alignment Forum. ### Sources ## Rethink Priorities (RP) Epistemic status: Only talking about explicitly longermist-branded parts of their research. ### Questions • How many FTEs are currently working using a longtermist perspective at Rethink Priorities? • Will Rethink Priorities be able to produce research in the long-termist space similar in quality to the research they have produced on invertebrate welfare? • Will Rethink Rethink Priorities be able to productively expand into the longtermist sphere? How will it do so? • How many FTEs producing high-quality longtermist research will RP employ by 2025? ### Tentative answers Rethink Priorities has recently been expanding into the longtermist sphere, and it did so by hiring Linch Zhang and Michael Aird, the latter part-time, as well as some volunteers/interns. At this point, I feel that the number of longtermist FTEs is so small that I wouldn't be evaluating an organization, I would be evaluating individuals. All in all, Zhang and Aird haven't spent enough time at RP that I feel that their output would be representative. This is in contrast to, e.g., FHI's Research Scholars program, which is large enough that I feel it would make more sense to talk about the average quality of a researcher. That said, some of RP's recent inputs can be found under their EA Forum tag. With regards to the expected quality of future research, on the one hand, past high quality research is predictive of future quality. On the other hand, research into invertebrate sentience feels foundational for animal-focused ethics and activism in a way which seems hard to upstage, so one might expect some regression to the mean. ### Sources ## Simon Institute for Long-Term Governance (SILG) Epistemic status: Brief and cursory. Considerations apply to other new organizations. ### Questions • What does the prior distribution of success for new longermist organizations look like? • When will we have a better estimate of the Simon Institute for Long-Term Governance's input? • Is funding SILG better than OpenPhilanthropy's last longtermist dollar? ### Tentative answers I imagine that the prior distribution of success for new organizations is pretty long-tailed (e.g., a Pareto distribution). This would lead to a high initial expected value for new organizations, which most of the time sharply drops off after some initial time has passed and there is more information about the promisingness of the project. I imagine that ~two years might be enough to determine if a new organization is promising enough to warrant further investment. If that was the case, the optimal move would look like funding a lot of new organizations, most of which are then deprived of funding shortly after an initial grace period. It's not clear how to create a functional culture around that dynamic. Silicon Valley aguably seems to be able to make it work, but they have somewhat reliable proxies of impact (e.g., revenue, user growth), whereas long-termists would have to rely on uncertain proxies. The above considerations are fairly generic, and would apply to organizations other than SILG. Overall, I estimate that funding SILG for the first two years of existence and seeing how they fare seems valuable, but I'm not very certain. ### Sources ## 80,000 hours Epistemic status: Deferring a lot to 80,000h's evaluation of itself. ### Questions • Can I generally defer to Benjamin Todd's judgment? • Will 80,000 hours continue to keep similar levels of cost-effectiveness as it scales? • Will 80,000 hours manage to keep its culture and ethos as it scales? • How does 80,000 hours compare to other, more speculative donation targets and career paths? • What percentage of 80,000 hours' impact is not related to career plan changes? • Will the percentage of 80,000 hours' impact not related to career plan changes remain constant as 80,000 hours scales? (so that thinking of 80,000 hours' impact as a multiple of the impact of its career changes "makes sense")? • What is a good way to think about 80,000 hours' aggregate impact? ### Tentative answers 80,000 hours has a clear evaluation of itself. For me, the gist is that 1. 80,000 hours appears to have reached a point of maturity: Each programme is working well on its own terms. There's a sensible, intuitive case for why each should exist, and their mechanisms for impact seem reasonably solid. They all seem to generate a reasonable number of plan changes or other value, and I expect them to compare well with alternatives. Big picture, 80,000 Hours seems likely to be among the biggest sources of talent into longtermist EA over the last couple of years, and it seems great to capitalize on that. 2. The CEO is keen on expanding: "Two years ago, I felt more uncertain about cost effectiveness and was more inclined to think we should focus on improving the programmes. My views feel more stable now, in part because we've improved our impact evaluation in response to critical feedback from 2018, clarified our views on the one-on-one programmes, and taken steps to limit negative side effects of our work. So, I think it makes sense to shift our focus toward growing the programmes' impact. Below I propose a two-year growth plan in which we aim to add 4.5 FTE in 2021, and 7.5 in 2022, though we plan to fundraise for 3.75 and 6.5, as we expect to hire no more than that many over the next two years in practice." Now, normally I'd think that the key questions were something like: • How many impact-adjusted career plan changes will 80,000 hours produce in 2021? • How many impact-adjusted career plan changes will 80,000 hours produce in 2021 per100,000 in funding?

And indeed, most of 80,000 hours' impact tracking and quantification is done with regards to career plan changes (operationalized as "discounted, impact-adjusted peak years"). However, per the 80,000 hours review:

We remain unsure that plan changes are the best framework for thinking about 80,000 Hours' impact, and we think they capture only a minority of the value, especially for the website and podcast. For example, I think it's plausible that most of our past impact has come from getting the EA movement more focused on longtermism and spreading other important ideas in society. An analysis I did this year confirmed my previous impression that 80,000 Hours is among the biggest and most effective ways of telling people about EA (though I expect less cost effective than the most successful written content, such as Doing Good Better and Slate Star Codex).

It is possible that further estimation of non-career plan change related impact would be clarifying, even if the estimation is very fuzzy. In particular, to the extent that most of 80,000 hours' impact comes from influencing the EA community, and this sounds plausible, having most of their evaluation focus on career plan changes feels misguided (cf. Streetlight effect).

(Despite feeling comfortable with the guess above, in practice, I've found that estimating total impact by estimating the impact of a measurable part and the fraction of value it represents leads to large errors)

With regards to cost-efficiency, 80,000 hours had a budget in 2020 of approximately $3M, and around 19 FTEs. In short, 80,000 hours' career changes seem valuable, but most of the organization's impact might come from fuzzier pathways, such as moving the EA community and 80,000 hours' followers in a more longtermist direction. I'm uncertain about the value of expansion. ### Sources ## Observations I don't have any overarching conclusions, so here are some atomic observations: • The field seems pretty messy, and very far from GiveWell style comparison and quantification. • That said, it still seems plausible that some organizations are much more valuable than others (per unit of resources, etc.) • A core proposition of longtermism is that by focusing on regions in which impact is less measurable, we might attain more of it. This is as we might expect from e.g. Goodhart's law (optimizing for impact will diverge from optimizing for measurable impact.) However, this plays badly with evaluation efforts, and perhaps with prioritization efforts among different longtermist opportunities. • Many organizations have a large number of "affiliates", or "associates", some of which may be pursuing PhDs somewhere else, be affiliated with more than one organization, or work only part-time. This makes it harder to know how many full-time equivalents are working for each organization, and how productive the organization is given its budget. • Many of these organizations have done a good job having prestigious people in their board of advisors, such that e.g., having Elon Musk or Nick Bostrom seems like a weaker signal that it could be. I'd welcome comments about the overall method, about whether I'm asking the right questions for any particular organization, or about whether my tentative answers to those questions are correct, and about whether this kind of evaluation seems valuable. For instance, it's possible that I would have done better by evaluating all organizations using the same rubric (e.g., leadership quality, ability to identify talent, working on important problems, operational capacity, etc.) I'd also be interested in algorithms to allocate funding supposing one had answers to all the questions I pose above, but did not have a clear way of comparing the impact of organizations working on different domains. Thanks to Ozzie Gooen, Gustavs Zilgavis, Kelsey Rodriguez, Tegan McCaslin for comments and suggestions. # Appendix: Organizations about whose evaluations I'm less sure ## Center on Long-term Risk (CLR) Epistemic status for this section: Confused. In particular, I get the sense that for CLR, more than for other organizations, a fair evaluation probably requires deeply understanding what they do, which I don't. ### Questions • Is most of their research only useful from a suffering-focused ethics perspective? • Is there a better option for suffering-focused donors? • Is the probability of astronomical suffering comparable to that of other existential risks? • Is CLR figuring out important aspects of reality? • Is CLR being cost-effective at producing research? • Is CLR's work on their "Cooperation, conflict, and transformative artificial intelligence"/"bargaining in artificial learners" agenda likely to be valuable? • Will CLR's future research on malevolence be valuable? • How effective is CLR at leveling up researchers? ### Tentative answers Previously, Larks briefly reviewed CLR on his 2020 AI Alignment Literature Review and Charity Comparison. Sadly, CLR's work on AI Safety related problems seems hard to judge as an outsider on the merits, and I get the impression that they are fairly disconnected from other longtermist groups (though CLR moved to London last year, which might remedy this.) This Alignment Forum post makes the case that multi-agent reinforcement learning, which CLR plans to explore in 2021, isn't particularly neglected. Their Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda series on the Alignment forum didn't get many comments. Fortunately, one of CLR's aims for the year is to "elicit feedback from outside experts to assess the quality and impact of our work"; I'm curious to see how that goes. I'm not sure about whether further work on malevolence would be fruitful. In particular, it seems to me that the original post was very interesting and engaging. However, possible conclusions or proposals stemming from this kind of project are probably not implementable in the current political system. For instance, requiring psychopathy tests for politicians, or psychological evaluation, seems very unrealistic. That said, perhaps one possible longer-term strategy might be to have proposals ready which can be implemented in the ensuing policy window following unexpected turmoil (e.g., pushing for psychopathy tests for politicians might have been more feasible in the aftermath of the Nürnberg trials, or after Watergate.) I imagine that people who interface with policy directly probably have better models about the political feasibility of anti-malevolence proposals. Maybe considering CLR's research agenda isn't a good way to think about its potential impact. Daniel Kokotajlo's work on AI timelines strikes me as valuable, and is outside that research agenda. I have the subjective impression that CLR has historically been good at providing mentorship/funding for junior people trying to jump into EA research, e.g., for Michael Aird, Jaime Sevilla, even when their ethics were not particularly suffering-focused. I found CLR particularly transparent with respect to their budget; their expected budget for 2021 was$1,830,000, and they expect to have 13.7 FTEs for the year. Commenters pointed out that this was surprisingly large compared to other organizations, e.g., 80,000 hours has around 19 FTEs (on a ~3M budget). In short, I don't feel particularly enthused about their research agenda, but overall I'm not sure how to think about CLR's impact. ### Sources ## Future of Humanity Institute Epistemic status for this section: Arguably shouldn't exist; FHI was just too large to be evaluated in a short time, so instead I rely mostly on status as a lagging indicator of impact. ### Questions • Is FHI figuring out important aspects of reality? • How valuable is additional funding for FHI likely to be? What proportion of donations to FHI goes to Oxford University? • Is it better to evaluate FHI as a whole, or team by team? • Is FHI's status proportionate to its current impact? That is, can we trust status as a measure of impact, or is it too laggy a measure? Does FHI get all or almost all of its status from a handful of very valuable projects? • How much x-risk reduction can we expect from FHI's research? Does it make sense to express this as a percentage, or as a distribution over percentages? • Besides x-risk reduction, can we also expect some dampening in the badness of the catastrophes that do happen? Can we expect that the value of the far future, conditional on not having an x-risk, is better? • Is FHI causing policy change? Will FHI's research and advocacy influence Britain's or the EU's AI policy? • Does/Will the vast majority of FHI's impact come from current senior researchers (Bostrom, Drexler, etc.)? • FHI has expanded a lot recently and seems to be continuing to do so. How well can it maintain quality? • What does the future of FHI operations look like? Will this substantially bottleneck the organization? • What are FHI's main paths to impact? Do other longtermist organizations find their continuing work highly valuable? • FHI researchers have historically helped identify multiple "crucial considerations" for other longtermists (like flagging X-risks). Do we think it's likely to continue to do so? ### Tentative answers Per their team page, FHI is divided into the following teams: • Macrostrategy Research Group • AI Safety Research Group • Biosecurity Research Group • Centre for the Governance of AI • Research Scholars Programme • Some number of associates and affiliates. Despite living under the FHI umbrella, each of these projects has a different pathway to impact, and thus they should most likely be evaluated separately. Note also that, unlike most other groups, FHI doesn't really have consistent impact accounting for the organization as a whole. For instance, their last quarterly report, from their news section is from January to March 2020 (though it is possible that they have yet to publish their annual review for 2020.) Consider in comparison 80,000 hours' annual review, which outlines what the different parts of the organization are doing, and why each project is probably valuable. I think having or creating such an annual review probably adds some clarity of thought when choosing strategic decisions (though one could also cargo-cult such a review solely in order to be more persuasive to donors), and it would also make shallow evaluations easier. In the absence of an annual review to build upon, I'm unsatisfied with my ability to do more than a very shallow review in a short amount of time. In particular, I start out with the strong prior that FHI people are committed longtermists doing thoughtful work, and browsing through their work doesn't really update me much either against or in favor. I imagine that this might change as I think more about this, and maybe come up with an elegant factorization of FHI's impact. In any case, below are some notes on each of the groups which make up FHI. In the meantime, it seems that FHI doesn't seem to be hurting for money, but that Open Phil is hesitant to donate too much to any particular organization. If one thinks that appeasing Open Phil's neurosis is particularly important, which, all things considered, might be, or if one thinks that FHI is in fact hurting for money, FHI might be a good donation target. ### Macrostrategy and AI Safety Research Groups Some of the outputs from these two groups were favorably reviewed by Larks here. ### Biosecurity Research Group Some publications can be found in FHI's page for the research group's members (Gregory Lewis, Cassidy Nelson, Piers Millett). Gregory Lewis also has some blog posts on the EA forum. I browsed their publications, but I don't think I'm particularly able to evaluate them, given that they are so far outside my area of expertise. In the medium term (e.g., once the pandemic has subsided), some outside expert evaluation in Open Philanthropy's style might be beneficial. Nonetheless, I'm somewhat surprised by the size of the team. In particular, I imagine that to meaningfully reduce bio-risk, one would need a bigger team. It's therefore possible that failing to expand is a mistake. However, commenters on a draft of this post pointed out that this isn't straightforward; expanding is difficult, and brings its own challenges. ### Centre for the Governance of AI (GovAI) Some of the outputs from the Centre for the Governance of AI were favorably reviewed by Larks here (same link as before). In addition, GovAI has its own 2020 Annual Report. It also has a post on the EA forum outlining its theory of impact, which is outlined with extreme clarity. ### Research Scholars Programme, DPhil Scholars A review of FHI's Research Scholars Programme can be found here. The page for the DPhil Scholarship can be found here. FHI also has a Summer Research Fellowship, a review of which can be found here. Overall, I'd guess that these programs have similar pathways to impact to some of the LTF grants to individual researchers, but the advantage that the participants gain additional prestige through their association with Oxford (as in the case of Research Scholars), or become more aligned with longtermist priorities (perhaps as in the case of the DPhil program). ### Other associates and affiliates. Associates and affiliates could contribute a small but significant part of FHI's impact, but in the absence of very detailed models, I'm inclined to consider them as a multiplier (e.g. between x 1.05 and x 1.5 on FHI's base impact, whatever that may be). ### Conclusion In conclusion, FHI's output is fairly large and difficult to evaluate, particularly because they don't have a yearly review or a well organized set of outputs I can bootstrap from. GovAI seems to be doing particularly valuable work. I still think highly of the organization, but I notice that I'm relying on status as a lagging indicator of quality. ### Sources ## Global Priorities Institute Epistemic status: Uncertain about how valuable GPI's work is, and about my ability to evaluate them. ### Questions • How promising is GPI's strategy of influencing reputable academics over the long term? • Is GPI discovering new and important truths about reality? • Is GPI conducting research which answers the question "What should an agent do with a given amount of resources, insofar as her aim is to do the most good?"? • Is their advocacy paying out? • Will GPI be able to get promising economists in the future? ### Tentative answers GPI's 2020 annual report is fairly short and worth reading in full. It describes GPI's aims as: The Global Priorities Institute (GPI) exists to develop and promote rigorous academic research into issues that arise in response to the question "What should an agent do with a given amount of resources, insofar as her aim is to do the most good?". The investigation of these issues constitutes the enterprise that we call global priorities research. It naturally draws upon central themes in (in particular) the fields of economics and philosophy; the Institute is interdisciplinary between these two academic fields. Overall, I see various pathways to impact which could arise from this kind of philosophy work: 1. Philosophical clarity might be needed to optimally allocate donations. At the donation volume of an organization like OpenPhilanthropy or the Gates Foundation, relatively subtle changes in philosophical stances could lead to large changes in funding allocation. Further, some empirical considerations, such as those relating to the hinge of history hypothesis could also have more than marginal impact. 2. Academic consensus could lead to policy change, by building the philosophical backbone of longtermism which would support and allow for future policy work. 3. In particular, acquiring prestige in an academic field to then later influence policy may not require the academic field to be useful (i.e., it could be prestige about abstruse philosophical disputes). For example, testimony on future generations to the UK Parliament by an Oxford professor may be listened to because of the Oxford professorship, independent of its field. 4. Trailblazing philosophy might pave the way for future practical developments. Exploring the moral landscape could lead to understanding the shape of our values, and realizing that e.g., invertebrates may hold some moral weight, or that most of the value of humanity may lie in its far away future. Organizations could later be created to work on the issues identified. A particularly striking example of this might be Trammell's work on patient philanthropy, which might lead to a Patient Philanthropy fund. Another example might be Brian Tomasik's essays on reducing suffering. 5. Good philosophy might facilitate movement building, particularly inside academia. For instance, university professors might give courses on longtermism. 6. Understanding ethical truths and decision theories at an extreme level of clarity would allow for the development of safer AI. This doesn't seem to be GPI's focus. It is possible that I am missing some paths to impact. Right now, I see GPI as mostly aiming for 2., and growing its contingent of economists to allow for 3. 5. also seems to be happening, but it's unclear what role GPI plays there (though potentially it could be a substantial role). Readers might want to browse GPI's list of publications (note that the list also contains papers which are relevant to GPI's research agenda by authors not affiliated with GPI). I'm personally confused about their object level value, though some people I respect tell me that some are great. In short, I'm fairly uncertain about GPI's pathway to impact. Acquiring prestige and status might enable future policy work. Economics research, which GPI has been expanding into, seems more valuable. ### Sources ## Notes [1]. One common unit might be "Quality-Adjusted Research Projects'', which could capture how efficiently an organization produces valuable research. However, that unit might be unsatisfactory, because research in different areas probably leads to differentially different outcomes. A different unit might be a "microtopia", which according to oral lore was defined by Owen Cotton-Barratt to represent one millionth of the value of an ideal long-termist utopia. One might also try to compare the value of additional funding to a threshold, like the value of OpenPhilanthropy's last (longtermist) dollar, or to compare to a given level of formidability. [2]. Initially, I thought that the result of this project might be a GiveWell-style evaluation of longtermist organizations, just many, many orders of magnitude more uncertain. For instance, if organization A produces between 1 and 10^6 "utilons'' per unit of resources (attention, effort, money, etc.), and organization B produces between 0.01 and 10^3 "utilons" per unit of resources, we would want to choose organization A over organization B, even though the impact estimates overlap and are very uncertain. [3]. Below is a list of perhaps notable organizations which I could have evaluated but didn't. As mentioned, because of their additional complexity, and to bound the scope of this post, I decided to exclude meta organizations. • Alcor Life Extension Foundation. Though cryonics has been proposed as an EA cause area in the past, it hasn't acquired mainstream acceptance as such. • Alpenglow. They recently rebranded as the Centre for Long-Term Resilience, and I feel that the information on their webpage/online is too little to conduct an informed evaluation. • Berkeley Existential Risk Initiative. It's a meta-organization. • CEELAR (formerly the EA Hotel). It's a meta-organization. • CFAR. Private. • Center for Election Science. Time limits, and too solid a pathway to impact. Though estimating the impact on governance of better voting systems would be difficult, I feel like most other organizations in this list have an impenetrable fog in their pathway to impact which CES doesn't really have. This is the organization I feel most uncertain about not having added. • Emergent Ventures. It's a meta-organization. • Future of Humanity Foundation. In the medium to long run, I can imagine this becoming an attractive donation target. In the short run, its value would depend on what FHI staff would do with money unaccountable to Oxford University, which I don't have much insight about. • Long-Term Future Fund. It's a meta-organization. • Nonlinear Fund. It's a meta-organization. Also, their webpage is down. (edit: was down for me at the time; it's up now (archive link.)) • Open Philanthropy Fund. It's a meta-organization. • Qualia Research Institute. Its pathway to impact appears implausible and overly ambitious. • Quantified Uncertainty Research Institute. I was planning to do an evaluation at the end of the year. • Sentience Institute. It's between the longtermist and the animal rights/suffering spheres. [4]. Which suggests a bias, perhaps because I'm reticent to assign probabilities lower than 1%, even if it's per year. In the estimates later in the section, I ended up going mostly with yearly estimates based on my 100 year estimates. [6]. Manhattan Project. "The Manhattan Project began modestly in 1939, but grew to employ more than 130,000 people and cost nearly US2 billion (equivalent to about $23 billion in 2019)." [7]. Lockheed Martin F-35 Lightning II development. "The program received considerable criticism for cost overruns during development and for the total projected cost of the program over the lifetime of the jets. By 2017 the program was expected over its lifetime (until 2070) to cost$406.5 billion for acquisition of the jets and $1.1 trillion for operations and maintenance." [8]. general purpose grants are likely less valuable per dollar than the best way to spend the marginal dollar for longtermist impact. [9]. For instance, Exceeding expectations: stochastic dominance as a general decision theory makes the point that stochastic dominance (A stochastically dominates B if 1) for all events x the probability for equal or better events is greater or equal in A than in B, and 2) there is at least one possible event for which the inequality is strict) generalizes even to comparisons of events with infinite or undefined expected value. Further, in the presence of "background uncertainty", stochastic dominance provides similar results to expected value, which might convince expected value skeptics to take some Pascalian-seeming wagers if the probability on which they depend is small, but not too small. Note that the paper doesn't word things that way. It also suggests in the latter sections that stochastic dominance stands as a decision theory on its own, which I'm very skeptical about. # 170 34 comments, sorted by Click to highlight new comments since: New Comment Quick bits of info / thoughts on the questions you raise re CLR (I spent 3 months there as a Summer Research Fellow, but don't work there anymore, and am not suffering-focused, so might be well-positioned to share one useful perspective.) • Is most of their research only useful from a suffering-focused ethics (SFE) perspective? • I think all of the research that was being done while I was there would probably be important from a non-SFE longtermist perspective if it was important from a SFE longtermist perspective • It might also be important from neither perspective if some other premise underpinning the work was incorrect or the work was just low-quality. But: • I think it was all at least plausibly important • I think each individual line of work would be unlikely to turn out to be important from an SFE longtermist perspective but not from a non-SFE longtermist perspective • This is partly because much of it could be useful for non-s-risk scenarios • E.g., much of their AI work may also help reduce extinction risks, even if that isn't CLR as an organisation's focus (it may be the focus of some individual researchers, e.g. Daniel K - not sure) • This is also partly because s-risks are also really bad from a non-SFE perspective (relative to the same future scenario but minus the suffering) • All that said, work that's motivated by a SFE longtermist perspective should be expected to be higher priority from that perspective than from another perspective, and I do think that that's the case for CLR's work • That said, if CLR had a substantial room for more funding and I had a bunch of money to donate, I'd seriously consider them (even if I pretend that I give SFE views 0 credence, whereas in reality I give them some small-ish credence) • Is there a better option for suffering-focused donors? • I think the key consideration here is actually room for more funding rather than how useful CLR's work is • I haven't looked into their room for more funding • Is the probability of astronomical suffering comparable to that of other existential risks? • Is CLR figuring out important aspects of reality? • I think so, but this is a vague question • Is CLR being cost-effective at producing research? • I haven't really thought about that • Is CLR's work on their "Cooperation, conflict, and transformative artificial intelligence"/"bargaining in artificial learners" agenda likely to be valuable? • I think so, but I'm not an expert on AI, game theory, etc. • Will CLR's future research on malevolence be valuable? • I think so, conditional on them doing a notable amount of such work (I don't know their current plans on that front) • And this I know more about since it was one of my focuses during my fellowship • How effective is CLR at leveling up researchers? • I think I learned a lot while I was there, and I think the other summer research fellows whose views I have a sense of felt the same • But I haven't thought about that from the perspective of "Ok, but how much, and what was the counterfactual" in the way that I would if considering donating a large amount to CLR • (I've noticed that my habits of thinking are different when I'm just having regular conversations or reading stuff or whatever vs evaluating a grant) • Two signals of my views on this: • I've recommended several people apply to the CLR summer research fellowship (along with other research training programs) • I've drawn on some aspects of or materials from CLR's summer research fellowship when informing the design of one or more other research training programs • "I get the impression that they are fairly disconnected from other longtermist groups (though CLR moved to London last year, which might remedy this.)" • I don't think CLR are fairly disconnected from other longtermist groups • Some data points: • Stefan Torges did a stint at GovAI • Max Daniel used to work there, still interacts with them in some ways, and works at FHI and is involved in a bunch of other longtermist stuff • I worked there and am still in touch with them semi-regularly • Daniel Kokotajlo used to work at AI Impacts • Alfredo Parra, who used to work there, now works at Legal Priorities Project • Jonas Vollmer used to work there and now runs EA Funds • I know of various of other people who've interacted in large or small ways with both CLR and other longtermist orgs I am not intending here to convince anyone to donate to CLR or work for CLR. I'm not personally donating to them, nor working there. Though I do think they'd be a plausibly good donation target if they have room for more funding (I don't know about that) and that they'd be a good place to work for many longtermists (depending on personal fit, career plans, etc.). Personal views only, as always. I think I learned a lot while I was there, and I think the other summer research fellows whose views I have a sense of felt the same +1. I'd say that applying for and participating in their fellowship was probably the best career decision I've made so far. Maybe 60-70% of this was due to the benefits of entering a network of people whose altruistic efforts I greatly respect, the rest was the direct value of the fellowship itself. (I haven't thought a lot about this point, but on a gut level it seems like the right breakdown.) Thanks for both comments here. Personal anecdotes are really valuable, and I assume would be useful to later people trying to get some idea of the value from CLR. Sadly, I imagine there's a significant bias for positive comments (I assume that people with negative experiences would be cautious of offending anyone), but positive comments still have signal. Sadly, I imagine there's a significant bias for positive comments (I assume that people with negative experiences would be cautious of offending anyone), but positive comments still have signal. Yeah, I think that this is true and that it's good that you noted it. Though that brings to mind another data point, which is that several people who did the summer research fellowship at the same time as me are now still working at CLR. I also think that there might be a bias against the people who still work at an org commenting, since they wouldn't want to look defensive or like they're just saying it to make their employer happy, or something. But overall I do think there's more bias towards positive comments. (And there are also other people I haven't stayed in touch with and who aren't working there anymore, who for all I know could perhaps have had worse experiences.) Thanks Michael, beautiful comment. Thanks for considering ALLFED. We try to respond to inquiries quickly. We have looked back, and have not be able to locate any such inquiries. We will be finalizing our 2020 report with financial details soon. Thanks a lot for the engagement in the cost-effectiveness model. To clarify, the cost of preparation does not include the scale up in a catastrophe. The idea is that the resilient foods (we are rebranding away from “alternative foods”) could be scaled up without large-scale preparation (e.g. countries would repurpose the paper factories to produce food after the catastrophe, rather than spending billions of dollars ahead of time). Most of the promising resilient foods have already been commercialized. In this paper, we found that if there were no resilient foods, expenditure on stored foods in a catastrophe would be approximately$90 trillion and about 10% of people would survive. However, if resilient foods could be produced at $2.5 per dry kilogram retail, 97% of people would survive but the total expenditure would only be ~$20 trillion. So one could argue that resilient foods would actually save money in a catastrophe. But we did not include that effect in the cost-effectiveness model.

I expect that affecting a large amount of the Earth's future impact (i.e., 3 to 50% of the future impact of humanity) would be very hard even in extreme circumstances.

Just to make sure we are on the same page, if there were a 10% probability of full-scale nuclear war in the next 30 years and there were a 10% reduction in the long-term future potential of humanity given nuclear war, and if planning and R&D for resilient foods mitigated the far future impact of nuclear war by 50%, then that would improve the long-term potential of humanity by 0.5 percentage points (the product of the three percentages).

[I'll put some thoughts on the ALLFED section here to keep discussion organised, but this is responding to Nuno's section rather than David's comment.]

I feel that that 50% is still pretty good, but the contrast between it and the model's initial 95% is pretty noticeable to me, and makes me feel that the 95% is uncalibrated/untrustworthy. On the other hand, my probabilities above can also be seen as a sort of sensitivity analysis, which shows that the case for an organization working on ALLFED's cause area is somewhat more robust than one might have thought.

[...]

In conclusion, I disagree strongly with ALLFED's estimates (probability of cost overruns, impact of ALLFED's work if deployed, etc.), however, I feel that the case for an organization working in this area is relatively solid. My remaining uncertainty is about ALLFED's ability to execute competently and cost-effectively; independent expert evaluation might resolve most of it.

I think this mostly sounds similar to my independent impression, as expressed here, though I didn't specifically worry particularly about their ability to execute competently and cost-effectively. (I'm not saying I felt highly confident about that; it just didn't necessarily stand out much to me as a key uncertainty, for whatever reason.)

E.g., I wrote in the linked comment:

• Their cost-effectiveness estimates seem remarkably promising (see here and here).
• But it does seem quite hard to believe that the cost-effectiveness is really that good. And many of the quantities are based on a survey of GCR researchers, with somewhat unclear methodology (e.g., how were the researchers chosen?)
• I also haven’t analysed the models very closely
• But, other than perhaps the reliance on that survey, I can’t obviously see major flaws, and haven’t seen comments that seem to convincingly point out major flaws. So maybe the estimates are in the right ballpark?

One thing I'd add is that most of your (Nuno's) section on ALLFED sounds like it's seeing ALLFED's impact as mostly being about their research & advocacy itself. But I think it's worth also giving a fair amount of emphasis to this question of yours: "Given that ALLFED has a large team, is it a positive influence on its team members? How would we expect employees and volunteers to rate their experience with the organization?"

I'd see a substantial fraction of the value of ALLFED as coming from how it might work as a useful talent pipeline. And I think that this could also be a source of nontrivial downside risk from ALLFED, e.g. if their training is low-quality for some reason, or if people implicitly learn bad habits of thinking/research/modelling, or if their focuses aren't good focuses and they make their volunteers more likely to stay focused on that long-term.

(I'm not saying that these things are the case. I'd currently guess that ALLFED produces notable impact as a talent pipeline. But I haven't looked closely and think it'd be worth doing so if one wanted to do a "thorough" evaluation of ALLFED.)

Thanks for considering ALLFED. We try to respond to inquiries quickly. We have looked back, and have not be able to locate any such inquiries. We will be finalizing our 2020 report with financial details soon.

This is most likely my fault; I think I got confused between allfed.org and allfed.info

To clarify, the cost of preparation does not include the scale up in a catastrophe

For clarity:
1. Your guesstimate model: 3% to 50% mitigation of the impact of war with a 30M to 200M, a war which has a probability 0.02% to 5% per year. You also say that so far, you've already mitigated the impact of such a war by 1% to 20%.
2. My model: a 0% to 15% mitigation (previously 0 to 5%, see below) of the impact of such a war with a 50M to 50B investment, where this is maybe not being fully prepared, but does include some serious paranoid preparation, some factories running, supply chains established, etc.
3. Objection: You're planning to go with the 30M to 200M path; my estimates should be for that path.
4. Answer: I'd have to think about it. Maybe 2x to 10x lower. Essentially I'd expect any preparation to at least fail partially, fail to get implemented, be ignored, not survive in institutional memory, etc.

In this paper, we found that if there were no resilient foods, expenditure on stored foods in a catastrophe would be approximately $90 trillion and about 10% of people would survive. However, if resilient foods could be produced at$2.5 per dry kilogram retail, 97% of people would survive but the total expenditure would only be ~$20 trillion. So one could argue that resilient foods would actually save money in a catastrophe I'll read the paper. Just to make sure we are on the same page, if there were a 10% probability of full-scale nuclear war in the next 30 years and there were a 10% reduction in the long-term future potential of humanity given nuclear war, and if planning and R&D for resilient foods mitigated the far future impact of nuclear war by 50%, then that would improve the long-term potential of humanity by 0.5 percentage points (the product of the three percentages). I see, thanks, I think I was getting this wrong (I've changed this in the guesstimate, but not in the post). With that in mind, your estimates now seem less high (but still very high). It changes my estimates slightly. Separately, your numbers still seem fairly high. Suppose that in 1980 you had$100M and knew that there was going to be a pandemic (or another global financial crisis) in the next 100 years, but didn't knew the details; it seems unlikely that you could have made the covid pandemic or the 2008 financial crisis more than 10% better.

This is most likely my fault; I think I got confused between allfed.org and allfed.info

We tried to buy the .org domain, but unfortunately it was not for sale.

Essentially I'd expect any preparation to at least fail partially, fail to get implemented, be ignored, not survive in institutional memory, etc.

There are definitely a lot of failure modes, though part of the money should go to updating institutions as staff turn over.

Thanks for updating the Guesstimate.

Separately, your numbers still seem fairly high. Suppose that in 1980 you had $100M and knew that there was going to be a pandemic (or another global financial crisis) in the next 100 years, but didn't knew the details; it seems unlikely that you could have made the covid pandemic or the 2008 financial crisis more than 10% better. Good question. I think these are quite different because billions of dollars had been put into preparedness, at least for a pandemic. Though billions of dollars have been put into preventing a nuclear war (and reducing weapon stockpiles), we could not find anything preparing for feeding populations for a multiyear catastrophe. I think generally there are logarithmic returns, which means the first amount of money spent on a problem has much greater marginal cost effectiveness. [re: FHI Bio] Nonetheless, I'm somewhat surprised by the size of the team. In particular, I imagine that to meaningfully reduce bio-risk, one would need a bigger team. It's therefore possible that failing to expand is a mistake. Specifically on the point about FHI's bio team, as a semi-insider providing information that isn't on the web site but isn't private, I'll note that the team is actually larger, in several ways. First, they have summer fellows and Oxford PhD students not officially hired by FHI that they work with. They also have people working jointly at/with other organizations (e.g. Piers is at iGem, as is Tessa, and they both talk with FHI folks a lot. Greg is a CHS ELBI fellow, and works with lots of people at CHS/NTI/etc. I work with/for them as a contractor. And Andrew Snyder-Beattie at OpenPhil used to be in charge of the team, and has coordinated projects with other groups.) Lastly, they have also recently brought on board additional people, not reflected on the web page. (Not sure about timing or announcements, so I won't say anything.) Strong upvoted - I found this very interesting, both for various parts of the specific evaluations and more generally as an example of one way to do longtermist charity evaluation (which currently seems to be rarely done in anything beyond a cursory or solely qualitative way, at least in public writings). One question I have is what you saw as the key purposes of this post. Some possibilities: 1. Inform decisions about donations that are each in something like the$10-$5000 dollar range 2. Inform decisions about donations/grants that are each in something like the >$50,000 dollar range
• (Obviously I'm missing the $5,000-$50,000 range; I have a vague sense that the more interesting question is which of those two buckets I pointed to you're more focused on, if either)
3. Inform decisions about which of these orgs (if any) to work for
4. Provide feedback to these orgs that causes them to improve
5. Provide an accountability mechanism for these orgs that causes them to work harder or smarter so that they look better on such evaluations in future
6. Just see if this sort of evaluation can be done, learn more about how to do that, and share that meta-level info with the EA public
7. [something else]

• My sense is that, traditionally, public charity evaluations are mostly focused on informing decisions by individual donors giving non-huge sums each
• But this seems somewhat less relevant in longtermism than in other EA cause areas
• Longtermism seems somewhat less funding-constrained than other EA cause areas
• In longtermism compared to in other areas, evaluation seems harder for various reasons, and so the case for giving to a donation lottery or a fund whose dollars are distributed by specialist grantmakers seems stronger
• But I guess your post, or other things like it, could mitigate this
• But I still think some of the bottlenecks aren't addressed, e.g. I think nonpublic info is more often relevant for evaluating longtermist orgs than for evaluating animal welfare orgs
• Also, you rarely talked about room for more funding, which seems to imply you weren't focused primarily on informing donors?

(To be clear, "what did you see as the key purposes of this post?" is a sincere rather than rhetorical question, and I think this post is great.)

(I work for two of the orgs discussed in this post and as a grantmaker for a fund, but this comment - as usual - expresses personal views only.)

I actually think it would be cool to have more posts that explicitly discuss which organizations people should go work at (and what might make it a good personal fit for them).

Thanks Michael. Going through your options one by one.

1. Inform decisions about donations that are each in something like the $10-$5000 dollar range. Not an aim I had, but sure, why not.
2. Inform decisions about donations/grants that are each in something like the >$50,000 dollar range. So rather than inform those directly, inform the kind of research that you can either do or buy with money to inform that donation.$50,000 feels a little bit low for commissioning research to make a decision, though (could a $5k to$10k investment in a better version of this post make a $50k donation more than 10-20% better? Plausibly. • That said, I'd be curious if any largish donations are changed as a result of this post, and why, and in particular why they didn't defer to the LTF fund. 3. Inform decisions about which of these orgs (if any) to work for. Not really for myself, but I'd be happy for people to read this post as part of their decisions. Also, 80,000 hours exists. 4. Provide feedback to these orgs that causes them to improve. Sure, but not a primary aim. 5. Provide an accountability mechanism for these orgs that causes them to work harder or smarter so that they look better on such evaluations in future. No, not really. 6. Just see if this sort of evaluation can be done, learn more about how to do that, and share that meta-level info with the EA public. Yep. 7. [Something else]. Show the kind of thing that an organization like QURI can do! In particular, you can't do this kind of thing using software other than foretold (Metaculus is great, but the questions are too ambiguous; getting them approved takes time & in the case of a tournament, money, and for this post I only needed my own predictions (not that you can't run a tournament on foretold.)) 8. [Something else]. Learn more about the longtermist ecosystem myself 9. [Something else]. So this was sort of on the edges of this project, but for making large amounts of predictions, one does need a pipeline, and improving that pipeline has been on my mind (and on Ozzie Gooen's). For instance, creating the 27 predictions one by one would be kind of a pain, so instead I use a Google doc script which feeds them to foretold. I also think that 4. and 5. are too strongly worded. To the extent I'm providing feedback, I imagine it's more of a) of the sanity check variety or b) about how a relatively sane person perceives these organizations. For instance, if I don't get pushback about it in the comments, I'll think that its a good idea for the APPGFG to expand, but I doubt it's something that they themselves haven't thought about. +1, to both the questions and the answers. In an ideal world we'd have intense evaluations of all organizations that are specific to all possible uses, done in communications styles relevant to all people. Unfortunately this is an impossible amount of work, so we have to find some messy shortcuts that get much of the benefit at a decent cost. I'm not sure how to best focus longtermist organization evaluations to maximize gains for a diversity of types of decisions. Fortunately I think whenever one makes an evaluation for one specific thing (funding decisions), these wind up relevant for other things (career decisions, organization decisions). My primary interest at this point are evaluations of the following: • How much total impact is an organization having, positive or negative? • How can such impact be improved? • How efficient is the organization (in terms of money and talent) • How valuable is it to other groups or individuals to read / engage with the work of this organization? (Think Yelp or Amazon reviews) My guess is that such investigations will help answer a wide assortment of different questions. To echo what Nuño said, some of my interest in this specific task was in attempting a fairly general-purpose attempt. I think that increasingly substantial attempts is a pretty good bet, because a whole lot could either go wrong (this work upsets some group or includes falsities) or new ideas could be figured out (particularly by commenters, such as those on this post). In the longer term my preference isn't for QURI/Nuño to be doing the majority of public evaluations of longtermist orgs, but instead for others to do most of this work. Perhaps this could be something of a standard blog post type, and/or there could be 1-2 small organizations dedicated to it. I think it really should be done independently from other large orgs (to be less biased and more isolated), so it probably wouldn't make sense for this work to be done as part of a much bigger organization. Also, I'd agree that <$1Mil funding decisions aren't the main thing I'm interested in. I think that talent and larger allocations are much more exciting.

For example, perhaps it's realized that one small nonprofit's work is much more valuable than expected, so future donors wind up spending \$200Mil in related work down the line. Or, there are many systematic effects, like new founders are inspired by trends identified in the evaluations and make better new nonprofits because of it.

Hey! Thanks for doing this, strong-upvoted.

I just wrote a post about the terms "outside view" and "inside view" and I figured I'd apply my own advice and see where it leads me. I noticed you used the term here:

CSER

Epistemic status for this section: Unmitigated inside view.

and so I thought I'd try my hand at saying what I think you meant, but using less ambiguous terms. You probably didn't just mean "I'm not using reference classes in this section," because that's true of most sections I'd guess. You also probably didn't mean that you are using a gears-level model, though arguably you are using a model of some sort? Idk, could also classify it as intuition. My guess is that you meant "This is just how things seem to me," i.e. this section doesn't attempt to defer to anyone else or correct for any biases you might have.  How does all this sound? What would you say you meant?

So what I specifically meant was: It's interesting that the current leadership probably thinks that CSER is valuable (e.g., valuable enough to keep working at it, rather than directing their efforts somewhere else, and presumably valuable enough to absorb EA funding and talent). This presents a tricky updating problem, where I should probably average my own impressions from my shallow review with their (probably more informed) perspective. But in the review, I didn't do that, hence the "unmitigated inside view" label.

Hmmm, this surprises me a bit because doesn't it apply to pretty much all of your evaluations on this list? Presumably for each of them, the leadership of the org has somewhat different opinions than your independent impression, and your overall view should be an average of the two. I didn't get the impression that you were averaging your impression with those of other org's leadership.

Sure, but it was particularly salient to me in this case because the evaluation was so negative

Ah, OK, that makes sense.

I guess (p=.75) Nuño would say that the following interpretation is mostly reasonable: "inside view" here means that Nuño presents his impressions which rely a lot on stories he tells himself about various research directions being valuable or not, which others might reasonably disagree with him about.

I am thinking that because Nuño uses a simple model to estimate a fraction of researchers doing "valuable" work, the subjectivity is rooted in his takes on how valuable their individual research directions are.

[Phrasing this kinda weirdly as I want to get a visceral update on my belief in "when thinking is clearly described, I can guess that the author means by inside/outside view." I also think that (p=.33) Nuño was just not very careful and will say something like "I have no idea what I really meant at the time of writing it."]

To resolve that prediction, yes, I would say that that interpretation is correct.

Just one guy, but I have no idea how I would have gotten into AGI safety if not for LW ... I had a full-time job and young kids and not-obviously-related credentials. But I could just come out of nowhere in 2019 and start writing LW blog posts and comments, and I got lots of great feedback, and everyone was really nice. I'm full-time now, here's my writings, I guess you can decide whether they're any good :-P

Thanks for this, it's really interesting. I'm curious around how much time you spent on

1. Each org you evaluated
1. Maybe an average time you spent per org, and you can also mention any outliers. It feels like you spent more time on a couple of organizations compared to the others?
2. On this project as a whole so far

Charity Entrepreneurship usually does this for their reports, so people get a sense of how much time was put in and how deep the evaluation/effort was. Maybe you can include these time estimates into the post?

This is a good question, and in hindsight, something I should have recorded. For the project as a whole, maybe two weeks to a month, but not of full-time work. I don't remember the times for each organization.

I'm providing numerical context for RP's longtermism team here because it's both a) easier to evaluate the costs than research(er) quality when you have the data, and b) that the costs are by default more invisible when you don't have the data.

Rethink Priorities has recently been expanding into the longtermist sphere, and it did so by hiring Linch Zhang and Michael Aird, the latter part-time, as well as some volunteers/interns.

Just for some numerical context, I'm full-time at RP right now and Michael is half-time at RP and half-time at FHI RSP (so we have 1.5 FTEs). At the beginning of 2021, DaveRhysBenard was half-time and Michael was full-time (2.5 FTEs). That said, David was poached internally for some neartermist work (so not sure how to count that, FTE wise). I also spent ~1.5 months on neartermist work (not public). In total, this will be about 1.9 FTE-years by EOY (unless you count neartermist work as not part of the longtermist team), not including new hires.

We currently have 4 longtermist summer interns (all paid), 2 of which are ~half-time (3 FTEs total). They're here for approximately 3 months and are paid for by the EAIF + an additional donation which we earmarked for longtermist interns. Our internal theory of change for the internship focuses more on (a) helping the interns test and improve their fit for EA-style research and (b) helping their managers built management experience to facilitate further scaling, rather than on producing longtermist research outputs. 3 FTEs x 3 months ~= .75 FTE-years.

Finally, we have one volunteer, Charles Dillon (special circumstances). He's approximately half-time for us and started in May. We may have had other volunteers but if so I think they dropped off quickly without time investment on either our part or theirs, so low cost on both ends.* If we assume Charles will be with us until EOY (I sure hope so!), then we have .5 x 2/3 ~= .33 FTE-years in volunteers by EOY 2021. We do not expect to take on additional volunteers.

In total, by EOY 2021 on the RP longtermist team we'll have ~1.9 FTE years if you only count employees, ~2.65 if you also count paid interns, and ~3 FTE years if you count both interns and unpaid volunteers. This is not including additional hires, which we may want to make in late Fall 2021 (See Footnote 9).

*(If I'm missing someone, sincere apologies! Let me know and I'll add you)

For instance, requiring psychopathy tests for politicians, or psychological evaluation, seems very unrealistic.

Seems like you could do polling and start a ballot initiative where it looks promising, if anywhere. Starting small can get the momentum rolling and more attention to the issue, and then pick up support elsewhere.

Is there any particular reason you think it would be too unpopular or not work well? People might not like it in case it becomes a weapon used by the state to shut out political opponents, but maybe there are ways to prevent this, with bipartisan testers, or letting the subject choose at least one of the testers (who must have appropriate credentials). It could be like jury selection, with subjects allowed to challenge/strike potential testers (see strike for cause, peremptory challenge).

Also, we wouldn't need to require them to pass these tests; we could just publish the results so the public can be informed.

Maybe, in the US, it wouldn't be very effective other than in primaries, given how partisan things are.

Or do you think no useful tests could be made?

I'd welcome comments about the overall method, about whether I'm asking the right questions for any particular organization, or about whether my tentative answers to those questions are correct, and about whether this kind of evaluation seems valuable. For instance, it's possible that I would have done better by evaluating all organizations using the same rubric (e.g., leadership quality, ability to identify talent, working on important problems, operational capacity, etc.)

FWIW:

• I think I thought the questions you asked about each org seemed good
• I say "I think I thought" because I wasn't actively trying to find questions I thought weren't useful, come up with more relevant questions, etc.
• I think it seems reasonable to use different questions for each org
• It seems reasonable for the questions to be guided by what the org's theory of change is, what seem the major plausible upside scenarios for the org, what seems the major plausible downside risks for the org, what seem the major uncertainties about or potential weaknesses of the org, etc., and these things differ a lot between orgs
• It might be useful to also have some questions or rubric that is used across all orgs (I'm not sure), but I think it'd still be good to have questions for each org tailored to that specific org
• Or perhaps the common questions/rubric elements could be broad enough that the tailored questions all fit under one question/element
• Toy example: You have a broad question about each of importance, tractability, and neglectedness for each org, and then tailored sub-questions for each of those factors for each org

Superb work. Thanks for doing it. Would l love to see more of this kind of thing.

Would you consider reviewing the Center for Reducing Suffering? They are an organization similar to the Center on Long-Term Risk in the sense that their main focus is reducing S-risks, i.e. risks of astronomical suffering, but are less focused on AI. CRS is currently Brian Tomasik's top charity recommendation.

In what capacity are you asking? I'd be more likely to do so if you were asking as a team member, because the organization right now looks fairly small and I would almost be evaluating individuals.

Despite living under the FHI umbrella, each of these projects has a different pathway to impact, and thus they should most likely be evaluated separately. [...]

Consider in comparison 80,000 hours' annual review, which outlines what the different parts of the organization are doing, and why each project is probably valuable. I think having or creating such an annual review probably adds some clarity of thought when choosing strategic decisions (though one could also cargo-cult such a review solely in order to be more persuasive to donors), and it would also make shallow evaluations easier.

I think I agree with this, and it reminds me of my question post Do research organisations make theory of change diagrams? Should they? and some of the views expressed by commenters there (e.g., by Max Daniel). (Though really the relevant thing here is more like "explicit, clear theory of change", rather than it necessarily being in the form of a diagram.)

(Personal view only.)

There has been discussion about this for FHI, and I have spoken to a number of people there. They do have some specific ideas, but I agree that it would be beneficial for it to be 1) public, 2)  explicit, and 3) actually used for evaluation. Unfortunately, I think that doing so would require a lot of work on their part, and it hasn't been a big priority.