Lifetime Impact of a GiveWell Researcher?

by agent1824 min read16th Aug 20213 comments

11

GiveWellImpact assessmentCareer choice
Frontpage

Please leave your kind feedback on the essay (on writing, logic, relevance etc.) in the comments section or through private message. Any feedback is much appreciated.

Entry question

What is could be the lifetime estimated impact (in $ donated) of being a EA GiveWell researcher?

TL;DR

This is a fermi estimate and shall be treated like that. Everything is an estimate, there are many uncertainties and geometric means have been used to bring the limits down to one number.

Lifetime Impact of a GiveWell Researcher: $8m

Epistemic Status: Off by a factor 10 (#fermi-estimates)

Introduction

I am curious to know what sort of impact a EA GiveWell Researcher has. I choose GiveWell, as there is a very "clear" dollar figure on how much money they "influenced" (at least according to them). With this essay, the lifetime impact of an entry level researcher starting in 2021, and working for another 35 years, is estimated. The following components help put together the final figure:

Organization Level

  1. Impact of GiveWell in 2021
  2. Growth rate of GiveWell's impact for the next 35 years
  3. Counterfactual impact of GiveWell

Researcher Level

  1. Impact attributed to an entry level researcher in the first year (2021)
  2. Growth rate of researcher's impact
  3. Counterfactual impact of a Researcher

The rest of the essay deals with the above components and the final lifetime impact calculation.

Note1: Fermi estimates are in general factor 2 or even factor 10 off. In many cases this might just be enough.

Note2: This estimate does not consider several factors such as probability of EA loosing its momentum, probability of getting fired and not being able to contribute as a researcher etc.


Impact and Growth Rate of GiveWell in 2021

GiveWell estimates how much money they move every year and publishes them on their website. For example, in the year 2019 GiveWell estimated that they moved $172m. In order to estimate the impact of GiveWell in 2021, a bit of extrapolation of data is needed. For this extrapolation, the growth rate is used.

To predict the growth rate in the coming years I believe it would be reasonable to base it on the money moved from 2015 to 2019 instead of 2007 to 2019. This is because it appears that in the recent years the growth has sort of "plateaued" (see table below) and this might be more representative of the future.

Thus, the compound annual growth rate for the money moved between 2015 and 2019 comes out to be 9.4% ((172/120)^(1/4)-1). It's really hard to predict the change in growth rate for the coming years. For simplicity's sake, a constant growth rate of 9.4% for the next 35 years, is assumed.

Year Money Moved ($) [1] Growth
2010 1.5m -
2011 5m 233%
2012 10m 100%
2013 16m 60%
2014 28m 75%
2015 120m 329%
2016 118m -2%
2017 149m 26%
2018 161m 8%
2019 172m 7%

Impact of GiveWell in 2021

As for the impact of GiveWell in 2021 it is estimated to be $205m (172*(1.094)^2), based on a growth rate of 9.4% and the money moved in 2019.

Counterfactual Impact of GiveWell

The counterfactual impact of GiveWell is the difference in impact between the "Actual Scenario" (world with GiveWell since 2007) and the "Counterfactual Scenario" (world without GiveWell since 2007).

In the counterfactual scenario, it is expected that a GiveWell-like organization would have formed anyway, 3-10 years later (than when GiveWell was founded)[2]. It appears likely because, many "similar" organizations (such as Charity Science Health, Rethink Priorities, Open Philanthropy, The Life You Can Save etc.) have formed since then (between 2010 and 2020), involving cost-effectiveness estimates and philosophies such as "doing the most-good for a dollar".

While comparing the actual and counterfactual scenario, we compare the two scenarios with the same amount of money but only varying cost-effectiveness depending on the scenario. As a result it is expected that when money doesn't go to GiveWell-like organizations, it goes to some random organization which is guessed to be within 1 to 1/50th times as cost-effective as GiveWell.

The counterfactual impact for the year 2021 is calculated as follows: As part of the actual scenario, GiveWell is estimated to move $206m with cost-effectiveness of 1. As part of the counterfactual scenario GiveWell-like organization is expected to move $118m [3] with cost-effectiveness of 1, in that year. The remainder $88m (206m - 118m$) is used by "other" organizations at one-seventh (geometric mean of 1 and 1/50) the cost-effectiveness of that of GiveWell. The difference between the actual and the counterfactual situation comes to be $75m. And finally, the counterfactual impact as a function of the money moved in the actual scenario is 36%.

The below table summarizes the counterfactual impact for the different years. For years beyond 2021 the counterfactual impact essentially starts to become a constant value of 31%. This 31% is used in the final calculation as a constant.

Year AS (m$)[1:1] CS GW-like (m$) CS non-GW-like (m$) AS-CS CS as % of AS
... ... ... ... ... ...
2015 120 1.5 118.5 101.6 84.64%
2016 118 5 113.0 96.9 82.08%
2017 149 10 139.0 119.1 79.96%
2018 161 16 145.0 124.3 77.20%
2019 172 28 144.0 123.4 71.76%
2020 188 120 68.2 58.5 31.06%
2021 206 118 87.9 75.4 36.6%
... ... ... ... ... ...
2029 423 270 153.3 131.4 31.06%
2030 463 295 167.7 143.8 31.06%
... ... ... ... ... ...
2059 6,295 4,014 2,281.2 1,955.3 31.06%
2060 6,888 4,392 2,496.0 2,139.4 31.06%
2061 7,537 4,806 2,731.1 2,340.9 31.06%
... ... ... ... ... ...

AS --> Actual scenario with cost-effectiveness 1
CS GW-like --> Counterfactual scenario with cost-effectiveness 1
CS non-GW-like --> Counterfactual scenario with cost-effectiveness 1/7

Note: Cost of operation of GiveWell is not taken into account as it is a small sum. For example, it is estimated that the cost of operation for 45 people ($110k per person) shall be $5m. Contrast that to it's 206m expected to be moved in 2021 (a mere 3%). So this is skipped confidently.

Note: Also the effects of the displacement chain are not considered for simplicity's sake.

Impact Attributed to an Entry Level Researcher in the First Year

In the year 2015, GiveWell estimates that an entry level researcher had 10% of the impact of a co-founder and that there were 9 co-founder equivalents [4]. In other words an entry level researcher was estimated to be responsible for 1.1% (10%/9) of the total impact (in 2015). But this is not the whole story.

With more people joining GiveWell, this percentage gets diluted. In 2015 there were 25 "equivalent staff members", where as in 2021 there are 45 "equivalent staff members" [5]. The resulting contribution would thus be much less. Let's assume this contribution is inversely proportional to the total number of staff. Then, the impact attributed to an entry level researcher in the first year (2021) is 0.6% (1.1%x25/45).

Growth Rate of a Researcher

The growth rate of a researcher is split into 2 factors. The first being the growth rate associated with the 'increase in experience', which leads to higher researcher's contribution. And the second being the growth rate associated with 'increase in number of staff' of GiveWell, which decreases the researcher's contribution as years pass.

For the "growth rate associated with increase in experience", the year-variable is held constant (and hence the number of staff is constant) and the difference between entry level researcher and co-founder is observed.

In 2015, a co-founder is estimated to be responsible for 11%[4:1] of the impact that year and an entry level researcher at 1.1% (as mentioned above). If it takes an entry level researcher 10-20 years (geometric mean: 14 years) to reach co-founder level, then growth rate per year due to increase in experience (alone) is a whopping 19% ((11%/1.1%)^(1/n)-1). (It is assumed further that the growth rate falls sharply to 0% after reaching co-founder level.)

Item Value
Entry level researcher 1.1%
Co-founder equivalent 11%
Number of years 14
Growth rate due to increase in experience 19%

For the "growth rate attributed to increase in staff", the average increase in the number of people per year in GiveWell is estimated. This increase is assumed to be negatively proportional to growth rate attributed to increase in staff.

From 2015 to 2021 the number of staff has increased from 25 to 45 [5:1]. This implies an increase per year of 10% per year ((45/25)^(1/6)-1). This implies a growth rate attributed to increase in staff of -10%.

Counterfactual Impact of a Researcher

The researcher positions seem very competitive now (with ~1% acceptance rate). If the entry level researcher manages to be 5-15% (geometric mean: 9%) better than the next researcher in line (perhaps by Deliberately Practicing), then that would be the factor multiplied to the impact attributed to a researcher. This would then produce the counterfactual impact of the researcher.

Note: I tried considering a "complex model" taking into account displacement chains (explained in the appendix), but the improvement was almost nothing (few $100k) compared to the final value, so it has been moved to the appendix. Adding it here only made the explanation unnecessarily complicated.

Bringing it all together for 2021 to 2056

The different factors computed so far are listed below:

Organization level

  1. Impact of GiveWell in 2021 = $205m

  2. Constant growth rate of GiveWell = 9.4%

  3. Counterfactual impact = 31%

Researcher Level

  1. Impact attributed to an entry level researcher in the first year (2021) = 0.6%

  2. Growth Rate due to increase in experience = 19% between 2021 and 2035, and 0 otherwise.

  3. Growth Rate due to increase in staff members = -10%

  4. Counterfactual contribution of Researcher = 9%

Taking them into account brings the counterfactual impact of a researcher to $8m (could be off by a factor of 10). The following table shows part of the Google Sheets document used to compute the final outcome.

Year Impact of GW (m$) Counterfactual of GW (%) Impact attributed to a researcher (%) Counterfactual attributed to a researcher (%) Total (m$)
2021 206 31% 0.60% 9% 0.03
2022 225 31% 0.64% 9% 0.04
2023 246 31% 0.69% 9% 0.05
2024 270 31% 0.74% 9% 0.06
2025 295 31% 0.79% 9% 0.07
... ... ... ... ... ...
2033 605 31% 1.37% 9% 0.24
2034 662 31% 1.46% 9% 0.28
2035 724 31% 1.57% 9% 0.32
2036 792 31% 1.41% 9% 0.32
... ... ... ... ... ...
2055 4,367 31% 0.19% 9% 0.25
2056 4,777 31% 0.17% 9% 0.25

Note: Milan Griffes estimates here that as an entry level researcher he expects to make an impact of $90k (back in 2016). This is one-third the estimate for 2021 as per the above calculations.

Why I can only be a factor 10 confident

First of all these are all forecasts, there are a ton of assumptions, such as the constant growth rates, the entire counterfactual scenario being "made up", counterfactual impact of a researcher being 9%, etc. I don't think a more complex model is the solution. Even a really complex model of the growth rate will have it's own un-testable assumptions by virtue of it being a forecast.

It is very hard to verify this on any level. For example, GiveWell estimates that in 2015 an entry level researcher contributes 10% of a co-founder. I don't know how accurate this is and how they are testing this assumption over time.

Many numbers are crazy sensitive. Just consider that the counterfactual contribution of a researcher changes to 1% instead of the 9%, then the final estimate falls from $8m to $800k. And we will "never really know" what the "true" value is. The estimation gets even harder with organizations like Rethink Priorities for example, where they produce research and are not really "moving money".


Appendix: Counterfactual Impact of a Researcher

The counterfactual impact of a researcher is the difference in impact between the "Actual Scenario" where there is an extra researcher and the "Counterfactual Scenario" which is without that extra researcher.

A model table is shown below attempting to estimate the difference in impact between the 2 scenarios. In the actual scenario we have researchers $R_{1}$ to $R_{n}$ and people who Earn-To-Give: $R_{n+1}$ followed by $E_{2}$ to $E_{n}$. In the counterfactual scenario we are missing $R_{1}$ and the whole chain displaces upwards accordingly. EA is the amount of impact associated with being a researcher and ETGX is the Earning-To-Give impact of different people. And the final counterfactual impact is given by $R_{n}-R_{n+1}$ x EA + $R_{n+1}$x ETG1.

Actual Counterfactual Impact Difference
$R_{1}$ $R_{2}$ 1x EA $R_{1}-R_{2}$ x EA
$R_{2}$ $R_{3}$ 1x EA $R_{2}-R_{3}$ x EA
... ... ... ...
$R_{n-1}$ $R_{n}$ 1x EA $R_{n-1}-R_{n}$x EA
$R_{n}$ $R_{n+1}$ 1x EA $R_{n}-R_{n+1}$x EA
$R_{n+1}$ 0 ETG1 $R_{n+1} - 0$ x ETG1
$E_2$ $E_2$ ETG2 0
... ... ... 0
$E_n$ $E_n$ ETGN 0
- - Total $R_{1}-R_{n+1}$x EA + $R_{n+1}$x ETG1

There are two components to the counterfactual impact of a researcher. The difference between the extra ($R_{1}$) and the displaced researcher is the first one: $R_{n+1}$) ($R_{n}-R_{n+1} x EA$. The researcher positions seem very competitive now (with ~1% acceptance rate). If the extra researcher ($R_{1}$) manages to be 5-15% (geometric mean: 9%) better than the displaced researcher, then the impact will be 0.09 x EA.

In addition, the displaced researcher's Earning-To-Give ($R_{n+1}$ x ETG1) will also contribute to the counterfactual impact. Assuming the person is European and is willing to donate 10% of his salary, this would roughly mean an Earning-To-Give base of $4000 with a growth rate of 5%.


Footnotes


  1. 2014
    2015
    2016
    2017
    2018
    2019 ↩︎ ↩︎

  2. (80000hours also does something similar to estimate their counterfactual impact by assuming they changed someone's career path by a few years). ↩︎

  3. The impact of GiveWell-like organization is simulated by shifting GiveWell's impact by 5 years (geometric mean of 3 and 10 years). So in 2021 GiveWell-like organizations' impact is basically GiveWell's 2016 impact. ↩︎

  4. "(Co-founder time was given a score of 1 but time spent by newer staff was scaled down significantly; for example, we multiplied time spent by entry-level staff in their first year at GiveWell by 0.1.)"--- GiveWell

    "The table shows that our overall capacity has increased significantly since 2012 (from approximately 3 co-founder equivalents then to 9 today)."--- GiveWell

    1/9 = 11% ↩︎ ↩︎

  5. Well in 2015 there were 21 people and 10 board members (not sure how much work they do). Let's round it to 25 people.

    In 2021 GiveWell has 41 staff and 6 board members. We round it up to 45 people.---GiveWell ↩︎ ↩︎

11

3 comments, sorted by Highlighting new comments since Today at 1:55 PM
New Comment

Thanks! Glad you did this analysis. You might also be interested in the numbers here, where surveyed EA leaders said they would be willing to sacrifice $250k in donations to keep their most recent junior hire ($1m for senior).

That's not the question you're asking exactly, but it's another interesting angle.

Thanks a lot for your response. I think 80000hours has actually "sort of withdrawn their conclusions" from that post about extra donations and recent hires.

It wouldn’t be surprising if the respondents were simply wrong. Our impression is that most of the answers were given with just a couple of minutes of reflection, and so mainly reflect a gut intuition. There’s not much reason to expect these intuitions to be accurate on average in this kind of domain.---80000hours

Thus I am not sure we should pursue these numbers anymore, from the survey. Your thoughts?

I sort of interpret that post as typical EA scrupulosity. They write:

Overall, the more suspect the estimates, the less you should update on the results and the more weight you should put on your prior.

But I didn't really have a strong prior to begin with. Maybe the hire's salary, but that's really just the lower bound.