Proposal: Impact List -- like the Forbes List except for impact via donations

4

Also happy to give support on this if I can.

Conor Barnes 🔶

2

There's good discussion happening in the Discord if you want to hop in there!

4

Awesome! (Ideopunk and I are chatting on discord and likely having a call tomorrow.)

9

Twin challenges: constructing a metric that (1) we (EA) will not hate, that (2) will not confuse the public

Comparability across causes and across outcomes is very difficult

What credit do I get for things like those below, and how can we compare these in a way that is satisfying to us, and understandable to the larger public, including billionaires?

I donate $1 billion ...

FIRST-WORLD: To elderly hospices in the US and the UK
GIVEWELL: To GiveWell charities,
GIVEWELL*: one of which is later found to have been funding a program whose impact is thrown in doubt
GIVEWELL-ESQUE: To a non-Givewell charity working to prevent malaria. They use method similar to AMF but Givewell didn't have the resources to evaluate them, and they were 'too similar' to be worth a separate evaluation
OXFAM
ANIMALS: to successfully promote legislation to end prawn eyestalk ablation in Chile
LONGTERMIST: to fund AI safety research, generating research papers deemed very interesting
GOOD-FAILED-BET: To fund research into a possible cure for alzheimers disease, which looked good but turned out unsuccessful.

It would be very hard to resolve, even amongst ourselves, about GIVEWELL vs ANIMALS vs LONGTERMIST.

So, should we limit it to 'GH&D only'? But that would drag attention away from animals and LT causes that many/most EAs value above all else.

Perhaps a good first pass would simply be to sum "donations to all plausibly-high-impact charities"... maybe "all of the above except for FIRST-WORLD?" But then, we would probably want to discount OXFAM ... relative to the others but by how much? And how can we be claiming to measure the GIVEWELL, ANIMALS, and LONGTERMIST benefits by the same units? Unless the importance of 'value of prawns' sentient life/suffering' happens to add up to just the sweet spot where the expected good accomplished just equals a top Givewell charity, one will vastly outweigh the other.

The evidence synthesis base is thin

Even within global public health/development, we have basically a single source of a public evaluations that we trust (GiveWell), or at most a handful (public OP reports? Founders' pledge) These give rigorous assessments in accounting of a handful of interventions and charities, backed by strong academic evidence. I think we can be fairly confident that these interventions (like bednets, micronutrients) are in fact very likely to have strong positive effects.

But what do we do with charities such as Oxfam, Doctors without Borders etc, where, I guess, most of the GPHD giving goes. As far as I know there has been no credible comparison effectiveness rating for of these, because it's very difficult, because they do many things, and because their theories of change involves some things that are harder to measure. GiveWell does not say that 'AMF is 10.6 times more impactful than Oxfam'. They just don't report on this.

For reasons including 'the ability to make an impact list' I've been advocating that we do more to try to come up with credible reasoning transparent metrics of the effectiveness of charities

'going down the list' where the evidence is thinner
stating our assumptions clearly, including our uncertainty, and giving measures of its calibration

I was hopeful ImpactMatters would go in this direction, but I didn't see it. SoGive might still, if the funding is there. There are also some good initiatives coming out of QURI which would need alternate funding, and I think HLI is working in this area also.

3

I also mention this in my response to your other comment, but in case others didn't notice that: my current best guess for how we can reasonably compare across cause areas is to use something like WALYs. For animals my guess is we'll adjust WALYs with some measure of brain complexity.

In general the rankings will be super sensitive to assumptions. Through really high quality research we might be able to reduce disagreements a little, but no matter what there will still be lots of disagreements about assumptions.

I mentioned in the post that the default ranking might eventually become some blend of rankings from many EA orgs. Nathan has a good suggestion below about using surveys to do this blending. A key point is that you can factor out just the differences in assumptions between two rankings and survey people about which assumptions they find most credible.

I think you highlight something really important at the end of your post about the benefit of making these assumptions explicit.

technicalities

8

Really glad this is happening!

I sympathise with wanting to be less contentious and so ignore non-donation impact. But I expect most philanthropists to have most of their impact in their day job, and so the list really is missing a huge amount without it. And then the incentives we're offering aren't right.

It's also not impossible to do well. Keyword is "consumer surplus estimation".

Either way, good luck

5

Interesting idea but I don't think that's challenging to do well and I'm not sure it's merited:

Determining consumer surplus generated is very difficult

... Particularly as we are in a situation far from perfect competition in most industries (IMO). Did Bill Gates' Microsoft/Windows add consumer surplus, did it add more welfare than his charitable work? Hard to say. Obviously the DOJ was claiming he cost the world a tremendous amount of innovation and surplus.

Incentives: Non-donation impact may be accidental

Was Gates trying to make the world better by building Windows? Was it his main aim? I doubt it. But you might say 'why does it matter'? Maybe it matters because the impacts of people trying to make money can be good, bad, or neutral.

Consumer surplus/income and distribution

In principle all the income that is accrued the global wealthy and upper middle consumer surplus as a result of Bill Gates (etc.) could be passed on the neediest causes. But it wasn't/won't. It's hard to know how much diminishing returns to income to put into the welfare function.

\

technicalities

2

None of these seem fatal to me (but then I'm not the one proposing to do the heavy inference).

Yeah, Gates is an exception.

4

But if we look at the other big billionaires, like Bezos/Amazon, the same issues come up. How much value did amazon bring? Probably a lot compared to 'no central web commerce site'. But if it weren't for them, presumably something else comparable would have arisen.

Matt Putz

8

This seems really exciting!

I skimmed some sections so might have missed it in case you brought it up, but I think one thing that might be tricky about this project is the optics of where your own funding would be coming from. E.g. it might look bad if most (any?) of your funding was coming from OpenPhil and then Dustin Moskovitz and Cari Tuna were very highly ranked (which they probably should be!). In worlds where this project is successful and gathers some public attention, that kind of thing seems quite likely to come up.

So I think conditional on thinking this is a good idea at all, this may be an unusually good funding opportunity for smaller earning-to-givers. Unfortunately, the flip-side is that fundraising for this may be somewhat harder than for other EA projects.

Nathan Young

6

On the canonical list point.

I'd recommend getting the key ways in which different orgs weight the donations differently, then polling the public and getting the average moral weights and then showing that. Then give people the option to shift those around.

Eg sliders for AI risk, chicken lives to human lives. These two sliders would probably significantly affect the top spots.

4

It's a great idea but the devil is definitely in the details. You get at much of this, but maybe underestimate the challenge, especially for things like 'getting customization without overwhelming people and dissolving the impact'.

For the traditional global development stuff this could be somewhat tractable; the Cgdev/CDI took a stab at this for a country-level attempt at this.

... Although even there there are still loads of moral weights and 'moonshot project' issues to consider

But going further, including animals, x-risk/s-risk, lt-ism, cause prioritization itself, opens up huge worm cans. We may be to debate this stuff intelligently, but it might look like a huge black box to outsiders, or "that's like, just your opinion man".

A big challenge, but seems worth attempting to me. And the process of trying to do this itself, and engage wider audiences, seems valuable.

brb243

3

Is the CGD's Commitment to Development Index an expression of policy and spending quantifications within different sectors? I also wonder if 'Commitment' is the right way to call it, since some countries may be advantaged differently. For example, nations have limited control over how much, absolutely, they can spend on international peacekeeing, one of the Security's three subcomponents due to different income levels.

This index can but does not have to motivate countries to optimize for an 'ideal approach.' This can be valuable when countries make better choices (e. g. realize that they get a better score if they divest from arms trade and invest into fishing alternatives) but disvaluable when the intent of the index is different from the effect of the calculation (e. g. if it would be favorable to take large sums from subsidized organic agriculture to arms trade within the context of 'peacekeeping').

A similar consideration should be applied in this list. There should be no way to 'trick' the index. A way to address this is enable adjustments alongside/based on feedback. This, in addition to the possibly reduced perception absoluteness can enable partiality based on convenience or strategical attraction of donors (for example, utility monsters can be weighted down if it would make someone extremely better than the others or up, if there is a large monsters donor who could be attracted to EA by their listing). But, this could be addressed by actually inviting the top people converse about possible improvements in the metrics, implementing impartiality considerations as they are applying courtesy to each other, maybe. Just some thoughts.

2

Yeah, I've lately been considering just three options for moral weights: 'humans only', 'including animals', and 'longtermist', with the first two being implicitly neartermist.

It seems like we don't need 'longtermist with humans only' and 'longtermist including animals' because if things go well the bulk of the beings that exist in the long run will be morally relevant (if they weren't we would have replaced them with more morally relevant beings).

3

but even within 'humans only' (say, weighted by 'probability of existing' ... or only those sure to exist).

There are still difficult moral parameters, such as:

suffering vs happiness
death of a being with a strong identity vs suffering
death of babies vs children vs adults

(Similar questions 'within animals' too).

2

Agreed. I guess my intuition is that using WALYs for humans+animals (scaled for brain complexity), humans only, and longtermist beings will be a decent enough approximation for maybe 80% of EAs and over 90% of the general public. Not that it's the ideal metric for these people, but good enough that they'd treat the results as pretty important if they knew the calculations were done well.

2

Do you mean all three separately (humans, animals, potential people) or trying to combine them in the same rating?

My impression was that separate could work but combining them ‘one of the three will overwhelm the others’.

Linch

2

If you do a linear weighting, this is expected. But one approach to worldview diversification is that you can normalize.

2

Linch, that sounds like a reasonable approach. I think something like that could work.

Ultimately, I guess a high value per dollarwould then be assigned to anyone who donated ~'maximally our best guess impactfully' in either of the three categories,

... and then the value would scale ~linearly with the amount donated 'max-impactfully' to either of the three categories.

Might be somewhat difficult to explain this to a smart popular audience, but I suspect it might be doable.

2

My suspicion is that there will only be a very narrow and “lucky” range of moral and belief parameters where the three cause areas will have cost effectivenesses in the same orders of magnitude.

But I should dig into this.

WilliamKiely🔸

3

Misc thoughts:

Doing credible cost-effectiveness estimates of all the world's top (by $ amount) philanthropists (who may plausibly make the list) seems very time-intensive.

Supposing the list became popular, I imagine people would commonly ask "Why is so-and-so not on the list?" and there'd be a need for a list of the most-asked-about-people-who-are-unexpectedly-not-on-the-list with justifications for why they are not on the list. After a few minutes of thinking about it, I'm still not sure how to avoid this. Figuring out how to celebrate top philanthropists (by impact) without claiming to be exhaustive and having people disagree with the rankings seems hard.

3

Yeah it will be very time intensive.

When we evaluate people who don't make the list, we can maintain pages for them on the site showing what we do know about their donations, so that a search would surface their page even if they're not on the list. Such a page would essentially explain why they're not on the list by showing the donations we know about and which recipients we've evaluated vs. those who we've assigned default effectiveness values for their category.

I think we can possibly offload some of the research work on people who think we're wrong about who is on the list, by being very willing to update our data if anyone sends us credible evidence about any donation that we missed, or persuasive evidence about the effectiveness of any org. The existence of donations seems way easier to verify than to discover. Maybe the potential list-members themselves would send us a lot of this data from alt accounts.

I think Impact List does want to present itself as a best-effort attempt at being comprehensive. We'll acknowledge that of course we've missed things, but that it's a hard problem and no one has come close to doing it better. Combined with our receptivity to submitted data, my guess is that most people would be OK with that (conditional on them being OK with how we rank people who are on the list).

WilliamKiely🔸

3

A very simple expected value calculation

Your estimate seems optimistic to me because:

(a) It seems likely that even in a wildly successful case of EA going more mainstream Impact List could only take a fraction of the credit for that. E.g. If 10 years from now the total amount of money committed to EA (in 2022 dollars) increased from its current ~$40B to ~$400B, I'd probably only assign about 10% or so of the credit for that growth to a $1M/year (2022 dollars) Impact List project, even in the case where it seemed like Impact List played a large role. So that's maybe $36B or so of donations the $10M investment in Impact List can take credit for.

(b) When we're talking hundreds of billions of dollars, there's significant diminishing marginal value of the money being committed to EA. So turn the $36B into $10B or something (not sure the appropriate discount). Then we're talking a 0.1%-1% chance of that. So that's $10M-$100M of value.

If a good team can be assembled, it does seem worth funding to me, but it doesn't seem as clear-cut as your estimate suggests.

3

Thanks for the feedback!

Regarding (a), it doesn't seem clear to me that conditional on Impact List being wildly successful (which I'm interpreting as roughly the $110B over ten years case), we shouldn't expect it to account for more than 10% of overall EA outreach impact. Conditional on Impact List accounting for $110B, I don't think I'd feel surprised to learn that EA controls only $400B (or even $200B) instead of ~$1T. Can you say more about why that would be surprising?

(I do think there's a ~5% chance that EA controls or has deployed $1T within ten years.)

I think (b) is a legit argument in general, although I have a lot of uncertainty about what the appropriate discount should be. This is also highlighting that using dollars for impact can be unclear, and that my EV calculation bucketed money as either 'ineffective' or 'effective' without spelling out the implications.

A few implications of that:

There's a 'free parameter' in the EV calculation that isn't obvious: the threshold we use to separate effective from ineffective donations. We might pick something like 'effective = anything roughly as or more effective than current GiveWell top charities'.
That threshold influences whether our probability estimates are reasonable. For instance depending on this threshold someone can object "A 1 in 1000 chance for $110B to be moved to things as effective as GiveDirectly seems reasonable, but 1 in 1000 for $110B to be moved to things as effective as AMF? No way!"
As noted in footnote 4, we assume that the donations in the 'ineffective' bucket are so much less effective than the donations in the 'effective' bucket that we can ignore them. Alternatively, we can assume that enough of the 'effective' donations are far enough above the minimum effectiveness threshold that they at least cancel out all the ineffective donations.
The threshold we pick also determines what it means when we talk about expected value. If we say the expected value of Impact List is $X it means roughly $X being put into things at at least the level of effectiveness of our threshold. We could be underestimating if Impact List causes people to donate a lot to ultra-effective orgs (and it might, if people try hard to optimize their rankings), but I didn't try to model that.

Given the bucketing and that "$X of value" doesn't mean "$X put into the most effective cause area", I think it may be reasonable to not have a discount. Not having a discount assumes that we'll find enough (or scalable enough) cause areas over the next ten years at least as effective as whatever threshold value we pick that they can soak up an extra ~110B. Although this is probably a lot more plausible to those who prioritize x-risk than to those who think global health will be the top cause area over that period.

WilliamKiely🔸

3

Considerations in the opposite direction:

Value of information of initial investments in the project. If it's not looking good after a year, the project can be abandoned when <<$10M has been spent.
80/20 rule: It could influence one person to become the new top EA funder, and this could represent a majority of the money moved to high-cost-effectiveness philanthropy.
It could positively influence the trajectory of EA giving, such that capping the influence at 10 years doesn't capture a lot of the value. E.g. Some person who is a child now becomes the next SBF in another 10-20 years, in part due to the impact the list has on the culture of giving.

brb243

3

Some questions; you do not have to reply to most.

1) Have you considered that in addition to one's absolute impact, their relative impact (e. g. to their income or capacity) would be shown? These lists can be inspirational and insightful, for example, could show that many extremely poor people compete with Bill Gates while other extremely poor people do not. This can motivate cooperation in inclusive global development which can make everyone involved in this area feel great and others wanting to join.

2) Since this is lifetime, younger people would be disadvantaged. Would you apply Bayesian updating?

3) Why are these lists of large philanthropes not so popular? Are there some regional/cause area/industry lists that are better known? Can they be aggregated? What are their current effects and how could EA make them better in doing good, such as motivating people to donate or develop market solutions that are better sustainable in the long term?

4) Are you considering the counterfactual impact of the people's alternative spending?

5) How are you going to make sure you do not forget anyone but do not discourage people who are motivated by not being recognized for their charity? (If you start excluding them, maybe they will not stop donating but get recognized?) Have you considered effects on networks, e. g. where some donations are a norm but no one individual donates a large sum? Is there a way to highlight them in a way that is not biased by the number of people?

6) Why would you include net worth? It seems inconsistent with the other columns, which celebrate donating rather than gaining status within the socioeconomic framework. Status is in impact, right.

7) I would also suggest deleting the amount donated so far/in a given year, since there should be intrinsic motivation of the donor/trust that they keep up with their pledge or when they do not there are valid reasons. If this is public, it seems like they have to do it, which can be demotivating. Thus, I would just keep an updated estimate of a pledge, updated by external experts, possibly after consultations with the donor. If the experts are sufficiently cool and serious, everyone will be excited to increase their expected impact value, even asking how they can do it even better?

8) I would either enable granular filtering, almost by output or intermediate outcome or no filtering, all converted to some sentience-adjusted wellbeing-adjusted life year. Because, if you want to compare who makes the most impact in animal welfare, for example, you have to measure which charity frees the most chicken from cages, or supports the implementation of policies or alternatives with the same effect regarding confined farm animals. If you just filter by cause area, then $1b to cute puppy mill awareness is the same as $1b to effective dynamic policy advocacy and development considering animals' experiences.

The one metric I am talking about is measuring 'active neural complexity' (Being You: A New Science of Consciousness, Chapter 2) and multiplying this by weighted adjustment in wellbeing. I am suggesting this first eliminating suffering and weighting suffering by an exponent of 2.5. Of course, externalities should be considered, so always one's contributions to changes of total weighted wellbeing in the universe should be estimated. This should be a prediction into infinity. This can seem like a challenge but if institutional inertia is considered and effects on the very long-term future of destructive actions known well and of constructive actions known little, it can be possible. Here, not only donations but also decisions should be considered.

9) Are you considering the counterfactual of what wealth and impact capacity people could have developed, within what they perceive as their free will?

10) Would you want to wait with heavily publicizing this list until it is possible to make impact cool in public, competing with other media competing for attention by other means (shaming, fear, impulsive appeal, ...)? Or, are you planning to draft it in a way that supports the occurrence of this environment?

11) Yes, I was talking to GWWC a while ago that they should enable people showcase their donations and sometimes I keep mentioning it. Is this already occurring? Maybe, it is not consistent with the spirit of the community to brag about donations. But, a pilot list could just include a few volunteers disclosing their donations (especially more speculative ones) and arguing for their impact. Others could comment, in lieu of organizations submitting other calculations of impact.

This could provide valuable feedback on even more impactful donations consideration as well as support 'donation specializations' within EA. Then, additional donors could use a donations ITN to research their philanthropic investment best options. Also, feedback prior to donating could be valuable.

12) Do you know of the Charity Navigator's Impact Unit (previously ImpactMatters) list of top charities within causes? Maybe the organization will not like it, but perhaps they could list top effective donors by multiplying the amount they donated by the unit effectiveness they list (for some they list cost per unit output, such as tonne of CO2 sequestered).

13) Trying to invite 'external' people by highlighting them on 'our' list can be ineffective because either they are already effective and so probably using at least some of EA-related resources or funding programs, so there is no need to maybe invite them to list their profile somewhere or there would be people who would like to make the list strategic so that it includes some people who would pay the most attention if included, some if excluded, also depending on the changing relationships among billionaires, so would either have to be well planned in advance considering possible scenarios of credible changing of metrics (but if billionaires just chat about improvements of the impact index under the standards of impartiality, it can work), a bit (tacitly) transparent about this objective of advancing a discourse, or not done.

14) But the list could also discourage some people, imagine that you were like number 3 or 4 in your local group and you realize you are number 135,000 - maybe you forget malaria or your local group before you start realizing that systemic change cannot happen if you just throw nets and money at the problem? So, I would maybe, in addition to considering impact relative to one's capacity and counterfactuals and including non-financial impact conduct a network analysis of decisionmaking. This can further make people comfortable and wanting to join EA. Anyone can be competitive, they just have to have high positive impact.

15) Will you consider the counterfactual of if one is preventing others to do what they are doing better?

16) Would the entire list be more interesting if it is something like a 4D vector space? It can still be possible to comprehend, but slightly challenging and all these colored images just attract one's attention. I would also make it VR compatible because it can be more interesting to explore this space of philanthropy with this headset.

17) Before you spend much resources, what is the metric? Is it QALY? WALY? Some other metric we have yet to come up with or discover, such as the prevalent spirit of virtuous progress? Are there any conditions (progress yes but no suffering or wellbeing yes and no suffering but no specific drugs, etc).

18) Would you be interested in spending an amount on researching the complex interactions of the millions of charities in the world and their donors to see what support increases efficiencies the most, considering donor interactions, the impact of counterfactual spending, and the charity's existing capital that makes the marginal cost of various programs complementary in systemic change different?

19) How do the billionaires get into EA? I trust that less and less will be interested in big numbers and more and more in sincere impact. Then, an impact list should primarily facilitate cooperation in making impact but also, of course, highlight some billionaires by its structure.

20) The rankings have to be publicly acceptable. It is a check for narratives use in EA, also, e. g. to attract donors. For example, if the public would be skeptical about AI safety, then sound arguments, that relate to the effects of AI safety, should be understandable.

21) How would you engage representatives of public groups and networks who have to buy into this list in creating it without introducing partiality biases? This can prevent any perceptions of arrogance by actual learning.

Would the participation of the public discourage large donors who will then perceive less exclusivity/something special for them developed by special people? Either, the representatives of the public have to be special, e. g. by their ability to think about impact, or this list should not be public public, more like EA-related networks public. But then, this could lower epistemics in EA, or improve them, depending on the calculation.

22) How are you going to include impact due to market-based innovations, coordination, efficiencies, and policy negotiations?

23) Will you differentiate situations when real income increases and when it does not (redistribution takes place)?

To answer your questions:

A lot of the community's funds, really at least $100m per year in the first year should go toward creating the environment that would make this idea seem plausible, aiming for sincerity, collaboration, and impartial and thoughtful definition of metrics/calculations. This is because if you hire like 5 people for $100k, pay maybe for articles in newspapers and popularize this list and it will be like who gives the most bednets then you end up with a world literally polluted by bednets (or with AI safety issues) and everyone thinking it is quite shameful to be thinking about stuff like systemic change or a cooperation toward it.

I would sum the contributions toward a better impact trajectory of EA of all the sub-parts of this project and update this sum as sub-parts occur and alternatives in EA that can achieve the same objective in a different way appear. It is an updated difference of two integrals. I would use my weighted WALY but I am biased.

In short, you should include externalities in an MVP. Also, consider making an actual spreadsheet.

Test it voluntarily with EAs. Do not publish it by Vox. This would go deeper the hole of 'more bednets and OpenAI' no thinking, because as it seems presented, people could experience negative emotions, such as fear, powerlessness/threat of submission, or anger, when seeing the list, which reduces critical thinking abilities, even within the environment they would be due to that list.

Vasco Grilo🔸

2

SoGive has rated about 100 top UK charities with the goal of increasing the amount of effective donations via popularising impact ratings (note this is not the only SoGive project). I can see some similarities with what you are describing:

As the impact of the donations of individuals is very much connected to the cost-effectiveness of the organisations to which they donate, I guess producing Impact List might also involve assessing lots of organisations.
Both theories of change involve influencing the donations of high-net-worth individuals.
Both projects seem quite time-intensive.
Ideally, Impact List aims to compare the impact of donations going to organisations in different cause areas, in the same way that SoGive uses the same rating scale for all cause areas.

I encourage you to talk with Sanjay Joshi. I am also volunteering at SoGive, so we can also chat!

Kaleem

3y

1

Eliot, is anyone working on this yet?