TLDR: The shortest version of this argument is very simple: your expectations for an organization should be higher where their budget and staff size are higher. In other words, we should have different expectations for a 20-person organization with a $1.5 million budget than a 2-person $150,000 budget organization.
While this seems pretty clear in the abstract, I find that people tend not to update nearly enough on this when they should. For example, I often see people comparing the total research output of two organizations, yet when I ask about it, they will not know the yearly budget or staff size of either. This is a big problem. As a movement, we want to support efficient and effective organizations, not just organizations that are the biggest, most salient or currently the highest funded.
Budgets and staff
When considering how impressive an organization's output is, one useful framing is the following:
1) What has this organization accomplished over the year (perhaps pulling this information from an annual report)
2) How much budget and how many staff would I expect the organization to need to do what they are presently doing.”
It is easy to then look up an organization’s budget or staff size and adjust your conclusions accordingly. If the budget and staff size are much lower than expected, you end up being impressed with the organization. On the flip side, if the organization and its budget are much higher than expected, it makes sense to be less impressed with the organization. Budgets are often public or findable, as are the organization's activities over the last year, so it is quite possible to compare them.
To pick on my own organization as an example, you could think about the actions and activities of Charity Entrepreneurship/AIM over the last year. Most significantly, we’ve launched ~13 charities, built out our 2 other new programs (grantmaking and research training), and laid the foundations for a fourth program (our Founding to Give incubator launching early next year).
Intuitively, what might you expect a reasonable budget and/or staff size to be in order to accomplish this work? You do not need to have an amazing sense of organizational budgets to do at least a rough estimate. I am pretty sure that if AIM/CE achieved these outcomes with an annual budget of $100k, everyone would (rightfully) think that is an amazing deal, and perhaps one that is too good to be true. On the other side, if it cost us $20 million, I expect people would generally (again rightfully) be a lot less excited.
Comparative size
Another way to get a sense of this is to look at comparative size. If you have a bunch of donation options you are considering, you can map out their comparative size and think about how they directly trade-off.
For example, let’s take a hypothetical funding proposal I receive for something costing $450k. I know the average seed grant for the CE incubated charities is about ~$150k. This means that a useful way of assessing the value of the funding proposal is to think of this as the equivalent of three seed grants. Thinking about the proposal in this way is crucial: it’s easy to compare two funding applications 1:1, or at least not adequately scale expectations with the scale of difference in funding ask. If I was not careful, I think my default would be to compare that $450k application 1:1 vs one of the seed grants when this is not the accurate point of comparison at all. This is true even if I am only covering a portion of the funding, as the net funding is still typically pulled from sources that fund other promising projects. This difference in expectation can be even more dramatic with organizations that have budgets that are different by orders of magnitude.
In theory, organizations with ~2x the budget should produce about ~2x the output. There are, of course, reasons in practice to believe that output does not scale quite so linearly with increases in resources. However, these reasons crucially fall on both sides of our possible expectations, suggesting a linear increase in outputs is a reasonable base expectation. Perhaps cofounder staff tend to be stronger than later hires, thus resulting in less output per budget with bigger organizations. On the flip side, principles of specialisation and economies of scale from the for-profit world suggest we might expect growth in outputs to outpace increases in budget size.
Why this matters
Personally, I tend to find that smaller organizations are effectively penalized by people not really having a sense of what the costs of a given organization are in comparison to each other, or not adequately factoring in this information when comparing outputs.
Grantmakers and donors being more sensitive to organizational size could lead to a more effective marketplace for donations. I’d expect people's exact judgments on how growth output and resources correlate will vary a lot depending on breadth vs depth intuitions and several other such differences. Regardless, having more direct comparisons and explicit consideration of resources relative to impact should lead to more informed donation choices.
P.s. If you found this interesting, you may want to check out Ambitious Impact's Impactful Grantmaking Training Program or our Charity Start-Up Incubation Program.
Useful post.
I think basic accounting ratios and financial analysis are quite helpful for getting a basic view on return on donations and high level valuations of charities.
This is a decent high level overview: https://www.investopedia.com/terms/f/fundamentalanalysis.asp
However, I do think this post underrates the value of assets.
Increased cost effectiveness over time usually comes from investment in quality assets (I'd include staff costs as assets - for management accounting purposes not for external financial reporting purposes).
Does AIM publish any estimates on the valuation of its IP and other intangible assets?
I think there was a conservative counterfactual estimate of the value of a founder somewhere too.
I'd say there's a decent number of highly effective charities with very valuable IP that are too leveraged on the output of very few staff members.
This makes them riskier and more exposed to shocks.
Quite frankly I'd rather they fundraised more than they needed and hired extra staff / had more in reserves for contractors than continue to run lean.
This is an important argument and makes a lot of sense.
"I'd say there's a decent number of highly effective charities with very valuable IP that are too leveraged on the output of very few staff members."
I agree with this, apart from the word "too". I've got no issue with a few talented people carrying an organisation - I think that's just how both charity and business often work for a long long time. How much of Apple's value was in Jobs and Wosniacki even after they were worth billions? How much of their current value is still a legacy of the style and philosophy they built?
This next statement though I think makes sense in theory and I agree to some extent. But I've seen the opposite happen most of the time.
"Increased cost effectiveness over time usually comes from investment in quality assets (I'd include staff costs as assets - for management accounting purposes not for external financial reporting purposes)."
I think this is more the case in the business world as incentives are unfortunately so cooked in the charity world that as organisations grow , it's hard to avoid very poorly performing staff getting paid way too much and for too long. We need to invest in staff to grow an organisation, and it's probably just the right thing to do by our fellow human, but unfortunately I'm not convinced it always makes us more cost effective. I would be interested to hear what examples your have of charities that you think have become clearly more cost effective through investing unusually heavily in staff? Although this might be an unfair question so it's hard to separate that it out.
Often as well I've seen in my org in UgAnda, if I invest heavily in good staff (which we do) they are more likely to get a higher paying job in the NGO or government sector so often the investment doesn't pay off for us. I've seen a number of cases too not in our org where super talented staff were funded for extra study which then directly enabled them to get a better paying job somewhere else. It's complicated. In Western EA orgs where you might have the luxury of relying partly on value alignment this situation might be very different. There's loyalty here but often more to to individual people than a value structure or org.
In my experience with NGOs, staff salaries and "investment" in staff usually end up increasing often to the detriment of the cost effectiveness of the org. Again I'm not saying it's the wrong thing to do. Just observing.
Pretty interesting consideration; it is one I have not thought about/modelled that much. I wonder if someone could do a simple version of this by considering willingness to pay—e.g., how much would a different charity would pay for a given piece of IP? My guess, though, is that many things would be relatively low value compared to the yearly org costs (e.g., I’m not sure someone would pay more than one year of our AIM yearly budget for all our IP).
The biggest intangible asset that comes to mind, which I have not seen modelled much, is the implicit staff training that happens in certain jobs. E.g., if the average staff member goes on to a career five times higher than the jobs they were getting offered before, something quietly valuable has probably happened over their tenure. Shoutout to Founders Pledge for this, as I feel like I see a ton of really valuable people entering the EA job world via Founders Pledge. I think often the counterfactual is that they would get far less impactful jobs than they do post-working for FP.
Strongly agree. When I took a financial accounting course, learning about the ratios was the moment when my eyes really opened up and I saw how I could use these tools to make comparisons.
Good post, thanks for writing it!
A quibble:
I know this is a sketch, but even if 100% of costs are labor both of these come out to fully-loaded costs of $50k/employee which seems quite low to me?
We had a budget of around 100k at more than 20 employees, depends on the country the employees are in!
Good thought I like putting things into close to life scale - made the change to $1.5m and $150k to be in line with ~average CE/AIM charity.
This is a fantastic post and I (unusually) think I agree with basically all of it.
Although I agree with this in principle...
"principles of specialisation and economies of scale from the for-profit world suggest we might expect growth in outputs to outpace increases in budget size."
I can't think of many non profit organizations that I'm convinced become more cost effective as they grow, especially when compared with the first 100k-300k they spend. It's often very hard in the non profit world to take advantage of efficiencies. Also there's a problem I think as funding increases orgs kind of find ways to spend it to justify those donations.
Often bigger scope means more middle management, higher salaries at the top and less efficiency, while also as organisations grow with mission creep and widening of scope can also often introduce interventions which might be less cost-effective than what they did originally.
Many non profits as they grow claim second order effects to justify these extra costs, like influencing government or building up other organizations doing similar things. The community health worker organizations are classic for this.
"I can't think of many nonprofit organizations that I'm convinced become more cost-effective as they grow in their core job."
I can't say I have many great examples of this either, at least past the first ~3-5 years or ~$1-3m budget. With AIM/CE charities, I think they tend to become more cost-effective in years 3-5 than they are in years 1-2, so there are some gains from very early-stage growth.
Although, I guess one mitigating factor here is that I think early-stage organizations are sometimes effectively cost-offset by a dedicated, high-talent founder. So maybe early 'on-paper numbers' don't fully reflect the counterfactual costs, and that cost would reduce with growth.
Aren't both AMF and GiveDirectly examples of charities that became more cost effective after scaling into the $millions?
My sense is that AMF has gotten a little less cost-effective over time due to working in slightly less ideal countries. GD might be pretty close, as I am less sure how the low-hanging fruit affects them. It looks like their percentage of funding that goes to beneficiaries has been pretty similar over time from a quick Google search.
I'd say this is missing where GiveDirectly is extremely cost effective.
Their corporate and government friendly brand.
If they can turn the tide on cash-transfers being the benchmark for foreign aid (and maybe even internal government policy) then that might change the game in terms of political efficacy.
Brilliant I love this.
Give Directly driving attention and attitudes towards cash is good for pushing the aid worlds' thinking in terms of cost effectiveness, normalizing benchmarking, which has positive spillovers to effective altruism in general
That's a good shout thanks Ian. From having a brief look, I would say they're decent examples of orgs that have maintained their cost-effectiveness fairly well but I really doubt they've become much more cost-effective over time. They've done this through continuing to do 1 thing and 1 thing well which I love. In AMF's case the cost have nets have come down which helps their cost effectiveness, but that's not much to do with specialisation or economies of scale within their org specifically..
In my view, this will also depend on how exactly the charity is delivering its intervention. Two examples come to mind where I'd expect much higher cost-effectiveness later on than early on:
(1) Tech-enabled charities (e.g. delivering digital healthcare training or facilitating migration) - you need to initially invest a lot into building the tech product, getting users, building partnerships etc. but then, if things go well, you might scale your impact exponentially while only growing your costs linearly.
(2) Government advocacy and technical-assistance-type projects (e.g. advocacy to introduce tobacco taxes or assistance with rolling out syphilis screening) - you need to initially invest money into building partnerships and/or running pilots with no or minimal impact but, if your work is successful, you'll have big impact later on, with comparable annual costs.
Yep I agree with the Tech example, except even as you scale there you're likely going to need to continue to invest in government partnerships etc. After the initial investment there might be a real spike in cost effectiveness, which might then level out as growth continues.
I would consider the Goverment advocacy and to some extent technical assistance project more of a "hits based approach" (which is great) more than thinking about the framing of long term cost-effectiveness
I like these examples. Maybe someone could do a series of Graphs to illustrate how cost-effectiveness over time could work with different types of orgs? This could help donors and investors understand how their investment functions at different stages of org growth.
Maybe less so in EA than in other charities, but at the ~100K point a hypothetical charity may rely more significantly on volunteer labor compared to the ~1M version of that charity. One could argue that the volunteer labor is a non-economic cost that should be factored into the cost-effectiveness analysis, or could view it as essentially a freebie. From a counterfactual perspective, the correct answer will probably vary.
I think that's a factor like Joey says between the early and mid stage mark. But after that it's more the beuracracy, bloat and mission drift which honestly are hard to avoid.
I'm curating this post. This was my favourite post from Funding Strategy Week. It makes a straightforward but important point that is useful to keep in mind.
What groups of people do you see this most commonly with?
On priors, I would expect most ea aligned donors (or researchers / evaluators) to take things like this into account because they seem pretty fundamental.
Mostly in EA meta...
I think people take this into account but not enough or something? I strongly suspect when evaluating research many people have a vague, and not sufficiently precise, sense of both the numerator and denominator, and their vague intuitions aren't sufficiently linear. I know I do this myself unless it's a grant I'm actively investigating.
This is easiest to notice in research because it's both a) a large fraction of (non-global health and development) EA output and b) very gnarly. But I don't think research is unusually gnarly in terms of EA outputs or grants, advocacy, comms, etc have similar issues.
I’d phrase this section a little differently. I think as a prior you should assume that charities become less cost-effective as they scale. However, the organisations that do grow should be the ones with above-average cost-effectiveness for their size. So even if a charity is less cost-effective than when it was smaller, if funders properly consider size, an average large charity should be equally cost-effective to an average small charity.
Thanks for writing this!
In a vaguely financial markets/efficient market hypothesis train of thought, if you can identify the 'undervalued' teams/organizations, there is (presumably?) some kind of benefit in that.
This may be ridiculously simplistic: I often think about how a person who invests $100 million and earns $1 million will be praised much more than a person who invests $100 and earned $20. In some contexts it matters "what are the absolute results" and in some contexts what really matters is "what were your results given what you had." Sure, I would rather have $1 million than $20, but I'd want to find the person that can get a 20% return and provide them with additional resources.
I do not think there is much reward in the charity sector for identifying undervalued organizations, particularly by criteria that differ from what the market as a whole aims for. Which sadly is not cost-effectiveness, etc. I think that’s part of why it’s a lot easier to find promising missed opportunities compared to the for-profit sector.
I do think it’s much harder (assuming ~equal time) for someone to spend $100 million cost-effectively compared to $100, due to systemic differences. However, I would predict there are many people who could spend $100,000 and get a higher ROI than many people spending $10 million, due to a lack of efficient delegation/regranting/communication between the two.