Is Neglectedness a Strong Predictor of Marginal Impact?

The neglectedness criterion is commonly used within effective altruism as a way of determining whether a cause area is high impact or not. As The Open Philanthropy Project puts it, “all else being equal, we prefer causes that receive less attention from others”. And why should we have that kind of preference? The given rationale is that most opportunities are subject to diminishing returns, where the impact of additional resources declines with the total number of resources already being put toward that opportunity. However, as I will attempt to show in this post, diminishing returns is not sufficient to guarantee a positive relationship between neglectedness and the marginal impact of more funding. Furthermore, diminishing returns may not be as common as is usually assumed. Overall, the results cause concern that the neglectedness heuristic is not useful in many contexts.

The words “neglectedness” and “crowdedness” get used a lot in effective altruism, sometimes to refer to very different ideas. In this post, I am using crowdedness to mean the amount of resources being put toward a cause (or project, opportunity, intervention, etc.), and I take neglectedness to mean the opposite of crowdedness. This is how the terms are defined in 80,000 Hours’ framework. By contrast, I’ve occasionally heard neglectedness be used to mean “less resources are going toward this cause than they should” or “the results of this intervention are valued less by other funders than they are to me”. Those are not the meanings that I have in mind in this post.

The post is organized in the following way: first is a qualitative discussion of the use of the neglectedness heuristic in situations with diminishing returns, then a short discussion of the possibility of increasing returns, then a section on how these ideas relate to the 80,000 hours cause prioritization framework. At the end I present the simple mathematical model that I used to generate my conclusions (this can be skipped by most readers).

Key Points:

· Even under diminishing returns, neglectedness is expected to be a weak proxy for marginal impact in cases where other actors have similar values to us, are well-informed, and are somewhat rational (this is probably clear to most people, but it provides a good starting point).

· Cause selection based on other factors that influence neglectedness can be more efficient than selection based on neglectedness itself. One such factor is the importance of an opportunity according to your value system relative to the value systems of others.

· Relative to other contexts (such as firms operating in market environments), there is less reason to expect diminishing returns for altruistic opportunities; the possibility of increasing returns further calls into doubt the general usefulness of the neglectedness heuristic.

When is neglectedness correlated with marginal impact?

The neglectedness heuristic depends on the idea that cause areas with less resources invested in them will have a larger marginal impact (adding a small amount of additional resources will make a larger difference). At first glance, this may seem to be implied by diminishing marginal returns. And in fact, if the amount of resources going toward each cause is randomly determined, then diminishing returns does guarantee that neglectedness correlates strongly with marginal impact.

However, in most circumstances, we wouldn’t expect other funders to be acting in a completely random fashion. Yes, I realize that effective altruism is motivated by the fact that people frequently attempt to do good in sub-optimal ways, but that doesn’t mean that people aren’t at least somewhat thoughtful when making decisions on how to do good (see GiveWell’s article: https://blog.givewell.org/2013/05/02/broad-market-efficiency/).

To see why this matters, let’s consider an extreme case: you are in a situation where all other donors share your values and are acting to maximize their marginal impact. This might be close to the decision of choosing between causes that are highly regarded within EA, or of choosing between GiveWell’s top charities. In these cases, would we expect to see a correlation between neglectedness and marginal impact? Probably not. The other funders are fully aware of diminishing returns, and they will take that into account when making decisions. If one cause had a higher marginal impact than another, we’d expect funders to move money away from the latter and toward the former. Thus, in equilibrium, we’d expect for all causes that receive positive amounts of funding to have the same marginal impact, and all other causes to receive zero funding and have marginal impacts lower than the equilibrium level. This means that the neglectedness heuristic is useless in such a world. Among causes that receive a positive amount of funding, there is no correlation between neglectedness and marginal impact. If we include causes that receive zero funding, the correlation actually becomes negative (the zero-funding causes are the most neglected precisely because they have a low marginal impact).

Of course, the situation above was an extreme case. In very few cases can we expect other funders to be perfectly rational and to perfectly share our values. But in many cases we can expect them to be at least partially rational and to partially share our values. In these situations, causes with more funding would tend to be more important or tractable, which would weaken the link between neglectedness and marginal impact. Specifically, the expected marginal impact of a cause would increase with neglectedness at a rate lower than the rate of diminishing returns. The more that other funders share our values, the weaker the relationship will become. In the extreme case, already covered, there is no correlation.

Now, one may try to defend the neglectedness heuristic by appealing to the fact that it is meant to be used in conjunction with other factors. Rather than looking at the general link between existing funding and marginal impact between any two causes, we should look at the link between existing funding and marginal impact between any two causes that are of similar tractability and importance. Sure, there may some selection going on where causes with more funding tend to be more important, but if we control for importance and other factors, we would expect the link between neglectedness and marginal impact to be just as strong as the rate of diminishing returns.

Such an argument works well in cases where you are sure that other funders are equally or less informed than you are. However, in many cases, there’s at least a possibility that others have some information that we don’t. So, if we have two causes that we expect are equally tractable and important, but for some reason cause A is receiving more funding than cause B, can we be completely sure that they are actually equally important and tractable? It could be the case that cause A is receiving more funding because of differences in other funders’ values relative to your own or due to irrationalities in others’ behavior. But it also could be that the other funders know something about the importance or tractability of the causes that you don’t. This will tend to weaken the link between neglectedness and marginal impact, even after controlling for the perceived importance and tractability of separate causes.

Selection Based on Alternative Factors

Even in cases where there is a positive relationship between neglectedness and marginal impact, other heuristics may be more useful than neglectedness. Ideally, we would like to not simply select causes that are neglected, but to select causes that are neglected for reasons other than their impact. This is possible in cases where we can predict some of the factors that cause variation in funding. For example, we may be able to predict which causes are likely to be underweighted by others’ value systems relative to ours (this is the idea behind Paul Christiano’s article). This predicted “value conflict” measure will be more positively correlated with marginal impact than neglectedness because it won’t correlate with the “additional information” factor. Furthermore, after controlling for the value conflict measure, the link between neglectedness and marginal impact will be weakened even more. The value conflict heuristic thus can serve as an upgrade to the neglectedness heuristic.

The Possibility of Increasing Returns

The analysis so far has depended on the assumption that all causes are subject to diminishing returns, which seems to be the default assumption in most EA cause-prioritization work that I have seen. Now I’d like to present a brief argument that calls into question the reliability of that assumption.

In intro economics courses, I was taught that although increasing returns to scale are common, in any competitive market equilibrium we can expect to see firms operating at a point where diminishing or constant returns exist. Why should that be the case? Because if one firm faces increasing returns, they can simply expand (which lowers their marginal costs), charge a lower price than their competitors, and capture more of the market. Thus, if there are multiple firms existing in equilibrium, we can expect that the firms have expanded to a point where they are subject to diminishing (or constant) returns to scale. This gives a clear theoretical reason for assuming diminishing returns to scale in competitive markets. (still, natural monopolies exist, where the point of diminishing returns never comes and the only possible equilibrium is with one firm facing increasing returns).

Does the same mechanism exist to motivate the assumption that charities (or cause-areas, research areas, etc.) will always be operating at an output level where they face diminishing returns? In some cases, maybe. For example, if the against malaria foundation were facing increasing returns, it could borrow money to expand, advertise that it can now do good for a much lower cost, attract donations away from competing charities, and use those donations to pay off the expansion. This scenario seems plausible because many AMF donors are rational and strategic in their donation decisions, and so they will predictably increase donations if the charity has increased cost-effectiveness. In other cases, however, it seems less likely that donors will be so strategic. Let’s say that a certain type of climate change mitigation research is facing increasing returns to scale. Are we confident that research institutes in this area can convince enough donors to finance an expansion? It’s very difficult to find out in the first place whether increasing returns exist, and it would be even more difficult to convince donors, many of whom do not make philanthropic decisions in the most rational way, to increase their donations in response to this.

I’d also note that constant returns to scale is a common assumption to make within economics, particularly at the industry level, which may be comparable to the cause-level for altruistic opportunities.

These considerations make me skeptical of assuming diminishing returns for altruistic opportunities. I’d like to hear if there’s been any relevant work done on this topic (either within EA organizations or within general academia). Increasing returns is a fairly common topic within economics, so I figure there is plenty of relevant research out there on this.

Relation to 80,000 Hours Cause Prioritization Framework

The commentary above leads us to question the usefulness of the neglectedness heuristic. Now I’d like to examine how this consideration would impact cause-prioritization when viewed through 80,000 Hours’ problem framework, presented here.

The 80,000 approach breaks marginal impact into three ratios corresponding to scale, solvability, and neglectedness:

This decomposition will always work as long as you can correctly estimate the values for scale, solvability, and neglectedness (whether it is the most useful way to break down the problem is another story, but it will certainly work if the numbers can be estimated correctly). Now, looking a bit closer at the neglectedness score, this can be rewritten as:

So our value for neglectedness is determined completely by how many resources are going toward the cause or problem. This means that it is not affected by any judgements about the link between neglectedness and marginal impact. Instead, such judgements will have to impact our estimates of solvability. If there is a sufficiently strong correlation between neglectedness and marginal impact, then we would expect there to be zero correlation between neglectedness and solvability. (Why zero? Because solvability is defined in terms of percentage increase in resources. If all problems yield the same amount of good when funding is doubled, then problems with less funding will have a larger marginal impact because it is easier to double their funding). If there is no correlation between neglectedness and marginal impact, then we would expect for there to be a negative correlation between neglectedness and solvability.

If we look at the scores that 80,000 hours gives to various problems, we see that measures of solvability are nearly constant. This leads to nearly no correlation between neglectedness and solvability, which leads to a large positive correlation between neglectedness and marginal impact (in the article "total score" = marginal impact). While I’m not sure how the solvability scores were determined, I would expect that the idea that there’s a strong link between neglectedness and marginal impact due to diminishing returns played a role, as it is mentioned multiple times in the general cause prioritization article.

If my argument presented above is correct, and there’s reason to doubt the link between neglectedness and marginal impact, then we may want to use a prior that neglectedness and solvability are negatively correlated. Furthermore, we may want to use other heuristics, such as the value conflict measure mentioned earlier, to better estimate solvability. Finally, if we allow for increasing returns to scale, we would want to adjust our priors on the correlation between neglectedness and solvability to be even more negative (not to mention the fact that prioritization based on marginal impact may not be the best option when one faces increasing returns).

Model

To illustrate the ideas in this post, I’ll use a simple model of cause selection. While the functional forms are chosen to make things nice and linear, I expect that the conclusions would still hold if more complex functional forms were used. There may be errors in the reasoning or algebra below; if you notice anything please let me know.

In this situation you are attempting to choose to invest a small amount of resources. There a number of causes open to you, indexed by i, which each have some fixed amount of resources r_i invested in them by other actors regardless of what decision you make. Each cause has a positive impact on the world (relative to your value system), which is a function of r_i:

where $PI{T}_i$ is the perceived importance and tractability of the cause, and $u_i$ is the unknown (to you) importance and tractability of the cause.

Because you only have a small amount of resources available, you are interested in the marginal impact of giving to each cause, which is the derivative of the previous equation with respect to r_i:

where neglectedness $n_i=1/r_i$. We're interested in predicting the marginal impact for a cause area based on the known values $PI{T}_i$ and $n_i$, so we'll look at the expected value of marginal impact conditional on $PI{T}_i$ and $n_i$:

First, let's imagine that the amount of funding for each cause is independent of $u_i$ (this could happen if others randomly choose what causes to fund, but it could also happen if they rationally choose but have values that are uncorrelated with ours). In this case, the conditional expectation becomes:

where I assume that $E\left[u_i|PI{T}_i\right]=0$ (if we are doing a good job of estimating importance and tractability, then this should be true). As a measure of the usefulness of the neglectedness heuristic, we can look at the slope of the conditional expectation curve:

This result shows the motivation behind the neglectedness heuristic. When other’s decisions on what to fund are uncorrelated with the impact of the opportunity, we would expect there to be a positive relationship between neglectedness and marginal impact that is equal to a cause area's importance and tractability.

However, in many cases we cannot expect for neglectedness to be completely independent of $u_i$, and instead would expect for there to be a negative correlation ($\frac{\partial }{\partial n_i}E[u_i|n_i,PI{T}_i]<0$ ). Here we can write:

which will usually be lower than $PI{T}_i$ because the third term is negative. Using the previous assumption that $E\left[u_i|PI{T}_i\right]=0$ , we find that the average slope of the conditional expectation curve is less than $PI{T}_i$:

This means that the usefulness of the neglectedness heuristic is weakened when neglectedness is correlated with unobserved aspects of importance or tractability, which is to be expected when others have at least some information that we don't and share some of our values. In the extreme case, where others are rational, equally informed, and share our values, the slope of the conditional expectation will be zero.

Next I’d like to include the decision-making of others into the model. Specifically, we’ll assume that there is some observable aspect of causes which influences how much people invest in them but has nothing to do with their marginal impact. As an example, I’ll use $\Delta V_i$, which is the difference in value that others place on cause i relative to your values. To model this, in addition to our marginal impact equation, we now need an equation for how the number of resources going toward a cause is determined:

where $f$ is the conditional expectation function of neglectedness given value conflict and $v_i$ are other determinants of outside funding that are uncorrelated with $\Delta V_i$. Plugging this into the marginal impact equation and taking the derivative gives:

where $E[u_i|\Delta V_i,PI{T}_i]=0$, $E[v_i|\Delta V_i,PI{T}_i]=0$, $\frac{\partial }{\partial \Delta V_i}E[u_i{v}_i|\Delta V_i,PI{T}_i]=0$ by the assumption that $\Delta V_i$ is uncorrelated with marginal impact and by the definition of $v_i$ as factors that are uncorrelated with $\Delta V_i$.

The result in the previous equation has a nice interpretation. $f^{\prime }\left(\Delta V_i\right)$ is the rate at which a value conflict causes an increase (or decrease) in neglectedness, while $PI{T}_i$ is the rate at which this neglectedness reduces marginal impact. Furthermore, we could potentially estimate the function $f\left(\Delta V_i\right)$ from the data (we can estimate the level of neglectedness and how much value others place on each cause relative to us). Using this function, we can calculate a predicted level of neglectedness ${\hat{n}}_i=f\left(\Delta V_i\right)$ for each cause. We then find:

which is the same slope that we got in the case where funding was randomly allocated across causes. This is what I meant when I said that selecting on other determinants of neglectedness (like value conflict) can be more useful than simply selecting based on neglectedness.