Points all well-taken. I'd love to share with FP's journal club, though I hasten to add that I'm still making edits and modifications based on your feedback, @smclare's, and others.
With respect to uncertainty in the CE calculation, my thinking was (am I making a dumb mistake here?) that because
Var(XY)=E(X2Y2)−E(XY)2 and Cov(X2Y2)=E(X2Y2)−E(X2)E(Y2) , then Var(XY)=Cov(X2,Y2)+E(X2)E(Y2)−E(XY)2. So if covariance is nonzero, then (I think?) the variance of the product of two correlated random variables should be bigger than in the uncorrelated counterfactual.
To me, the main value of the CE model was in the sensitivity analysis - working through it really helped me think about what "effective lobbying" would have to be able to do, and where the utility would lie in doing so. I think if it doesn't serve this purpose for the reader, then I agree this document would have been better off without the model altogether.
Thanks for your thoughts on money in politics. Vis (1) I have to think more about this, but I do definitely view the topic a little differently. For instance, it's not obvious to me that economic arguments and political representation do the necessary work of regulatory capture. Boeing is in Washington and Northrop Grumman is in Virginia. It seems clear that the representatives of the relevant districts are prepared to argue for earmarks that will benefit their constituents... but these companies are still in direct competition, and it seems like there's still strategic benefit to each in getting the rest of Congress on their side. I might misunderstand- maybe we're reaching the limits of asynchronous discussion on this topic.
Vis (2), the "inside view" I was talking about was actually yours, as someone who thinks about this professionally- so thank you for your thoughts!
I'm replying again here to note that I've struck the salience point from my conclusions. I've noted why up top. I now have a lot of uncertainty about whether this is the case or not, and don't stand by my suggestion that salience is a good guide to resource allocation.
Thanks for your response!
With respect to your first point, I'm considering striking this conclusion upon reflection - see my discussion with @jackva elsewhere in this thread. In any case, my confidence level here is certainly too high given the evidence, and I really appreciate your close attention to this.
With respect to your second point, I don't mean to imply that the lack of organized opposition is the only thing that justifies lobbying expenditure, and think my wording is sloppy here as well. I used "lack of an organized opposition" to refer broadly to oppositions that are simply doing less of the (ostensibly) effective things — lower "organizational strength" as in Caldeira and Wright (1998), number of groups, as in Wright (1990), or simply lower relative expenditure, as in Ludema, Mayda, and Mishra (2018).
The evidence in Baumgartner et al that you reference about the apparent association between lack of countermobilization and success is also related to @jackva's concern about my underemphasis on potential lobbying equilibria here. On the one hand, I think this is clearly evidence in favor of the hypothesis that there is some efficiency in the market for lobbying- perhaps most lobbyists have a good idea of which efforts succeed, and don't bother to countermobilize against less sophisticated opposition. On the other hand, lobbying is a sequential game, and, since the base rate for policy enactment is so low to start with, it makes sense that opposition wouldn't appear until there's a more significant threat.
EDIT: I've actually struck the first bit, with a note. I wanted to add one more thing, which is that I don't know how much you've adjusted your prior on lobbying, but I wouldn't say this has made me "optimistic" about lobbying. The core thing I've come away with is that lobbying for policy change is extraordinarily unlikely to succeed, but that marginal changes to increase the probability of success are (1) plausible, based on the research and (2) potentially cost-effective, based on the high value of some policies.
I like this spreadsheet idea and think I may kick it off (if you haven't already done so!)
I took the project on because I got interested in this topic, went looking for this, couldn't find it, and decided to make it so that it might be useful to others. I wasn't feeling very useful in my day job, so it was easy to stay motivated to spend time on this for a while. I tend to be most interested in generalizable or flexible approaches to improving welfare across different domains, and this seemed like it might be one of those.
Some areas I'm thinking about exploring. These are pretty rough thoughts:
Hello and thank you for your response!Your criticism of the cost-effectiveness model is fair. Thematically, I guess it does contradict the spirit of my prior analysis in that it avoids the concerns of strategic choice. I was actively trying to be as general as possible, and actively trying to err on the side of greater uncertainty by not including any assumptions about correlatedness, though it occurs to me now that making such an assumption (e.g. a correlation between expenditure and likelihood of success) would actually have increased the variance of the final estimate, which would have been more in line with my goals. When I have time, I may comment here with an updated CEA.I also agree that the only useful way to do this analysis is, as you've described, with a suite of models for different scenarios. I don't have a defense for not having done this beyond my own capacity constraints, though I hope it's more useful to have included the flawed model than not to have one at all (what do you think?).
I also think that the conclusion which, I believe, mostly draws from Baumgaertner " (80%) Well-resourced interest groups are no more or less likely to achieve policy success, in general, than their less well-resourced opponents." is quite surprising and I would be curious to find out why you think that / in how far you trust that conclusion.
Thanks for this, in particular. I think your surprise stems from a lack of clarity on my part. The reason I have high confidence in this conclusion is that it's a much weaker claim than it might seem. It does stem primarily from Baumgartner et al and from Burstein and Linton (2002). The claim here is that resource-rich groups are no more or less likely to get what they want--holding all else equal, including absolute expenditure and the spending differential between groups and their opponents.There are three types of claim that are closely related:1) Groups that spend more relative to their opposition on a given policy are likelier to win2) Groups that spend more in absolute terms are likelier to win3) Groups that have more money to spend are likelier to winSo I found fairly consistent evidence for (1), some evidence for (2), and no real evidence for (3). It's not obvious to me that (3) should be the case irrespective of (1): why would resource-rich groups succeed in lobbying if they deploy those resources poorly? It seems like the success of resource-rich groups is dependent upon (1), and that (3) should not be true when in isolation, unmediated by (1). Although Baumgartner et al conduct an observational study, the size of their (to me, convincingly representative) sample to me suggests that if such an effect exists, it should be observable as a correlation in their analysis. The association they observe is pretty small.I have to say, though, that in writing this comment, my confidence in this conclusion has eased up a bit, so I'm curious to hear your response. I also think that since Baumgartner et al do find a small effect, I probably overstate the case here.Baumgartner et al offer a theoretical take on this: "...organizations rarely lobby alone. Citizen groups, like others, typically participate in policy debates alongside other actors of many types who share the same goals. For every citizen group opposing an action by a given industrial group, for example, there may also be an ally coming from a competing industry with which the group can join forces" (p.12). So it's important to recognize that the finding here is about individual parties, not "sides" or coalitions advocating a given policy.
Finally, I'm curious to hear your take on the two potential money-in-politics explanations you mentioned. I've never found (1) particularly convincing—it's not clear to me that firms and their employees have the same interests, or that (if they do) the marginal value of regulatory capture isn't still high. But I agree that I underemphasized (2) and think it would be useful to have in this thread the "inside view" on lobbying equilibria from someone who works in the field.
(1) I spent something like 100 hours on this over the course of several months. I think I could have cut this by something like 30-40% if I'd been a little bit more attentive to the scope of the research. I decided on the scope (assessing the effectiveness of national-level legislative lobbying in the U.S.) at the beginning of the project, but I repeatedly wound up off track, pursuing lines of research outside of what I'd decided to focus on. I also spent a good chunk of time on the GitHub repo with the setup for analyzing lobbying data, which wasn't directly related to the lit review but which I felt served the goal of presenting this as a foundation for further research.
If I had 40 more hours, I'd intentionally pursue an expanded scope. In particular, I'd want to fully review the research on lobbying of (a) regulatory agencies and (b) state and local governments. I explicitly excluded studies along those lines, some of which were very interesting.
(2) Thanks for asking for clarification on this. Baumgartner et al mean that it takes a long time for policy change to be observed on any given issue. After starting to pursue a policy goal, lobbyists are more likely to see success after four years than after two.
Baumgartner et al include a chapter that is mostly critical of the incrementalist idea of policy change, which they trace to Charles Lindblom's 1959 article The Science of "Muddling Through". Incrementalism is tied to Herbert Simon's idea of "bounded rationality." Broadly, the incrementalist idea is that policymakers face a broad universe of possible policy options, and in order to reduce the landscape to a manageable set, they choose from only the most available options, e.g. those closest to the status quo: "incremental" changes.
Frank Baumgartner, with Bryan Jones, is now well-known for their theory of "punctuated equilibrium." This is a partial alternative to incrementalism which uses the analogy of friction to understand policy change. Basically: the pressure builds on an issue over a period of time, during which no change occurs. After the pressure is overwhelming, policy shifts in a major way.
I say that punctuated equilibrium is a "partial" alternative because Baumgartner and Jones actually collected data that seems to demonstrate that policy change follows a steeply peeked, fat-tailed distribution. Their overall takeaway is that very small changes are overwhelmingly common, but moderate changes are relatively uncommon, and very large changes are surprisingly common. To come back to your question, Baumgartner et al might say that although most policy change is incremental—like year-to-year changes in agency budgets—meaningful policy change happens in a big way, all of a sudden.
(3) I agree with you. I think some of my suggested policies are not likely to be those most effectively advocated for, and I included them just to give a flavor of the types of things we might care about lobbying for. Coming up with more practicable ideas is, I think, a much bigger, much longer-term project.
I also think that although lobbying for the status quo is more effective all other things being equal, it may not be the best use of EA resources to focus exclusively on that side of things. That's because (per the counteractive lobbying theory) on many issues there is are latent interests that will arise to lobby against harmful proposals. It's hard to identify beforehand which proposals will stimulate this opposition, so there's a lot of prior uncertainty as to whether funding opposition to policy change is marginally useful in expectation.
(4) There are a lot of takes on the Tullock paradox, but I'll present two broad possible explanations.
Given the evidence here, I'm starting to be a lot more inclined toward Explanation B. I think it's demonstrably not the case, as you have noted with respect to the Clean Air Task Force, that organizations that lobby are wasting their money. For both altruistic and self-interested interest groups, the rewards to be captured are very large, and they make it worth the risk of wasting money. Alexander, Scholz, and Mazza (2009), for example, find a 22,000% return on investment.
If Explanation B holds, then the question is really just why the market for policy isn't efficient. Why hasn't the price of lobbying been bid up to the value of the rewards to be captured? I think it seems likely that this is down to multiple layers of information asymmetry (between legislators and their staffs, between these staffers and lobbyists, between lobbyists and their clients, etc.), which create multiple layers of uncertainty and drive the expected value of lobbying down from the standpoint of those in a position to purchase it.
I agree with you that a normal distribution is probably not the best choice to model the expected incremental change in probability. I felt like, given my CI for this figure and my sense that values closer to 0% and values closer to 5% were each less likely than values in the middle of that range, this served my purposes here - but please take my code and modify as you see fit!
Perhaps we want to start with a low prior chance of policy success, and then update way up or down based on which policy we're working on. Do you think we'd be able to identify highly-likely policies in practice?
I don't know. I think it's worth investigating. It seems like, given an already-existing basket of policies we'd be interested in advocating for, we can make lobbying more cost-effective just by allocating more resources to (e.g.) issues that are less salient to the public.
I have a sense that lobbyists, do, in fact, do something like what you're describing, and that this is part of the resolution to the Tullock paradox. Money spent on lobbying is not spent all at once: lobbyists can make an effort, check their results, report to their clients, and identify whether or not they're likely to meet with success in continued expenditure. If lobbying expenditure on a given topic seems unlikely to make a difference, then it can just stop. I wasn't able to find anything on how this process actually works, so the next step in this research is to actually talk to some lobbyists.
I think perhaps something that's missing here is a discussion of incentives within the civil service or bureaucracy
I agree with this too. I'd love for an EA with a public choice background to tackle this topic. I didn't consider it as part of my scope, but I do want to note something:
A policy proposal like taking ICBMs off hair-trigger alert just seems so obvious, so good, and so easy that I think there must be some illegible institutional factors within the decision-making structure stopping it from happening.
I think this is probably true in many if not most cases of yet-to-be-implemented policy changes that are obvious, good, and easy. It is probably true in this case. But I want to warn against concluding that, because some obvious, good, and easy policy change has not been implemented, that means that there is some illegible institutional factor that is stopping it from happening. It could just be that no one has been pushing for it. In EA terms, it's an important and tractable policy change that's neglected by the policy community. Given what I know about the policy community, it's not at all difficult for me to imagine that such policies exist.
I refer you to Sindy's comment (she is actually an expert) but I want to note and verify that it sounds as if you may not actually be thinking of collecting individual-level data, and that you're thinking of making observations at the village level (e.g. what % of people in this village wear masks?). So it's not just the case that you wouldn't have enough clusters to make a statistical claim, but you may actually be talking about doing an experiment in which the units are villages... so n = 6 to 12. Then of course you'd have considerable error in the village-level estimate, and uncertainty about the representativeness about the sample within each village. I agree with Sindy that you probably don't want an RCT here.
If you don't already have it, I would strongly recommend getting a copy of Gerber & Green's Field Experiments. I would also very strongly recommend that you (or EA Cameroon) engage an experimental methodology expert for this project, rather than pose the question on the forum (I am not such an expert).
It is very difficult to address all of these questions in a broad way, since the answers depend on:
I'm a little confused about the setup. You say that there are 6 groups— so how would it be possible to have "6 intervention + 3 non-intervention?" Sorry if I'm misunderstanding.
In general, and particularly in this context, it makes sense to split your clusters evenly between treatment and control. This is the setup that minimizes the standard error of the difference between groups. When the variance is larger, smaller effect sizes are difficult to detect. The smaller the number of clusters in your control group, for example, the larger the effect size that you would have to detect in order to make a statistically defensible claim.
With such a small number of clusters, effect sizes would have to be very large in order to be statistically distinguishable from zero. If indeed 50% of the population in these groups is already masked, 6 clusters may not be enough to see an effect.
Can we get some clarification on some of your questions? Particularly:
How important, in terms of statistical power is to include all clusters
If you have only 6 to choose from, then the answer is very important. But I'm not sure this is the sense in which you mean this.
How many persons should be observed at each place?
My inclination here is to say "as many as possible." But this is constrained by your resources and your method of observation. Can you say more about the data collection plan?
I also thought this when I first read that sentence on the site, but I find it difficult (as I'm sure its original author does) to communicate its meaning in a subtler way. I like your proposed changes, but to me the contrast presented in that sentence is the most salient part of EA. To me, the thought is something like this:
"Doing good feels good, and for that reason, when we think about doing charity, we tend to use good feeling as a guide for judging how good our act is. That's pretty normal, but have you considered that we can use evidence and analysis to make judgments about charity?"
The problem IMHO is that without the contrast, the sentiment doesn't land. No one, in general, disagrees in principle with the use of evidence and careful analysis: it's only in contrast with the way things are typically done that the EA argument is convincing.
I don't work in physical goods (I'm a data scientist) but I am definitely interested in leveling up my skillset in this way. I'm probably only available for 3 to 4 hours a week to start, but that will probably change soon.
Thanks for making this post! This is an interesting observation.