Hi Nick, thanks for the thoughtful response. I think you make a lot of good points and I agree that there are numerous incentives can can lead an M+E provider to bias results positively. That's why there is a ton of bad M+E out there.
One main reaction: for an employee who works in an M+E org, there is arguably no worse situation than being pressured to skew your results positively, or even worse, taking on projects where you know a certain results is expected by your clients. It makes you feel you work is meaningless, and really sucks. And when you are put in this situations, you sure as hell don't want to work for the same client again.
Yes, i hear you that for bean-counters in an organization (or those who get dividends in a for-profit org), there are strong incentives to make clients happy and get more contracts. But I think that the job-satisfaction incentive for rank-and-file employees skews the other way. And in the course of my experience, I think it is this latter incentive toward truth-telling that has dominated in most cases.
[Disclaimer: I'm the Chief Economist of IDinsight, an M+E provider who has worked with GiveWell and many others. I have a LOT of experience with evaluators being pressured to sugarcoat results, or lack thereof. ]
Strong disagree on this conclusion that M+E providers are inherently biased.
Yes, there are situations where M+E have incentives that can lead to bias. For instance, if an NGO hires an M+E provider to do an external evaluation of themselves, the NGO is therefore the 'client' of the researchers. This can be problematic, since the NGO will need to approve deliverables before payments are made. I've been involved in these situations and it is tricky.
But in general, arrangements can be made to align incentives with the truth. For instance if a funder (like GiveWell) hires an M+E provider to do an evaluation of one of its grantees, the incentives of the M+E provider are aligned with the funder, who hopefully would like to know the unvarnished truth. We've done numerous evaluations for GiveWell (most notably the New Incentives RCT) and have never felt any incentive to skew results one way or another.
From an organizational perspective, a well-run evaluation organization has much stronger long-term incentives to have a reputation for being honest, transparent, and truth-seeking, rather than getting repeat business from any particular client.
Is it still true though that FTX Foundation Inc has not filed for bankruptcy? If that's true, returning any funds from FTX Foundation through this mechanism seems premature. But well, I'm no lawyer.
By "very careful", I mean they shouldn't make the case that their org is higher-impact than the current org unless they are damn sure. And this is an extremely difficult judgement call to make, when comparing two organizations whose mission is social impact. Given that impact is integral to an EA's worldview, it would be a pretty incendiary accusation for a headhunter to make the case that org X is higher-impact than org Y, so someone should switch jobs. It's one thing to make this case if hiring someone away from Exxon, but another to make the case within a community of arguably impactful organizations. I think these kinds of tactics have potential to cause major rifts within the community so should be avoided.
OP here. Thanks for all of the engagement with this post and for the varying opinions. People have brought up some important points on the benefits of headhunting (increased information, better outcomes for employees, overall better job matches, etc), and I agree with a lot of what is said. After taking these into account and mulling what has been said, here's where I stand (subject to change):
Good points- I take back my earlier "Clearly..." statement, and agree it needs to also include utility gains for the worker in the calculation.
Just to clarify, I wouldn't be advocating that orgs don't hire from peer orgs. Of course, post jobs, make them widely known, take and consider applications from all place. But I think it's different to spend money on dedicated staff to directly target and aggressively recruit staff from friendly orgs within your ecosystem.
Oh that's very interesting! I had no idea, seems relevant. Also not a lawyer, but I think that this would just apply to agreements not to hire others' employees, as opposed to an agreement not to aggressively recruit.
Thanks for the comment- I see where you are coming from. As noted in a previous reply, I think a lot has to do with how much the headhunter informs vs convinces. There are a lot of parallels with advertising. Do we think that advertising performs a positive social function? Well, it could if it simply provides information about a new product and allows consumers to make more informed choices. But also the advertiser has incentives to increase sales, so why would we trust them to be truthful and have everyone's best interests at heart? Headhunters/recruiters have incentives to fill roles, so I don't think we should assume that they are playing a neutral, information-providing role.
Thanks for the comment- I understand where you are coming from, and see how this could go either ways. But I think I'd tend to disagree. I'm always happy for people to be aware of other opportunities and consider them, but I think there's a difference when there are paid professionals targeting specific people to switch jobs. These professions tend to not just inform, but also convince. So in the situation of a job switch, you end up with a situation where the recruiting organization gains, the recruited organization loses, and actual job-seeker perhaps gains but this isn't totally clear, depends on the amount that their decision was motivated by information vs convincing. And there's a deadweight loss from the salary of the headhunter. Therefore, I think that the net effect of a headhunter could be positive or negative. Certainly it seems like they would have a higher impact if they recruited people from low-impact orgs to move to high-impact orgs.