Dan Stein

502 karmaJoined Jul 2020


Hi Nick, thanks for the thoughtful response. I think you make a lot of good points and I agree that there are numerous incentives can can lead an M+E provider to bias results positively. That's why there is a ton of bad M+E out there. 

One main reaction: for an employee who works in an M+E org, there is arguably no worse situation than being pressured to skew your results positively, or even worse, taking on projects where you know a certain results is expected by your clients. It makes you feel you work is meaningless, and really sucks. And when you are put in this situations, you sure as hell don't want to work for the same client again. 

Yes, i hear you that for bean-counters in an organization (or those who get dividends in a for-profit org), there are strong incentives to make clients happy and get more contracts. But I think that the job-satisfaction incentive for rank-and-file employees skews the other way. And in the course of my experience, I think it is this latter incentive toward truth-telling that has dominated in most cases. 

[Disclaimer: I'm the Chief Economist of IDinsight, an M+E provider who has worked with GiveWell and many others. I have a LOT of experience with evaluators being pressured to sugarcoat results, or lack thereof. ]

Strong disagree on this conclusion that M+E providers are inherently biased. 

Yes, there are situations where M+E have incentives that can lead to bias. For instance, if an NGO hires an M+E provider to do an external evaluation of themselves, the NGO is therefore the 'client' of the researchers. This can be problematic, since the NGO will need to approve deliverables before payments are made. I've been involved in these situations and it is tricky. 

But in general, arrangements can be made to align incentives with the truth. For instance if a funder (like GiveWell) hires an M+E provider to do an evaluation of one of its grantees, the incentives of the M+E provider are aligned with the funder, who hopefully would like to know the unvarnished truth. We've done numerous evaluations for GiveWell (most notably the New Incentives RCT) and have never felt any incentive to skew results one way or another. 

From an organizational perspective, a well-run evaluation organization has much stronger long-term incentives to have a reputation for being honest, transparent, and truth-seeking, rather than getting repeat business from any particular client. 

Is it still true though that FTX Foundation Inc has not filed for bankruptcy? If that's true, returning any funds from FTX Foundation through this mechanism seems premature. But well, I'm no lawyer. 

By "very careful", I mean they shouldn't make the case that their org is higher-impact than the current org unless they are damn sure. And this is an extremely difficult judgement call to make, when comparing two organizations whose mission is social impact.  Given that impact is integral to an EA's worldview, it would be a pretty incendiary accusation for a headhunter to make the case that org X is higher-impact than org Y, so someone should switch jobs.  It's one thing to make this case if hiring someone away from Exxon, but another to make the case within a community of arguably impactful organizations. I think these kinds of tactics have potential to cause major rifts within the community so should be avoided. 

OP here. Thanks for all of the engagement with this post and for the varying opinions. People have brought up some important points on the benefits of headhunting (increased information, better outcomes for employees, overall better job matches, etc), and I agree with a lot of what is said.   After taking these into account and mulling what has been said, here's where I stand (subject to change):

  1. It's clearly ok for EA orgs to hire employees who work at other friendly orgs. Refusing to do so is illegal and I'd say also unethical. 
  2. I think it's ok if you work at an EA org and you know someone who might be a great fit for a role at your org, totally find to reach out to them and let them know. 
  3. It's not ok to use misinformation. I'll be clear that in the headhunting instances  I witnessed from EA orgs, I did not see any misinformation, and all communication was respectful. 
  4. I think recruiters and headhunters within the community should aim to inform rather than persuade recruits, which I do not think is the norm in the headhunting world. This can be a very fine line. The headhunting I witnessed was mostly in the "information" column, yet there was some stuff on the margin. For instance, I think a headhunter should be very careful about insinuate that they job they are recruiting for is higher impact than someone's current job. In one of the emails I saw, the recruiter did not explicitly say this, but it was implied (or at least that was my interpretation). I'm not sure if this was purposeful. 
  5. I think it would be a positive norm for friendly orgs to alert one another if they plan to headhunt, especially if reaching out to multiple people for a role. This allows the original org an opportunity to do what they can to ensure that the employee is having their needs met in the current job. This did happen in one of the instances of headhunting that IDinsight encountered, and it was appreciated.  
  6.  I think foundations/donor orgs have an extra layer of responsibility, as they wield a lot of power in the ecosystem, and in most cases pay more money than their grantees. When considering headhunting the staff of non-profits, they should be extra sure to follow best practices and ensure that they don't destabilize the orgs they are poaching from. I'm not saying they shouldn't recruit from NGOs, but I think they should be very careful about avoiding persuasion and alerting the recruitee orgs. 
  7. I think the bar for paid headhunters is even higher. For instance, if the EA Infrastructure Fund is going to fund headhunters for the EA ecosystem, I think that these headhunters should have a strong bias toward headhunting from lower-impact jobs. (Agreed this is not so easy to figure out what is high and low impact.) If they are instead just contributing to churn among high-impact orgs, I think they have an ambiguous amount of total social impact. Could be high if it increases information, improved job matches, improves employee satisfaction. But could be low if those positive elements are outweighed by switching costs of jobs and the cost of the headhunter themselves. 

Good points- I take back my earlier "Clearly..." statement, and agree it needs to also include utility gains for the worker in the calculation. 

Just to clarify, I wouldn't be advocating that orgs don't hire from peer orgs. Of course, post jobs, make them widely known, take and consider applications from all place. But I think it's different to spend money on dedicated staff to directly target and aggressively recruit staff from friendly orgs within your ecosystem. 

Oh that's very interesting! I had no idea, seems relevant. Also not a lawyer, but I think that this would just apply to agreements not to hire others' employees, as opposed to an agreement not to aggressively recruit. 

Thanks for the comment- I see where you are coming from. As noted in a previous reply, I think a lot has to do with how much the headhunter informs vs convinces. There are a lot of parallels with advertising. Do we think that advertising performs a positive social function? Well, it could if it simply provides information about a new product and allows consumers to make more informed choices. But also the advertiser has incentives to increase sales, so why would we trust them to be truthful and have everyone's best interests at heart? Headhunters/recruiters have incentives to fill roles, so I don't think we should assume that they are playing a neutral, information-providing role. 

Thanks for the comment- I understand where you are coming from, and see how this could go either ways. But I think I'd tend to disagree. I'm always happy  for people to be aware of other opportunities and consider them, but I think there's a difference when there are paid professionals targeting specific people to switch jobs. These professions tend to not just inform, but also convince. So in the situation of a job switch, you end up with a situation where the recruiting organization gains, the recruited organization loses, and  actual job-seeker perhaps gains but this isn't totally clear, depends on the amount that their decision was motivated by information vs convincing. And there's a deadweight loss from the salary of the headhunter.  Therefore, I think that the net effect of a headhunter could be positive or negative. Certainly it seems like they would have a higher impact if they recruited people from low-impact orgs to move to high-impact orgs. 

Load more