Some surprising hiring practices I follow (as a hiring manager and grantmaker in EA)

The best approach to take to hiring differs by industry. 

This means that best practice differs across startups, think tanks, video production, non-profits, academia, and news outlets will all have different practices. The practices that work best in the EA ecosystem will be different again – but unfortunately the effective altruism organisation landscape is much smaller than all of those, and so people haven't written books on how to do it.

I have been working as a hiring manager and sometimes grant maker in this space for over ten years, and I have developed some views on which practices work well in this industry. And I’ve noticed that others have (semi-)independently converged on these.

Below I've written out a few of the ones which I follow. I’ve especially tried to list those which might seem most surprising to people coming from industries like the ones listed above. 

I’m trying to do a pretty quick job of this so that it actually gets out there. [Having written that sentence, I proceeded not to publish the post for 2 years. So if you’re wondering why there are errors in it, it’s because this time round I’m actually going to post it rather than finesse it. Really.] I’m not trying to give a well rounded picture of how I do all hiring, partly because I just want to write about a few things that will be interesting to people and partly because hiring varies a lot by role and organisation. 

Two resources I’ve found particularly useful for thinking about how I should hire are Sam Altman’s post on How to Hire and Who: The A Method of Hiring.

Run unstructured interviews

I periodically hear people say that everyone should run ‘structured’ interviews: ones where the questions are set in advance and the same is asked of everyone. That does seem like an improvement on some interviews. It means that the interview difficulty is uniform between different candidates. It also forces you to think in advance about what you most want to learn from the interviews. 

On the other hand, there’s a limit to how much you can learn in a structured interview, because you can’t adapt your questioning on the fly if you notice some particular strength or weakness of a candidate. It can also be difficult to really get a good sense of how they reason and how good their judgement is. 

A case I think of as being at the other extreme is university philosophy admissions interviews. When I’ve done these, the setup aimed to be as close as possible to getting a sense of how well people ‘do philosophy’ by probing in depth the argument a candidate makes. This style of interview is a pretty useful tool to have your toolkit.

Variety between candidates amongst interviews can also be really important. For high skill jobs, you actually want to know a whole host of things about a person before deciding to hire them. It’s likely you’ll have different uncertainties about different candidates. In some cases that will occur as the interview goes, so you’ll want the flexibility to change the questions as you. Sometimes it will be based on their performance in earlier rounds of the hiring process, so you might want to have interviews about different things with different candidates. 

Work tests can be feasible and useful even for high-skill roles

Work tests seem to be the kind of thing which is usually thought of as primarily used for either hard skills like programming or lower skill, rote tasks. But I’ve found work tests pretty useful even for roles requiring high levels of soft skills like being an 80,000 Hours Advisor. 

Designing work tests that can assess skills like high level management is indeed often harder than designing work tests for hard skills and rote tasks. But it’s very much not impossible. 

It’s possible that one thing going on here is really a confusion of terms. The best ways of testing soft skills often include conversation, so might normally be thought of as interviews. For example, 80,000 Hours’ work test for its advising team involves a ~2 hour work test where the bulk of the time is spent on a call with a staff member, discussing possible advisees. It might be that this would be considered ‘best practice’, but referred to as an interview rather than a work test. We distinguish ‘work tests’ from ‘interviews’ in part by there being reading/writing involved in addition, and in part by the fact that it’s intended to mimic the work the person would do on the job.

Perhaps the weirdest ‘work test’ I’ve administered was a grant interview for a therapist. I hadn’t previously directly evaluated therapists for grants, and felt at sea on how to do it. I realised that typically when hiring I got most info from seeing a person having a go at something that was as close as possible to the actual job, and that in this case since I’ve seen a few different therapists in the past I had some sense of what ‘doing a good job at it’ looked like. It worked surprisingly well to talk through a few representative seeming issues with her to get a sense of how she would handle them.

My guess is that these types of work tests are underinvested in because they’re quite a lot of work to prepare for. They often vary role by role rather than being able to reuse them for lots of vacancies, and are tricky to design so that they provide useful information without being prohibitively time consuming for both parties. 

Informal references are useful

The types of references people expect vary markedly across sectors:

• In large companies and local governments, it’s often accepted practice to provide only written references, and for those to only say ‘the person worked in this role from x date to y date’. This avoids people taking issue with bad references, and can help avoid both abuse (threatening someone with a bad reference) and nepotism.
• In academia, long reference letters are common for early stage academics who don’t have much of a publication record. There’s wide variety in them, but they try to get reasonably specific, including making comparative claims like ‘this is the best student I’ve taught this year’.
• The book ‘Who’ recommends asking specifically for verbal references so the referee is more willing to be candid and so that you can ask specific questions. 

Who to ask for references from also seems to vary by sector. The easy default is to ask the applicant to provide suggested referees and only reach out to them. ‘Who’ suggests specifically telling candidates you want to talk to their boss and that you’ll do that after interviewing them, in order that you get more accurate answers from them. Start-up founders sometimes recommend talking to essentially as many people as you can about a potential hire, to really get a picture of them. 

Given these differences in advice, it seems useful to know what causes the differences to get a sense of which advice is likely most applicable for your situation. 

One thing going on is likely to be the stakes involved in particular hiring decisions. If a large company or local council hires one mediocre candidate, the overall system can deal with it. By contrast, a start-up having one mediocre early employee could make the difference between success and failure. 

Another difference is likely to be the overall size of the ecosystem an organisation is in. A recruiter for high level legal executives is working with a reasonably small, specialist pool. They would know people who have worked with particular candidates well enough to ask for a take on the person. They’ll be able to trust people to tell the truth because they’ll work together in future so need to keep trust. 

I’ve found getting a broad range of references very useful in the past. Ideally, I can find people who I trust to give a reasonably honest take and have worked with both me and the applicant. The reason is that if I’ve worked with someone, I likely have a goodish sense of how good they are at the skills I’m evaluating, and also some sense of how effusive they typically are. That makes it easier to tell how good a reference actually is.

People in the EA ecosystem are often touchingly generous with their time and willingness to provide references. 

Get information from people with conflicts of interest

Another type of reference I think sometimes gets an unwarranted blanket ban is from people with a conflict of interest. As a stark example, in one organisation I worked for, if a person was quitting or going on parental leave, they were not allowed to be involved in hiring for their replacement due to potential conflict of interest. (For example, they might want to ensure that their replacement is ‘like them’ rather than focusing on whether the person would do a good job). That seemed kind of crazy to me. After all, the person who had been in the role had the most information about what it was like to do the job and what was needed to do it well. 

Unfortunately, correlation between people having information on a candidate and having a conflict of interest is pretty frequent. Someone who has worked extensively and closely with a candidate can provide detailed information on their strengths and weaknesses. But they also likely get on well and feel positively disposed to them. After all, they know each other well and often will have deliberately chosen to continue working together. Someone who works in the same lab as a grant applicant likely has a good sense of how good they are at their job. They also likely have a vested interest in their lab getting more grants since it will have spillover prestige effects. 

Avoiding getting references or other information from people with a conflict of interest is plausibly better than totally ignoring conflicts of interest. But those usually aren’t the only options - you can typically get some sense of how serious a person’s conflict of interest is, and take their information into account appropriately downweighted. 

You can put numbers on things (but don’t trust them too much)

Putting numbers on the qualities people have feels pretty gross, which is probably why using quantification in hiring is rather polarising. On the one hand, there’s some line of thinking that the different ways in which people are well and ill suited to particular roles isn’t quantifiable and if you try to quantify it you’ll just be introducing bias. On the other hand, people in favour of quantification tend to strongly recommend that you stick exactly to the ranking your weightings produced. 

I think I probably use quantification of this kind a little less than I endorse, because it feels so difficult to decide what the weightings should be and then to actually figure out for each person how to assign them. I do think it’s often a useful exercise for getting me to be explicit about the types of traits I care more and less about, and what evidence I have that different candidates have those traits. 

On the other hand, I think that it’s mostly impossible to get formulae you fully endorse and that actually capture everything you want and nothing else. For that reason, I would use the quantification as a tool for guiding my thinking, rather than as the only input. 

Bonus suggestion for applicants: Ask about the people you’ll be working with

My impression is that it’s not very common for applicants to try to talk to people who might be able to give candid views of their potential future colleagues (including their manager). In many cases, that’s too difficult to do. In others, it’s reasonably straightforward. For example a PhD applicant could ask their prospective supervisor’s current grad students what it’s like to work with the supervisor. Yet, at least when I was applying to grad school, this was not very common. 

Colleagues, particularly managers, make a huge difference to your work experience. The extent to which you have compatible styles, and the extent to which you trust the judgement and integrity of those you’ll be working with, is incredibly important. So it can be really useful to get a sense of those things before accepting a job offer. 

You should make these enquiries as sensitively as possible. You might want to ask people who used to work at the company rather than (or as well as) those who currently do, since the former will feel more free to speak their mind. 

Getting advice

I think it’s very important to get advice from people with significant hiring experience. That includes both consuming content about it and talking to people individually. 

A significant risk of not following ‘best practice’ is trusting your own intuitions too much. Worse still would be just not putting much thought into hiring. I think we should in fact be putting more rather than less time into hiring. We should try to get a sense of what practices are used in a range of contexts, design a process that draws on the elements most sensible for a particular situation, then get feedback on it from people with relevant experience. 

Of course, doing things entirely in a bespoke way is not always possible - in cases where you have limited time, borrowing an existing template is likely to work better. For those cases, it seems useful to have a sense of whose templates you want to borrow - likely depending on what context your situation is most similar to. 

Personally, I’ve typically found start-up advice the most useful, since it’s aimed at small, young organisations with large ambitions.

I’ve also found it useful to reach out to people who seem to have relevant expertise when I have a specific quandary, to talk it through it with them. To find people it might be useful to talk to and would likely be happy to, you could go to an EA Global or reach out to people on the EA UK directory. 

I’m grateful for comments from a number of people, but Brenton Mayer particularly helped shape the post and Will MacAskill's haranguing caused it to actually get published.