If you lose your top choice due to insufficient salary, how good do you expect the replacement to be? 

For CEA, I'd likely guess they'd be indistinguishable for most roles most of the time. 

I'm CEA's main recruitment person and I've been involved with CEA's hiring for 6+ years. I've also been involved in hiring rounds for other EA orgs.

I don't remember a case where the top two candidates were "indistinguishable." The gap very frequently seems quite large (e.g. our current guess is the top candidate might be twice as impactful, by some definition of expected impact). There have also been many cases where the gap is so large we don't hire for the role at all and work we feel is important  simply doesn't happen. There have also, of course, been cases where we have two candidates we are similarly excited about. This is rare. If it does happen, we'd generally be happy to be transparent about the situation, and so if you have been offered a role and are wondering about your own replacability, I'd encourage you to just ask.

Fully agree about false negatives vs. false positives! As you say, I was indeed taking that for granted. I was aiming to encourage people to think more carefully about tradeoffs between false negatives and other factors like speed.

Context: I’m an advisor to the search committee, not a member so I have some insight but not complete visibility.

My guess is the set of people with significant EA context who have been involved in appointing 3+ leaders of EA and EA-aligned organisations is a very small set of people, many of whom would not have been willing and/or appropriate members of the search committee. I also think Max's description may have been misleading in terms of how much/what kind of experience was brought to bear during this process:

• Max has previously run the hiring rounds for two executive directors (EA Funds and GWWC). Claire participated in decision-making for both of those decisions. These were smaller, less complex processes, though.
• Max consulted ~a dozen people with more experience hiring executives.
• Max met ~weekly with an advisor who had more experience hiring executives.
• We paid an external consultant to help generate candidates and found it a bit useful, although we think we would have found most of the top candidates without their assistance.
• In terms of recruitment generally, Max and I have both been involved in dozens of hiring rounds at CEA and the others folks involved also bring significant recruitment experience (which I know a bit less about).

Still, I agree that this is still not the same as having someone who has done this activity at this scale many times actually leading the process.

If you are wondering about whether we should have hired an external firm to run the process/help us run the process, what I’d say is that when we’ve tried to hire professional firms or experts for guidance in high context decision making in the past, it has often been either unhelpful or actively harmful. My personal experience has been that their normal customers have such different desires/incentives that their guidance to us feels wildly off base. HR people, for example, assume we are in an adversarial relationship with our employees and become confused when we want to e.g. share more of certain types of information with them. Similarly, we often don’t seem to be able to get on the same page about valuing integrity or transparency for their own sake, not just the appearance of such things. So, if prior experience is applicable, I personally expect a big headhunting firm would have had some very different ideas about what makes a good CEO and we’d have spent a lot of time trying to close a large inferential gap for a small outcome difference.

All of that said, I think it's possible we should have explored what executive search firms might be able to offer in more depth. While I expect them to be of limited use in terms of candidate generation, and expect some of their guidance to be wildly off target, it's possible they might have brought advice about conducting the recruitment process or about candidate assessment that might have complimented the perspectives the committee members brought to the process. That said, given the degree to which this increases coordination costs, it seems plausible but not obvious it would have been the right call to invest effort into this.

Also pulled from Max’s quick notes:

As Caitlin says context is pretty useful and a lot of my experience trying to hire external people to do stuff has been net-unhelpful (e.g. PR firms).

• E.g. I could imagine an exec search firm recommending that we edit the job ad to be less transparent, and this putting some of our top candidates off.
• I’ve been part of hiring rounds that did use an exec search firm, and the exec search firm added little-to-no value (didn’t make much progress, wasted time)

On the more general point - does EA undervalue expertise? - I think that this is directionally true, but complicated for the reasons that Caitlin says. I think it’s a pretty complicated question when and where and how to use external experts vs. people with more context. For instance, I think ex-consultants can often be quite good at quickly gathering context, stakeholder management, and crisis management, but are generally less well-suited to being agile and building longer-term working relationships. (Obviously consultants vary a tonne, these are just patterns that I think I've seen.) I’m probably not handling this question perfectly - I think that with more experience and expertise, I’ll handle it better!

CEA has grown so much under your guidance. Your leadership has been patient, curious, and insightful. I've been immensely impressed by the humility you've shown while leading CEA, and which show in this post and this decision. I've personally learned a great deal from working with you and being managed by you. Thank you for all you've done for CEA!

I recently wrote a note in CEA's work slack and thought it might be worth writing up for the Forum, too. All of these are my own perspectives and not those of my employer.

Here’s a shower thought about evaluating people/what happens during high-pressure interactions:

• Talking to ~powerful people can be scary
• When I'm stressed or feeling judged, my reasoning and communication abilities seem to get noticeably worse, at least some of the time
• I think this is common
• And hard to control for in evaluative contexts (like job interviews)

So, if Powerful Person and Being Evaluated Person are talking, and BEP is having trouble articulating their thoughts, I often see one of three things from PP:

A. Silence
B. Quickly moving on to another topic, mentally tagging BEP’s answer as insufficient/poor
C. Putting words in BEP's mouth

• and sometimes continuing as though the answer BEP parroted was generated by BEP

I think C can be pretty hard to avoid, but avoiding it is vital.

What I try to do instead:

1b. (common tactic:) Be generally reassuring and soothing.  

• I think this tactic can be helpful, if you don’t let it lead you into B or C.

2b. (less common:) try to help them scaffold their arguments without actually providing the answer. It takes a lot of curiosity and patience. Poke, explore, ask. I don’t think I’m great at this, and I don’t have bullet-proof guidance for how to do it, but I try to:

• Reflect people’s positions back to them to check I’ve actually understood what they were trying to say
• Ask “why” a lot
• Explore reasoning chains step by step, digging in when I notice confusion or disagreement
• If they say something hyperbolic or bizarre, I check in to give them a chance to tweak what they’ve said (“All EAs should do X immediately!” “All EAs?” “Well, ok, what I really meant was. . . .”)

I want more PPs who are in evaluation mode - formally or informally! - to remember how nerves can throw noise into the interaction. I want people to be doing way more of 2b.