Thanks for clarifying your thinking! It’s great to see you being responsive to community feedback in this way; for instance, the discussion about low acceptance rates should help give job seekers a more accurate picture of the landscape.

I’ve got a few thoughts to share in the spirit of offering more constructive criticism that will hopefully driving further improvements.

1. After a lot of time spent by 80K and the EA community trying to interpret the results of this question, we’re at a place where:

we do not think the precise numbers are a reliable answer to decision-relevant questions for job seekers, funders, or potential employers. We think it’s likely that mistakes are driving up these estimates. Even ignoring the high probability of mistakes, the implications of the data depend heavily on exactly what is driving the results. We are very uncertain about the magnitude of various considerations, so we recommend against leaning on these numbers when making career decisions.

So why not just ditch this question in favor of something more helpful, instead of continuing to pour more resources into it? If you look at things from a cost-benefit perspective, this question clearly has a high cost due to all the clarification its required. Is the benefit high enough to warrant that? Even if you worked out all the kinks and knew all the respondents were thinking about the question the same way, and investing the time to produce an answer they trusted, what do we really learn by asking: “For a typical recent Senior/Junior hire, how much financial compensation would you need to receive today, to make you indifferent about that person having to stop working for you or anyone for the next 3 years?” Which parties would take which different actions on the margins because of that information?

2. I suspect one of the reasons why people are having trouble interpreting this question is because it doesn’t correspond to a real world decision people have to make. Confusing hypothetical questions are likely to produce non-actionable results. I suggest thinking carefully about exactly what you’re trying to capture (the value of an employee? The urgency and/or difficulty of hiring?) and looking at whether you can measure that dynamic via a clearly labeled Likert scale (e.g. a 1-7 scale rating with values like “employee is transformative in allowing us to do important new things”, “employee lets us do what we currently do, but a tiny bit better”, “it would be very easy to replace an employee of this type”, etc.) I’ve posted some specific suggestions in a separate comment.

And relatedly, I’d recommend as strongly as possible not using this question (or any question involving genies). I think its very likely to lead to the same sorts of interpretation problems.

Below is a very rough draft of one version of the question we are considering asking. We hope it would be more decision-relevant than the question used above, but we haven’t yet had time to pilot it or vet it for any issues:

Imagine that sometime in the next year you are about to hire your next junior (senior) hire. A genie appears and offers you the following choice. You can have one of the following:

1. The genie will create a person and applicant for the job from thin air. They will be as much more productive (in % terms) than the next best applicant in the pool, as your last junior (senior) hire appeared to be at the point when you were evaluating whether to hire them. This person will live out the rest of their life like any other staff member, and may well go on to do other useful work outside of your organisation later on. You should consider the benefits of that for the world as well.

2. The genie will distribute $X among whichever organisations or people you nominate – which can include you and your organisation – to be used to improve the world as much as possible. Consider all the benefits for the world this would generate.

At what value of X would you be indifferent between these two options?

In addition to asking EA organizations more questions about their talent wants/needs, it’d be nice to get more information about their funding gaps. I suggest asking organizations how much they’d like to raise over 1,3, and 5 year timeframes, and also asking them to rate how difficult they found their last fundraising round.

Great to hear you’re considering new questions for the talent survey! I’ve suggested some specific questions to help get a better understanding of how much EA organizations are paying.

As 80K puts it, “Skill bottlenecks are a matter of degree” and these degrees can vary significantly depending on the specific skills in question. But we have little hard data to quantify skill gaps across various areas.

I suggest adding new questions to the next talent survey of EA organizations to help capture some of these nuances.[9] These questions will hopefully make it easier to answer questions like: Are EA organizations talent constrained? If so, which sorts of organizations and which sorts of talent? How large are these constraints? What can be done about them?

New questions (which should continue past surveys’ practice of asking the same questions about both a junior and senior hire and anonymizing organizational responses due to the sensitivity of the information involved):

● Generally speaking, how easy/difficult do you currently find it to fill roles at your organization? (Scale of 1 = Very difficult to 5 = Very easy)

● What do you pay current employees relative to what they could earn on open market, including jobs in the for-profit sector? (Multiple choice: More; about the same; 0-10% less; 11-20% less, etc)

● What do you pay current employees relative to what they could otherwise earn in the nonprofit sector (including all nonprofits not just EA organizations)? (Multiple choice: More; about the same; 0-10% less; 11-20% less, etc)

● What do you pay current employees relative to what they could otherwise earn at another EA organization? (Multiple choice: More; about the same; 0-10% less; 11-20% less, etc)

● How much do you plan to offer future hires relative to current employees in similar roles? (Multiple choice: More; Less; About the same. If someone answers “more”: Do you plan to increase salaries for existing employees? What factors into this decision?[10]

● Would any of the following steps be helpful in closing your organization’s talent gaps? (Rate the following options on a Scale of 1 =Not at all helpful to 5 = Extremely helpful):

○ Increasing salaries

○ Investing in recruiting

○ Increased publicity of position

○ Better access to interested candidates

○ Other (please describe)

This information is relatively easy to collect and interpret, corresponds to a real-world decision organizations make (how much to pay), is grounded in observable data (market wages), and is comparable across roles, organizations, geographies, and causes.[11] To mitigate the cost of data collection, I suggest abandoning survey questions about non-traditional labor metrics that seem to be producing noisy data and to generally be causing confusion.[12] If we collect compensation data and respondents indicate which types of roles they’re thinking about as junior and senior (which will vary across organizations), we’ll have a rich and actionable picture how EA compensation stacks up to the competition for various types of skills.

Possible ambiguity in the survey question: If the person stops working "for you or anyone for 3 years" that plausibly negates most of their life's impact, unless they find a great way to build career capital without working for anyone. So with this interpretation, the answers would be something close to the NPV of their life's impact divided by 3 (ignoring discounting).

Also, did you control for willingness to accept vs pay?

Sorry if this addressed, I skimmed the post.