Yeh, I definitely agree that asking multiple questions per object of interest to assess reliability would be good. But also agree that this would lengthen a survey that people already thought was too long (which would likely reduce response quality in itself). So I think this would only be possible if people wanted us to prioritise gathering more data about a smaller number of questions.

Fwiw, for the value of hires questions, we have at least seen these questions posed in multiple different ways over the years (e.g. here) and continually produce very high valuations. My guess is that, if those high valuations are misleading, this is driven more by factors like social desirability than difficulty/conceptual confusion. There are some other questions which have been asked in different ways across years (we made a few changes to the wording this year to improve clarity, but aimed to keep the same where possible), but I've not formally assessed how those results differ. 

Thanks Vasco!

This bullet plus the other I quoted above suggest typical junior and senior hires have lifetimes of 40.2 (= 2.04*10^6/(50.7*10^3)) and 16.1 roles (= 7.31*10^6/(455*10^3)), which are unreasonably long. For 3 working-years per junior hire, and 10 working-years per senior hire, they would correspond to working at junior level for 121 years (= 40.2*3), and at senior level for 161 years (= 16.1*10).

We took a different approach to this here, where we looked at the ratio between the value people assigned to a role being filled at all and the value of a person joining the community, rather than the value of the first vs second most preferred hire.

If we look at those numbers, we only get a ratio of ~5 (for both junior and senior hires), i.e. however valuable people think a role being filled is, they think the value of getting a 'hire-level' person to the community is approximately 5x this. 

This seems more in line with the number of additional roles that we might imagine a typical hire goes onto after being hired for their first role. That said, people might also have been imagining (i) that people's value produced increases (perhaps dramatically) after their first role, (ii) that people create value for the community outside the roles they're hired to. 

Thanks for the comment Jessica! This makes sense. I have a few thoughts about this:

• More time for people to answer, and in particular to reflect, sounds like it could have been useful (though I wasn't at the event, so I can't comment on the tradeoffs here).
• My impression is that the difficulty of the survey is mostly due to the inherent difficulty of the questions we were asked to elicit judgements about (either/both because the questions were substantively difficult and required a lot of information/reflection- e.g. what is the optimal growth rate for EA- or because they're very conceptually difficult/counterintuitive- e.g. how much value do you assign to x relative to y controlling for the value of x's converting into y's), and less because of the operationalization of the questions themselves (see the confusion about earlier iterations of the questions).
  • One possible response to this, which was mentioned in feedback, is that it could be valuable to pose these questions to dedicated working groups, who devote extensive amounts of time to deliberating on them. Fwiw, this sounds like a very useful (though very costly) initiative to me. It would also have the downside of limiting input to an even smaller subset of the community: so perhaps ideally one would want to pose these questions to a dedicated group, presenting their findings to the wider MCF audiences, and then ask the MCF audience for their take after hearing the working group's findings. Of course, this would take much more time from everyone, so it wouldn't be valuable.
  • Another possible response is to just try to elicit much simpler judgements. For example, rather than trying to actually get a quantitative estimate of "how many resources do you think think each cause should get?", we could just ask "Do you think x should get more/less?" I think the devil is in the details here, and it would work better for some questions than others e.g. in some cases, merely knowing whether people think a cause should get more/less would not be action-guiding for decisionmakers, but in other cases it would (we're entirely dependent on what decisionmakers tell us they want to elicit here, since I see our role as designing questions to elicit the judgements we're asked for, not deciding what judgements we should try to elicit). 

Hey Manuel,

I think the public posts should start coming out pretty soon (within the next couple of weeks). 

That said I would strongly encourage movement builders and other decision-makers to reach out to us directly and request particular results when they are relevant to your work. We can often produce and share custom analyses within a day (much faster than a polished public post).

Many people believe that AI will be transformative, but choose not to work on it due to factors such a (perceived) lack of personal fit or opportunity, personal circumstances, or other practical considerations.

There may be various other reasons why people choose to work on other areas, despite believing transformative AI is very likely, e.g. decision-theoretic or normative/meta-normative uncertainty.

Thanks for asking ezrah. We currently plan to leave the survey open until December 31st, though it’s possible we might extend the window, as we did last time. 

I think the possibility that outreach to younger age groups^[1] might be net negative is relatively neglected. That said, the two possible reasons suggested here didn't strike me as particularly conclusive.

The main reasons why I'm somewhat wary of outreach to younger ages (though there are certainly many considerations on both sides):

• It seems quite plausible that people are less apt to adopt EA at younger ages because their thinking is 'less developed' in some relevant way that seems associated with interest in EA.
  • I think something related to but distinct from your factor (2) could also be an influence here, namely reaching out to people close to the time when they are making relevant decisions might be more effective at engaging people.
• It also seems possible (though far from certain) that the counterfactual for many people engaged by outreach to younger age groups, is that they could have been reached by outreach targeted at a later date, i.e. many people we reach as high schoolers could simply have been reached once they were at university. 

These questions seem very uncertain, but also empirically tractable, so it's a shame that more hasn't been done to try to address them. For example, it seems relatively straightforward to compare the success rates of outreach targeting different ages. 

We previously did a little work to look at the relationship between the age when people first got involved in EA and their level of engagement. Prima facie, younger age of involvement seemed associated with higher engagement, though there's a relative dearth of people who joined EA at younger ages, making the estimates uncertain (when comparing <20s to early 20s, for example), and we'd need to spend more time on it to disentangle other possible confounds.

1. ^{^}

   Or it might be that 'life stages' are the relevant factor rather than age per se, i.e. a younger person who's already an undergrad might have similar outcomes when exposed to EA as a typical-age undergrad, whereas reaching out to people while in high school (regardless of age) might be associated with negative outcomes.