SamiPetersen

Wiki Contributions

Comments

LSE EA’s Fellowship Application Scores Moderately Predicted Engagement and Discussion Quality

Thanks for writing this!

This is very reasonable; 'no predictive power' is a simplification.

Purely academically, I am sure a well-reasoned Bayesian approach would get us closer to the truth. But I think the conclusions drawn still make sense for three reasons.

  1. I did not specify in the table, but the p-values for the insignificant coefficients were very high; often around p=0.85. I think this constitutes so little evidence that it would be too minor a Bayesian update to have to formally conduct.
  2. Given that we do have evidence of some other variables being predictive, updating in favour of weighting those higher still makes sense (although maybe to a lesser degree than I implied in the post).
  3. The time applicants and facilitators spend on the many different criteria we used is a cost (and a meaningful one for smaller groups). I would guess that cutting down the number of variables used would increase productivity more than what can be outweighed by the small updates we could make with little (but non-zero) predictive power.
LSE EA’s Fellowship Application Scores Moderately Predicted Engagement and Discussion Quality

Thanks for the comment! 

I think it's completely plausible that these two measures were systematically measuring something other than what we took them to be measuring. The confusing thing is what it indeed was measuring and why these traits had negative effects.

(The way we judged open-mindedness, for example, was by asking applicants to write down an instance where they changed their minds in response to evidence.)

But I do think the most likely case is the small sample.