Ah no, we didn’t, though results from a perceived-probability question may have been interesting. Whatever participants believed about their chances should be the same across arms, so I'm confident in the relative ranking of modules even if the absolute $10 figure may be an upper limit.

No, they were told 20 people would be selected to receive their choice for real, but not how many people were in the study overall.

Amazing to see! Publishing your full costs, preregistering the next round and committing to track non-attendees is fantastic, and I hope other talent programmes follow suit.

Fair point on the “massively oversubscribed” condition. Historically you didn’t have a big pool of near misses, but with a ~15% acceptance rate in 2026, it's likely you do now? A few low-cost steps taken at the point of selection would make future comparisons much stronger:

• Record applicant scores. If reviewers give each applicant a numeric score (even a rough one), you can later compare people just above and just below the cutoff. That gets you close to a regression discontinuity design without changing who you admit. It would also show, as OscarD says, whether your selection judgements predict career outcomes.
• Measure both groups the same way. Like you said, the response rate for the comparison group will probably be lower. Still worth doing, but I’d make sure to code current role from public profiles for everyone (both alumni and comparison group), ideally by someone who doesn’t know which group each person is in.
• Check the dropouts are comparable before relying on them. Illness is plausibly random. Visa problems probably aren’t: they will cluster by nationality, and therefore by region, and you’ve shown that region strongly affects outcomes. Comparing visa dropouts with alumni from the same bootcamp, or at least the same region, would deal with most of that.

The comparison group also gives you a way to test the self-reported counterfactual. Your respondents imply that about half of their AI safety work is down to ML4Good (0.53). If waitlisted applicants turn out to be working in AI safety at roughly half the rate of alumni, the self-reports hold up well! If waitlisted applicants are nearly as likely to be in AI safety, that you can calculate how much to discount the self-report. (Useful for every programme in the field.)

Really glad to see this. Happy to look over the comparison design if useful.

Good to see this open. I’ve put some thoughts in a quick take on what my previous evaluation of Blackfish suggests about when films lead to change. If any grantees want help working out how to evaluate their film’s impact, I’d be glad to talk it through.

With Who Let the Docs Out now taking applications, here are a couple of quick thoughts from an evaluation of Blackfish.

Blackfish is the film everyone points to. A few years ago I looked at whether it actually caused SeaWorld’s troubles, using stock market data and interviews with 26 people from marine parks, zoo associations, animal welfare NGOs, the film crew, marine science and narrative research. (Blackfish paper; plain-English write-up in The Conversation). We used General Elimination Methodology to test the competing explanations. Some didn’t hold up. In 2013 SeaWorld’s Orlando and California parks lost visitors while Universal and Disney gained them, so tighter household budgets can’t explain the drop.

Blackfish did contribute to the fall in attendance and share price, the redesigned orca show and the end of the breeding programme. Interviewees put its impact down to four things:

• Distribution. CNN aired it roughly 40 to 50 times, and the CNN branding made it read as news to a lot of viewers.
• Content. Empathy for Tilikum, plus anger at a company that had not been straight with the public.
• Timing. It landed on decades of campaigning, along with Free Willy, The Cove and the book Death at SeaWorld, and campaigners were ready to use it.
• SeaWorld’s own response, which was slow and defensive.

Some advice for farmed animal documentary filmmakers:

1. Ask who will pick the film up and use it. Blackfish had a single company to target and campaigns ready to go.
2. Budget for distribution as seriously as production.
3. Views and social engagement tell you about reach. Plan from the start how you’d find out whether anything changed, even if the answer is qualitative.

Happy to talk to any filmmakers or funders thinking about evaluation.

Nice to see the causal chain out laid out!

The counterfactual impact of (1) is that individuals "do good" they otherwise wouldn't have, but not necessarily that more good is done in the world overall. That relies on the pool being talent-constrained, so jobs would either sit empty, or be filled by demonstrably less effective candidates.

I would lean to the employer-side evaluation. Might be the costly option, but it's more informative, and could potentially be done via asking hiring managers at the point of hire, who else was in the final set and what they would have done if this candidate had not applied. That's close to a one-question survey. 

Also worth surveying hiring managers on what are the biggest blockers - e.g., contextual knowledge? Specific skill sets? Could be useful for lighter touch evaluations of the programme down the road (i.e., are participants actually leaving with better contextual knowledge?)
 

I'm less sure about putting a money value on months saved early on - quite a few assumptions would be involved. 

I would definitely recommend pre-registering the outcome definitions and the analysis before anyone sees the data, timestamped somewhere public. Ideally the outcome coding would also be done by someone blind whether participants were part of the cohort or not.

I think the wartime framing actually argues for the retrospective design - you wouldn't need to randomise anything, ask a mentor to take a fellow they did not choose, or make a single cohort worse.

Ah, very fair point! Screening is a real service to employers, and even if programmes produced no counterfactual placements at all they may still save the ecosystem a lot of search cost. It is measurable too, at least crudely, by asking hiring managers what they would have spent finding the same person.

I keep coming back to this question too. The whole case for spending heavily on talent pipelines rests on the field being talent constrained rather than absorption constrained, and I haven't seen public tests of that assumption. I think it would make another super interesting study, and is very testable! Vacancy durations, re-advertisement rates, applicant to hire ratios by role type, and hiring managers asked directly why a search failed would get you a long way.

I'm tempted to look at this properly as a follow up, so if anyone reading has hiring data they would be willing to share, or strong priors in either direction, I'd love to hear from you!