Great ideas! 
++ for "scouting, not converting" and cause-area-specific outreach. Do you have ideas on how to build the funnel here?
Have you also considered working with local funders (not necessarily EA-adjusted?)
Side note: I believe School of Moral Ambitions (early attempts with local circles) has some aspects of "captains school"

What would you recommend for evaluating general high-impact career courses?

Say, the course aims to bring new, qualified people to hard-to-fill high-impact roles (or founder paths) and improve their chances of getting hired. 

The program's target is framed as:
(1) "qualified people start pursuing HI roles who otherwise wouldn't have, or make this transition faster" 
(2) "these candidates are ready to be hired"

(1) More/faster HI transitions: we have quite weak evidence - self-reported "I wasn't aware of these roles or founder paths / wasn't trying these before the program." But people have multiple sources of information and might have found these paths anyway.

(2) I'd like to look at this from the employer side.
additional qualified candidates → better/faster hiring → value for HI orgs

On better/faster hiring for hard-to-fill roles, I'd want to estimate something like:
months saved + quality uplift of the hire
- value of months saved = HR costs saved + the new hire starts to bring value earlier ("value of the role" defined by the hiring orgs). 
- quality uplift of the hire = the new hire brings more value (defined by the hiring orgs, again)

Let's say we have historical time-to-fill & hire quality data for similar roles, and we could identify the trade-offs made in the past (a less-qualified candidate was eventually hired).

But how to measure the program impact? 
- We could gather data on the alumni hired for roles identified as "hard-to-fill". If we have all qualified/near-missess applications & timestamps, we could check whether the hiring manager had another equally qualified candidate in this group at the target time. And we could check for trade-offs - there should be fewer if the program works.
That seems doable but costly (get data & gather additional info from hiring orgs).

- Or we could try to analyze all applications for "hard-to-fill" roles (not necessarily filled by the program alumni) - to check whether the program expanded the pool of qualified candidates  (if program alumni were equally qualified/near-misses). So we'll see whether the program actually adds qualified candidates to the funnel. It's even more costly. 
But it still doesn't answer whether the program saves time / improves hire quality compared to the existing candidate pool.

I guess we could see whether we could save time-to-hire: if the program's alumni appear in the funnel earlier than equally qualified candidates. But if they weren't eventually hired, it seems harder to argue that this time saving was actually valuable to the hiring org.

So how should we answer this question - is the pool enlargement worth the money?

Sorry, rough thoughts, and I have no proper MEL experience.