This makes me think of the contrast between systems analysis and net assessment/strategy. Yes, Fermi calculations are a valuable input into the discussion, but the nature of the problem is likely too complex to give sole weight to that calculation (in most circumstances - I think your example of comparing different research papers is a relatively simple/constrained environment).
In net assessment/strategy, the nature of the choice and your assessment of that nature determine the best methods of analysis. In systems analysis, a one-size-fits-all approach of measurement and analysis is taken. In certain circumstances that method of measurement is hugely valuable (why it became so popular in the US DoD), but in many other circumstances (like the Vietnam War) that method is deeply flawed.
It takes a solid assessment of the problem space to identify the key aspects of the competition. From there you can assess which methods and measures are most useful to judge the decision upon. And from there you can search for the data available to match your assessment.
Although choosing a career isn't a competition against a competitor, your search to do the most good is a clearer objective than most adversaries can articulate regarding their objectives for long-term competition. The same meta-methods of analysis (to determine the best methods of analysis to fit the problem) apply. And then you can go on to finding the right data to fit those methods you've deemed most effective (maybe a Fermi calculation, but maybe 5 expert opinions in your career field or maybe your own gut feeling). I also think that in a career decision (unless it's super major and clearly between two choices), the data you are going to be looking for is closer to a startup's lean experiment than a longitudinal study. As you mentioned, Fermi calculations being readily available and usable doesn't mean that is an effective tool for your decision.
Looking back at my answer, this didn't really answer your question. The most effective technique I have found for deciding career choices though is to talk to as many career-related people as you can - because 1. they'll be able to help you better understand the nature of the decision; and 2. they'll open up doors for future opportunities, potentially even avoiding the need to make your current decision.
Thanks for asking this! I'm looking forward to reading discussion of it. I feel similar to you, I think. I'm trying to decide between career options in global health, health security, and global catastrophic biological risk reduction (GCBR). There's a lot of different inputs, both personal and external, but one aspect I've struggled with is the tension between being mostly convinced of the arguments for GCBR work (and trusting the many smart people convinced by them) and feeling the probabilities of me making a difference on low-probability/high-consequence events are too small when multiplied together.
Regardless of whether the math 'works out' in a Fermi calculation of a GCBR career (whether by me or others), it still feels sort of 'thin' to base a major career change on.
Here's an example to try to capture the feeling of thinness (or fragility): it seems plausible that a few papers (or blog posts) might come along that are devastatingly clever, featuring arguments or evidence I hadn't thought of, that showed with high confidence that the risk of synthetic pandemics is extremely low (this specific example might not be plausible, but it captures my psychology at least). If that came along after I'd spent 20ish years narrowly focused on synthetic pandemic risk, without major transferable career capital, I'd feel like I'd made a mistake (if not ex ante, at least ex post).
The reduction in importance wouldn't necessarily have to be dramatic for it to be consequential for individual career choices . If, given my previous experience, it's a close race between my options in terms of projected ethical impact, a more minor reduction could reveal my eventual choice was actually a distant second. Also, the reduction wouldn't necessarily have to deflate the entire problem - arguments might have revealed that people similar to me in the field actually has vanishingly little (or negative) impact on the issue.
Traditional career choices seem to be based more on personal preference , social connection, tradition, or chance. And there's plenty wrong with those approaches. One positive however is that those ways of making choices are more resistant to this kind of intellectual deflation by others.
Of course, there's also plenty of regret and uncertainty in traditional careers and career-choices. It's not clear to me whether this feeling of 'thinness' is just a bias I need to work through, or is actually tracking something important. And, like you say, it's not clear what we should do otherwise.