This makes me think of the contrast between systems analysis and net assessment/strategy.  Yes, Fermi calculations are a valuable input into the discussion, but the nature of the problem is likely too complex to give sole weight to that calculation (in most circumstances - I think your example of comparing different research papers is a relatively simple/constrained environment).  

In net assessment/strategy, the nature of the choice and your assessment of that nature determine the best methods of analysis.   In systems analysis, a one-size-fits-all approach of measurement and analysis is taken.  In certain circumstances that method of measurement is hugely valuable (why it became so popular in the US DoD), but in many other circumstances (like the Vietnam War) that method is deeply flawed.  

It takes a solid assessment of the problem space to identify the key aspects of the competition.  From there you can assess which methods and measures are most useful to judge the decision upon.  And from there you can search for the data available to match your assessment.  

Although choosing a career isn't a competition against a competitor, your search to do the most good is a clearer objective than most adversaries can articulate regarding their objectives for long-term competition.  The same meta-methods of analysis (to determine the best methods of analysis to fit the problem) apply.  And then you can go on to finding the right data to fit those methods you've deemed most effective (maybe a Fermi calculation, but maybe 5 expert opinions in your career field or maybe your own gut feeling).  I also think that in a career decision (unless it's super major and clearly between two choices), the data you are going to be looking for is closer to a startup's lean experiment than a longitudinal study.  As you mentioned, Fermi calculations being readily available and usable doesn't mean that is an effective tool for your decision.

Looking back at my answer, this didn't really answer your question.   The most effective technique I have found for deciding career choices though is to talk to as many  career-related people as you can - because 1. they'll be able to help you better understand the nature of the decision; and 2. they'll open up doors for future opportunities, potentially even avoiding the need to make your current decision.

Firstly, Monte Carlo simulations (such as on https://www.getguesstimate.com/ ) is likely more precise/useful than pure Fermi estimates, as they make uncertainty, including your consideration of the value possibly being off by factor 10, explicit and thus you can have greater confidence in the results. One advantage Fermi estimates definitely have is that they force you to think about the different components of the problem, or in this case how different careers contribute to your impact. But they are generally speaking primarily helpful to estimate orders of magnitude, and are thus not all that useful in comparing different options unless they lie very far apart.

Secondly, I agree with shawzach that it makes a lot of sense to talk your career considerations through with other people. I used EAGx Virtual for that purpose and looked for all the people who might have something to contribute to my career considerations. But then again, it differed from your situation in that I didn't have several alternatives but rather one preexisting plan and wanted to figure out whether it was "good" or I should look further for alternatives. Still, people might be able to add a few crucial considerations or just arguments you hadn't thought of before that affect your estimates.

I recently made a big career change, and I am planning to write a detailed post on this soon. In particular, it will touch this point.

I did use use Fermi calculation to estimate my impact in my career options.
In some areas it was fairly straightforward (the problem is well defined, it is possible to meaningfully estimate the percentage of problem expected to be solved, etc.). However, in other areas I am clueless as to how to really estimate this (the problem is huge and it isn't clear where I will fit in, my part in the problem is not very clear, there are too many other factors and actors, etc.).

In my case, I had 2 leading options, one of which was reasonable to amenable to these kind of estimates, and the other - not so much. The interesting thing was that in the first case, my potential impact turned out to be around the same order of magnitude as EtG, maybe a little bit more (though there is a big confidence interval).

All in all, I think this is a helpful method to gain some understanding of the things you can expect to achieve, though, as usual, these estimates shouldn't be taken too seriously in my opinion.

Thanks for asking this! I'm looking forward to reading discussion of it.  I feel similar to you, I think. I'm trying to decide between career options in global health, health security, and  global catastrophic biological risk reduction (GCBR). There's a lot of different inputs, both personal and external, but one aspect I've struggled with is the tension between being mostly convinced of the arguments for GCBR work (and trusting the many smart people convinced by them) and feeling the probabilities of me making a difference on low-probability/high-consequence events are too small when multiplied together. 

Regardless of whether the math 'works out' in a  Fermi calculation of a GCBR career (whether by me or others), it still feels sort of 'thin' to base a major career change on. 

Here's an example to try to capture the feeling of thinness (or fragility): it seems plausible that a few papers (or blog posts) might come along that are devastatingly clever, featuring arguments or evidence I hadn't thought of, that showed with high confidence that the risk of synthetic pandemics is extremely low (this specific example might not be plausible, but it captures my psychology at least). If that came along after I'd spent 20ish years narrowly focused on synthetic pandemic risk, without major transferable career capital, I'd feel like I'd made a mistake (if not ex ante, at least ex post).

The reduction in importance wouldn't necessarily have to be dramatic for it to be consequential for individual career choices . If, given my previous experience, it's a close race between my options in terms of projected ethical impact, a more minor reduction could reveal my eventual choice was actually a distant second. Also, the reduction wouldn't necessarily have to deflate the entire problem - arguments might have revealed that  people similar to me in the field actually has vanishingly little (or negative) impact on the issue. 

Traditional career choices seem to be based more on personal preference , social connection, tradition, or chance. And there's plenty wrong with those approaches. One positive however is that those ways of making choices are more resistant to this kind of intellectual deflation by others. 

Of course, there's also plenty of regret and uncertainty in traditional careers and career-choices.  It's not clear to me whether this feeling of 'thinness' is just a bias I need to work through, or is actually tracking something important. And, like you say, it's not clear what we should do otherwise.