I figured this was important to keep separate:

I DO believe that preventing extinction is valuable and there are some projects promising enough to prioritize over work in other cause areas.

I DON'T believe that longtermism dominates all other cause areas in expectation, or that the argument you have made in favor of longtermism is convincing.

I agree that rejecting both A and B would not make sense, if you are informed of both. I think the author is wrong to treat A and B as separate decisions, when the agent knows about both in advance.

Knowing that you have the option to take bet B later fundamentally changes the considerations for bet A. As a result, we are not making 2 independent decisions (A: yes or no, and B: yes or no). We are making 4 (A, B, BOTH, NEITHER).

When considering that list, we can see that BOTH is strictly greater than NEITHER in all worlds and rule out NEITHER. We are left with A, B, and BOTH to choose from, all of which might make sense depending on the agent's choices.

At no point did I need to employ NARROW, PLAN, or SEQUENCE. I didn't even consider the probability of H, let alone whether that probability is sharp. I just considered the available options differently.

EDIT: I think this is close in effect to SEQUENCE. As a result, there might be the objection, "What if, of the 4 options, you choose B? Could you change your mind after rejecting A and then reject B as well?" To this I would say that a rational actor does not change their mind without new information. They would only choose B if they believe B > BOTH > NEITHER. Any rational actor who believes B > NEITHER would end up betting B. They would never bet NEITHER.

What might have muddied the waters:

I separately considered how I might deal with these probabilities separately, WITHOUT knowledge that one will follow the other. This is a distinct problem from the original dilemma. However, I think it's the only situation where a rational actor who follows UNSHARP might behave differently.

Without knowledge beforehand, if you hold UNSHARP, the following can happen:

You receive A, evaluate it, conclude it's optional due to UNSHARP probabilities, and reject it. Then, you are offered B, evaluate it, conclude it's optional, and reject it. You look back and think "I wish I would have known beforehand. I would have taken advantage of the arbitrage. Oh well. I guess rational actors with less information make worse decisions."

I think it is rational for an actor to hold unsharp probabilities for some hypotheses.^[1] I think it's rational to not engage in sports gambling when no arbitrage exists. My initial example was designed to connect the two.

1. ^{^}

   I haven't made my mind up on whether it's necessary to hold unsharp probabilities in theory but I'm much more confident in practice.

   When you see a new opportunity that you know very little about that might be massively valuable, using your minimally informed baseline model to direct action seems irresponsible. Upon further investigation, everything regresses to the mean.

   In the sports gambling example I gave, you should reject unless you see arbitrage because ~all available information is priced in. In the case of impact, new opportunities look more exciting than reality due to (e.g.) selection effects and stable equilibria.

   This discussion of whether or not we should have unsharp probabilities is beside the point. My argument is about whether we can have unsharp probabilities without sacrificing rationality. I believe we can.

This seems to me to be another instance of the 1% fallacy (or the 0.1% fallacy, or the 10^-18 fallacy).

In your post, you talk about being skeptical of arguments where infinities cancel. I would argue that uncanceled infinities are generally a sign of a model being applied beyond its range of efficacy.

If you start off with a rough model and extrapolate it out to get a big enough number, all you have to do is come up with a set of conditions under which that number could plausibly be achieved, no matter how improbable. Then, say "Zero isn't a probability, and you don't have enough evidence to show that the probability cancels my big number." from the start and suddenly you have a large expected value.

However, when you refine the model (considering higher order effects, more thoroughly treating counterfactuals, turning exponential curves into more accurate s-curves), the big number drops out.

The base rate for the infinite is 0. As a result, I am much more skeptical of models where infinities don't cancel.

Could you please provide any concrete grounding for the probability of counterfactually shifting from extinction to a vast future (not delaying extinction temporarily) that is not based on a very small subjectively "conservative" probability?

I agree that accepting both bets is consistent with a sharp probability at 50%, though I'm just trying to give an example of a case where I would have an unsharp probability range where I would reject both bets in isolation but take them when they arrive together.

I don't employ any of the 3 strategies. My argument is that you don't need a fancy strategy because, in the example, you know that bet B is coming when you're asked about bet A. I think it's reasonable for a rational actor to reject bet A and reject bet B if the two are presented separately but accept them both if they are presented together. My example is intended to demonstrate that. A rational actor doesn't need NARROW, PLAN, or SEQUENCE. They need to consider the future: "Bet B is coming, so there's an arbitrage opportunity regardless of the probability." The article seems to disagree, treating every action in isolation and requiring that we make the right decision without global thinking.

My recommendation for portfolios is not an argument for, but an implication of, unsharp probabilities. A lot of cause prioritization is about the core philosophical positions you hold underpinning it. If you have a sharp probability, you might be comfortable investing all in one cause. If you have an unsharp one, you might not be convinced that investing in any one cause is net positive. However, you might find a combination of causes that seems robustly better than no action.

For example, you might be concerned about climate policy's constraints on growth as well as growth's effect on the climate. If you believe that the second order effects of investing in growth on the climate are smaller than the direct benefits of donating to climate policy (and vice versa), it is strictly better to donate to both in some combination than to do nothing. Someone with a sharp probability might be comfortable donating to just one in a way someone with unsharp probabilities would not.

As a result, portfolios are better (i.e. are more often optimal) in a world where UNSHARP is true.

I see and agree with your point about marginal returns. Depending on how strong that effect is, portfolios are also good in a world with sharp probabilities only.

Related to/expandng upon @huw 's objection, you not only need to show that longtermist organizations make a difference in the probability of extinction, but that it lasts for a long time. Even if we assume that your $10 Billion would reduce the probability of extinction in 2026, we could counterfactually face extinction in 2027, or 2028, or... etc.

In effect, your calculation assumes that your reduction in existential risk lasts forever. This is a classic example of the "1% fallacy," where the tiny probability is really hard to achieve.

I would say any intervention that moves us from the path of extinction to the path of never going extinct ever for the next 10^(big number) years is exceptionally valuable. But that's a different category of interventions than simply "reducing the probability of extinction."

Even if you believe that the value of utility in the future is equal to the value of utility in the present, you should always discount by a minimum of your probability of extinction per unit time.

This has some really counterintuitive implications.

Let's consider the value of preventing extinction entirely for a year, which would be an absurdly large claim for any organization to make by orders of magnitude.

Maybe the probability of extinction per year is really high. That means that reducing it to zero is a massive change in our odds of survival (yay!). It also means that our probability that everyone will die in the next few years anyway is also large (darn!).

Maybe the probability of extinction per year is really low. That means reducing it to zero is unlikely to actually do anything (darn!). But if it does, we'll probably stick around for a long time (yay!).

It turns out these effects exactly cancel out mathematically. Using an annual probability of extinction x, the value of preventing extinction is:

That means that preventing extinction for 1 year is equal to 1 year of utility. Granted, 1 year of utility is a lot, but it is very different from 10^40 or 10^58 lives.

Using the numbers you used in your post,^[1] each dollar reduces the probability of extinction by one in a hundred trillion. We should multiply that by the value of one year of utility. I'll use 10 Billion DALYs (representing 1 year of life for ~10B people) for simplicity, which results in a cost-effectiveness of $10,000 per DALY. That's orders of magnitude worse than Global Health, your "worst" EA Cause Area.

Caveats/objections:

Accounting for population growth/improving livelihoods, the value of each year does increase over time, but unless you assume unconstrained exponential growth and a very small probability of extinction every year, that effect is relatively mild compared to the discounting and that won't shift the endline result much.

You could claim that the extinction probabilities are heavily correlated so by preventing extinction in one year you're also preventing it every year afterwards. I think there's a reasonable argument to be made that probabilities are correlated across short time spans, but I find it much less plausible across long time spans. As a result, this could shift the value up, but I don't think it resolves the underlying issue.

You could claim that the probability of extinction is particularly high this year but will drop off later. However, that requires a very specific view of probabilities that is not argued for here in your post. Even if you believe we are at a "hinge of history," you would need to make the claim that the duration of the hinge is very short and the probability of extinction after the hinge is tiny.

1. ^{^}

   I acknowledge that the numbers in your post are intended to be very conservative. I also have my own reasons to object to your choice of conceptual approach to considering what cause is the best cause area. However, I am considering your numbers as listed for the sake of argument.

This seems to be a breakdown with the consideration of actions in complete isolation rather than with having coarse probability estimates.

At least in practice, there's a clear difference between considering bet A in isolation and considering bet A when you know bet B is coming. If you told me about a sports game between the Snofuls and the Fleertis and offered me 2:1 odds on the Snofuls to win, I wouldn't take it. But if you told me you would also give me 2:1 odds on the Fleertis to win, I would take both bets, guaranteeing a profit.

As a rational actor with no useful information, I have a very broad range of potential probabilities for this bet, and it is permissible to do neither bet in isolation. However, when we consider our options simultaneously, that changes the calculus.

To apply this to altruistic action, there might be actions that we are uncertain about in isolation, but we are willing to pursue as a part of a portfolio approach.

I struggle to see practical cases where it makes sense to spend significant time on WFMs. I would rather improve cost-effectiveness analyses (CEA).

I think that is a reasonable decision. I think WFMs are very useful for certain types of decisions, but not always. I use CEAs much more often. My claim is *not* that more people should be using WFMs. If anything, my post should be seen as a warning to those who do.

My claim is that people should take time to understand their tools and account for their weaknesses. Accounting for weaknesses should happen not just within the tool, but outside of it when making the final decision.

I think GiveWell is a good example of this. If CEAs made up 100% of their decision making process, their decisions would be heavily influenced by the weaknesses of CEAs as a method. However, GiveWell acknowledges these weaknesses and uses CEAs as a primary deciding factor, while also incorporating other factors as well.

You are correct that there are ways to mitigate these issues. However, that does not mean that the issues completely disappear or that the method is without weakness.

The fundamental problem remains. Like I mentioned in my original post, any system for decision making is going to be trading away truth for practicality.

A more refined method means that some weaknesses will be less pronounced, though they frequently introduce new types of errors (like the WFM example in my post). We still need to account for methodological bias into our final decision.

You cite GiveWell as an example of an organization that takes EV estimates "close to literally". I assume by this you mean the EV estimates they make with respect to cost-effectiveness. However, GiveWell outlines 5 things they keep in mind when considering cost-effectiveness here, including the following:

Because of the many limitations of cost-effectiveness estimates, we consider other factors when recommending programs or grants. For example, confidence in an organization's track record and the strength of the evidence for an intervention generally also carry significant weight in our investigations.

In other words, GiveWell seems to believe that cost-effectiveness is a useful tool, but it is not perfect. There are methodological biases with that method, so they acknowledge those limitations and incorporate other factors before making a final decision.

I think EV is one valuable (but incomplete) metric for evaluating charities. WFMs can capture EV as well as other variables that are harder to incorporate quantitatively. However, creating BOTECs to estimate EV is a lot faster than making a full WFM. Which one to use is, in my view, a question of whether the importance of your decision justifies that extra effort or whether your time would be better spent on other decisions/work.

Regardless which one you choose, you should be careful not to rely on just the one tool. EV reasoning is vulnerable to Pascal's Mugging and the Optimizer's Curse. WFM is vulnerable to the issues I talked about in my post and more. The underlying point is that we need to supplement our tools with critical thinking to ensure we're not falling victim to their weaknesses.