The set of all possible futures is infinite, regardless of whether we consider the life of the universe to be infinite. Why is this? Add to any finite set of possible futures a future where someone spontaneously shouts “1”!, and a future where someone spontaneously shouts “2”!, and a future where someone spontaneously shouts

Wait are you assuming that physics is continuous? If so, isn't this a rejection of modern physics? If not, how do you respond to the objection that there is a limited number of possible configurations for atoms in our controllable universe to be in? 

I think, however, that longtermism has the potential to destroy the effective altruism movement entirely, because by fiddling with the numbers, the above reasoning can be used to squash funding for any charitable cause whatsoever. The stakes are really high here.

I'm worried that this will come across as a bravery debate, but are you familiar with the phrase "anything is possible when you lie?" 

I don't find it particularly problematic that sufficient numerical fiddling (or verbal fiddling for that matter) can produce arbitrary conclusions. 

Your critique reminds me of people who argue that consequentialism can be used to justify horrific conclusions, as if consequentialism had an unusually bad track record, or if other common ethical systems have never justified any terrible actions ever. 

Fwiw, while I have some sympathy with the arguments you advance (and haven't managed to form an opinion on much of it), I find the level of snark in your writing offputting.

I think that some of your anti-expected-value beef can be addressed by considering stochastic dominance as a backup decision theory in cases where expected value fails.

For instance, maybe I think that a donation to ALLFED in expectation leads to more lives saved than a donation to a GiveWell charity. But you could point out that the expected value is undefined, because maybe the future contains infinite amount of both flourishing and suffering. Then donating to ALLFED can still be the superior option if I think that it's stochastically dominant.

There are probably also tweaks to make to stochastic dominance, e.g., if you have two "games",

• Game 1: Get X expected value in the next K years, then play game 3
• Game 2: Get Y expected value in the next K years, then play game 3
• Game 3: Some Pasadena-like game with undefined value

then one could also have a principle where Game 1 is preferable to Game 2 if X > Y, and this also sidesteps some more expected value problems.

I like your comparisons with other historical cases when people thought they had inevitable theories about society, and it is a thing I think about.

I do have a pet peeve though about the following claim.

Expected values were being used by the authors inappropriately (that is, without data to inform the probability estimates).

 Let's consider a very short argument for strong longterminism (and a tractable way to influence the distant future by reducing x-risk):
- There is a lot of future ahead of us.
- The universe is large
-  humans are fragile/the universe is harsh (most planets are not inhabitable for us (yet). We don't survive in most space by default)
 ⇒ Therefore expected outcomes  of your actions for the near future become rounding errors compared to future expected outcomes by making sure humanity survives.
All three of these points (while more might be necessary for a convincing case for longterminism) are very much informed by physical theories which in turn have been informed by data about the world we live in (observing through a telescope, going to the moon)!
To illustrate:

- Had I been born in a universe where physicists were predicting with high degrees of certainty (through well-established theories like thermodynamics in our world) that the universe (all of which already inhabited) would be facing an inevitable heat death in 1000 years from now, then I would think that the arguments for longterminism were weak since they would not apply to the universe we live in.

I am not convinced by your arguments around epistemology. I don't understand your fascination with Popper. Popper's philosophy seems more like an informal way to make bayesian updates. You did not provide sufficient evidence for me to convince me to the contrary. While I agree that rigid Bayseanism has flaws, my current best guess means more subjectivism, not less.
 

Thanks for collating all of this here in one place. I should have read the later posts before I replied to the first one. Thank you too for your bold challenge. I feel like Kant waking from his 'dogmatic slumber'. A few thoughts:

1. Humanity is an 'interactive kind' (to use Hacking's term). Thinking about humanity can change humanity, and the human future.
2. Therefore, Ord's 'Long Reflection' could lead to there being no future humans at all (if that was the course that the Long Reflection concluded). 
3. This simple example shows that we cannot quantify over future humans, quadrillions or otherwise, or make long term assumptions about their value. 
4. You're right about trends, and in this context the outcomes are tied up with 'human kinds', as humans can respond to predictions and thereby invalidate the prediction. Makes me think of Godfrey-Smith's observation that natural selection has no inertia, change the selective environment and the observable 'trend' towards some adaptation (trend) vanishes. 
5. Cluelessness seems to be some version of the Socratic Paradox (I know only that I know nothing).
6. RCTs don't just falsify hypotheses, but also provide evidence for causal inference (in spite of hypotheses!)