Thank you so much for this paper! I literally made a similar argument to someone last weekend (in the context of economic growth), glad to have a canonical/detailed source to look at so I can present more informed views/have an easy thing to link to.
I will read the rest of the paper later, but just flagging that I don't find your response to "incomplete understanding of physics" particularly persuasive:
Perhaps our understanding of physics is incorrect. That is, it is possible that our understanding of any of the assumed correct disciplines discussed here, from cosmology to computation. This is not merely an objection to the authors’ personal grasp of the subjects, but a claim that specific premises may, in the future, be found to be incorrect.
I think the strongest version of the "we don't understand physics" argument is that we (or at least I) have nonzero credence in physics as we know it to be mistaken in a way that allows for infinities. This results in an infinite expected value.
Now, perhaps we can exclude arbitrarily exclude sufficiently small probabilities ("Pascal's mugging"). But at least for me, my inside-view credence in misunderstanding the finitude of physics is >0.1%, and I don't think Pascal's mugging exceptions should be applicable to probabilities at anywhere near that level.
Michael Dickens has a different issue where finite distributions can still have infinite expected value, but I have not read enough of your paper to know if it addresses this objection.
Thanks David, this looks like a handy paper!
I don't agree with the argument that infinite impacts of our choices are of Pascalian improbability, in fact I think we probably face them as a consequence of one-boxing decision theory, and some of the more plausible routes to local infinite impact are missing from the paper:
The main reason for taking the simulation hypothesis seriously is the simulation argument, but that argument needs to assume that our physical models are broadly correct about reality itself and not just the "physics" of the simulation. Otherwise, there would be no warrant for drawing inferences from simulated sense data about the behavior of agents in reality, including whether these agents will choose to run ancestor simulations.
There is some effect in this direction, but not a sudden cliff. There is plenty of room to generalize, not an in. We create models of alternative coherent lawlike realities, e.g. the Game of Life or and physicists interested in modeling different physical laws.
Thanks, and thanks for posting this both places. I've responded on the lesswrong post, and I'm going to try to keep only one thread going, given my finite capacity to track things :)