All of Johannes_Treutlein's Comments + Replies

One way I imagine dealing with this is that there is an oracle that tells us with certainty, for two algorithms and their decision situations, what the counterfactual possible joint outputs are. The smoothness then comes from our uncertainty about (i) the other agents' algorithms (ii) their decision situation (iii) potentially the outputs of the oracle. The correlations vary smoothly as we vary our probability distributions over these things, but for a fully specified algorithm, situation, etc., the algorithms are always either logically identical or not.

U... (read more)

Not to trigger you, but I think by now it's probably more than 4%. The reason being America's soft power :/

Thanks for this meta-advice! Will try to adhere to it when asking for advice the next time :)

Just as a side note, Harsanyi's result is not directly applicable to a formal setup involving subjective uncertainty, such as Savage's or the Jeffrey-Bolker framework underlying evidential and causal decision theory. Though there are results for the Savage setup too, e.g., https://www.jstor.org/stable/10.1086/421173, and Caspar Oesterheld and I are working on a similar result for the Jeffrey Bolker framework. In this setup, to get useful results, the indifference Axiom can only be applied to a restricted class of propositions where everyone agrees on beliefs.

I don't think Romeo even has to deny any of the assumptions. Harsanyi's result, derived from the three assumptions, is not enough to determine how to do intersubjective utility comparisons. It merely states that social welfare will be some linear combination of individual utilities. While this already greatly restricts the way in which utilities are aggregated, it does not specify which weights to use for this sum.

Moreover, arguing that weights should be equal based on the veil of ignorance, as I believe Harsanyi does, is not sufficient, since ut... (read more)

And it turns out that the utilitarian approach of adding up utilities is *not* a bargaining solution, because it violates Pareto-optimality in some cases. Does that "disprove" total utilitarianism?

I'm not sure this is right. As soon as you maximize a weighted sum with non-negative coefficients your solution will be weakly Pareto optimal. As soon as all coefficients are strictly positive, it will be strongly Pareto optimal. The axioms mentioned above don't imply non-negative coefficients, so theoretically they are also satisfied by "... (read more)

Your argument seems to combine SSA style anthropic reasoning with CDT. I believe this is a questionable combination as it gives different answers from an ex-ante rational policy or from updateless decision theory (see e.g. https://www.umsu.de/papers/driver-2011.pdf). The combination is probably also dutch-bookable.

Consider the different hingeynesses of times as the different possible worlds and your different real or simulated versions as your possible locations in that world. Say both worlds are equally likely a priori and there is one real version of you

EV stands for Expected Value. (I think I actually meant Expected Utility more precisely)

Thanks a lot for this article! I just wanted to link to Lukas Gloor's new paper on Fail-Safe AI, which discusses the reduction of "quality future-risks" in the context of AI safety. It turns out that there might be interventions that are less directed at achieving a perfect outcome, but instead try to avoid the worst outcomes. And those interventions might be more tractable (because they don't aim at such a tiny spot in value-space) and more neglected than other work on the control problem. https://foundational-research.org/wp-content/uploads/2016/08/Suffering-focused-AI-safety.pdf

(Edit: I no longer endorse negative utilitarianism or suffering-focused ethics.)

Thank you! Cross-posting my reply as well:

If we adopt more of a preference-utilitarian view, we end up producing contradictory conclusions in the same scenarios that I discussed in my original essay—you can't claim that AMF saves 35 DALYs without knowing AMF's population effects.

Shouldn't this be fixed by negative preference utilitarianism? There could be value in not violating the "preference-equivalent" of dying one year earlier, but no value in cre... (read more)

(Edit: I no longer endorse suffering-focused ethics.)

Regardless of your stance on population ethics, I think in general it makes sense to take DALYs as a heuristic for how much good you can do with your money. Clearly all population ethical views consider improving existing lives in quality (decreasing YLDs, years lived with disability) a good thing. Preventing deaths expressed through reducing YLLs (Years of Life Lost) is probably overall good as well, although different views will assign more or less value to it. I agree with Michael Dickens that if the ... (read more)