All of EJT's Comments + Replies

You should read the post! Section 4.1.1 makes the move that you suggest (rescuing PAVs by de-emphasising axiology). Section 5 then presents arguments against PAVs that don't appeal to axiology. 

I think my objections still work if we 'go anonymous' and remove direct information about personal identity across different options. We just need to add some extra detail. Let the new version of One-Shot Non-Identity be as follows. You have a choice between: (1) combining some pair of gametes A, which will eventually result in the existence of a person with welfare 1, and (2) combining some other pair of gametes B, which will eventually result in the existence of a person with welfare 100. 

The new version of Expanded Non-Identity is then the same as ... (read more)

Here's my understanding of the dialectic here:

Me: Some wide views make the permissibility of pulling both levers depend on whether the levers are lashed together. That seems implausible. It shouldn't matter whether we can pull the levers one after the other.

Interlocutor: But lever-lashing doesn't just affect whether we can pull the levers one after the other. It also affects what options are available. In particular, lever-lashing removes the option to create both Amy and Bobby, and removes the option to create neither Amy nor Bobby. So if a wide view has ... (read more)

Thanks! I'd like to think more at some point about Dasgupta's approach plus resolute choice. 

In Parfit's case, we have a good explanation for why you're rationally required to bind yourself: doing so is best for you.

Perhaps you're morally required to bind yourself in Two-Shot Non-Identity, but why? Binding yourself isn't better for Amy. And if it's better for Bobby, it seems that can only be because existing is better for Bobby than not-existing, and then there's pressure to conclude that we're required to create Bobby in Just Bobby, contrary to the claims of PAVs.

And suppose that (for whatever reason) you can't bind yourself in Two-Shot Non-Ident... (read more)

Interesting, thanks! I hadn't come across this argument before.

Yes, nice points. If one is committed to contingent people not counting, then one has to say that C is worse than B. But it still seems to me like an implausible verdict, especially if one of B and C is going to be chosen (and hence those contingent people are going to become actual). 

It seems like the resulting view also runs into problems of sequential choice. If B is best out of {A, B, C}, but C is best out of {B, C}, then perhaps what you're required to do is initially choose B and then (once A is no longer available) later switch to C, even if doing so is costly. And that seems like a bad feature of a view, since you could have costlessly chosen C in your first choice.

Taken as an argument that B isn't better than A, this response doesn't seem so plausible to me. In favour of B being better than A, we can point out: B is better than A for all of the necessary people, and pretty good for all the non-necessary people. Against B being better than A, we can say something like: I'd regret picking B over C. The former rationale seems more convincing to me, especially since it seems like you could also make a more direct, regret-based case for B being better than A: I'd regret picking A over B.

But taken as an argument that A is permissible, this response seems more plausible. Then I'd want to appeal to my arguments against deontic PAVs.

Yes, nice point. I argue against this kind of dependence in footnote 16 of the paper. Here's what I say there:

Here’s a possible reply, courtesy of Olle Risberg. What we’re permitted to do depends on lever-lashing, but not because lever-lashing precludes pulling the levers one after the other. Instead, it’s because lever-lashing removes the option to create both Amy and Bobby, and removes the option to create neither Amy nor Bobby. If we have the option to create both and the option to create neither, then creating just Amy is permissible. If we don’t have

Ha, thanks!

Thanks, fixed now!

Thanks!

https://newscience.org/ ?

I'm quite surprised that superforecasters predict nuclear extinction is 7.4 times more likely than engineered pandemic extinction, given that (as you suggest) EA predictions usually go the other way. Do you know if this is discussed in the paper? I had a look around and couldn't find any discussion.

That all sounds approximately right but I'm struggling to see how it bears on this point:

If we want expected-utility-maximisation to rule anything out, we need to say something about the objects of the agent's preference. And once we do that, we can observe violations of Completeness.

Can you explain?

The only thing that matters is whether the agent's resulting behaviour can be coherently described as maximising a utility function.

If you're only concerned with externals, all behaviour can be interpreted as maximising a utility function. Consider an example: an agent pays $1 to trade vanilla for strawberry, $1 to trade strawberry for chocolate, and $1 to trade chocolate for vanilla. Considering only externals, can this agent be represented as an expected utility maximiser? Yes. We can say that the agent's preferences are defined over entire histories of ... (read more)

Thanks, Danny! This is all super helpful. I'm planning to work through this comment and your BCA update post next week.

I think this paper is missing an important distinction between evolutionarily altruistic behaviour and functionally altruistic behaviour.

• Evolutionarily altruistic behaviour: behaviour that confers a fitness benefit on the recipient and a fitness cost on the donor.
• Functionally altruistic behaviour: behaviour that is motivated by an intrinsic concern for others' welfare.

These two forms of behaviour can come apart.

A parent's care for their child is often functionally altruistic but evolutionarily selfish: it is motivated by an intrinsic concern for the child'... (read more)

I wouldn't call a small policy like that 'democratically unacceptable' either. I guess the key thing is whether a policy goes significantly beyond citizens' willingness to pay not only by a large factor but also by a large absolute value. It seems likely to be the latter kinds of policies that couldn't be adopted and maintained by a democratic government, in which case it's those policies that qualify as democratically unacceptable on our definition.

suggests that we are not too far apart.

Yes, I think so!

I guess this shows that the case won't get through with the conservative rounding off that you applied here, so future developments of this CBA would want to go straight for the more precise approximations in order to secure a higher evaluation.

And thanks again for making this point (and to weeatquince as well). I've written a new paragraph emphasising a more reasonable, less conservative estimate of benefit-cost ratios. I expect it'll probably go in the final draft, and I'll edit the post here to incl... (read more)

Thanks for this! All extremely helpful info.

Naively a benefit cost ratio of >1 to 1 suggests that a project is worth funding. However given the overhead costs of government policy, to governments propensity to make even cost effective projects go wrong and public preferences for money in hand it may be more appropriate to apply a higher bar for cost-effective government spending. I remember I used to have a 3 to 1 ratio, perhaps picked up when I worked in Government although I cannot find a source for this now.

This is good to know. Our BCR of 1.6 is bas... (read more)

Though I agree that refuges would not pass a CBA, I don't think they are an example of something that would be extreme cost to those alive today-I suspect significant value could be obtained with $1 billion.

I think this is right. Our claim is that a strong longtermist policy as a whole would place extreme burdens on the present generation. We expect that a strong longtermist policy would call for particularly extensive refuges (and lots of them) as well as the other things that we mention in that paragraph.

We also focus on the risk of global catastrophes,

Maybe an obvious point, but I think we shouldn't lose sight of the importance of providing EA funding for catastrophe-preventing interventions, alongside attempts to influence government. Attempts to influence government may fail / fall short of what is needed / take too long given the urgency of action.

Yep, agreed! 

Should we just get on with developing refuges ourselves?

My impression is that this is being explored. See, e.g., here.

Second, the argument overshoots.

The argument we mean to refer to here is the one that we call the ‘best-known argument’ elsewhere: the one that says that the non-existence of future generations would be an overwhelming moral loss because the expected future population is enormous, the lives of future people are good in expectation, and it is better if the future contains more good lives. We think that this argument is liable to overshoot.

I agree that there are other compelling longtermist arguments that don’t overshoot. But my concern is that governments c... (read more)

But CBA cares about marginal cost effectiveness and presumably the package can be broken into chunks of differing ex-ante cost-effectiveness (e.g. by intervention type, or by tranches of funding in each intervention). Indeed you suggest this later in the piece. Since the average only just meets the bar, if there is much variation, the marginal work won’t meet the bar, so government funding would cap out at something less than this, perhaps substantially so.

Yes, this is an important point. If we were to do a more detailed cost-benefit analysis of catastroph... (read more)

On the first, I think we should use both traditional CBA justifications as well as longtermist considerations

I agree with this. What we’re arguing for is a criterion: governments should fund all those catastrophe-preventing interventions that clear the bar set by cost-benefit analysis and altruistic willingness to pay. One justification for funding these interventions is the justification provided by CBA itself, but it need not be the only one. If longtermist justifications help us get to the place where all the catastrophe-preventing interventions that cl... (read more)

at other times you seem to trade on the idea that there is something democratically tainted about political advocacy on behalf of the people of the future — this is something I strongly reject.

I reject that too. We don’t mean to suggest that there is anything democratically tainted about that kind of advocacy. Indeed, we say that longtermists should advocate on behalf of future generations, in order to increase the present generation’s altruistic willingness to pay for benefits to future generations.

What we think would be democratically unacceptable is gov... (read more)

Thanks, these comments are great! I'm planning to work through them later this week. 

I agree with pretty much all of your bulletpoints. With regards to the last one, we didn't mean to suggest that arguing for greater concern about existential risks is undemocratic. Instead, we meant to suggest that (in the world as it is today) it would be undemocratic for governments to implement polices that place heavy burdens on the present generation for the sake of small reductions in existential risk.

Thanks for the tip! Looking forward to reading your paper.

but surely to be authorized by wider consultation

What do you mean by this?

Thanks for the comment!

There are clear moral objections against pursuing democratically unacceptable policies

What we mean with this sentence is that there are clear moral objections against governments pursuing [perhaps we should have said 'instituting'] democratically unacceptable policies. We don't mean to suggest that there's anything wrong with citizens advocating for policies that are currently democratically unacceptable with the aim of making them democratically acceptable.

Great post!

For example, maybe, according to you, you’re an “all men are created equal” type. That is, you treat all men equally. Maybe you even write a fancy document about this, and this document gets involved in the founding of a country, or something.

There’s a thing philosophy can do, here, which is to notice that you still own slaves. Including: male slaves. And it can do that whole “implication” thing, about how, Socrates is a man, you treat all men equally, therefore you treat Socrates equally, except oh wait, you don’t, he’s your slave.

Charles Mills... (read more)

I think all of these objections would be excellent if I were arguing against this claim: 

• Agents are rationally required to satisfy the VNM axioms.

But I’m arguing against this claim:

• Sufficiently-advanced artificial agents will satisfy the VNM axioms.

And given that, I think your objections miss the mark. 

On your first point, I’m prepared to grant that agents have no reason to rule out option A- at node 2. All I need to claim is that advanced artificial agents might rule out option A- at node 2. And I think my argument makes that... (read more)

There's a complication here related to a point that Rohin makes : if we can only see an agent's decisions and we know nothing about its preferences, all behavior can be rationalized as EU maximization.

But suppose we set that complication aside. Suppose we know this about an agent's preferences: 

• There is some option A such that the agent strictly prefers A+$1

Then we can observe violations of Completeness. Suppose that we first offer our agent a choice between A and some other option B, and that the agent chooses A. Then we give the agent the chance to ... (read more)

Ah I see! Yep, agree with that.

Nice point. The rough answer is 'Yes, but only once the agent has turned down a sufficiently wide array of options.' Depending on the details, that might never happen or only happen after a very long time. 

I've had a quick think about the more precise answer, and I think it is: 

• The agent's preferences will be functionally complete once and only once it is the case that, for all pairs of options between which the agent has a preferential gap, the agent has turned down an option that is strictly preferred to one of the options in the pair.

I didn’t mean to suggest it was new! I remember that part of your book.

Your second point seems to me to get the dialectic wrong. We can read coherence arguments as saying:

• Sufficiently-advanced artificial agents won't pursue dominated strategies, so they'll  have complete preferences.

I’m pointing out that that inference is poor. Advanced artificial agents might instead avoid dominated strategies by acting in accordance with the policy that I suggest.

I’m still thinking about your last point. Two quick thoughts:

• It seems like most humans aren’t consequentialists.
• Advanced artificial agents could have better memories of their past decisions  than humans.

it seems like we agree on the object-level facts

I think that’s right.

Often a lot of the important "assumptions" in a theorem are baked into things like the type signature of a particular variable or the definitions of some key terms; in my toy theorem above I give two examples (completeness and lack of time-dependence). You are going to lose some information about what the theorem says when you convert it from math to English; an author's job is to communicate the "important" parts of the theorem (e.g. the conclusion, any antecedents that the reader may no

Thanks for the comment! In this context, where we're arguing about whether sufficiently-advanced artificial agents will satisfy the VNM axioms, I only have to give up Decision-Tree Separability*:

Sufficiently-advanced artificial agents’ dispositions to choose options at a choice node will not depend on other parts of the decision tree than those that can be reached from that node. 

And Decision-Tree Separability* isn't particularly plausible. It’s false if any sufficiently-advanced artificial agent acts in accordance with the following policy: ‘if I pre... (read more)

So, you would agree that the following is an English description of a theorem:

If an agent has complete, transitive preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.

Yep, I agree with that.

I feel pretty fine with justifying the transitive part via theorems basically like the one I gave above.

Note that your money-pump justifies acyclicity  (The agent does not strictly prefer A to B, B to C, and C to A) rather than the version of transitivity necessary for the VNM and Complete Class the... (read more)

Theorems are typically of the form "Suppose X, then Y"; what is X if not an assumption?

X is an antecedent.

Consider an example. Imagine I claim:

• Suppose James is a bachelor. Then James is unmarried.

In making this claim, I am not assuming that James is a bachelor. My claim is true whether or not James is a bachelor.

I might temporarily assume that James is a bachelor, and then use that assumption to prove that James is unmarried. But when I conclude ‘Suppose James is a bachelor. Then James is unmarried’, I discharge that initial assumption. My conclusion no lo... (read more)

Two points, made in order of importance:

(1) How we define the term ‘coherence theorems’ doesn’t matter. What matters is that Premise 1 (striking out the word ‘coherence’, if you like) is false.

(2) The way I define the term ‘coherence theorems’ seems standard.

Now making point (1) in more detail:

Reserve the term ‘coherence theorems’ for whatever you like. Premise 1 is false: there are no theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other avai... (read more)

Using 'coherence theorems' with a meaning that is as standard as any, and explaining that meaning within two sentences, seems fine to me.

I would have hoped you reached the second sentence before skimming! I define what I mean (and what I take previous authors to mean) by 'coherence theorems' there.

I cite John in the post!

I’m following previous authors in defining ‘coherence theorems’ as

theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.

On that definition, there are no coherence theorems. VNM is not a coherence theorem, nor is Savage’s Theorem, nor is Bolker-Jeffrey, nor are Dutch Book Arguments, nor is Cox’s Theorem, nor is the Complete Class Theorem.

there are theorems that are relevant to the question of agent coherence

I'd have no proble... (read more)

Sounds good!

I haven't read this post yet, but it sounds like you might be interested in this paper on existential risks from a Thomist Christian perspective if you haven't seen it already.

Nice post! Consider this a vote for more summaries.

All good points, but Tarsney's argument doesn't depend on the assumption that longtermist interventions cannot accidentally increase x-risk. It just depends on the assumption that there's some way that we could spend $1 million  that would increase the epistemic probability that humanity survives the next thousand years by at least 2x10^-14.

Thanks! This is valuable feedback.

By 'persistent difference', Tarsney doesn't mean a difference that persists forever. He just means a difference that persists for a long time in expectation: long enough to make the expected value of the longtermist intervention greater than the expected value of the neartermist benchmark intervention.

Perhaps you want to know why we should think that we can make this kind of persistent difference. I can talk a little about that in another comment if so.