Comments
Thanks to @eigengender and especially Chris Lakin and Simon Dima for valuable comments on a draft.
In this post, I will present an alternate framing[1] of LessWrong-style decision theory. I believe it should make its concrete recommendations much more palatable to people who find them absurd. Very little of the theoretical substance here is original to me, and the core claim will be obvious to many, but I don’t think anyone has cleanly framed it all like this for easy consumption.
TLDR: Framing the thorny issues of decision theory[2] as stemming from a novel, non-decision-theoretic consideration (something like “cooperating with your past self”) - instead of forcing the relevant intuitions and claims into a decision theory frame - is much less counterintuitive, and gives the exact same recommendations.
Let’s briefly check out Will MacAskill’s “Bomb*” scenario[3]:
You’re alone at home at the end of the universe. Omega - an amazing predictor of human behavior, but long since dead - gave you one last challenge. You face two boxes, Left and Right. You must take one of them.
In the Left box, Omega put either nothing or a live bomb; if there’s a live bomb, taking this box will set it off, setting you ablaze, and you will burn slowly to death. In the Right box, Omega put a glitter bomb; taking this box will set it off, covering you and your whole living room in glitter, and you’ll have to spend hours cleaning it up - very inconvenient.
Both boxes are transparent, so you can actually see the contents of the Left box - there is a live bomb.
Omega predicted whether you would choose Left or Right when faced with exactly this situation - of seeing a bomb in the Left box. If he predicted that you would choose Right, he put a bomb in Left (like you’re seeing now). If he had predicted that you would choose Left, he wouldn’t have put a bomb in Left, and the box would be empty.
You know that Omega has a failure rate of one in a trillion trillion in this situation.
You are the only person left in the universe. You have a happy life, but you know that you will never meet another agent again, nor face another situation where any of your actions will have been predicted by another agent. What box do you choose?
Functional decision theory (FDT), the main LessWrong-style decision theory, recommends taking the Left box - in the full knowledge that as a result you will slowly burn to death. Why? Because if your decision process were to output ‘Left’, then Omega would have predicted that. So there would be no bomb in the box, and you could save yourself the effort of having to clean by taking the Left box[4] - even though that would mean changing the past.
Proponents also sometimes say, by putting the same math into words differently[5], that FDT chooses to take the Left box and burn to death because it imagines that you must be in the reality where Omega made a mistake, and that you can currently affect 999,999,999,999,999,999,999,999 other versions of you and cause them to not have to clean - even though we have no reason to think these other versions exist.
Many people find this kind of recommendation, and either of these framings, extremely implausible - both MacAskill and Bentham’s Bulldog see it as a knock-out blow to FDT, and Wolfgang Schwarz, an academic decision theorist who was a referee of the FDT paper, calls it “insane”. LessWrongers, on the other hand, insist that this way of thinking actually makes sense.[6]
The debate has landed at an impasse - both sides have planted their flags and refuse to budge from their deeply held intuitions.
But I believe that there is a way out.
Many people have made the observation that LessWrong-style decision theory is trying to do something different, in some deep worldview-level way, from academic decision theory - and that this is what leads to the persistent and seemingly intractable disagreement between the two camps.
For example, Paul Christiano:
I think a lot of [academic decision theorists not being interested in LessWrong-style decision theory] is this semantic disagreement / this understanding of "what is the project of decision theory?"…like a difference in, "What are the questions that are interesting, and how should we use language?" Like, "What do concepts like 'right' mean?"
I think that there is this (obvious to LessWrongers, because it is deeply entangled with the entire LessWrong philosophy) ontology…that most decision theorists haven’t really considered.
FDT is not playing the same game as CDT or EDT... So it’s odd to have a whole paper comparing them side-by-side as if they are rivals.
So… let’s run with this. Let’s assume that LessWrongers were wrong all along when they thought they were doing decision theory. Let’s come up with a different name for whatever it is that they’ve been doing: call it commitment theory. And let’s reexamine the topic with a fresh eye.
Let’s step back from the extremity of “Bomb*” for a minute - take a simple blackmail scenario:
Omega, a near-perfect predictor, knows a piece of embarrassing information about you. He threatens you that he will release it to the public if you don’t pay him 1,000,000$. However, making the threat is slightly costly to him, and he wouldn’t have done it if he hadn’t predicted you would pay.
Everyone agrees that if you can precommit (somehow force your future self) to not pay before you get into this situation, you should - since that would make it so Omega won’t blackmail you and you don’t get into this situation in the first place.
So there’s a class of situations where you would like to previously commit yourself to a choice, to get better outcomes overall. That’s interesting.[7]
So maybe we should do a kind of "generalized precommitment" for those situations, where we commit now to always doing whatever it would have been good to have precommitted to?
Yeah - it seems like, by definition, it would be good to do that. It’s not clear yet what exactly it means for something to be “good to have been precommitted to”, but… we can figure out the details later. I mean, this is tautological: It just has to be good to precommit to… the actions that would have been good to have precommitted to.
In fact, we should do this ASAP, before we end up in such a situation. So let’s do it right now, together, on three: one... two... three!
Well done! Let’s call whatever we just did, this “generalized precommitment”, the great commitment.
What does it mean concretely, that we made this “great commitment”? Well, we’ll get better outcomes if we get put into the blackmail scenario above in the future - so that’s good.
What about other cases? What about, say… Parfit’s Hitchhiker?
You’re in the desert, running out of water and soon to die, when Omega, a near-perfect predictor, finds you. You don’t have any money on you - but he’s not going to save you for free. He’ll only bring you to the city if he predicts you’ll pay him 1,000$ upon reaching it. But once he brings you to the city, Omega can’t force you to pay.
Omega brings you to the city. You are standing in front of the ATM.
Do you pay?
In the past, we obviously wouldn’t have paid, since it would lose us 1,000$ for no reason. But now, as commitment theorists, we think:
Oh man… I wish I could save the 1,000$ by just not paying. But… in advance, I probably would have precommitted to paying, since otherwise Omega wouldn’t save me. So I guess this is covered by the great commitment, and I have to pay.
Darn, I really wish I could just leave and not pay… but it just doesn’t feel “right” somehow. And not even in a moral sense, just that… I can’t imagine doing it? Even though it’s obviously irrational. I guess this is just the kind of guy I am, ever since I made the great commitment.
Actually, I guess that’s why I’m even here and Omega saved me in the first place. Phew, thank God I happened to make the great commitment before I went into the desert.
Okay, that’s interesting. It seems like this might be pretty useful!
But what does this all mean? What did we just commit ourselves to, exactly?
That’s exactly what commitment theory research consists of - the study of what situations are covered, and how much, by our vague intuitive notion of the “great commitment”.
And since it seems like the great commitment is tautological and like all agents should make it, commitment theory research seems like an important part of the study of ideal decisionmaking - even if it may be distinct from decision theory as such. One might say that decision theory is concerned with what choices are rational, and commitment theory with how, precisely, we should force our future selves to be irrational.[8]
So how does this relate back to LessWrong-style decision theory?
Well, you might have guessed where this was going already - sorry for tricking you into it. When you made the great commitment earlier, you actually, for all intents and purposes… became an FDT agent.[9] Congratulations!
Yes - commitment theory gives the same concrete recommendations as FDT[10], seemingly without requiring any big metaphysical claims about changing the past or alternate realities[11], or any big normative claims about seemingly absurd choices actually being “rational”.
(The only normative claim is that you should make the great commitment ASAP, if you haven’t already - again, it’s tautologically good!)
This also finally gives us a way of reinterpreting some of the complicated technical machinery[12] that people have come up with over the years in their study of LessWrong-style “decision theory” - all this kind of stuff:
In effect, it’s exactly the commitment theory research that I referred to before: trying to pin down and formalize our intuitive notion of “what would have been good to have precommitted to” - trying to figure out what exactly the great commitment covers.
Lastly, let’s reexamine “Bomb*” from our new perspective:
You’re alone at home at the end of the universe. Omega - an amazing predictor of human behavior, but long since dead - gave you one last challenge. You face two boxes, Left and Right. You must take one of them.
In the Left box, Omega put either nothing or a live bomb; if there’s a live bomb, taking this box will set it off, setting you ablaze, and you will burn slowly to death. In the Right box, Omega put a glitter bomb; taking this box will set it off, covering you and your whole living room in glitter, and you’ll have to spend hours cleaning it up - very inconvenient.
Both boxes are transparent, so you can actually see the contents of the Left box - there is a live bomb.
Omega predicted whether you would choose Left or Right when faced with exactly this situation - of seeing a bomb in the Left box. If he predicted that you would choose Right, he put a bomb in Left (like you’re seeing now). If he had predicted that you would choose Left, he wouldn’t have put a bomb in Left, and the box would be empty.
You know that Omega has a failure rate of one in a trillion trillion in this situation.
You are the only person left in the universe. You have a happy life, but you know that you will never meet another agent again, nor face another situation where any of your actions will have been predicted by another agent. What box do you choose?
As a commitment theorist, your thoughts in this situation might now go along these lines:
OH MY GOD THAT’S AN ACTUAL BOMB. Holy fuck. Am I going to die??
Jesus Christ. Oh my god. Why would Omega do this?? Oh my god.
I can just set off the glitter bomb, right? I don’t care about the cleanup, I just don’t want to die!!
But wait - Omega was predicting me, as usual, so this is definitely covered by the great commitment. So I need to act correctly.
Okay, okay, let’s stay calm - what is actually going on here? What would my past self have precommitted to?
(you take a deep breath)
Well, obviously I’d like to take the Right box - if I had committed to that and Omega predicted it, what would have happened? There would be a live bomb in the Left box, I would take the Right box like it predicted… and I’d have to do a few hours of cleaning. Okay, that’s not that bad.
What if I had committed to taking the Left box? Omega would have predicted that, there’d be no bomb in the Left box, and… I could just take the empty Left box and I wouldn’t have to clean.
…
(you start feeling a knot in your stomach)
Wait, wait, wait - there is a one in a trillion trillion chance of Omega being wrong. So for the Right box commitment, there’d actually be a one in a trillion trillion chance of getting an empty Left. Although I would have committed to taking the Right box in that case, so it wouldn’t matter. And for the Left box commitment, there’d be a one in a trillion trillion chance of actually getting a bomb in the Left box, which I would have committed to take, and burning to death.
So, summing up, from my past self’s perspective: Right box, 100% chance of having to clean up for a few hours. Left box, one in a trillion trillion chance of burning to death.
That means... the question is if my past self would have valued not having to clean over a one in a trillion trillion chance of burning to death.
(tears well up in your eyes as you start realizing)
I’ve always been a bullet-biting utilitarian type of guy.[13]
One in a trillion trillion is extremely small.
(the tears are streaming down your face now)
I… I don’t want to die.
But there is a bomb in the Left box - that means… Omega predicted I would choose Right in this situation.
What?? But why??? I just said that I wouldn’t do that!!!
…why am I here?
Did Omega make a mistake? Is this a prank? But no, I know that he doesn’t lie.[14]
(screaming at your empty room in disbelief) OMEGA!!! I know you’re long dead but why did you think I would choose Right???? Why????
(you collapse on the ground and cry)
(20 minutes pass)
I don’t want to, I don’t want to, I don’t want to.
You incompetent asshole. Why couldn’t you just be perfect at predicting me, like you usually are? One in a trillion trillion. I really had to get that unlucky.
…
(you stare at the bomb and imagine it exploding as you take the Left box, drenching you in flames)
(you think about how burning to death is said to be among the most painful ways to die)
God, I’m so scared, I’m trembling. It’s so obvious to me now. The incommensurabilists were right all along. This is worse than any amount of cleaning could be good. I was such an idiot. Such a naive, utilitarian moron.
…I don’t actually believe that. I’d be happy with my choice if I had gotten luckier and didn’t have to clean.
(you turn to look at the Right box with the glitter bomb)
Maybe…? I could just…?
(a wordless desperation runs through you)
(you lunge forward to grab the Right box - but something stops you, with your hand only inches away)
…I can’t.
(you drop your hand, sadly)
It’s too deep in my bones. I have to act correctly. I’ve done it for so long. I can’t betray myself like that.
It doesn’t make any sense, but… I could never live with that. I would hate myself.
Sometimes you just get unlucky. That doesn’t mean you get to betray your past self.
…
Yeah. There’s an honor in that. A righteousness.
(you pace around for another hour, reflecting on your life, mourning)
(finally, you go to stand in front of the Left box, and take a final deep breath)
(the tears on your face have long since dried)
(you take the Left box, the bomb explodes, and you burn to death)
Where FDT was doing weird stuff, commitment theory makes the equivalent decisionmaking seem more understandable. Of course, it remains counterintuitive - as Bomb* was designed to be maximally counterintuitive. But hopefully this makes it at least conceivable that a real person might actually take the Left box.
I do believe the great commitment is possible. Things like this are normal: for example, you might tell yourself to get up early tomorrow - and that itself makes you to do it, even if you don’t feel like it in the morning. Human brains are physical systems and, via internal physical manipulations (cognitive activity), can affect their future mental state.[15]
In my mind, it’s a psychological change, some deep realization of a new, non-decision-theoretic notion of “acting correctly” - something like “cooperating with”, being “loyal to” and “coherent with” your past self.
Whether it’s psychologically possible for you to follow such a principle as far as in Bomb*, when the stakes are life-and-death, is an empirical question. But that doesn’t change the fact that the situation is clearly covered by the great commitment, i.e. that you would want to precommit to choosing to burn to death in Bomb*, if you could.
And… you can.
Here’s what I think your takeaways from this post should be:
To be clear, it’s not my personally preferred framing, and in my opinion it ultimately fails for conceptual/metaphysical reasons, and LessWrong thought is much closer to correct - but I think it goes quite far, as I’ll argue in this post.
Those about updatelessness, specifically, not those about counterfactuals (which are already present in academic decision theory).
I’ve rewritten it to make it clearer, since the original is a bit confusing and underspecified (see this Stuart Armstrong comment), and including that Omega specifically simulates you to predict you is unnecessary. My version preserves the structure and should meet the core desideratum of its proponents (that FDT chooses to burn to death for seemingly no reason) - but maybe call it Bomb* (with a star at the end) for clarity, if you reference it.
It’s kind of like the Transparent Newcomb’s Problem, but optimized for counterintuitiveness - for example, that you are in an “impossible reality” and not in a “possible reality that you just need to make real”.
at least with extremely high probability. (also, for convenience this paragraph is taken from Will MacAskill and modified for my context - I couldn’t find a better way of explaining it than his, but I also needed to make too many small changes to make it a direct quote.)
This is a more UDT-style framing.
e.g. there are a variety of arguments in the comments on Will’s post.
This is the academic discussion on “binding” and dynamic choice.
One might also gesture to some intuition that this is central to what “being an agent” means - you only have the ability to make plans if you can lock your future self into a set of actions, if you can expect your future self to “cooperate” with your current self’s commitments. Decision theory and commitment theory seem like two sides of the coin of ideal decisionmaking - how to maximize for your preferences (an adversarial position towards the world) versus how to intentionally ignore and blind yourself to your preferences (to cooperate with the world). (This framing is inspired by Richard Ngo).
This is the famous “Non-FDT agents self-modify to be FDT agents” argument, put into practice.
This is one of the core ways that UDT was originally characterized. The clearest statement of that I’ve been able to find is Demski and Garrabrant (2019), who straight-up define it as: “UDT [recommends] that the agent do whatever would have seemed wisest before—whatever your earlier self would have committed to do”.
(and FDT is an umbrella term for “UDT-ish approaches to decision theory”. Although there are some terminological subtleties - there’s a longer comment by Rob Bensinger in that post that’s quite clarifying. But it also comes up several times in the FDT paper as a core property, just ctrl-F “committed”.)
Of course, you still need the correct counterfactuals - e.g. when deciding what to precommit to in a situation with an agent that is similar but not identical to you. But counterfactuals are the “decision theory dimension” (EDT vs CDT), not the “commitment theory dimension” (updateless vs updateful), so they are a separate issue. (But I think e.g. commitment theory EDT just works).
It’s buried pretty deep in the post, so I’ll quote it - Christiano says: “I’m not sure if I’m thinking about worlds that don’t exist, or if it’s us who don’t exist and there is some real world somewhere thinking about us.”
There’s one caveat here (this is complicated and not well-argued, feel free to skip): When agents throughout the universe realize that they should make the great commitment, they might already be in the midst of executing strategies that they might together have precommitted to not doing in advance (e.g. cooperating with other value systems in the universe too little - ECL), if they were making the great commitment from the perspective “outside of time” of the simple agent structure that is inside of them (cf Nate Soares’ discussion of being “wrapped around The Algorithm” here). This is the debate about “how far back” to be updateless in UDT - there’s an analogous debate in commitment theory about how simple of a version of yourself you imagine making the great commitment (and therefore how much what you commit to is determining outcomes all throughout the universe). So you can’t avoid weird metaphysics after all, (“outside of time” is obviously metaphysical reasoning), no matter how hard you try. That’s part of why I think the commitment theory framing is ultimately insufficient. But it should at least make clear that people shouldn’t be disagreeing about stuff like Newcomb’s problem and Bomb* (anything other than ECL, really) - you don’t even need to get into metaphysics to see the right answer there!
Not all of it - Some of it is better classified as EDT vs CDT stuff (i.e. the “decision theory dimension”), and some as updateless vs updateful stuff (i.e. the “commitment theory dimension”).
If this makes you disconnect from the scenario because your past self wouldn’t be this utilitarian, just imagine some other less bad consequence. Something like this works no matter what your preferences are - the point is just that FDT can make you choose a locally bad outcome for seemingly no reason.
Of course, if you face such an unlikely situation in real life, you should completely rethink your assumptions and seriously wonder if you’ve gone crazy. But as it’s a hypothetical decision problem, we’ve fixed the agent’s ex hypothesi beliefs about the situation he’s in - so I’ve written this snippet like you just happen to not think of that possible response.
Once again, this is the academic discussion on binding.
I want to briefly examine other attempts to do something similar, and explain why I think they fail. First, Will MacAskill’s “Global CDT”, as it’s closest in the social graph:
Even as it succeeds at replicating a lot of FDT’s recommendations, it leaves less of its conceptual attractiveness intact. Global CDT is framed as simply evaluating with CDT many different “evaluative focal points”: different personalities, dispositions, rules, and so on. For example, it gets as specific as “[the] sort of person… who cooperates in prisoner’s dilemmas”. But we’d prefer to retain the intuitive sense that the “FDT insight” is one singular thing (“act as you would have precommitted to”), and still avoid the associated metaphysical and normative claims - which we can, with commitment theory.
More importantly, I think it fails on counterfactual mugging with a deterministic coin, since the reality where the coin lands heads is not real and could never have been real, and so doesn’t get taken into account in any updateful decision theory after you learn it lands tails (even if we evaluate bigger things like dispositions or rules). (You can understand dispositions and rules as leaky human approximations of theoretical “true” updatelessness - which just evaluates anew the correct precommitment to have taken for every situation)
For attempts from academia (this is going to be uninteresting if you’re not in the weeds on this stuff, so feel free to skip, and I haven’t fully read any of these papers so take it with a grain of salt): I expect the worries above to also apply to Fisher’s and Gauthier’s disposition-based decision theories, Parfit’s “rational irrationality”, and Spohn (”Reversing 30 Years of Discussion”, 2012). McClennen’s “resolute choice” (“it is rational to follow through on plans even when they become locally harmful”) is insufficient because it fails on situations that you didn’t anticipate, like Bomb* - the thing you need to be “resolute” about is much deeper than plans. The same goes for Bratman (Intentions, Plans, and Practical Reason, 1987). Poellinger (“Unboxing the Concepts in Newcomb's Paradox”, 2013) and Hedden (”Counterfactual Decision Theory”, 2023) are about counterfactuals (the “decision theory dimension”), and not updatelessness (the “commitment theory dimension“), so they’re orthogonal to this discussion.
Meacham’s cohesive decision theory (“do as you would have bound yourself to do”, quite similar) is the closest academic relative to FDT (footnote 34 is genuinely fascinating as an early discussion of the idea of updatelessness) and has seemingly the exact same recommendations (modulo a theory of counterfactuals) - but as it also has acts as its evaluative focal point, it incurs the same intuitive drawback of calling it “rational” to choose to burn to death in Bomb*. Also, since it lacks the intuition pumps I’ve built up around it, and tries to fit it into the unnatural frame of decision theory, it ends up looking a bit unmotivated (which is maybe why it didn’t get much traction). It also doesn’t explicitly open up the question of what exactly it means for an action to “have been good to have been bound to” (which ends up being a rich field of study).
Especially because, in my opinion, you inevitably run into metaphysical questions (changing the past, alternate realities, platonism about computations) if you really try to justify why e.g. taking the Left box and burning to death in Bomb* is the “correct” decision, in the decision-theoretic sense of maximizing your utility. (Not sure how much of a hot take that is, I will justify it more in a future post). And it’s understandable that people are reluctant to wade into that.
Finally, a brief and dense sidenote (don’t worry if this doesn’t make sense to you): Commitment theory is actually slightly better than naive TDT-style FDT (which is the way many people conceive of FDT), because it performs better on counterfactual mugging with a deterministic coin. This is the sense in which FDT lacks a theory of anthropics, in the “UDT = FDT + a theory of anthropics” way of carving up the space - another way of seeing that metaphysics comes into play.
(Of course, commitment theory doesn’t fully avoid metaphysics at the end of the day either, as I argue in footnote 11)
(If you want to immediately skip to why I believe past attempts to incorporate LessWrong-style intuitions into standard decision theory are insufficient, go to footnote 16)