Hide table of contents

Epistemic status: Not an expert on moral uncertainty.
 

The Model

There have been many models proposed to resolve moral uncertainty, but I would like to introduce one more. Instead of acting in accordance to the moral theory we are most confident in (my favorite theory) or making complex electoral systems (MEC, the parliamentary model), we might want to pick a moral theory at random. Just assign to every moral theory you know the probability of how confident you are about this theory, put them in a row from least to most likely (or any sequence really) and pick a random real number between 0 and 100. E.g: Say you have 1% credence in 'Kantian ethics', 30.42% in 'Average utilitarianism' and 68.58% in 'Total utilitarianism' and you generate the random number 31, you will therefore pursue 'Average utilitarianism'. Whenever you update your probabilities you can reroll the dice (another version would be that you reroll at a fixed frequency of intervals, e.g every day). Here are some of the advantages and disadvantages of this model.
 

The Good

  1. It represents your probabilities
  2. It is fair, every theory gets an equal chance
  3. It is easy to understand
  4. It is fast
  5. It stops a moral theory from dominating even though it only has slightly more credence than the second largest theory (e.g 49%, 50%)
  6. It stops a moral theory from dominating even though it has a minority credence (e.g 20%, 20%, 20%, 40%)
  7. It stops the problem of theory-individuation
  8. It has no need for intertheoretic comparisons of value
  9. It makes you less fanatical
  10. It is cognitively easy (no need to do complex calculations in your head)
     

The Bad

  1. Humans need to use something other than their brain (dice/computers) to choose randomly (for an A.I this would not be a problem)
  2. You're not considering a lot of information about the moral theories. This could lead to you violating “moral dominance” e.g picking a theory that decides on an option that it doesn't have much stake in while another theory screams from the sideline (this problem could potentially be solved by making the 'stakes' an additional metric for deciding any given option, but that increases the complexity)
  3. It makes you more inconsistent and therefore harder to cooperate with
  4. Someone might get a wrong impression of you because they met you on a day of very low probability


Overall I'm not really convinced this is the path to a better model of moral uncertainty (or value uncertainty, since this model could also be applied there). I think some variation of MEC is probably the best route. The reason I posted this was because:

  1. Maybe someone with more expertise in moral uncertainty could expand upon this model to make it better
  2. Maybe sortition elements could be included in other theories to improve them
  3. Maybe sortition elements could be included in other theories to make them more useful in practice, since sortition is so easy without sacrificing fairness
Comments5


Sorted by Click to highlight new comments since:

Interesting idea!  However, not too sure about the simple version you've presented. As you mention, the major problem is that it neglects information about 'stakes'. You could try weighting the decision by the stakes somehow, but in cases where you have that information it seems strange to sometimes randomly and deliberately choose the option which is sub-optimal by the lights of MEC.

Also, as well as making you harder to cooperate with, inconsistent choices might over time lead you to choose a path which is worse than MEC by the lights of every theory you have some credence in. Maybe there's an anology to empirical uncertainty: suppose I've hidden $10 inside one of two envelopes and fake money in the other. You can pay me $1 for either envelope, and I'll also give you 100 further opportunities to pay me $1 to switch to the other one. Your credences are split 55%-45% between the envelopes. MEU would tell you to pick the slightly more likely envelope and be done with it. But, over the subsequent 100 chances to switch, the empirical analogue of your sortition model would just under half the time recommend paying me $1 to switch. In the end, you're virtually guaranteed to lose money. Even picking the less likely envelope would represent a better strategy, as long as you stick to it. In other words, if you're unsure between  states of the world A and B, constantly switching between doing what's best given A and doing what's best given B could be worse in expectation than just coordinating all your choices around either A or B, irrespective of which is true. I'm wondering if the same is true where you're uncertain between moral theories A and B.

That said, I'm pretty sure there are some interesting ideas about 'stochastic choice' in the empirical case which might be relevant. Folks who know more about decision theory might be able to speak to that!

Regarding stakes, I think OP's point is that it's not obvious that being sensitive to stakes is a virtue of a theory, since it can lead to low credence-high stakes theories "swamping" the others, and that seems, in some sense, unfair. Bit like if you're really pushy friend always decides where the your group of friends goes for dinner, perhaps. :)

I'm not sure your point about money pumping works, at least as stated: you're talking about a scenario where you lose money over successive choices. But what we're interested in is moral value, and the sortition model will simply deny their's a fixed amount of money in the envelope each time one 'rolls' to see what one's moral view is. It's more like there's $10 in the envelope at stage 1, $100 at stage 2, $1 at stage 3, etc. What this brings out is the practical inconsistency of the view. But again, one might think that's a theoretical cost worth paying to avoid other theories costs, e.g. fanaticism.

I rather like the sortition model - I don't know if I buy it, but it's at least interesting and one option we should have on the table - and I thank the OP for bringing it to my attention. I would flag the "worldview diversification" model of moral uncertainty has a similar flavour, where you divide your resources into different 'buckets' depending on the credence you have in each bucket. See all the bargaining-theoretic model, which treats moral uncertainty as a problem of intra-personal moral trade. This two models also avoid fanaticism and leave one open to practical inconsistency.

Got it. The tricky thing seems to be that sensitivity to stakes is an obvious virtue in some circumstances; and (intuitively) a mistake in others. Not clear to me what marks that difference, though. Note also that maximising expected utility allows for decisions to be dictated by low-credence/likelihood states/events. That's normally intuitively fine, but sometimes leads to 'unfairness' — e.g. St. Petersburg Paradox and Pascal's wager / mugging.

I'm not entirely sure what you're getting at re the envelopes, but that's probably me missing something obvious. To make the analogy clearer: swap out monetary payouts with morally relevant outcomes, such that holding A at the end of the game causes outcome  and holding B causes . Suppose you're uncertain between  and .   says  is morally bad but  is permissible, and vice-versa. Instead of paying to switch, you can choose to do something which is slightly wrong on both  and , but wrong enough that doing it >10 times is worse than  and  on both theories. Again, it looks like the sortition model is virtually guaranteed to recommend taking a course of action which is far worse than sticking to either envelope on either   or  — by constantly switching and causing a large number of minor wrongs.

But agreed that we should be uncertain about the best approach to moral uncertainty!

What about integrating this into a Monte Carlo method?

I think you highlight some potentially good pros for this approach and I can't say I've thoroughly analyzed this approach. However, quite a few of those pros seem non-unique to this particular model of moral uncertainty vs. other frameworks that acknowledge uncertainty and try to weigh the significance of the scenarios against each other. For example, such models already have the pros related to "It stops a moral theory from dominating...," "it makes you less fanatical," etc. (but there are some seemingly unique "pros," such as "It has no need for intertheoretic comparisons of value").

Still, I am highly skeptical of such a model even in comparison to just simply "going with whatever you are most confident in" because of things like complexity (among other things). More importantly, I think this model has a few serious problems along the lines of failing to weight the significance of the situation and thus wouldn't perform well under basic expected value tests (which you might have been getting at with your point about choosing theories with low "stake"): suppose your credences are 50% average utilitarian, 50% total utilitarian. You are presented with a situation where choice A mildly improves average utility such as by severely restricting some population's growth rate (imagine it's for animals)--but this is drastically bad from a total utilitarian viewpoint in comparison to choice B (do nothing / allow the population to rise). To use simple numbers, we could be talking about choice A = +5,-100 (utility points under "average, total"), vs. choice B = 0,0. If the decisionmaker is operating on average utilitarianism, it would be drastically bad. This is why (to my understanding), when your educated intuition says you have the time, knowledge, etc. to do some beneficial analysis, you should try to weight and compare the significance of the situations under different moral frameworks.

Curated and popular this week
Paul Present
 ·  · 28m read
 · 
Note: I am not a malaria expert. This is my best-faith attempt at answering a question that was bothering me, but this field is a large and complex field, and I’ve almost certainly misunderstood something somewhere along the way. Summary While the world made incredible progress in reducing malaria cases from 2000 to 2015, the past 10 years have seen malaria cases stop declining and start rising. I investigated potential reasons behind this increase through reading the existing literature and looking at publicly available data, and I identified three key factors explaining the rise: 1. Population Growth: Africa's population has increased by approximately 75% since 2000. This alone explains most of the increase in absolute case numbers, while cases per capita have remained relatively flat since 2015. 2. Stagnant Funding: After rapid growth starting in 2000, funding for malaria prevention plateaued around 2010. 3. Insecticide Resistance: Mosquitoes have become increasingly resistant to the insecticides used in bednets over the past 20 years. This has made older models of bednets less effective, although they still have some effect. Newer models of bednets developed in response to insecticide resistance are more effective but still not widely deployed.  I very crudely estimate that without any of these factors, there would be 55% fewer malaria cases in the world than what we see today. I think all three of these factors are roughly equally important in explaining the difference.  Alternative explanations like removal of PFAS, climate change, or invasive mosquito species don't appear to be major contributors.  Overall this investigation made me more convinced that bednets are an effective global health intervention.  Introduction In 2015, malaria rates were down, and EAs were celebrating. Giving What We Can posted this incredible gif showing the decrease in malaria cases across Africa since 2000: Giving What We Can said that > The reduction in malaria has be
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f