All of tobycrisford's Comments + Replies

I 'disagreed' with this, because I don't think you drew enough of a distinction between purchasing animals raised on factory farms, and purchasing meat in general.

While there might be an argument that the occasional cheeseburger isn't that "big of a deal", I think purchasing a single chicken raised on a factory farm is quite a big deal. And if you do that occasionally, stopping doing that will probably be pretty high up on the list of effective actions you can take, in terms of impact to effort ratio.

Thanks for this write up!

You might already be aware of these, but I think there are some strong objections to the doomsday argument that you didn't touch on in your post.

One is the Adam & Eve paradox, which seems to follow from the same logic as the Doomsday argument, but also seems completely absurd.

Another is reference class dependence. You say it is reasonable for me to conclude I am in the middle of 'humanity', but what is humanity? Why should I consider myself a sample from all homo sapiens, and not say, apes, or mammals? or earth-originating life? What even is a 'human'?

Makes sense, thank you for the reply, I appreciate it!

And good to know you still want to hear from people who don't meet that threshold of involvement, I wasn't sure if that was the case or not from the wording of the post and survey questions. I will fill it in now!

Are you wanting non "actively involved" EAs to complete the survey?

The definition of "active involvement" is given as working >5 hours per week in at least one EA cause area, and it reads like the $40 is only donated for people in that category? Suggesting maybe these are the only people you want to hear from?

This seems quite strict! I've taken the GWWC pledge and I give all my income to EA causes above a cap. I also volunteer with the humane league, probably spending a few hours a month on average doing stuff with them. And I check the EA forum pretty ... (read more)

4
Cameron Berg
1mo
Thanks for your thoughtful comment—I think that is indeed a better way of phrasing it. For the sake of continuity of the survey we probably won't update it now, but I agree that it probably should have been phrased this way to begin with. However, it does seem perfectly reasonable to add in the option that Jason suggested below, so we will go ahead and do this. Ultimately, we sought some unambiguous cutoff to qualify for the donation amount, and we thought that 5hr/week devoted to meaningfully working on an EA cause would be a reasonable criterion for this. We definitely still want to hear from you and are certainly not going to discount or exclude your responses, but we are still currently planning to donate $40 * number of actively/directly involved EAs who take the survey to the selected charities.
2
Jason
1mo
I thought that question should have an option for ">= 5 hr/wk but not >= 5 hr in any specific cause area," and should be called "significant time involvement" or somesuch.

You've made some interesting points here, but I don't think you ever discussed the possibility that someone is actually voting altruistically, for the benefit of some group or cause they care about (either helping people in their local area, people in the rest of the country, everyone in the world, future generations, etc).

Is it really true that most voters' behavior can be explained by either (i) self-interest, or (ii) an 'emotionally rewarding cheer for their team'..? I find that a depressing thought. Is no one sincerely trying to do the right thing?

If y... (read more)

Thanks for this summary, a really interesting read!

This is an interesting analysis that I haven't properly digested, so what I'm about to say might be missing something important, but something feels a bit strange about this type of approach to this type of question.

For example, couldn't I write a post titled "Can AI cause human extinction? not on priors" where I look at historical data on "humans killed by machines" (e.g. traffic accidents, factory accidents) as a fraction of the global population, show that it is tiny, and argue it's extremely unlikely that AI (another type of machine) will wipe us all o... (read more)

3
Vasco Grilo
2mo
Great reflection, Toby! I agree that would be the mistake. I think one can use an analysis similar to mine to conclude, for example, that the probability of car accidents causing human extinction is astronomically low. However, I would say using past war deaths to estimate future war deaths is much more appropriate than using past car accident deaths to estimate the risk of human extinction due to advanced AI. I assume increasing capability to cause damage is the main reason for people arguing that future wars would belong to a different category. Yet: * I think war capabilities have been decreasing or not changing much in the last few decades: * "Nuclear risk has been decreasing. The estimated destroyable area by nuclear weapons deliverable in a first strike has decreased 89.2 % (= 1 - 65.2/601) since its peak in 1962" (see 1st graph below). * Military expenditure as a fraction of global GDP has decreased from 1960 to 2000, and been fairly constant since then (see 2nd graph below). * Taking a broader view, war capabilities do have been increasing, but there is not a clear trend in the deaths in conflicts as a fraction of the global population since 1400 (see last figure in the post). * Increases in the capability to cause damage are usually associated with increases in the capability to prevent damage, which I guess explains what I said just above, so one should not forecast future risk based on just one side alone. Something like deaths in car accidents does not capture the relevant factors which can lead to AI causing human extinction. I think the best reference class for this is looking into how species have gone extinct in the past. Jacob Steinhardt did an analysis which has some relevant insights: Advanced AI is much more analogous to a new species than e.g. cars, so per the 1st point it makes sense that extinction risk from advanced AI is much higher than from e.g. cars. I would still claim deaths in past wars and terrorist attacks provide a

I don't know much about the Nestle example, but in principle yes I think so.

I think the same would apply to any case where the production of each individual product does marginal harm. In that case a single individual can choose not to purchase the product and therefore have a marginal impact.

And maybe these kind of boycotts are more common than I suggested in the original answer, but it definitely applies to veganism.

1
dstudioscode
2mo
But boycotts where you are trying to make a policy change require mass organization then?

This is just a quick answer to point out that veganism (which you mention in the question) is a bit different to other kinds of boycotts.

In a conventional boycott, you refuse to purchase certain kinds of products until the organisation(s) who sell them change their ways. I don't know much about how effective those kind of boycotts tend to be (although I think there are some famous examples of where large scale well organised boycotts seem to have produced some powerful results, e.g. montgomery bus boycott).

But veganism isn't just about pressuring organisat... (read more)

1
dstudioscode
2mo
Could the same logic be applied elsewhere?  E.g. boycotting Nestle results in reduced money for Nestle which causes Nestle to exploit people in Africa less?

Thanks for this, a really nice write up. I like these heuristics, and will try to apply them.

On the intuition behind how to interpret statistical power, doesn't a bayesian perspective help here?

If someone was conducting a statistical test to decide between two possibilities, and you knew nothing about their results except: (i) their calculated statistical power was B (ii) the statistical significance threshold they adopted was p and (iii) that they ultimately reported a positive result using that threshold, then how should you update on that, without knowi... (read more)

1
Nathan_Barnard
2mo
Yeah, I think a Bayesian perspective is really helpful here and this reply seems right.  

I have a personal anecdote that I can use as a knock down argument against anyone on the EA forum who tells me that I am wasting my time by reading the news: I discovered EA through an article on the BBC News website.

I first heard about Effective Altruism from this article, which I read in December 2010, before the concept was even called "Effective Altruism". The article was a profile of Toby Ord, and his decision to give away most of his lifetime income to effective causes. It made a big impact on me at the time, because I knew that I wanted to do the sa... (read more)

Thanks for this great post! Really fascinating!

Sorry if this was already asked, but I couldn't see it: how likely is it that pathogens would be able to develop resistance to UVC, and how quickly might that happen? If it did happen, how big a concern would it be? E.g. would it just be a return to the status quo, or would it be an overcorrection?

I really like the way Derek Parfit distinguishes between consequentialist and non-cosequentialist theories in 'Reasons and Persons'.

All moral theories give people aims. A consequentialist theory gives everyone the same aims (e.g. maximize total happiness). A non-consequentialist theory gives different people different aims (e.g. look after your own family).

There is a real important difference there. Not all moral theories are consequentialist.

Thanks for writing this up, this is a really interesting idea.

Personally, I find points 4, 5, and 6 really unconvincing. Are there any stronger arguments for these, that don't consist of pointing to a weird example and then appealing to the intuition that "it would be weird if this thing was conscious"?

Because to me, my intuition tells me that all these examples would be conscious. This means I find the arguments unconvincing, but also hard to argue against!

But overall I get that given the uncertainty around what consciousness is, it might be a good idea to use implementation considerations to hedge our bets. This is a nice post.

1
Derek Shiller
8mo
I'm not particularly sympathetic with arguments that rely on intuitions to tell us about the way the world is, but unfortunately, I think that we don't have a lot else to go on when we think about consciousness in very different systems. It is too unclear what empirical evidence would be relevant and theory only gets us so far on its own. That said, I think there are some thought experiments that should be compelling, even though they just elicit intuitions. I believe that the thought experiments I provide are close enough to this for it to be reasonable to put weight on them. The mirror grid, in particular, just seems to me to be the kind of thing where, if you accept that it is conscious, you should probably think everything is conscious. There is nothing particularly mind-like about it, it is just complex enough to read any structure you want into it. And lots of things are complex. (Panpsychism isn't beyond the pale, but it is not what most people are on board with when they endorse functionalism or wonder if computers could be conscious.) Another way to think about my central point: there is a history in philosophy of trying to make sense of why random objects (rocks, walls) don't count as properly implementing the same functional roles that characterize conscious states. There are some accounts that have been given for this, but it is not clear that those accounts wouldn't predict that contemporary computers couldn't be conscious either. There are plausible readings of those accounts that suggest that contemporary computers would not be conscious no matter what programs they run. If you don't particularly trust your intuitions, and you don't want to accept that rocks and walls properly implement the functional roles of conscious states, you should probably be uncertain over exactly which view is correct. Since many views would rule out consciousness in contemporary computers, you should lower the probability you assign to that.

I think this is an interesting question, and I don't know the answer.

I think two quite distinct ideas are being conflated in your post though: (i) 'earning to give' and (ii) the GWWC 10% pledge.

These concepts are very different in my head.

'Earning to give': When choosing a career with the aim of doing good, some people should pick a career to maximize their income (perhaps subject to some ethical constraints), and then give a lot of it away to effective causes (probably a lot more than 10%). This idea tells you which jobs you should decide to work in.

GWWC ... (read more)

3
DirectedEvolution
10mo
Thank you for your response. I completely agree that earning to give and the GWWC pledge are conceptually distinct. Ideally, anyone dealing with these ideas would treat them as such. Where I disagree with you is that my post is conceptually 'conflating' these two ideas. Instead, my post is identifying that a bundle of associated ideas, including the GWWC pledge and earning to give, are prominent parts of EA's reputation. Here is an analogy to the point I am making: * When people think of engineering, they think of math, chemicals and robots. * When people think of Effective Altruism, they think of earning to give and donating 10% of your income to effective charities The abstract relationship I am drawing with this analogy is that people who are not part of a specific community often have a shallow, almost symbolic view of major topics in the community. They do not necessarily come to a clear understanding of how all the parts fit together into a cohesive whole. My post is not at all arguing the virtues of earning to give or a 10% pledge. It is arguing that these two topics are part of a bundle of ideas that people associate with EA's brand or reputation, in response to the debate suggested by the two seemingly contradictory claims I quoted at the top of the post. I don't think my post represents the critics it cites as saying donating 10% of one's income to charity is a bad thing to do. What they critique is a perception of absolutism and the tension inherent in setting any specific standard for such a pledge, given various forms of inequality. On the one hand, this doesn't exactly reflect the true beliefs of EA thought leaders: MacAskill calls for the ultra-wealthy to donate as much as 99% of their income, and Giving What We Can has a Trial Pledge option, which is a way to make a smaller and more time-limited commitment. Nobody is stopping you from donating 10% to an effective charity and an extra 2% to the opera. But psychologically, when people are pro

Thanks for this reply! That makes sense. Do you know how likely people in the field think it is that AGI will come from just scaling up LLMs vs requiring some big new conceptual breakthrough? I hear people talk about this question but don't have much sense about what the consensus is among the people most concerned about AI safety (if there is a consensus).

2
Riccardo
1y
Since these developments are really bleeding edge I don't know who is really an "expert" I would trust on evaluating it. The closest to answering your question is maybe this recent article I came across on hackernews, where the comments are often more interesting then the article itself: https://news.ycombinator.com/item?id=35603756 If you read through the comments which mostly come from people that follow the field for a while they seem to agree that it's not just "scaling up the existing model we have now", mainly because of cost reasons, but that's it's going to be doing things more efficiently than now. I don't have enough knowledge to say how difficult this is, if those different methods will need to be something entirely new or if it's just a matter of trying what is already there and combining it with what we have. The article itself can be seen skeptical, because there are tons of reasons OpenAIs CEO has to issue a public statement and I wouldn't take anything in there at face value. But the comments are maybe a bit more trustworthy / perspective giving.

I've seen people already building AI 'agents' using GPT. One crucial component seems to be giving it a scratchpad to have an internal monologue with itself, rather than forcing it to immediately give you an answer.

If the path to agent-like AI ends up emerging from this kind of approach, wouldn't that make AI safety really easy? We can just read their minds and check what their intentions are?

 Holden Karnofsky talks about 'digital neuroscience' being a promising approach to AI safety, where we figure out how to read the minds of AI agents. And for curr... (read more)

5
aogara
1y
Tamera Lanham is excited about this and is doing research on it: https://www.lesswrong.com/posts/FRRb6Gqem8k69ocbi/externalized-reasoning-oversight-a-research-direction-for
2
Riccardo
1y
The reasons you provide would already be sufficient for me to think that AI safety will not be an easy problem to solve. To add one more example to your list: We don't know yet if LLMs will be the technology that will reach AGI, it could also be a number of other technologies that just like LLMs make a certain breakthrough and then suddenly become very capable. So just looking at what we see develop now and extrapolating from the currently most advanced model is quite risky. For the second part about your concern about the welfare of AIs themselves, I think this is something very hard for us to imagine, we anthropomorphize AI, so words like 'exploit' or 'abuse' make sense in a human context where beings experience pain and emotions, but in the context of AI those might just not apply. But I would say in this area I still know very little so I'm mainly repeating what I read is a common mistake to make when judging morality in regards to AI.

I really like this argument. I think there's another way of framing it that occurred to me when reading it, that I also found insightful (though it may already be obvious):

  • Suppose the value of your candidate winning is X, and their probability of winning if you don't do anything is p.
  • If you could buy all the votes, you would pay X(1-p) to do so (value of your candidate winning minus a correction because they could have won anyway). This works out at X(1-p)/N per vote on average.
  • If p>1/2, then buying votes probably has diminishing returns (certainly this
... (read more)

Point taken, although I think this is analogous to saying: Counterfactual analysis will not leave us predictably worse off if we get the probabilities of others deciding to contribute right.

Thank you for this correction, I think you're right! I had misunderstood how to apply Shapley values here, and I appreciate you taking the time to work through this in detail.

If I understand correctly now, the right way to apply Shapley values to this problem (with X=8, Y=2) is not to work with N (the number of players who end up contributing, which is unknown), but instead to work with N', the number of 'live' players who could contribute (known with certainty here, not something you can select), and then:

  • N'=3, the number of 'live' players who are decidin
... (read more)
2
Vasco Grilo
1y
Since I was calculating the Shapley value relative to doing nothing, it being positive only means taking the action is better than doing nothing. In reality, there will be other options available, so I think agents will want to maximise their Shapley cost-effectiveness. For the previous situation, it would be: * SCE(N)=1−(1−p)NNVc. For the previous values, this would be 7/6. Apparently not very high, considering donating 1 $ to GWWC leads to 6 $ of counterfactual effective donations as a lower bound (see here). However, the Shapley cost-effectiveness of GWWC would be lower than their counterfactual cost-effectiveness... In general, since there are barely any impact assessments using Shapley values, it is a little hard to tell whether a given value is good or bad.

Edit: Vasco Grilo has pointed out a mistake in the final paragraph of this comment (see thread below), as I had misunderstood how to apply Shapley values, although I think the conclusion is not affected.

If the value of success is X, and the cost of each group pursuing the intervention is Y, then ideally we would want to pick N (the number of groups that will pursue the intervention) from the possible values 0,1,2 or 3, so as to maximize:

(1-(1/2)^N) X - N Y

i.e., to maximize expected value.

If all 3 groups have the same goals, they'll all agree what N is. If ... (read more)

7
Vasco Grilo
1y
Hi Toby, I do not think this is correct. If we consider a game with N players where each has to pay c to have a probability p of achieving a value of V: * The actual contribution of a coalition with size n is: * va(n)=(1−(1−p)n)V−nc. * The marginal contribution to a coalition with size n is (note it tends to -c as n increases, as expected): * vm(n)=va(n+1)−va(n)=(1−(1−p)(n+1))V−(n+1)c−(1−(1−p)n)V+nc==p(1−p)nV−c. * Since the marginal contribution only depends on the size of the coalition, not on its specific members, the Shapley value is (note it tends to -c as n increases, as expected): * S(N)=1N∑N−1n=0vm(n)=1N(pV1−(1−p)Np−Nc)=1−(1−p)NNV−c. In David's example, N = 3, and p = 0.5, so S = (7/24) V - c. For your values of V = 8, and c = 2, S = 1/3. This is not higher than 2. From the formula for the Shapley value, maximising it is equivalent to maximising: * f(N)=1−(1−p)NN. I have concluded in this Sheet the above is a strictly decreasing function of N (which tends to 0), so Shapley value is maximised for the smallest possible number of players. This makes intuitive sense, as there is less credit to be shared when N is smaller. The smallest possible number of players is 1, in which case the Shapley value equals the counterfactual value. In reality, N is higher than 1 in expectation, because it can only be 1 or higher. So, since the Shapley value decreases with N, assuming a single player game will tend to overestimate the contribution of that single player. I think David was arguing for this. In any case, I do not think it makes sense to frame the problem as deciding what is the best value for N. This is supposed to be the number of ("live") agents in the problem we are trying to solve, not something we can select.
4
Davidmanheim
1y
In a single person game, or one where we're fully aligned and cooperating, we get to choose N. We should get to the point where we're actively cooperating, but it's not always that easy. And in a game-theoretic situation, where we're only in control of one party, we need a different approach than either saying we can choose where to invest last, when we can't, and I agree that it's more complex than Shapley values.

To arrive at the 12.5% value, you were assuming that you knew with certainty that the other two teams will try to create the vaccine without you (and that they each have a 50% chance of succeeding). And I still think that under that assumption, 12.5% is the correct figure.

If I understand your reasoning correctly for why you think this is incoherent, it's because:

If the 3 teams independently arrive at the 12.5% figure, and each use that to decide whether to proceed, then you might end up in a situation where none of them fund it, despite it being clearly wo... (read more)

2
Vasco Grilo
1y
Hi Toby, Regarding: You are most likely aware of this, but I just wanted to clarify Shapley value is equal to counterfactual value when there is only 1 ("live") agent. So Shapley value would not leave us predictably worse off if we get the number of agents right.
5
Davidmanheim
1y
Let's make the problem as simple as possible; you have a simple intervention with 3 groups pursuing it. Each has an independent 50% chance of success,  per superforecasters with an excellent track record, and talking about it and coordinating doesn't help, because each one is a different approach that can't benefit from coordination. And I agree with you that there are cases where it's the wrong tool - but as you said, I think "EA is now often in situations like these," and we're not getting the answer right! 

Edited after more careful reading of the post

As you say in the post, I think all these things can be true: 


1) The expected counterfactual value is all that matters (i.e. we can ignore Shapley values).

2) The 3 vaccine programs had zero counterfactual value in hindsight.

3) It was still the correct decision to work on each of them at the time, with the information that was available then.

At the time, none of the 3 programs knew that any of the others would succeed, so the expected value of each programme was very high. It's not clear to me why the '12.5%... (read more)

6
Davidmanheim
1y
The most important thing about your decision theory is that it shouldn't predictably and in expectation leave you worse off than if you had used a different approach. My claim in the post is that we're using such an approach, and it leaves us predictably worse off in certain specific cases. For example, I strongly disagree with the idea that it's coherent to say that all three programs would have zero value in hindsight, and that the true value is 12.5% each, because it means that in many plausible cases, where return on investment from a single working solution is, say, only 3x the bar for funding, we should fund none of them.  And regarding Toby's comment, I agree with him - the problem I pointed to in the last section is specifically and exactly relevant - we're committing to things on an ongoing and variable basis, along with others. It's a game-theoretic setup, and as he suggests, "Shapley [should be] applied when the other agents' decisions are still 'live'" - which is the case here. When EA was small, this was less problematic. We're big enough to factor into other large player's calculus now, so we can't pretend we move last in the game. (And even when we think we know we are, in fact, moving last in committing funds after everyone else, it is an iterated game, so we're not actually doing so.)

I should admit at this point that I didn't actually watch the Philosophy Tube video, so can't comment on how this argument was portrayed there! And your response to that specific portrayal of it might be spot on.

I also agree with you that most existential risk work probably doesn't need to rely on the possibility of 'Bostromian' futures (I like that term!) to justify itself. You only need extinction to be very bad (which I think it is), you don't need it to be very very very bad.

But I think there must be some prioritisation decisions where it becomes relev... (read more)

On your response to the Pascal's mugging objection, I've seen your argument made before about Pascal's mugging and strong longtermism (that existential risk is actually very high so we're not in a Pascal mugging situation at all) but I think that reply misses the point a bit.

When people worry about the strong longtermist argument taking the form of a Pascal mugging, the small probability they are thinking about is not the probability of extinction, it is the probability that the future is enormous.

The controversial question here is: how bad would extinctio... (read more)

4
Tobias Häberli
1y
Thank you for your response – I think you make a great case! :)  I very much agree that Pascal's Mugging is relevant to longtermist philosophy,[1] for similar reasons to what you've stated – like that there is a trade-off between high existential risk and a high expected value of the future.[2] I'm just pretty confused about whether this is the point being made by Philosophy Tube.  Pascal's mugging in the video has as an astronomical upside that "Super Hitler" is not born - because his birth would mean that "the future is doomed". She doesn't really address whether the future being big is plausible or not. For me, her argument derives a lot of the force from the implausibility of the infinitesimally small chance of achieving the upside by preventing "Super Hitler" from being born. And maybe I watched too much with an eye for the relevance of Pascal's Mugging to longtermist work on existential risk. I don't think your version is very relevant unless existential risk work relies on astronomically large futures, which I don't think much of it does. I think it's quite a common sense position that a big future is at least plausible. Perhaps not Bostromian 10^42 future lives, but the 'more than a trillion future lives' that Abigail Thorn uses.  If we assume a long-run population of around 10 billion. Then we'd get to 1 trillion people who would have lived in 10*80 = 800 years.[3] That doesn't seem to be an absurd timeframe for humanity to reach. I think most of the longtermist-inspired existential risk research/efforts still work with futures that only have a median outcome of a trillion future lives. 1. ^ I omitted this from an earlier draft of the post. Which in retrospect maybe wasn't a good idea. 2. ^ I'm personally confused about this trade-off. If I had a higher p(doom), then I'd want to have more clarity about this. 3. ^ I'm unsure if that's a sensible calculation.
3
O Carciente
1y
I would like to add to this that there is also just the question of how strong a lot of these claims can be.  Maybe the future is super enormous. And maybe me eating sushi tomorrow night at 6pm instead of on Wednesday could have massive repercussions. But it could also have massive repercussions for me to eat sushi on Friday, or something. A lot of things "could" have massive repercussions. Maybe if I hadn't missed the bus last week, Super Hitler wouldn't have been born.  There are some obvious low-hanging fruits in the world such that they would reduce the risk of catastrophe (say, nuclear disarmament, or the  Seed Vault, or something). But there are also a lot of things that seem less obvious in their mechanisms, and which could go radically differently than how the people who outline them seem to think. Interventions to increase the number of liberal democracies on the planet and the amount of education could lead to more political polarization and social instability, for example. I'm not saying it would, but it could. Places that have been on the receiving end of "democratizing" interventions often wind up more politically unstable or dangerous for a variety of reasons, and the upward trend in education and longevity over the past few decades has also been an upward trend in polarization, depression, anxiety, social isolation...  Sure, maybe there's some existential risk to humanity, and maybe the future is massive, but what reason do I have to believe that my eating sushi, or taking public transit, or donating to one charity over another, or reading some book, is actually going to have specific effects? Why wouldn't the unintended consequences outweigh the intended ones? It's not just skepticism about the potential size of the future, it's skepticism about the cause-effect relationship being provided by the potential "mugger". Maybe  we're 100% doomed and nothing we do will do anything because an asteroid is going to hit us in 50 years that we will not be

When you write:

"I decide what the probability of the Mugger's threat is, though. The mugger is not god, I will assume. So I can choose a probability of truth p < 1/(number of people threatened by the mugger) because no matter how many people that the mugger threatens, the mugger doesn't have the means to do it, and the probability p declines with the increasing number of people that the mugger threatens, or so I believe. In that case, aren't people better off if I give that money to charity after all?"

This is exactly the 'dogmatic' response to the mugge... (read more)

1
Noah Scales
1y
Yes, I took a look at your discussion with MichaelStJules. There is a difference in reliability between: * probability that you assign to the Mugger's threat * probability that the Mugger or a third party assigns to the Mugger's threat Although I'm not a fan of subjective probabilities, that could be because I don't make a lot of wagers. There are other ways to qualify or quantify differences in expectation of perceived outcomes before they happen. One way is by degree or quality of match of a prototypical situation to the current context. A prototypical situation has one outcome. The current context could allow multiple outcomes, each matching a different prototypical situation. How do I decide which situation is the "best" match? * a fuzzy matching: a percentage quantity showing degree of match between prototype and actual situation. This seems the least intuitive to me. The conflation of multiple types and strengths of evidence (of match) into a single numeric system (for example, that bit of evidence is worth 5%, that is worth 10%) is hard to justify. * a hamming distance: each binary digit is a yes/no answer to a question. The questions could be partitioned, with the partitions ranked, and then hamming distances calculated for each ranked partition, with answers about the situation in question, and questions about identifying a prototypical situation. * a decision tree: each situation could be checked for specific values of attributes of the actual context, yielding a final "matches prototypical situation X" or "doesn't match prototypical situation X" along different paths of the tree. The decision tree is most intuitive to me, and does not involve any sums. In this case, the context is one where you decide whether to give any money to the mugger, and the prototypical context is a payment for services or a bribe. If it were me, the fact that the mugger is a mugger on the street yields the belief "don't give" because, even if I gave them the money, the

I still don't think the position I'm trying to defend is circular. I'll have a go at explaining why.

I'll start with aswering your question: in practice the way I would come up with probabilities to assess a charitable intervention is the same as the way you probably would. I'd look at the available evidence and update my priors in a way that at least tries to approximate the principle expressed in Bayes' theorem. Savage's axioms imply that my decision-describing-numbers between 0 and 1 have to obey the usual laws of probability theory, and that includes Ba... (read more)

3
MichaelStJules
1y
Ok, this makes more sense to me. FWIW, I think most of us go with our guts to assign probabilities most of the time, rather than formally picking priors, likelihoods and updating based on evidence. I tend to use ranges of probabilities and do sensitivity analysis instead of committing to precise probabilities, because precise probabilities also seem epistemically unjustified to me. I use reference classes sometimes.

I can see it might make sense to set yourself a threshold of how much risk you are willing to take to help others. And if that threshold is so low that you wouldn't even give all the cash currently in your wallet to help any number of others in need, then you could refuse the Pascal mugger.

But you haven't really avoided the problem, just re-phrased it slightly. Whatever the amount of money you would be willing to risk for others, then on expected utility terms, it seems better to give it to the mugger, than to an excellent charity, such as the Against Malaria Foundation. In this framing of the problem, the mugger is now effectively robbing the AMF, rather than you, but the problem is still there.

1
Noah Scales
1y
In my understanding, Pascal's Mugger offers a set of rewards with risks that I estimate myself. Meanwhile, I need a certain amount of money to give to charity, in order to accomplish something. Let's assume that I don't have the money sufficient for that donation, and have no other way to get that money. Ever. I don't care to spend the money I do have on anything else. Then, thinking altruistically, I'll keep negotiating with Pascal's Mugger until we agree on an amount that the mugger will return that, if I earn it, is sufficient to make that charitable donation. All I've done is establish what amount to get in return from the Mugger before I give the mugger my wallet cash. Whether the mugger is my only source of extra money, and whether there is any other risk in losing the money I do have, and whether I already have enough money to make some difference if I donate, is not in question. Notice that some people might object that my choice is irrational. However, the mugger is my only source of money, and I don't have enough money otherwise to do anything that I care about for others, and I'm not considering consequences to me of losing the money. In Yudkowsky's formulation, the Mugger is threatening to harm a bunch of people, but with very low probability. Ok. I'm supposed to arrive at an amount that I would give to help those people threatened with that improbable risk, right? In the thought experiment, I am altruistic. I decide what the probability of the Mugger's threat is, though. The mugger is not god, I will assume. So I can choose a probability of truth p < 1/(number of people threatened by the mugger) because no matter how many people that the mugger threatens, the mugger doesn't have the means to do it, and the probability p declines with the increasing number of people that the mugger threatens, or so I believe. In that case, aren't people better off if I give that money to charity after all? You wrote, "I can see it might make sense to set yourself a th

I am comfortable using subjective probabilities to guide decisions, in the sense that I am happy with trying to assign to every possible event a real number between 0 and 1, which will describe how I will act when faced with gambles (I will maximize expected utility, if those numbers are interpreted as probabilities).

But the meaning of these numbers is that they describe my decision-making behaviour, not that they quantify a degree of belief. I am rejecting the use of subjective probabilities in that context, if it is removed from the context of decisions.... (read more)

2
MichaelStJules
1y
I think this doesn't really answer my question or is circular. I don't think that you decide how to act based on probabilities that come from how you decide to act, but that seems to be what you're saying if I interpet your response as an answer to my question. It might also justify any course of action, possibly even if you fix the utility function (I think the subjective probabilities would need to depend on things in very weird ways, though). I think you still want to be able to justify specific acts, and I want to know how you'll do this. Maybe we can make this more explicit with an example. How do you decide which causes to prioritize? Or, pick an intervention, and how would you decide whether it is net positive or net negative? And do so without assigning probabilities as degrees of belief. How else are you going to come up with those probabilities? Or are you giving up probabilities as part of your procedure? On Savage’s axioms, if your state space is infinite and your utility function is unbounded, then completeness requires the axioms to hold over acts that would have infinite expected utility, even if none is ever accessible to you in practice, and I think that would violate other axioms (the sure thing principle; if not Savage’s version, one that would be similarly irrational to violate; see https://onlinelibrary.wiley.com/doi/full/10.1111/phpr.12704 ). If your state space is finite and no outcome has infinite actual utility, then that seems to work, but I'm not sure you'd want to commit to a finite state space.

I guess I am calling into question the use of subjective probabilities to quantify beliefs.

I think subjective probabilities make sense in the context of decisions, to describe your decision-making behaviour (see e.g. Savage's derivation of probabilities from certain properties of decision-making he thinks we should abide by). But if you take the decisions out of picture, and try to talk about 'beliefs' in abstract, and try to get me to assign a real number between 0 and 1 to them, I think I am entitled to ask "why would I want to do something like that?" E... (read more)

2
MichaelStJules
1y
If you aren't (ever) using subjective probabilities to guide decisions, then what would you use instead and why? If you're sometimes using subjective probabilities, how do you decide when to and when not to, and why? Unbounded utilities do violate Savage's axioms, though, I think because of St. Petersburg-like lotteries. Savage's axioms, because of completeness (you have to consider all functions from states to outcomes, so all lotteries), force your utility function and probability function to act in certain ways even over lotteries you would assign 0 probability to ever countering. But you can drop the completeness axiom and assume away St. Petersburg-like lotteries, too. See also Toulet's An Axiomatic Model of Unbounded Utility Functions (which I only just found and haven't read).

Lets assume for the moment that the probabilities involved are known with certainty. If I understand your original 'way out' correctly, then it would apply just as well in this case. You would embrace being irrational and still refuse to give the mugger your wallet. But I think here, the recommendations of expected utility theory in a Pascal's mugger situation are doing well 'on their own terms'. This is because expected utility theory doesn't tell you to maximize the probability of increasing your utility, it tells you to maximize your utility in expectat... (read more)

I think that's a very persuasive way to make the case against assigning 0 probability to infinities. I think I've got maybe three things I'd say in response which could address the problem you've raised:

  • I don't think we necessarily have to asign 0 probability to the universe being infinite (or even to an infinite afterlife), but only to our capacity to influence an infinite amount of utility with any given decision we're faced with, which is different in significant ways, and more acceptable sounding (to me).
  • Infinity is a tricky concept to grapple with. Ev
... (read more)
2
MichaelStJules
1y
I think the original Pascal's wager addresses your first point (assuming you think a higher probability of infinity is infinitely better than a lower but still nonzero probability of infinity). Also, if our descendents will create infinite (or undefined) value in expectation, then extinction risk reduction plausibly affects the probability here. There's also acausal influence over infinitely many agents in an infinite universe, but that might be more tractable to treat as finite EVs, after some kind of averaging. On your 2nd and 3rd points, I think we should aim for justified beliefs, and 0 to infinity and infinite EV subdistributions doesn't seem justifiable to me. Is it reasonable for someone to have an arbitrarily strong prior that the average number of heads per coin flip (for a specific coin, flipped a million times) is 0.9 with no particular reason favouring heads over tails (or only motivated reasons), and only barely adjusting their beliefs as the average comes out to about 0.5? I guess, from the outside, with "more reasonable" beliefs, this person is predictably losing by their own lights. The same seems true of those assigning 0 credence to infinities, although it's much harder (maybe impossible?) to get good enough feedback to show they're losing, and one-shots seem pretty different from events that are basically repeated many times. On my last sentence from my first comment, see my point 5 here: https://forum.effectivealtruism.org/posts/qcqTJEfhsCDAxXzNf/what-reason-is-there-not-to-accept-pascal-s-wager?commentId=Ydbz56hhEwxg9aPh8 Also, see this post and discussion: https://forum.effectivealtruism.org/posts/sEnkD8sHP6pZztFc2/fanatical-eas-should-support-very-weird-projects And https://reducing-suffering.org/lab-universes-creating-infinite-suffering/

You've pointed to a lot of potential complications, which I agree with, but I think they all also apply in cases where someone has done harm, not just in cases where they have not helped.

I just don't think the act/ommission distinction is very relevant here, and I thought the main claim of your post was that it was (but could have got the wrong end of the stick here!)

1
Dave Cortright
1y
Maybe I overcomplexifyed things in my previous response. If they have caused harm, or appear to have, I think the next step is to make that known to them plainly, but in a nonjudgmental way. Then be open and curious to their response. We can't go through all the scenarios here, but if someone is defiant about it, doesn't take ownership, doesn't make amends… then we can exclude them from future participation in the community. So yes, there is judgment taking place, but it is against the metric of harm and whether they are doing their best to minimize it. Thanks again for engaging. This is helping me clarify my stance.

If we know the probabilities with certainty somehow (because God tells us, or whatever) then dogmatism doesn't help us avoid reckless conclusions. But it's an explanation for how we can avoid most reckless conclusions in practice (it's why I used the word 'loophole', rather than 'flaw'). So if someone comes up and utters the Pascal's mugger line to you on the street in the real world, or maybe if someone makes an argument for very strong longtermism, you could reject it on dogmatic grounds.

On your point about diminishing returns to utility preventing reckl... (read more)

1
Noah Scales
1y
Hm, ok. Couldn't Pascal's mugger make a claim to actually being God (with some small probability or very weakly plausibly) and upset the discussion? Consider basing dogmatic rejection on something other than the potential quality of claims from the person whose claims you reject. For example, try a heuristic or psychological analysis. You could dogmatically believe that claims of godliness and accurate probabilism are typical expressions of delusions of grandeur. My pursuit of giving to charity is not unbounded, because I don't perceive an unbounded need. If the charity were meant to drive unbounded increase in the numbers of those receiving charity, that would be a special case, and not one that I would sign up for. But putting aside truly infinite growth of perceived need for the value returned by the wager, in all wagers of this sort that anyone could undertake, they establish a needed level of utility, and compare the risk involved to whatever stakeholders of taking the wager at that utility level against the risks of doing nothing or wagering for less than the required level. In the case of ethics, you could add an additional bounds on personal risk that you would endure despite the full need of those who could receive your charity. In other words, there's only so much risk you would take on behalf of others. How you decide that should be up to you. You could want to help a certain number of people, or reach a specific milestone towards a larger goal, or meet a specific need for everyone, or spend a specific amount of money, or whathaveyou, and recognize that level of charity as worth the risks involved to you of acquiring the corresponding utility. You just have to figure it out beforehand. If by living 100 years, I could accomplish something significant, but not everything, on behalf of others, that I wanted, but I would not personally enjoy that time, then that subjective decision makes living past 100 years unattractive, if I'm deciding solely based on m

I think this comes down to the question of what subjective probabilities actually are. If something is concievable, do we have to give it a probability greater than 0? This post is basically asking, why should we?

The main reason I'm comfortable adapting my priors to be dogmatic is that I think there is probably not a purely epistemological 'correct' prior anyway (essentially because of the problem of induction), and the best we can do is pick priors that might help us to make practical decisions.

I'm not sure subjective probabilities can necessarily be give... (read more)

1
Ryze
1y
Make sense!

Thanks for your comment, these are good points!

First, I think there is an important difference between Pascal's mugger, and Kavka's poison/Newcomb's paradox. The latter two are examples of ways in which a theory of rationality might be indirectly self-defeating. That means: if we try to achive the aims given to us by the theory, they can sometimes be worse achieved than if we had followed a different theory instead. This means there is a sense in which the theory is failing on its own terms. It's troubling when theories of rationality or ethics have this p... (read more)

1
Derek Shiller
1y
Can you expand on what you mean by this? I would think that expected utility maximization is doing well insofar as your utility is high. If you take a lot of risky bets, you're doing well if a few pay off. If you always pay the mugger, you probably think your decision theory is screwing you unless you find yourself in one of those rare situation where the mugger's promises are real. I don't . I'm more or less in the same boat that I wish there was a better justification, and I'm inclined to continue using it because I have to (because there is no clear alternative, because it is human nature, etc.)

The two specific examples that come to mind where I've seen dogmatism discussed and rejected (or at least not enthusiastically endorsed) are these:

The first is not actually a paper, and to be fair I think Hajek ends up being pretty sympathetic to the view that in practice, maybe we do just have to be dogmatic. But my impression was it was ... (read more)

I agree with some of what you say here. For example, from a mental health perspective, teaching yourself to be content 'regardless of your achievements' sounds like a good thing.

But I think adopting 'minimize harm' as the only principle we can use to make judgements of people, is far too simplistic a principle to work in practice.

For example, if I find out that someone watched a child fall into a shallow pond, and didn't go to help them (when the pond is shallow enough that that would have posed no risk to them), then I will judge them for that. I am not c... (read more)

2
Dave Cortright
1y
I appreciate the thoughtful response, Toby. The problem with judgment in this scenario is that it presumes complete knowledge of all the factors at play in that situation. There are a lot of scenarios that could account for what was seen. Perhaps that person… * doesn't know how to swim and doesn't know the pond is shallow and of no risk. * once saw their sibling drown when they were a child and is frozen reliving the trauma of that incident. * is impaired—sight, hearing, developmentally…—in a way where they cannot fully grasp the nature of the situation. * chose to call emergency services because they believed that was the best way they could intervene. * knows the boy has a painful and fatal condition and has chosen to end his life rather than continue to suffer. … Ultimately, what happened happened and we cannot change that. The question is, " How will we live together moving forward?" If we suspect someone is not minimizing harm in their actions, I believe we need to have a conversation with them focusing on the harm and suffering caused to us or others: “I" and "they" statements rather than "you."(If we are speaking for others, it's very important that we are truly representing their experience.) Coming at someone from an accusatory, judgmental stance does not set a tone for a constructive, healthy ongoing relationship. In fact, many people will get defensive, dig in and even push back.  [It's worth noting there is another person in this story: the observer. Are they a trustworthy reporter? Do they have all the facts? Why did they also choose to only observe and not act?]

Apologies, I misunderstood a fundamental aspect of what you're doing! For some reason in my head you'd picked a set of conjectures which had just been posited this year, and were seeing how Laplace's rule of succession would perform when using it to extrapolate forward with no historical input.

I don't know where I got this wrong impression from, because you state very clearly what you're doing in the first sentence of your post. I should have read it more carefully before making the bold claims in my last comment. I actually even had a go at stating the te... (read more)

2
NunoSempere
1y
Hey, I'm not in the habit of turning down free money, so feel free to make a small donation to https://www.every.org/quantifieduncertainty

Edit: This comment is wrong and I'm now very embarrassed by it. It was based on a misunderstanding of what the NunoSempere is doing that would have been resolved by a more careful read of the first sentence of the forum post!

Thank you for the link to the timeless version, that is nice! 

But I don't agree with your argument that this issue is moot in practice. I think you should repeat your R analysis with months instead of years, and see how your predicted percentiles change. I predict they will all be precisely 12 times smaller (willing to bet a small... (read more)

5
NunoSempere
1y
Consider a conjecture first made twenty years ago. If I look at a year as the trial period: * n=20, probability predicted by Laplace of being solved in the next year = 1/(n+2) = 1/22 ~= 4.5% If I look at a month at the trial period: * n = 20 * 12, probability predicted by Laplace of being solved in the next year = the probability that it isn't solved in any of twelve months = 1 - (1-1/(n+2))^12 = 4.8% As mentioned, both are pretty similar.

I'm confused about the methodology here. Laplace's law of succession seems dimensionless. How do you get something with units of 'years' out of it? Couldn't you just as easily have looked at the probability of the conjecture being proven on a given day, or month, or martian year, and come up with a different distribution?

I'm also confused about what this experiment will tell us about the utility of Laplace's law outside of the realm of mathematical conjectures. If you used the same logic to estimate human life expectancy, for example, it would clearly be v... (read more)

6
NunoSempere
1y
I model a calendar year as a trial attempt. See here (and the first comment in that post) for a timeless version. I think that this issue ends up being moot in practice. If we think in terms of something other than years, Laplace would give: 1−(1−1/(d⋅n+2))d where if e.g., we are thinking in terms of months, d=12 instead of  1/(n+2) But if we look at the Taylor expansion for  the first expression, we notice that its constant factor is  1/(n+2) and in practice, I think that the further terms are going to be pretty small when n reasonably large. Alternatively, you can notice that  (1−1/(d⋅n))d converges to the n-th root of e, and that it does so fairly quickly.

Thanks for the comment! I have quite a few thoughts on that:

First, the intention of this post was to criticize strong longtermism by showing that it has some seemingly ridiculous implications. So in that sense, I completely agree that the sentence you picked out has some weird edge cases. That's exactly the claim I wanted to make! I also want to claim that you can't reject these weird edge cases without also rejecting the core logic of strong longtermism that tells us to give enormous priority to longterm considerations.

The second thing to say though is th... (read more)

Thanks! Very related. Is there somewhere in the comments that describes precisely the same issue? If so I'll link it in the text.

3
MichaelStJules
1y
I don't have any specific comment in mind to single out.

I tried to describe some possible examples in the post. Maybe strong longtermists should have less trust in scientific consensus, since they should act as if the scientific consensus is wrong on some fundamental issues (e.g. on the 2nd law of thermodynamics, faster than light travel prohibition). Although I think you could make a good argument that this goes too far.

I think the example about humanity's ability to coordinate might be more decision-relevant. If you need to act as if humanity will be able to overcome global challenges and spread through the g... (read more)

This seems like an odd post to me.  Your headline argument is that you think SBF made an honest mistake, rather than wilfully misusing his users' funds, and most commenters seem to be reacting to that claim. The claim seems likely wrong to me, but if you honestly believe it then I'm glad you're sharing it and that it's getting discussed.

But in your third point (and maybe your second?) you seem to be defending the idea that even if SBF wilfully misused funds, then that's still ok. It was a bad bet, but we should celebrate people who take risky, but pos... (read more)

6
Sharmake
1y
Specifically, he talks about while the heuristic "don't do fraud" is a good heuristic to have, "don't do mistakes" is not a good heuristic at all, and this is trivially true. We can't expect people to be perfect and never make mistakes, so why are you disagreeing with this.

I am very confident that the arguments do perfectly cancel out in the sky-colour case. There is nothing philosophically confusing about the sky-colour case, it's just an application of conditional probability.

That doesn't mean we can never learn anything. It just means that if X and Y are independent after controlling for a third variable Z, then learning X can give you no additional information about Y if you already know Z. That's true in general. Here X is the colour of the sky, Y is the probability of a catastrophic event occurring, and Z is the number... (read more)

I'd like to spend more time digesting this properly, but the statistics in this paragraph seem particularly shocking to me:

"For instance, Hickel et al. (2022) calculate that, each year, the Global North extracts from the South enough money to end extreme poverty 70x over. The monetary value extracted from the Global South from 1990 to 2015 - in terms of embodied labour value and material resources - outstripped aid given to the Global South by a factor of 30. "

They also seem hard to reconcile with each other. If the global north extracts every year 70... (read more)

1
Matthew_Doran
2y
Hi Toby,  Thanks for flagging this, and apologies for the delayed reply (I've been on holiday since posting, and wanted to compose a full reply).  I've double-checked the paper and I believe I am reporting it accurately. I trust this journal, but I'm not a specialist in economics. I'd encourage you to check out their paper and methodology, for more details. Here's part of their abstract for clarification:  "Our results show that in 2015 the North net appropriated from the South 12 billion tons of embodied raw material equivalents, 822 million hectares of embodied land, 21 exajoules of embodied energy, and 188 million person-years of embodied labour, worth $10.8 trillion in Northern prices – enough to end extreme poverty 70 times over. Over the whole period, drain from the South totalled $242 trillion (constant 2010 USD). This drain represents a significant windfall for the global North, equivalent to a quarter of Northern GDP. For comparison, we also report drain in global average prices. Using this method, we find that the South’s losses due to unequal exchange outstrip their total aid receipts over the period by a factor of 30."  Here's the relevant explanation section on extreme poverty:  "This drain represents a significant loss for the South. For perspective, $10.8 trillion would have been enough to end extreme poverty 70 times over in 2015; i.e., with reference to the poverty gap at $1.90 per day in 2011 PPP, which is expressed in roughly the equivalent of Northern prices (World Bank 2021). It is worth noting that this result is larger than previous estimates of drain through unequal exchange (e.g., five times larger than in Hickel et al., 2021). This is because the footprint data we use here captures not only traded goods but also the upstream resources and labour embodied in the production of traded goods, which results in a larger North-South price differential (d)." Here's the section on aid: "Our results show that net appropriation by DAC countrie
Load more