Interesting. Thanks for sharing :)
Thanks for sharing. Fyi, I'm getting a "Page not found" error because of the "." at the end of the link. (But once I remove the full stop, it works fine.)
This is a quickly written note that I don't expect to have time to polish.
This note aims to bound reasonable priors on the date and duration of the next technological revolution, based primarily on the timings of (i) the rise of homo sapiens; (ii) the Neolithic Revolution; (iii) the Industrial Revolution. In particular, the aim is to determine how sceptical our prior should be that the next technological revolution will take place this century and will occur very quickly.
The main finding is that the historical track record is consistent with the next technological revolution taking place this century and taking just a few years. This is important because it partially undermines the claims that (i) the “most important century” hypothesis is overwhelmingly unlikely and (ii) the burden of evidence required to believe otherwise is very high. It also suggests that the historical track record doesn’t rule out a fast take-off.
I expect this note not to be particularly surprising to those familiar with existing work on the burden of proof for the most important century hypothesis. I thought this would be a fun little exercise though, and it ended up pointing in a similar direction.
There have been two technological revolutions since the emergence of homo sapiens (about 3,000 centuries ago): the Neolithic Revolution (started about 100 centuries ago) and the Industrial Revolution (started about 2 centuries ago).
Full calculations in this spreadsheet.
Full calculations in the spreadsheet.
Suppose technological revolutions arise as a Poisson point process, with time measured in human-years, so that it takes the same number of human-years for each technological revolution (on average). This seems like a reasonable way to form a prior in this case. If it takes N human-years for a technological revolution on average, and the number of human-years has been growing exponentially, then the time between each multiple of N should get shorter. But population hasn’t grown at a constant exponential rate, it’s more like the growth rate is proportional to the population level (until very recently, in macrohistorical terms).
Numerical simulations suggest that when population growth is proportional to population level, the time delay between each N human-years gets shorter by the same factor each time.
I'm happy to see more discussion of bargaining approaches to moral uncertainty, thanks for writing this! Apologies, this comment is longer than intended -- I hope you don't mind me echoing your Pascalian slogan!
My biggest worry is with the assumption that resources are distributed among moral theories in proportion to the agent's credences in the moral theories. It seems to me that this is an outcome that should be derived from a framework for decision-making under moral uncertainty, not something to be assumed at the outset. Clearly, credences should play a role in how we should make decisions under moral uncertainty but it's not obvious that this is the right role for them to play. In Greaves and Cotton-Barratt (2019), this isn't the role that credences play. Rather, credences feed into the computation of the asymmetric Nash Bargaining Solution (NBS), as in their equation (1). Roughly, credences can be thought to correspond to the relative bargaining power of the various moral theories. There's no guarantee that the resulting bargaining solution allocates resources to each theory in proportion to the agent's credences and this formal bargaining approach seems much more principled than allocating resources in proportion to credences, so I prefer the former. I doubt your conclusions significantly depend on this but I think it's important to be aware that what you described isn't the same as the bargaining procedure in Greaves and Cotton-Barratt (2019).
I like how you go through how a few different scenarios might play out in Section 2 but while I think intuition can be a useful guide, I think it's hard to how things would play out without taking a more formal approach. My guess is that if you formalised these decisions and computed the NBS that things would often but not always work out as you hypothesise (e.g. divisible resources with unrelated priorities won't always lead to worldview diversification; there will be cases in which all resources go to one theory's preferred option).
I'm a little uncomfortable with the distinction between conflicting priorities and unrelated priorities because unrelated priorities are conflicting once you account for opportunity costs: any dollars spent on theory A's priority can't be spent on theory B's priority (so long as these priorities are different). However, I think you're pointing at something real here and that cases you describe as "conflicting priorities" will tend to lead to spending resources on compromise options rather than splitting the pot, and that the reverse is true for cases you describe as "unrelated priorities".
The value of moral information consideration is interesting. It should be possible to provide a coherent account of the value of moral information for IB because the definition of the value of information doesn't really depend on the details of how the agent makes a decision. Acquiring moral information can be seen as an act/option etc. just like any other and all the moral theories will have views about how good it would be and IB can determine whether the agent should choose that option vs other options. In particular, if the agent is indifferent (as determined by IB) between 1. acquiring some moral information and paying $x and 2. not acquiring the information and paying nothing, then we can say that the value of the information to the agent is $x. Actually computing this will be hard because it will depend on all future decisions (as changing credences will change future bargaining power), but it's possible in principle and I don't think it's substantially different to/harder than the value of moral information on MEC. However, I worry that IB might give quite an implausible account of the value of moral information, for some of the reasons you mention. Moral information that increases the agent's credence in theory A will give theory A greater bargaining power in future decisions, so theory A will value such information. But if that information lowers theory B's bargaining power, then theory B will be opposed to obtaining the information. It seems likely that the agent will problematically undervalue moral information in some cases. I haven't thought through the details of this though.
I didn't find the small vs grand worlds objection in Greaves and Cotton-Barratt (2019) very compelling and agree with your response. It seems to me to be analogous to the objections to utilitarianism based on the infeasibility of computing utilities in practice (which I don't find very compelling).
On regress: perhaps I'm misunderstanding you, but this seems to me to be a universal problem in that we will always be uncertain about how we should make decisions under moral uncertainty. We might have credences in MFT, MEC and IB, but which of these (if any) should we use to decide what to do under uncertainty about what to do under moral uncertainty (and so on...)?
I think you have a typo in the table comparing MFT, MEC and IB: MEC shouldn't be non-fanatical. Relatedly, my reading of Greaves and Cotton-Barratt (2019) is that IB is more robust to fanaticism but still recommends fanatical choices sometimes (and whether it does so in practice is an open question), so a tick here might be overly generous (though I agree that IB has an advantage over MEC here, to the extent that avoiding fanaticism is desirable).
One concern with IB that you don't mention is that the NBS depends on a "disagreement point" but it's not clear what this disagreement point should be. The disagreement point represents the utilities obtained if the bargainers fail to reach an agreement. I think the random dictator disagreement point in Greaves and Cotton-Barratt (2019) seems quite natural for many decision problems, but I think this dependence on a disagreement point counts against bargaining approaches.
Another use of "consequentialism" in decision theory is in dynamic choice settings (i.e. where an agent makes several choices over time, and future choices and payoffs typically depend on past choices). Consequentialist decision rules depend only on the future choices and payoffs and decision rules that violate consequentialism in this sense sometimes depend on past choices.
An example: suppose an agent is deciding whether to take a pleasurable but addictive drug. If the agent takes the drug, they then decide whether to stop taking it or to continue taking it. Suppose the agent initially judges taking the drug once to have the highest payoff, continuing to take the drug to have the lowest payoff and never taking it to be in between. Suppose further though, that if the agent takes the drug, they will immediately become addicted and will then prefer to carry on taking it to stopping. One decision rule in the dynamic choice literature is called "resolute choice" and requires the agent to take the drug once and then stop, because this brings about the highest payoff, as judged initially. This is a non-consequentialist decision rule because at the second choice point (carry on taking the drug or stop), the agent follows their previously made plan and stops, even though it goes against their current preference to carry on.
I don't know how, if at all, this relates to what Yudkowsky means by "consequentialism", but this seems sufficiently different from what you described as "decision consequentialism" that I thought it was worth adding, in case it's a further source of confusion.
After a little more thought, I think it might be helpful to think about/look into the relationship between the mean and median of heavy-tailed distributions and in particular, whether the mean is ever exponential in the median.
I think we probably have a better sense of the relationship between hours worked and the median than between hours worked and the mean because the median describes "typical" outcomes and means are super unintuitive and hard to reason about for very heavy tailed distributions. In particular, arguments like those given by Hauke seem more applicable to the median than the mean. This suggests that the median is roughly logarithmic in hours worked. It would then require the mean to be exponential in the median for the mean to be linear in hours worked, in which case, working 20% less would lose exactly 20% of the expected impact (more if the mean is more convex than exponential in the median, less if it's less than exponential).
In the simple example above, the mean is linear in the median, so the mean is logarithmic in hours worked if the median is. But the lognormal distribution might not be heavy-tailed enough, so I wouldn't put too much weight on this.
Looking at the pareto distribution, it seems to be the case that the mean is sometimes more than exponential in the median -- it's less convex for small values and more convex for high values . You'd have to a bit of work to figure out the scale and whether it's more than exponential over the relevant range, but it could turn out that expected impact is convex in hours worked in this model, which would suggest working 20% less would lose more than 20% of the value. I'm not sure how well the pareto distribution describes the median though (it seems good for heavy tails but bad for the whole distribution of things), so it might be better to look at something like a lognormal body with a pareto tail. But maybe that's getting too complicated to be worth it. This seems like an interesting and important question though, so I might spend more time thinking about it!
I don't have a good object-level answer, but maybe thinking through this model can be helpful.
Big picture description: We think that a person's impact is heavy tailed. Suppose that the distribution of a person's impact is determined by some concave function of hours worked. We want that working more hours increases the mean of the impact distribution, and probably also the variance, given that this distribution is heavy-tailed. But we plausibly want that additional hours affect the distribution less and less, if we're prioritising perfectly (as Lukas suggests) -- that's what concavity gives us. If talent and luck play important roles in determining impact, then this function will be (close to) flat, so that additional hours don't change the distribution much. If talent is important, then the distributions for different people might be quite different and signals about how talented a person is are informative about what their distribution looks like.
This defines a person's expected impact in terms of hours worked. We can then see whether this function is linear or concave or convex etc., which will answer your question.
More concretely: suppose that a person's impact is lognormally distributed with parameters μ and σ, that μ is an increasing, concave function of hours worked, h, and that σ is fixed. I chose this formulation because it's simple but still enlightening, and has some important features: expected impact, eμ(h)+σ22, is increasing in hours worked and the variance is also increasing in hours worked. I'm leaving σ fixed for simplicity. Suppose also that μ(h)=logh, which then implies that expected impact is heσ22, i.e. expected impact is linear in hours worked.
Obviously, this probably doesn't describe reality very well, but we can ask what changes if we change the underlying assumptions. For example, it seems pretty plausible that impact is heavier-tailed than lognormally distributed, which suggests, holding everything else equal, that expected impact is convex in hours worked, so you lose more than 20% impact by working 20% less.
Getting a good sense of what the function of hours worked (here μ(h)) should look like is super hard in the abstract, but seems more doable in concrete cases like the one described above. Here, the median impact is eμ(h)=h, if μ(h)=logh, so the median impact is linear in hours worked. This doesn't seem super plausible to me. I'd guess that the median impact is concave in hours worked, which would require μ to be more concave than log, which suggests, holding everything else equal, that expected impact is concave in hours worked. I'm not sure how this changes if you consider other distributions though -- it's a peculiarity of the lognormal distribution that the mean is linear in the median, if σ is held fixed, so things could look quite different with other distributions (or if we tried to determine μ and σ from h jointly).
Median impact being linear in hours worked seems unlikely globally -- like, if I halved my hours, I think I'd more than half my median impact; if I doubled them, I don't think I would double my median impact (setting burnout concerns aside). But it seems more plausible that median impact could be close to linear over the margins you're talking about. So maybe this suggests that the model isn't too bad for median impact, and that if impact is heavier-tailed than lognormal, then expected impact is indeed convex in hours worked.
This doesn't directly answer your question very well but I think you could get a pretty good intuition for things by playing around with a few models like this.
Sorry for the slow reply. I don't have a link to any examples I'm afraid but I just mean something like this:
Prior that we should put weights on arguments and considerations: 60%Pros:Clarifies the writer's perspective each of the considerations (65%)Allows for better discussion for reasons x, y, z... (75%)Cons:Takes extra time (70%)
Prior that we should put weights on arguments and considerations: 60%
This is just an example I wrote down quickly, not actual views. But the idea is to state explicit probabilities so that we can see how they change with each consideration.
To see you can find the Bayes' factors, note that if P(W) is our prior probability that we should give weights, P(¬W)=1−P(W) is our prior that we shouldn't, and P(W|A1) and P(¬W|A1)=1−P(W|A1) are the posteriors after argument 1, then the Bayes' factor is
Similarly, the Bayes' factor for the second pro is 0.750.250.350.65≈1.62.
Good questions! It's a shame I don't have good answers. I remember finding Spencer Greenberg's framing helpful too but I'm not familiar with other useful practical framings, I'm afraid.
I suggested the Bayes' factor because it seems like a natural choice of the strength/weight of an argument but I don't find it super easy to reason about usually.
The final suggestion I made will often be easier to do intuitively. You can just to state your prior at the start and then intuitively update it after each argument/consideration, without any maths. I think this is something that you get a bit of a feel for with practice. I would guess that this would usually be better than trying to formally apply Bayes' rule. (You could then work out your Bayes' factor as it's just a function of your prior and posterior but that doesn't seem especially useful at this point/it seems like too much effort for informal discussions.)
Nice post! I like the general idea and agree that a norm like this could aid discussions and clarify reasoning. I have some thoughts that I hope can build on this.
I worry that the (1-5) scale might be too simple or misleading in many cases though and it doesn't quite give us the most useful information. My first concern is that this looks like a cardinal scale (especially the way you calculate the output) but is it really the case that you should weigh arguments with score 2 twice as much as arguments with score 1 etc.? Some arguments might be much more than 5x more important than others, but that can't be captured on the (1-5) scale.
Maybe this would work better as an ordinal ranking with 5 degrees of importance (the initial description sounds more like this). In the example, this would be sufficient to establish that the pros have more weight, but it wouldn't always be conclusive (e.g. 5, 1 on the pro side and 4, 3 on the con side).
I think a natural cardinal alternative would be to give the Bayes' factor for each alternative, and ideally give a prior probability at the start. Or similarly, give a prior and then update this after each argument/consideration, so you and the reader can see how much each argument/consideration affects your beliefs. I've seen this used before and found it helpful. And this seems to convey more important information than how important an argument/consideration is: how much we update our beliefs in response to arguments/considerations.