All of Maxime_Riche's Comments + Replies

Interesting and nice to read!

Do you think the following is right?

The larger the Upside-focused Colonist Curse, the fewer resources agents caring about suffering will control overall and the smaller the risks of conflicts causing S-risks?

This may balance out the effect that the larger the Upside-focused Colonist Curse, the more neglected S-risks are.

High Upside-focused Colonist Curse produces fewer S-risks at the same time as making them more neglected.

2
Jim Buhler
8mo
Thanks, Maxime! This is indeed a relevant consideration I thought a tiny bit about, and Michael St. Jules also brought that up in a comment on my draft. First of all, it is important to note that UCC affects the neglectedness -- and potentially also the probability -- of "late s-risks", only (i.e., those that happen far away enough from now for the UCC selection to actually have the time to occur). So let's consider only these late s-risks. We might want to differentiate between three different cases: 1. Extreme UCC (where suffering is not just ignored but ends up being valued as in the scenario I depict in this footnote. In this case, all kinds of late s-risks seem not only more neglected but also more likely. 2. Strong UCC (where agents end up being roughly indifferent to suffering; this is the case your comment assumes I think). In this case, while all kinds of late s-risks seem more neglected, late s-risks from conflict seem indeed less likely. However, this doesn't seem to apply to (at least) near-misses and incidental risks. 3. Weak UCC (where agents still care about suffering but much less than we do). In this case, same as above, except perhaps for the "late s-risks from conflict" part. I don't know how weak UCC would change conflict dynamics. The more we expect #2 more than #1 and #3, the more your point applies, I think (with the above caveat on near-misses and incidental risks). I might definitely have missed something, though. It's a bit complicated.

Thanks for your response! 

Yet, I am still not clearly convinced that my reading doesn't make sense. Here are some comments:

  • "respondents were very uncertain"
    This seems to be, at the same time, the reason why you could want to diversify your portfolio of interventions for reducing X-risks. And the reason why someone could want to improve such estimates (of P(Nth scenario|X-risk)). But it doesn't seem to be a strong reason to discard the conclusion of the survey (It would be, if we had more reliable information elsewhere).
  • "there's overlap between the sce
... (read more)

I am confused by this survey. Taken at face value, working on improving Cooperation would only be x2 less impactful than working on hard AI alignment (only looking at the importance of the problem). And working on partial/naive alignment would be as impactful as working on AI alignment (looking only at the importance).
Does that make sense?

(I make a bunch of assumptions to come up with these values. The starting point is the likelihood of the 5-6 X-risks scenarios. Then I associate each scenario with a field (AI alignment, naive AI alignment, Cooperation) t... (read more)

4
Sam Clarke
1y
Thanks for your comment! I doubt that it's reasonable to draw these kinds of implications from the survey results, for a few reasons: * respondents were very uncertain * there's overlap between the scenarios * there's no 1-1 mapping between "fields" and risk scenarios (e.g. I'd strongly bet that improved cooperation of certain kinds would make both catastrophic misalignment and war less likely) (though maybe your model tries to account for this, I didn't look at it) A broader point: I think making importance comparisons (between interventions) on the level of abstraction of "improving cooperation", "hard AI alignment" and "partial/naive alignment" doesn't make much sense. I expect comparing specific plans/interventions to be much more useful.

Thanks for this clarification! I guess the "capability increase over time around and after reaching human level" is more important than the "GDP increase over time" to look at how hard alignment is. It's likely why I assumed takeoff meant the former. Now I wonder if there is a term for "capability increase over time around and after reaching human level"...

Reading Eli's piece/writing this review persuaded me to be more sceptical of Paul style continuous takeoff[6] and more open to discontinuous takeoff; AI may simply not transform the economy much until it's capable of taking over the world[7].

From the post we don't get information about the acceleration rate of AI capabilities but on the impact on the economy. This argument is thus against slow takeoff with economic consequences but not against slow takeoff without much economic consequences.

 So updating from that towards a discontinuous takeoff d... (read more)

5
RobBensinger
1y
Paul Christiano operationalizes slow/soft takeoff as: Though there are other takeoff-ish questions that are worth discussing, yeah.
4
𝕮𝖎𝖓𝖊𝖗𝖆
1y
I guess I don't understand how slow takeoff can happen without economic consequences.   Like takeoff (in capabilities progress) may still be slow, but the impact of AI is more likely to be discontinuous  in that case. I was probably insufficiently clear on that point.