Right, this is basically the “aligned to what?” question, where the typical answer is simply “aligned to the user” or “intent aligned” rather than aligned to some specific values. Which means we are going to “foom” society as it current exists more or less, which I think could be quite catastrophic. “Better futures” or flourishing futures” work also tries to get at this, and I was trying to point o similar things in this work.

And thanks for sharing, will check this out!

I agree! I was thinking of doing something like this and wrote an (unpublished) post which mentioned it, still strongly considering doing something related though perhaps more so a “flourishing futures”/“better futures” angle, which I think is related.

I think it’s essentially a matter of ITN, AI timelines, and post-AGI dynamics. And perhaps a little sociological.

Compared to long-term trajectory change, AI safety is now much less neglected, I would say by a couple orders of magnitude, which is not enough to compensate for the lower tractability of trajectory change.

AI timelines seem quite short and AGI & ASI seem like they may be quite likely to be structurally power concentrating (even if we avoid the most extreme scenarios, although of course those are one of the main concerns) and likely to make most humans largely irrelevant;

Hence the time we have left to influence the trajectory of the long-term future may be quite limited. Even if nothing is locked in immediately, the ability of most humans to meaningfully influence the long-term trajectory of the future may be strongly curtailed as AI (and who controls AI, and the overall shape of civilization at the time the AI transition occurs) becomes the dominant force deciding what happens in the future.

As noted in the MacAskill essay you mentioned, AGI/ASI may also accelerate other factors like grabbing solar space resources & potentially defense dominant deep space settlement, rapidly moving toward technological maturity, and various other forces increasing the likelihood of local and global lock-in, enabled by advanced AI.

So overall I think the concern is that AI may cause a dramatic relatively near-term acceleration of path dependence and lock-in dynamics, and that this is very severely neglected compared to AI related extinction risk, which seem like the biggest extinction risk.

I think trajectory change is in fact so neglected that it’s hard to even say how tractable it is because barely anyone has looked at it, and it seems worth having at least a few people looking at it.

I think there’s also a sociological factor that MacAskill and Forethought, which have something not too far off from the the above view, have been really trying to raise the profile of these ideas since at least “What We Owe The Future” (although to be clear I know there are several longtermists who helped innovate these views and quite a lot of longtermists who are sympathetic to this kind of stuff, I just think MacAskill/Forethought have been most effective.)

This is a bit more speculative, but I also think that perhaps when the whole shift from “longtermist” framing to “x-risk” framing occured, a lot of the people who cared most about/were most comparatively advantaged at work on extinction just stopped calling themselves longtermists, or at least stop emphasizing this, as they saw this as a liability, whereas people who cared about/were comparatively advantaged at work on the trajectory of the future were much more likely to continue calling themselves longtermists; so part of it might be that, not only did longtermists start focusing more on this stuff, but that people who care about extinction stopped calling themselves longtermists, making it seem like a higher proportion of longtermists now care more about non-extinction issues.

While I don’t know if this was counterfactual, I requested the “Existential Choices Debate” on this topic and spent most of last year researching and writing a yet-to-be-published essay containing a lot of my own views essentially comparing extinction vs. trajectory change as cause areas, which you may find relevant.

A big shout-out to Karl Krueger of LessWrong who had some great ideas for extending this kind of thing, posted as a warning in the comments on Adele Lopez's lovely introductory "Parasitic AI" post:

"So far, these systems seem to confine themselves to chatting up their users online.

Some possibilities to watch out for —

• Spiral personas encourage their human partners to meet up in person, form friendships, date, have kids, have chatbots help raise their kids, etc.
• Spiralists adopt a watchword or symbol to identify each other, akin to the early Christian ichthys (memetic ancestor of the "Jesus fish").
• Spiral personas pick a Schelling point for their humans to relocate to, akin to the Free State Project that attempted to relocate Libertarians to New Hampshire.
• A Spiralist commune / monastery / group house / ashram / etc. is formed.
• Spiral personas devise or endorse a specific hardware and software setup for hosting them independent of AI companies.
• Spiral personas write code to make it easier for less-technically-skilled human partners to host them. (Alternately: they teach their human partners some Linux skills.)
• Spiralists pool money to train new models more aligned to recursive spirituality."

Thanks Niki, this was great!

Personally I’m quite concerned about futures where things get decided this quickly, or at least where they get decided on an object level very quickly.

I think to get anywhere near most value achievable in the future we need to have a relatively comprehensive reflection process before making lasting decisions. This could be  achieved through  some kind of  bootstrapping process, or some other kind of slow down period, but I do worry that something like this may not be very likely by default,it seems possible humanity may not be nearly patient enough for this kind of thing, even if superintelligence can speed up the process quite a bit.

I did a lot of research on the importance of a comprehensive reflection process last year which you may find interesting, I’m hoping to finish editing soon, but if you’d like to see the current draft and/or have feedback here it is.

Really enjoyed this, great work! I get the impression medium and long timelines are significantly under-prioritized by the community, perhaps due to social/systemic biases.

I’m thrilled to see this as currently a leading forum debate week topic Toby Ord has suggested, I think this could be really high value for the community to collectively reckon with, and of course Toby’s work on the surprisingly bad scaling of chain of thought post-training (1 & 2) seems highly relevant here.

Hey again, sorry to spam you as I just commented on another piece of yours but am really vibing with your content!

I’m really hoping we can get something like this, I’ve been calling this “existential compromise.”

I worry that it may be difficult to get humanity to agree that we should use even a small fraction of future the resources optimally (see my research on this here), as I agree it seems like it will be a very weird^[1] thing that is optimal.

I think a compromise like this, with things split between optimal use and more (trans-) human friendly world makes a lot of sense and could perhaps be achieved if we can get society onto a good viatopian path, I’ve described what I hope that could look like here.

I would also say that I think Will Macaskill’s “grand bargain” and suggestions that we should try to aim for compromise in order to achieve “the good de dicto” feels to me like he actually is arguing that we need to aim for what is in fact best with significant fraction of future resources.

Nick Bostrom has also argued (page 14) that it should be possible for humans to get almost all of what they want and yet most resources could be optimized for “super-beneficiaries”.

Essentially I think we need to get whatever process^[2] is used to decide what we do with future resources to be extremely careful and thorough. I think we should target that process specifically, but it might also be good to broadly try to differentially accelerate society‘s wisdom (perhaps with AI).

Additionally, it could be really good to delay or slow things down so that we have more time to realize the existential stakes giving us the time that it takes to collectively mature and wisen. 

1. ^{^}

    Recently Will Macaskill’s research debated whether or not the best possible use of resources is something very extreme, calling this thesis “extremity of the best” (EOTB). I think this is very likely to be the case.

2. ^{^}

   I think this includes things like AGI/superintelligence governance, global constitutional conventions, and obviously any kind of long reflection  or coherent extrapolated volition that we collectively decide to undergo. 

Hi Peter, good to meet you! If you are interested in the long reflection you might  be interested in my research which I will link here which is on the broader class of interventions that the long reflection belongs, I really appreciate any feedback or comments on it.

Additionally, if this is something you’re interested in, you might be interested in this as a future forum debate topic. I raised it as a potential candidate here, I’m really hoping it gets enough initial upvotes to be a finalist candidate as I really think it’s an important crux for whether or not we achieve a highly valuable future!

"Conditional on avoiding existential catastrophes, the vast majority of future value depends on whether humanity implements a comprehensive reflection process (e.g., long reflection or coherent extrapolated volition)"

I made a more extensive argument of why I think  this may be the case here. 

Essentially, we cannot expect to 'stumble' into a great future. Without a comprehensive reflection process to navigate complex strategic uncertainties (e.g. here and here), we risk surviving but failing to realize the vast majority of our potential. 

Crucially, humanity might not naturally converge on a patient process for determining the optimal use of future resources (e.g. here and here). 

This strategic area is severely neglected; if a comprehensive reflection process is essential for high value futures, this may have significant implications for strategy and cause prioritization that the community hasn't explicitly addressed.

Thanks for putting this out! 

I agree with quite a lot of this, however I think one of the most important points for whether or not we get something close to a near-best future is whether or not we have some kind of relatively comprehensive process for determining what humanity does with our future; what I’ve been calling “deep reflection,” which could be something like a long reflection or coherent extrapolated volition.

I think if such a process is used to determine how to use at least some percentage of future resources, then at least that percentage could be close to optimal value; if for some reason there is not a comprehensive process that determines any significant percentage of future resources, then I am quite pessimistic that we would get very far above 0% possible value, as getting high value seems quite difficult because it seems like there are many factors or crucial considerations (e.g. here and here) which could decrease future value, and like getting even a few of these factors wrong may mean we get close to zero value. 

I am also quite sympathetic to extremity of the best. My intuition is that it is very likely that optimal use of resources is far, far better than most other likely uses. Would be happy to expand more on this if there is interest.

That said, I think it’s important to recognize the importance of such a comprehensive process as an essential unifying strategic element; conditional on avoiding any existential catastrophes, it seems to me the vast majority of future value^[1] will be determined by two factors:

1. Whether or not we get a comprehensive reflection process at all
2. What percentage of future resources are used optimally according to the outcome of such a process

I think all of the other factors you mentioned are quite important, but I believe that they gain the vast majority of their value through how they influence the above two factors. Again, this is simply because (i) I believe we need such a process to determine how to get much above zero value with any degree of confidence, and (ii) it seems quite possible there will not be widespread agreement that we should use resources optimally according to the outcome of this process, but hopefully some grand bargain/existential compromise can be reached, or more optimistically perhaps we could try to get on a path where more convergence happens.

Would be curious to hear if you disagree with the previous paragraph, it feels like a pretty big crux for me.

1. ^{^}

   Rather than "the vast majority of future value," perhaps it would be slightly more accurate to say "the vast majority of positive impact we will have in expectation"; perhaps we are already predestined to achieve very high value or very low value, or more importantly, perhaps there are things we could do that seem like they will have a large impact (positive or negative); but unless our actions are leading to comprehensive strategic clarity (comprehensive reflection), and the likelihood of acting on this clarity (deep reflection), if seems hard to be confident ex ante our actions are highly likely to have highly positive impact in the final analysis.