Really enjoyed this, great work! I get the impression medium and long timelines are significantly under-prioritized by the community, perhaps due to social/systemic biases.

I’m thrilled to see this as currently a leading forum debate week topic Toby Ord has suggested, I think this could be really high value for the community to collectively reckon with, and of course Toby’s work on the surprisingly bad scaling of chain of thought post-training (1 & 2) seems highly relevant here.

Hey again, sorry to spam you as I just commented on another piece of yours but am really vibing with your content!

I’m really hoping we can get something like this, I’ve been calling this “existential compromise.”

I worry that it may be difficult to get humanity to agree that we should use even a small fraction of future the resources optimally (see my research on this here), as I agree it seems like it will be a very weird^[1] thing that is optimal.

I think a compromise like this, with things split between optimal use and more (trans-) human friendly wor... (read more)

Hi Peter, good to meet you! If you are interested in the long reflection you might  be interested in my research which I will link here which is on the broader class of interventions that the long reflection belongs, I really appreciate any feedback or comments on it.

Additionally, if this is something you’re interested in, you might be interested in this as a future forum debate topic. I raised it as a potential candidate here, I’m really hoping it gets enough initial upvotes to be a finalist candidate as I really think it’s an important crux for whet... (read more)

Thanks for putting this out! 

I agree with quite a lot of this, however I think one of the most important points for whether or not we get something close to a near-best future is whether or not we have some kind of relatively comprehensive process for determining what humanity does with our future; what I’ve been calling “deep reflection,” which could be something like a long reflection or coherent extrapolated volition.

I think if such a process is used to determine how to use at least some percentage of future resources, then at least that percentage... (read more)

And I might add – not just a deep understanding of how the world is, but of how the world could be:

• Large knowledge models for grounded, efficient information retrieval
• Decomposable tasks for superintelligent systems without superintelligent agents
• The potential for coordinated small models to outcompete large models on narrow tasks, making superintelligence potentially nearer but also safer
• Structured transparency enabling verifiable commitments and de-escalation of races
• Massively positive sum possibilities making coordination much more desirable

That is to sa... (read more)

Thanks for posting this Owen, couldn’t agree more! 

I often find myself referencing Eric’s work in specific contexts, in fact I just recommended it last night to someone working on AI control via task decomposition. I have been meaning to do a link-post on Why AI Systems Don’t Want Anything as soon as I get some free time, as it’s the biggest update I have had on AI existential risk since ChatGPT was released.

Eric has the keen ability to develop a unique, nuanced, first principles perspective. I agree his work is dense and I think this is one of its gr... (read more)

Max Tegmark explains it best I think. Very clear and compelling and you don’t need any technical background to understand what he’s saying.

I believe his third or maybe it was second appearance on Lex Fridman’s podcast where I first heard his strongest arguments, although those are quite long with extraneous content, here is a version that is just the arguments. His solutions are somewhat specific, but overall his explanation is very good I think:

Quick link-post highlighting Toner quoting Postrel’s dynamist rules + her commentary. I really like the dynamist rules as a part of the vision of the AGI future we should aim for:

“Postrel does describe five characteristics of ‘dynamist rules’:

As an overview, dynamist rules:

1. Allow individuals (including groups of individuals) to act on their own knowledge.
2. Apply to simple, generic units and allow them to combine in many different ways.
3. Permit credible, understandable, enduring, and enforceable commitments.
4. Protect criticism, competition, and feedback.
5. Establish

Hey Will, very excited to see you posting more on viatopia, couldn't agree more that some conception of viatopia might be an ideal north star for navigating the intelligence explosion.

As crazy as this seems, I just last night wrote a draft of a piece on what I have been calling primary and secondary cruxes/crucial considerations, (in previous work I also used a perhaps even more closely related concept of “robust viatopia proxy targets”) which seems closely related to your "societal version of Rawls' primary goods," though I had not been previously aware o... (read more)

Thanks Kat! Couldn’t agree more, I think self-care is essential, I wish there were more posts, or better yet more high quality comprehensive systematic health and mental health support for high-impact x-risk workers and a culture where this is acknowledged as important, I think this is an underrated crux for x-risk work.

It may seem like ‘fluff,’ but really I think the research is on our side! Would love to see what a quantitative case for exercise or other forms of self-care might look like.

I agree one way for supporting this is that you can read wok stuff... (read more)

I really like this. I’ve been thinking it would be good to have a market that measures positive and negative externalities and includes those in prices of goods, but I noticed that a shortcoming of this was that it didn’t really directly address existential risk and I think tour suggestion is a really interesting way of trying to have that included. After all, x-risk seems like the most impact of all and just incentivizing the positive externalities from, say, AI, could actually have negative impact if it doesn’t account for this.

I very much agree with this and have been struggling with a similar problem in terms of achieving high value futures, versus mediocre ones.

I think there may be some sort of a “Fragile Future Value Hypothesis,” somewhat related to Will MacAskill’s “No Easy Eutopia,” (and the essay which follows this one in the series) and somewhat isomorphic to “The Vulnerable World Hypothesis,” in which there may be many path dependencies, potentially leading to many low and medium value futures attractor states we could end up in, because, in expectation, we are somewhat ... (read more)

Thank you for sharing Arden! I similarly have been thinking longtermism is an important crux for making AI go well, I think it’s very possible that we could avoid x-risk and have really good outcomes in the short-term, but put ourselves on a path where we predictably miss out on nearly all value in the long-term.

I really enjoyed this! Very important crux for how well the future goes. You may be interested to know that Nick Bostrom talks about this, he calls them super-beneficiaries.

I think this is great. I designed an entire market structure/operating system for society, which tries to generalize this principle to all externalities, positive and negative. Happy to share it if you're interested.

I have been thinking that one solution to this could be people self-organizing and spending more of their off-time and casual hours working on these issues in self-organizing or crowd-sourced ways. Would be really curious to hear what your thoughts are on such an approach. I feel like there is enough funding that if people were able to collectively produce something promising, then this could really go somewhere. I have thought a lot about what kind of organizational structures would allow this;

Something like a weekly group meeting where people bring their... (read more)

Far-future effects are the most important determinant of what we ought to do

The future is extremely big, and if we work hard, we are in an unusually high leverage position to influence it. 

Interesting! I think I didn’t fully distinguish between two possibilities:

1. AW just has an understanding of wisdom
2. AW whose values are aligned to wisdom, or at least aligned to pursuing and acting on wisdom 

I think both types of AW are worth pursuing, but the second may be even more valuable, and I think this is the type I had in mind at least in scenario 3.

While there are different value functions, I believe there is a best possible value function. 

This may exist at the level of physics, something to do with qualia that we don’t understand perhaps, and I think it would be useful to have an information theory of consciousness which I have been thinking about. 

But ultimately, I believe that in theory, even if it’s not at the level of physics, I think you can postulate a meta-social choice theory which evaluates every possible social choice theory under all possible circumstance for every possible min

Wow, this is exciting. I agree this is one of the most important things we should be working on right now.

I very much agree that we need less deference and more people thinking for themselves, especially on cause prioritization. I think this is especially important for people who have high talent/skill in this direction, as I think it can be quite hard to do well.

It’s a huge problem that the current system is not great at valuing and incentivizing this type of work, as I think this causes a lot of the potentially highly competent cause prioritization people to go in other directions. I’ve been a huge advocate for this for a long time.

I think it is somewhat har... (read more)

Hey Trevor, it’s been a while, I just read Kuhan’s quick take which referred to this quick take, great to see you’re still active!

This is very interesting, I’ve been evaluating a cause area I think is very important and potentially urgent—something like the broader class of interventions of which “the long reflection” and “coherent extrapolated volition” are examples, essentially how do we make sure the future is as good as possible conditional on aligned advanced AI.

Anyways, I found it much easier to combine tractability and neglectedness into what I call... (read more)

You forgot to mention that Anthropic’s name literally means “as seen from a narrowly human point of view”, far cry from moral circle expansion or doing the most good possible

Thanks Tyler! I think this is spot on. I am nearing the end of writing a very long report on this type of work so I don’t have time at the moment to write a more detailed reply (and what I’m writing is attempting to answer these questions). One thing that really caught my eye was when you mentioned:

Populating and refining a list of answers to this last question has been a lot of the key work of the field over the past few years.

I am deeply interested in this field, but not actually sure what is meant by “the field.”   Could you point me to what search terms to use and perhaps the primary authors or research organizations who have published work on this type of thing?”

@William_MacAskill, what are the main characteristics we should aim for “Viatopia” to have?

Will MacAskill stated in a recent 80,000 hours podcast that he believes marginal work on trajectory change toward a best possible future rather than a mediocre future seems likely significantly more valuable than marginal work on extinction risk.

Could you explain what the key crucial considerations are for this claim to be true, and a basic argument for why think each of the crucial considerations resolves in favor of this claim?

Would also love to hear if others have any other crucial considerations they think weigh in one direction or the other.

Great, thank you!

Hi, hate to bother you again, just wondering where things are at with this contest?

Yes… So basically what you’re saying is this argument goes through if you make the summation of all bubble universes at any individual time step, but longtermist arguments would go through if you take a view from outside the metaverse and make the summation across all points of time in all bubble universes simultaneously?

I guess my main issue is that I’m having trouble philosophically or physically stomaching this, it seems to touch on a very difficult ontological/metaphysical/epistemological question of whether or not it is coherent to do the summation of... (read more)

Hey again quila, really appreciate your incredibly detailed response, although again I am neglecting important things and unfortunately really don’t have any time to write a detailed response, my sincere apologies for this! By the way, really glad you got more clarity from the other post, I also found this very helpful.

1. Yes, I think there is a constant time factor. It is all one unified, single space-time, as I understand it (although this also isn’t an area of very high expertise for me,) I think that what causally separates the universes is simply that sp

Hi Magnus, thank you for writing out this idea!

I am very encouraged (although, perhaps anthropically I should be discouraged for not having been the first one to discover it) that I am not the only one who thought of this (also, see here.) 

I was thinking about running this idea by some physicists and philosophers to get further feedback on whether it is sound. It does seem like adding at least a small element of this to a moral parliament might not be a bad idea, especially considering that making it only 1% of the moral parliament would capture the v... (read more)

Hi Hans, I found your post incredibly helpful and validating, and much clearer than my own in some ways. I especially like the idea of "living in the moment" as a way of thinking about how to maximize value, I actually think this is probably correct and makes the idea potentially more palatable and less conflicting with other moral systems than my own framing.

Thank you, I appreciate your comment very much.

I realized upon reading your response that I was relying very heavily on people either watching the video I referenced or already being quite knowledgeable about this aspect of physics.

I apologize for not being able to answer the entire detailed comment, but I’m quite crunched for time as I spent a few hours being nerd-sniped by myself by taking a few hours to write this post this morning when I had other important work to do haha…

Additionally, I think the response I have is relatively brief, I actually added ... (read more)

I am wondering if the winners of this contest are going to be publicly announced at some point?

I have been thinking about this kind of thing quite a lot and have several ideas I have been working on. Just to clarify, is it acceptable to have multiple entries, or are there any limit on this?

Mmm yeah, I really like this compromise, it leaves room for being human, but indeed, I’m thinking more about career currently. Since I’ve struggled to find a career that is impactful and I am good at, I’m thinking I might actually choose a career that is a relatively stable normal job that I like (Like therapist for enlightened people/people who meditate), and then I can use my free time to work on projects that could be maximally massively impactful.

Yes! This is helpful. I think one of the main places where I get caught up is taking expected value calculations very seriously even though they are wildly speculative; it seems like there is a very small chance that I might make a huge difference on an issue that ends up being absurdly important, and so it is hard to use my intuition on this kind of thing, whereas my intuitions very clearly help me with things that are close by and hence more easier to see I am doing some good but more difficult to make wild speculations that I might be having a hugely po... (read more)

Anyone else ever feel a strong discordance between emotional response and cognitive worldview when it comes to EA issues?

Like emotionally I’m like “save the animals! All animals deserve love and protection and we should make sure they can all thrive and be happy with autonomy and evolve toward more intelligent species so we can live together in a diverse human animal utopia, yay big tent EA…”

But logically I’m like “AI and/or other exponential technologies are right around the corner and make animal issues completely immaterial. Anything that detracts from ... (read more)

Good question. Like most numbers in this post, it is just a very rough approximation used because it is a round number that I estimate is relatively close (~within an order of magnitude) to the actual number. I would guess that the number is somewhere between $50 and $200.

Thanks Mo! These estimates were very interesting.

As to discount rates, I was a bit confused reading William MacAskill's discount rate post, it wasn't clear to me that he was talking about the moral value of lives in the future, it seemed like it might be having something to do with value of resources. In "What We Owe The Future" which is much more recent, I think MacAskill argues quite strongly that we should have a zero discount rate for the moral patienthood of future people.

In general, I tend to use a zero discount rate, I will add this to the backgroun... (read more)

Thank you so much for this reply! I’m glad to know there is already some work on this, makes my job a lot easier. I will definitely look into the articles you mentioned and perhaps just study AI risk / AI safety a lot more in general to get a better understanding of how people think about this. It sounds like what people call “deployment” may be very relevant, so well especially look into this.

Yes, I agree this is somewhat what Bostrom is arguing. As I mentioned in the post, I think there may be solutions which don’t require totalitarianism, i.e. massive universal moral progress. I know this sounds intractable, I might address why I think this maybe mistaken in a future post, but it is a moot point if a vulnerable world induced X-risk scenario is unlikely, hence why I am wondering if there has been any work on this.

Ah yes! I think I see what you mean.

I hope to research topics related to this in the near future, including in-depth research on anthropics, as well as on what likely/desirable end-states of the universe are (including that we may already be in an end-state simulation) and what that implies for our actions.

I think this could be a 3rd reason for acting to create a high amount of well-being for those close to you in proximity, including yourself.

Hey Carl! Thanks for your comment. I am not sure I understand. Are you arguing something like “comparing x-risk interventions to other inventions such as bed nets is invalid because the universe may be infinite, or there may be a lot of simulations, or some other anthropic reason may make other interventions more valuable”?

Highly Pessimistic to Pessimistic-Moderate Estimates of Lives Saved by X-Risk Work

This short-form supplements a post estimating how many lives x-risk work saves on average.

Following are four alternative pessimistic scenarios, two of which are highly pessimistic, and two of which fall between pessimistic and moderate.

Except where stated, each has the same assumptions as the original pessimistic estimate, and is adjusted from the baseline estimates of 10^16 lives possible and one life saved per hour of work or $100 donated.

1. It is 100% impossible to prevent ex

Thanks Spencer, really appreciated the variety of guests, this was a great podcast.

Is this contest still active after the FTX fiasco?