I am a research analyst at the Center on Long-Term Risk.
I previously studied maths at the University of Cambridge and University of Warwick.
Send me anonymous feedback or messages here https://www.admonymous.co/tristancook
I agree with what you say, though would note
(1) maybe doom should be disambiguated between "the short-lived simulation that I am in is turned of"-doom (which I can't really observe) and "the basement reality Earth I am in is turned into paperclips by an unaligned AGI"-type doom.
(2) conditioning on me being in at least one short-lived simulation, if the multiverse is sufficiently large and the simulation containing me is sufficiently 'lawful' then I may also expect there to be basement reality copies of me too. In this case, doom is implied for (what I would guess is) most exact copies of me.
Thanks for this post! I've been meaning to write something similar, and have glad you have :-)
I agree with your claim that most observers like us (who believe they are at the hinge of history) are in (short-lived) simulations. Brian Tomasik discusses how this marginally makes one value interventions with short-term effects.
In particular, if you think the simulations won't include other moral patients simulated to a high resolution (e.g. Tomasik suggests this may be the case for wild animals in remote places), you would instrumentally care less about their welfare (since when you act to increase their welfare, this may only have effects in basement reality as well as the more expensive simulations that do simulate such wild animals) . At the extreme is your suggestion, where you are the only person in the simulation and so you may act as a hedonist! Given some uncertainty over the distribution of "resolution of simulations", it seems likely that one should still act altruistically.
I disagree with the claim that if we do not pursue longtermism, then no simulations of observers like us will be created. For example, I think an Earth-originating unaligned AGI would still have instrumental reasons to run simulations of 21st century Earth. Further, alien civilizations may have interest to learn about other civilizations.
Under your assumptions, I don't think this is a Newcomb-like problem. I think CDT & EDT would agree on the decision,[1] which I think depends on the number of simulations and the degree to which the existence of a good longterm future hinges your decisions. Supposing humanity only survives if you act as a longtermist and simulations of you are only created if humanity survives, then you can't both act hedonistically and be in a simulation.
This tool is impressive, thanks! I like the framing you use of safety as a race against capabilities, though think don't really know what it would look like to have "solved " AGI safety 20 years before AGI. I also appreciate all the assumptions being listed at the end of the page.
Some minor notes
Thanks for this post! I used to do some voluntary university community building, and some of your insights definitely ring true to me, particularly the Alice example - I'm worried that I might have been the sort of facilitator to not return to the assumptions in fellowships I've facilitated.
A small note:
Well, the most obvious place to look is the most recent Leader Forum, which gives the following talent gaps (in order):
This EA Leaders Forum was nearly 3 years ago, and so talent gaps have possibly changed. There was a Meta Coordination Forum last year run by CEA, but I haven't seen any similar write-ups. This doesn't seem to be an important crux for most of your points, but thought would be worth mentioning.
This definitely sounds like a better approach than mine, thanks for sharing! This will be useful for me for any future projects
Thanks for your questions and comments! I really appreciate someone reading through in such detail :-)
- What is the highest probability of encountering aliens in the next 1000 years according to reasonable choices once could make in your model?
SIA (with no simulations) gives the nearest and most numerous aliens.
My bullish prior (which has a priori has 80% credence in us not being alone) with SIA and the assumption that grabby aliens are hiding gives a median of ~ chance in a grabby civilization reaching us in the next 1000 years.
I don't condition on us not having any ICs in our past light cone. When conditioning on not being inside a GC, SIA is pretty confident (~80% certain) that we have at least one IC (origin planet) in our past light cone. When conditioning on not seeing any GCs, SIA thinks ~50% that there's at least one IC in our past light cone. Even if there origin planet is in our light cone, they may already be dead.
- Sometimes you just give a prior, e.g., your prior on d, where I don't really know where it comes from. If it wouldn't take too much time, it might be worth it to quickly motivate them (e.g., "I think that any interval between x and y would be reasonable because of such and such, and I fitted a lognormal". It's possible I'm just missing something obvious to those familiar with the literature.
Thanks for the suggestion, this was definitely an oversight. I'll add in some text to motivate each prior.
Lots of the priors aren't super well founded. Fortunately, if you think my bounds on each parameter is reasonable, I get the same conclusions when taking a joint prior that is uniform on and log-uniform in all other parameters.
- Do you think your conclusion (e.g., around likelihood of observing GCs) would change significantly if "non-terrestrial" planets were habitable?
Good question. In a hack-y and unsatisfactory way, my model does allow for this:
If the ratio of non-terrestrial (habitable) planets to terrestrial (habitable) planets is , they replace the product of try-once steps with to account for the extra planets. (My prior on is bounded above by 1, but this could be easily changed). This approach would also suppose that non-terrestrial planets had the same distribution of habitable lifetimes as terrestrial ones.
Having said that, I don't think a better approach would change the results for the SIA and ADT updates. For SSA, the habitability of non-terrestrial planets makes civs like us more atypical (since we are on a terrestrial planet). If this atypicality applies equally in worlds with many GCs and worlds with very few GCs, then I doubt it would change the results. All the anthropic theories would update strongly against the habitability of non-terrestrial planets.
Typos:
Thanks!
Great to see this work!
Thanks!
Re the SIA Doomsday argument, I think that is self-undermining for reasons I've argued elsewhere.
I agree. When I model the existence of simulations like us, SIA does not imply doom (as seen in the marginalised posteriors for in the appendix here).
Further, the simulation case, SIA would prefer human civilization to be atypically likely to become a grabby civilization (this does not happen in my model as I suppose all civs have the same transition chance to become grabby).
Re the habitability of planets, I would not just model that as lifetimes, but would also consider variations in habitability/energy throughput at a given time
...
Smaller stars may have longer habitable windows but also smaller values for V and M. This sort of consideration limits the plausibility of red dwarf stars being dominant, and also allows for more smearing out of ICs over stars with different lifetimes as both positive and negative factors can get taken to the same power.
I'd definitely like to see this included in future models (I'm surprised Hanson didn't write about this in his Loud aliens paper). My intuition is that this changes little for the conclusions of SIA or anthropic decision theory with total utilitarianism, and that this weakens the case for many aliens for SSA, since our atypicality (or earliness) is decreased if we expect habitable planets around longer lived stars to have smaller volumes and/or lower metabolisms.
I'd also add, per Snyder-Beattie, catastrophes as a factor affecting probability of the emergence of life and affecting times of IC emergence.
I hadn't seen this before, thanks for sharing! I've skimmed through and found it interesting, though I'm suspicious that at times it uses SSA -with reference class of observers on planets as habitable as long as Earth - type reasoning.
Thanks, glad to hear it!
I wrote it in Google Docs, primarily for the ease of getting comments. I then copied it into the EA Forum editor and spent a few hours fixing the formatting - all the maths had to be rewritten, all footnote added back in, tables fixed, image captions added - which was a bit of a hassle.
I sadly don't have any neat tricks. I tried this Google Docs tool to convert to Markdown but it didn't work well.
The EA Forum editor now have the ability to share drafts and allow comments and collaborative editing, which I think I'll try for my next project. I'm also hoping Google Docs will add a better maths editor.
This looks great, thanks for creating it! I could see it becoming a great 'default' place for EAs to meet for coworking or social things.
I'd add another benefit that I've not seen in the other answers: deciding on the curriculum and facilitating yourself get you to engage (critically) with a lot with EA material. Especially for the former you have to think about the EA idea-space and work out a path through it all for fellows.
I helped create a fellowship curriculum (mostly a hybrid of two existing curricula iirc) before there were virtual programs or and this definitely got me more involved with EA. Of course, there may be a trade-off in quality.