Effective Altruism Forum
EA Forum

Hide table of contents
What values will control the Future?
by Jim Buhler
2 min read 0

27

Cause prioritizationForecastingCultural evolutionExtraterrestrial intelligenceGlobal priorities researchGrabby aliensLong-range forecastingLong-term futureMoral psychologyResearch agendas, questions, and project listsValue driftValue lock-in
Frontpage

(This post also has a Russian version, translated from the present original by K. Kirdan.)

In order to influence the longterm future, we often need to understand what it might look like. What it might look like partly depends on what we expect the relevant stakeholders to value. This is why the key question I ask (and started trying to answer) in the present sequence is: What values should we expect powerful civilizations/agents to hold and what does this imply for longtermist cause prioritization?

Key takes

  • Growing the (currently small and very informal) research field of axiological futurism would help answer this question.
    • My first post makes the case for working in the area and makes a literature review of related work.
  • Contrary to what many longtermists assume,  what the moral truth might be likely doesn't influence the answer to this question. (See my second post.)
  • We should expect values that are "grabby" (i.e., expansion-conducive) to be selected over the long run, at least in the worlds where there is more at stake. Call this the Grabby Values Selection Thesis (See my third post.)
  • Concern for suffering is particularly non-grabby and may therefore be strongly selected against. Call this the Upside-focused Colonist Curse. The stronger this curse, the more we might want to prioritize reducing s-risks (in long-lasting ways) over reducing other long-term risks. (See my fourth post.)

Some key remaining questions

  • What is the likelihood of an early[1] value lock-in (or of design escaping selection early and longlastingly, in Robin Hanson’s (2022) terminology)?
  • How could we try to quantify the significance of the Upside-focused Colonist Curse?
    • How is it most likely to play out? (See this comment for a typology of different forms it could take.)
  • If the Upside-focused Colonist Curse is decisively important, does that imply anything else than “today’s longtermists might want to prioritize s-risks”?
    • What does it imply for s-risks cause prioritization (other than "we should prioritize 'late s-risks'")?
  • What makes some values more adaptive than others? What features, other than upside-focusedness, might the most grabby values have?
    • Simplicity? Low conduciveness to uncertainty on how to maximize for these values? Where are they on the action-omission axis?
  • Should we expect biological aliens to be sentient or not?
  • How wide will the moral circle of our successors and that of grabby aliens be? Does the moral circle framework even make sense here? Would they even care about sentient beings and/or their experiences?
  • What things other than suffering could plausibly be disvalued by a grabby civilization/AGI?
  1. ^

    I.e., before grabby values get the time to be selected for within the civilization. (See my third post.)

27

0
0

Reactions

0
0
Previous:
Why we may expect our successors not to care about suffering
31 comments65 karma
Comments


No comments on this post yet.
Be the first to respond.
More from Jim Buhler
65
Why we may expect our successors not to care about suffering
Jim Buhler
· · 10m read
31
31
62
Future technological progress does NOT correlate with methods that involve less suffering
Jim Buhler
· · 4m read
12
12
63
Some governance research ideas to prevent malevolent control over AGI and why this might matter a hell of a lot
Jim Buhler
· · 19m read
5
5
View more
Curated and popular this week
Paul Present
 ·  · 28m read
 · 
Note: I am not a malaria expert. This is my best-faith attempt at answering a question that was bothering me, but this field is a large and complex field, and I’ve almost certainly misunderstood something somewhere along the way. Summary While the world made incredible progress in reducing malaria cases from 2000 to 2015, the past 10 years have seen malaria cases stop declining and start rising. I investigated potential reasons behind this increase through reading the existing literature and looking at publicly available data, and I identified three key factors explaining the rise: 1. Population Growth: Africa's population has increased by approximately 75% since 2000. This alone explains most of the increase in absolute case numbers, while cases per capita have remained relatively flat since 2015. 2. Stagnant Funding: After rapid growth starting in 2000, funding for malaria prevention plateaued around 2010. 3. Insecticide Resistance: Mosquitoes have become increasingly resistant to the insecticides used in bednets over the past 20 years. This has made older models of bednets less effective, although they still have some effect. Newer models of bednets developed in response to insecticide resistance are more effective but still not widely deployed.  I very crudely estimate that without any of these factors, there would be 55% fewer malaria cases in the world than what we see today. I think all three of these factors are roughly equally important in explaining the difference.  Alternative explanations like removal of PFAS, climate change, or invasive mosquito species don't appear to be major contributors.  Overall this investigation made me more convinced that bednets are an effective global health intervention.  Introduction In 2015, malaria rates were down, and EAs were celebrating. Giving What We Can posted this incredible gif showing the decrease in malaria cases across Africa since 2000: Giving What We Can said that > The reduction in malaria has be
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
Recent opportunities in Forecasting
79
HealthLearn: Impact Evaluation and Next Steps
2 authors
· · 12m read
1
1
20
Hiring pre-docs
Eva
· · 1m read
0
0
32
$300 Fermi Model Competition
Ozzie Gooen
· · 2m read
8
8
View more