Comment Permalink

Do you think "Most of these required community organising and protest as at least part of the process to achieve the concrete change." is that strong a statement? There is a pretty strong correlation between protest/organising and these changes. Elite consense is clearly very important, but I think that the voice of the masses can move the elite to consensus so there's some chicken and egg there. Also to mention a few cases here where I don't think elite consensus was strong at the time of change and their hand's were perhaps forced...

- Access to free HIV treatment (This I'm pretty sure of)
- Civil rights movement
- Women's suffrage

I do find this a tricky issue to keep a scout mindset on here on the forum, as I find EAs in general are unusually against protest and organising compared to other communities I am a part of. My feeling is this is largely because the nature of many EAs is more to be into research, debate and policy rather than social roles like organising and protest.

What makes you think it is overstated? I think its a tricky counterfactual question with a lot of room for conjecture....

...but is increasing the value of futures tractable?

by Davidmanheim

Mar 192 min read 22

45

Cause prioritizationExistential riskExistential Choices Debate WeekLongtermism

Frontpage

The central question being discussed in the current debate is whether marginal efforts should prioritize reducing existential risk or improving the quality of futures conditional on survival. Both are important, both are neglected, though the latter admittedly more so, at least within EA. But this post examines the tractability of shaping the long-term future if humanity survives, and the uncertainty about our ability to do so effectively.

I want to very briefly argue that given the complexity of long-term trajectories, the lack of empirical evidence, and the difficulty of identifying robust interventions, efforts to improve future value are significantly less tractable than reducing existential risk.

We have strong reasons to think we know what the likely sources of existential risk are - as @Sean_o_h's new paper lays out very clearly. The most plausible risks are well known, and we also have at least some paths towards mitigating them, at least in the form of not causing them. On the other hand, if we condition on humanity’s survival, we are dealing with an open-ended set of possible futures that is both not well characterized, and poorly explored. Exploration of futures is also not particularly tractable, given the branching nature and the complexity of the systems being predicted. And this problem is not just about characterizing futures - the tractability of interventions decreases as the system's complexity increases, especially over multi-century timescales. The complexity of socio-technological and moral evolution makes it infeasible, in my view, to shape long-term outcomes with even moderate confidence. It seems plausible that most interventions would have opposite signs in many plausible futures, and we seem unlikely to know the relative probabilities or the impacts.

And despite @William_MacAskill's book on the topic, we have very limited evidence for what works to guide the future - one of the few key criticisms I think should be generally convincing about the entire premise of longtermism. The exception, of course, is avoiding extinction.

And compared to existential risk, where specific interventions may have clear leverage points, such as biosecurity or AI safety, increasing the quality of long-term futures is a vast and nebulous goal. There is no singular control knob for “future value,” making interventions more speculative. So identifying interventions today that will robustly steer the future in a particular direction is difficult because, as noted, we lack strong historical precedent for guiding complex civilizations over thousands of years, and also, the existence of unpredictable attractor states (e.g., technological singularities, value shifts) makes long-term interventions unreliable. Work to change this seems plausibly valuable, but also more interesting than important, as I previously argued.

45 Reactions

Comments22

Sorted by

New & upvoted

Click to highlight new comments since: Today at 12:52 PM

NickLaing6d*14

I agree with this, and like to lean on what evidence we have rather than theory and speculation

There are many past events which increased the value of the future unintentionally, like free markets and most scientific advancements.

Previous examples of people and movements which intentionally planned to increase the value of the future mostly achieved it through concrete current change in laws and cultural norms. When MLK said "I have a dream", he fought for change then and there to make that dream a reality. Efforts like these have had a huge effect. This list is just what comes to mind, in no particular order and far from extensive. Most of these required community organising and protest as at least part of the process to achieve the concrete change.

- Abolition of slavery
- Women's suffrage
- Nuclear Treaties
- Universal declaration of human rights (which although breaking down a bit has been so successful that "human rights" are seen almost as an objective truth for many people)
- Banning warfare with poisonous gases
- Civil rights movement
- Access to free HIV treatment becoming a universal "right" through social movements and PEPFAR
- Animal welfare movement - specifically the cage free revolution
- Democratic transformation of nation states
- LGBTQ+ rights movements and legal changes
- Social welfare movements for the poorest part of the population in many rich countries

I think efforts to improve the value of the future through concrete current change right here right now are fantastic and EA can continue to focus on those where they are neglected and tractable, but I struggle to see a clear Theory of Change for other possible mechanisms due to, as you put it very well"The complexity of socio-technological and moral evolution". Good things locked in now can be really hard to change in democratic countries. I would (perhaps slightly controversially) cite Obamacare as an example of something that was so beneficial to so many it became extremely hard to reverse.

I'm very open to other pathways for creating change that don't involve a concrete change right here right now, but you'll need to convince me.

Davidmanheim5d4

This seems mostly correct, though I think the role of community organizing (versus elite consensus change) is strongly overstated.

NickLaing5d2

Mo Putera5d12

Tangentially re: protest, I think things are slowly shifting, due to the work of folks like James Özden founding Social Change Lab to understand how social change movements can be more evidence-based and effective. For instance, James changed my mind on the effectiveness of radical protest tactics in What’s everyone got against throwing soup?, which drew upon this literature review to conclude that

A nonviolent radical flank is likely to help, not hinder, a social movement. Specifically, we think there’s good evidence it can increase support for more moderate groups and increase the salience of an issue without harming support for the overall movement’s policy goals.

I'd also signal-boost James' article Protest Movements Could Be More Effective Than the Best Charities published in the Stanford Social Innovation Review. You should always take charts like the one below claiming superlative cost-eff with a metric ton of salt, but I mostly trust the general quality of his analysis and think his bottomline holds up.

That said, James seems to be the only person banging this drum, so I suppose your observation still broadly holds true.

NickLaing5d4

I think James's did is really important and shows the potential good if a radical flank

Traditionally though movements and organizing was often around issues which has decent public support already and wasnt necessarily that radical. The civil rights movement and the HIV medicine campaigners directly moved the elites towards their goal, and didn't just move the needle on public opinion for the better like the radical flank can

Davidmanheim5d4

I agree that the examples you list are ones where organizing and protest played a large role, and I agree that it's effectively impossible to know the counterfactual - but I was thinking of the other examples, several where there was no organizing and protest, but which happened anyways - which seems like clear evidence that they are contributory and helpful but not necessary factors. On the other hand, effectiveness is very hard to gauge!

The conclusion is that organizing is likely or even clearly positive - but it's evidently not required, if other factors are present, which is why I thought it was overstated.

NickLaing5d4

Yep I agree with that

tylermjohn6d10

(Low effort comment as I run out the door, but hope it adds value) To me the most compelling argument in favour of tractability is:

We could make powerful AI agents whose goals are well understood and do not change or update in ex ante predictable ways.
These agents are effectively immortal and the most powerful thing in the affectable universe, with no natural competition. They would be able to overcome potentially all natural obstacles, so they would determine what happens in the lightcone.
So, we can make powerful AI agents that determine what happens in the lightcone, whose goals are well understood and update in ex ante predictable ways.
So, we can take actions that determine what happens in the lightcone in an ex ante predictable way.

Derek Shiller6d5

This more or less conforms to why I think trajectory changes might be tractable, but I think the idea can be spelled out in a slightly more general way: as technology develops (and especially AI), we can expect to get better at designing institutions that perpetuate themselves. Past challenges to affecting a trajectory change come from erosion of goals due to random and uncontrollable human variation and the chaotic intrusion of external events. Technology may help us make stable institutions that can continue to promote goals for long periods of time.

tylermjohn6d4

Here's a shower thought:

If you think extinction risk reduction is highly valuable, then you need some kind of a model of what Earth-originating life will do with its cosmic endowment
Some of the parameters in you model must be related to things other than mere survival, like what this life is motivated by or will attempt to do
Plausibly, there are things you can do to change the values of those parameters and not just the extinction parameter

It won't work for every model (maybe the other parameters just won't budge), but for some of them it should.

Davidmanheim5d3

If you think extinction risk reduction is highly valuable, then you need some kind of a model of what Earth-originating life will do with its cosmic endowment

No, you don't, and you don't even need to be utilitarian, much less longtermist!

tylermjohn5d1

Any disagreement about longtermist prioritization should presuppose longtermism

Davidmanheim4d2

First, you're adding the assumption that the framing must be longtermist, and second, even conditional on longtermism you don't need to be utilitarian, so the supposition that you need a model of what we do with the cosmic endowment would still be unjustified.

tylermjohn4d1

You're not going to be prioritizing between extinction risk and long term trajectory changes based on tractability if you don't care about the far future. And for any moral theory you can ask "why do you think this will be a good outcome?" and as long as you don't value life intrinsically you'll have to state some empirical hypotheses about the far future

Davidmanheim4d2

There is a huge range of "far future" that different views will prioritize differently, and not all need to care about the cosmic endowment at all - people can care about the coming 2-3 centuries based on low but nonzero discount rates, for example, but not care about the longer term future very much.

tylermjohn4d1

I don't understand why that matters. Whatever discount rate you have, if you're prioritizing between extinction risk and trajectory change you will have some parameters that tell you something about what is going to happen over N years. It doesn't matter how long this time horizon is. I think you're not thinking about whether your claims have bearing on the actual matter at hand.

It would probably be most useful for you to try to articulate a view that avoids the dilemma I mentioned in the first comment of this thread.

Davidmanheim3d2

we can make powerful AI agents that determine what happens in the lightcone

I think that you should articulate a view that explains why you think AI alignment of superintelligent systems is tractable, so that I can understand how you think it's tractable to allow such systems to be built. That seems like a pretty fundamental disconnect that makes me not understand your )in my view, facile and unconsidered) argument about the tractablity of doing something that seems deeply unlikely to happen.

Davidmanheim5d2

Well-understood goals in agents that gain power and take over the lightcone is exactly the thing we'd be addressing with AI alignment, so this seems like an argument for investing in AI alignment - which I think most people would see as far closer to preventing existential risk.

That said, without a lot more progress, powerful agents with simple goals is actually just a fancy way of guaranteeing of a really bad outcome, almost certainly including human extinction.

NunoSempere6d5

We have strong reasons to think we know what the likely sources of existential risk are - as @Sean_o_h's new paper lays out very clearly.

Looked at the paper. The abstract says:

In all cases, an outcome as extreme as human extinction would require events or developments that either have been of very low probability historically or are entirely unprecedented. This introduces deep uncertainty and methodological challenges to the study of the topic. This review provides an overview of potential human extinction causes considered plausible in the current academic literature...

So I think you are overstating it a bit, i.e., it's hard to support statements about existential risks coming from classified risks vs unknown unknowns/black swans. But if I'm getting the wrong impression I'm happy to read the paper in depth.

Davidmanheim5d2

Not at all correct - and you clearly started the quote one sentence too late! "Potential causes of human extinction can be loosely grouped into exogenous threats such as an asteroid impact and anthropogenic threats such as war or a catastrophic physics accident. "

So the point of the abstract is that anthropogenic risks, ie. the ones that the next sentence calls "events or developments that either have been of very low probability historically or are entirely unprecedented," are the critical ones, which is why they are a large focus of the paper.

Maxime Riché 🔸6d3

I want to very briefly argue that given the complexity of long-term trajectories, the lack of empirical evidence, and the difficulty of identifying robust interventions, efforts to improve future value are significantly less tractable than reducing existential risk.
[...]

And compared to existential risk, where specific interventions may have clear leverage points, such as biosecurity or AI safety, increasing the quality of long-term futures is a vast and nebulous goal.

I guess, there is a misunderstanding in your analysis. Please correct me if I am wrong.

"Increasing the quality of long-term futures" reduces existential risks. When longtermists talk about "increasing the quality of long-term futures," they include progress on aligning AIs as one of the best interventions they have in mind.

To compare their relative tractability, let's look at the best intervention to reduce Extinction-Risks and, on the other hand, at the best interventions for "increasing the quality of long-term futures", what I call reducing Alignment-Risks.

Illustrative best PTIs for Extinction-Risks reduction: Improving AI control, Reducing AI misuses. These reduce the chance of AI destroying future Earth-originating intelligent agents.
Illustrative best PTIs for Alignment-Risks reduction: Technical AI alignment, improving AI governance. These improve the quality of the long-term futures.

Now, let's compare their tractability. How these interventions differ in tractability is not clear. These interventions actually overlap significantly. It is not clear if reducing misuse risks is actually harder than improving alignment or than improving AI governance.

Interestingly, this leads us to a plausible contradiction in arguments against Alignment-Risks: Some will say that the interventions to reduce Alignment-Risks and Extinction-Risks are the same, and some will say they have vastly different tractability. One of the two groups is incorrect. Interventions can't be the same and have different tractability.

Davidmanheim5d1

You make a dichotomy not present in my post, then conflate the two types of interventions while focusing only on AI risk - so that you're saying that two different kinds of what most people would call extinction reduction efforts are differently tractable - and conclude that there's a definition confusion.

To respond, first, that has little to do with my argument, but if it's correct, your problem is with the entire debate week framing, which you think doesn't present two distinct options, not with my post! And second, look at the other comments which bring up other types of change as quality increasing, and try to do the same analysis, without creating new categories, and you'll understand what I was saying better.