Lots of EA discourse focusses explicitly on the exponential capabilities of technology, and a lot more so than most other social movements. Be it longtermism or x-risk reduction or progress studies and so forth. It also focusses on opportunities that have been systematically neglected by existing social institutions - be it global aid or x-risks again. Yet EA as a whole seems to me to very techno-optimist, believing that new technology can be guided for good, extinction can be averted and so on. This belief seems to me more ideological than just scientific, and perhaps will always remain so. Hence I'm curious.

Question: Is EA compatible with technopessimism?

But for that let me try defining technopessimism first. I can broadly see it as encompassing two beliefs:

  1. New technology in our society causes more harm than good (including expected future harm and good).
  2. The more effective immediate action on learning this, is to to limit the creation of new technology itself, rather than improve policy and institutions to control it.

It is possible to believe neither 1 or 2, 1 and not 2; or 1 and 2 both. For the scope of this post, let's assume technopessimists are people who believe both 1 and 2. I'm sure there's also a doomerist set of beliefs here, namely that 1 is true, but no form of change is worth pursuing. I'll also ignore that for the scope of the post. (If you feel this is a bad framework, please let me know.)


Regarding the first component, I'm sure some EAs would say that the EA community is open to it if there's stronger proof that tech progress is a bad thing, and evidence-based reasoning is required. But I'm not sure what kind of evidence would suffice. It's hard to use RCTs and scientifically prove that technological progress is a bad thing, at best you'll get some policy frameworks that people will view through their usual mental filters. There's already plenty of work on various brands of economics (neoclassical, socialist, etc.), various types of social structures and governments, and so on - it's unlikely (albeit not impossible) we quickly develop a slam dunk case for one set of ideas being superior to another.

Regarding the second component, there is significant uncertainty shared by people in the EA community and outside of it as to how easy any form of regulation, control or guiding is. Whether it is by influencing existing institutions or building new ones, whether is via incremental changes or revolutionary ones. Yet both technopessimists (as I've defined) and most EAs share the belief that change is possible. The question is how to evaluate what forms of change are more tractable. There are for instance those in the AI alignment community who believe 1 but not 2 because they believe AGI cannot be regulated. But again you end up with same problem, it's much harder to definitely prove what works.

Hence an elaboration of the original question:

Is there a reasonable burden of proof that the EA community as a whole is willing to accept in favour of techno-pessimism?

P.S. Please try to avoid arguments in favour of techno-optimism or techno-pessimism in the comment section if you can. I'm more interested in understanding what the burden of proof is, and what current EA stances are, rather than debating in favour or against them. I totally understand that this still requires expressing these arguments, I'd just prefer if the intent is descriptive rather than persuasive.


New Answer
Ask Related Question
New Comment

3 Answers sorted by

In my experience, many EAs have a fairly nuanced perspective on technological progress and aren't unambiguous techno-optimists. 

For instance, a substantial fraction of the community is very concerned about the potential negative impacts of advanced technologies (AI, biotech, solar geoengineering, cyber, etc.) and actively works to reduce the associated risks. 

Moreover, some people in the community have promoted the idea of "differential (technological) progress" to suggest that we should work to (i) accelerate risk-reducing, welfare-enhancing technologies (or ideas generally) and (ii) decelerate technologies (or ideas) with the opposite effects. That said, studying the concrete implications of differential progress seems fairly neglected and deserves to be explored in much greater depth. In line with the above idea, it seems common for EAs to argue that technological progress has been very beneficial in some regards—improving human welfare, especially over the last hundreds of years (e.g. here)—while it has been harmful in other regards, such as factory farming having led to greater animal suffering.

Thanks this is super useful.

Although I guess the question now becomes, should we improve existing institutions or build new ones in ways that allow for differential tech progress, or is it better to prevent all progress.

The fact that risk from advanced AI is one of the top cause areas is to me an example of at least part of EA being technopessimist for a concrete technology. So I don't think there is any fundamental incompatibility, nor that the burden of proof is particularly high, as long as we are talking about specific classes of technology.

If technopessimism requires believing that most new technology is net harmful that's a very different question, and probably does not even have a well defined answer.

"risk from advanced AI is one of the top cause areas is to me an example of at least part of EA being technopessimist" 
...assuming that particular example is a concern of such an impact primarily on humans, could that be articulated as anthropocentric technopessimism ?

On a broader sidebar, there is discussion around technology (particularly computing) in regards to ecological and other limits - e.g. https://computingwithinlimits.org

1. Why would you want to describe it that way? 2. On reflection, I don't think it can be called anthropocentric, no. There are four big groups of beings involved here: Humanity, Animals, Transhumanist post-humanity (hopefully without value-drift), and Unaligned AI. Three of those groups are non-human. Those concerned with AI alignment tend to be fighting in favor of more of those non-human groups than they are fighting against. (It's a bit hard to tell whether we would actually like animals once they could speak, wield guns, occupy vast portions of the accessible universe etc. Might turn out there are fundamental irreconcilable conflicts. None apparent yet, though.)
In my mind animals are defined by their inability to use language. Check out this list of things mostly unique to human language [https://en.wikipedia.org/wiki/Animal_language#Aspects_of_human_language]. P.S. I assume you mean higher animals such as mammals here.
I'm referring to https://en.wikipedia.org/wiki/Uplift_(science_fiction) [https://en.wikipedia.org/wiki/Uplift_(science_fiction)] , sort of under the assumption that if we truly value animals we will eventually give them voice, reason, coherence. On reflection, I guess the most humane form of this would probably consist of just aligning an AI with the animals and letting it advocate for them. There's no guarantee that these beings adapted to live without speech will want it, but an advocate couldn't hurt.
I don't know, a lot of what you're saying still feels anthropocentric to me. (I'll assume mammals again since most animals don't have a brain or spinal chord.) Language isn't just about speech, it's about cognitive processing. "Do you want speech?" is likely not an object you can input to the algorithm of a mammal's brain and get a meaningful response, and the same applies to most of the mental objects humans manipulate as part of using language. Also there's no guarantee mammals even "want" consistent things, plenty of algorithms "want" nothing in particular, they just run. If at all they run in specific directions it's because outside forces terminated those running differently. (To some extent this also applies to humans.) Hence I have difficulty understanding what it means to align an AI to a mammal.
Yeah, that objection does also apply to humans, which is why, despite it being so difficult to extract a coherent extrapolated volition from a mammal brain, we must find a way of doing it, and once we have it, although it might not produce an agenty utility function for things like spiders or bacteria, there's a decent chance it'll work on dogs or pigs.
Right, I understand your opinion now. I do still have some intutions on why CEV shouldn't exist (even for humans), but I'm not sure what the best place to discuss is, it's probably not here.
Hm. Well feel free to notify me if you ever write it up.
Tbvh I'm not sure how to engage with alignment folk because I feel like I'm missing a lot of existing knowledge and mental models that they do. Like I don't get what people think of when they think of "aligned AI", that they go - you know what, we don't even know how to rigorously define this thing yet we're convinced it exists and it is a good thing. But I'll try: -- Bostrom mentions stuff like populating the cosmos with digital minds generating positive experiences, that seems like unusual and one of the things we could possibly want, but not necessarily the only thing we want. If it was the only thing we wanted we could actually work to explicitly specify that as the AI's goal, and that's CEV and hence problem solved. Basically humans want a lot of different things, and we're confused about which things we want more. That doesn't necessarily mean there exists some objective answer as to which of those we want more, that we need to "solve" it. Instead it could just be random - there are neurochemicals firing from different portions of the brain and sometimes one thing wins out, sometimes the other thing wins out. And if you input a sufficiently persuasive sequence of words into this brain it will prioritise some things more, but if you input a different sequence it will prioritise different things more. (An AI with sufficient human manipulation skills can find this easy imo.) Turing machines don't have "values" by default, they have behaviours based on their input. -- I also wrote up some stuff here a month back but idk how coherent it is: https://www.lesswrong.com/posts/uswN6jyxdkgxHWi7F/samuel-shadrach-s-shortform?commentId=phwddxuNNuumAbSxC [https://www.lesswrong.com/posts/uswN6jyxdkgxHWi7F/samuel-shadrach-s-shortform?commentId=phwddxuNNuumAbSxC]
This is just an aside, but it might be informative. I actually think that * single alignment: "This specific blob of meat here "is" an "agent". Figure out its utility function and do that" is going to be simpler to program than * Hard-coded: "make a large number of many "humanlike" things "experience" "happiness"" I think it's clear that.. there seem to be more things in the hard-coded solution that we don't know how to formalize, and we're much further from knowing how to formalize them (we're pretty deep into agency already). And it's fairly likely we'll arrive at a robust account of agency in the process of developing AGI, or as a result of stumbling onto it, it seems to be one of the short paths to it. I agree about the hmm, murcurial quality of of human agency. There seem to be a lot of triggers that "change" the utility function. Note, I don't know if anyone knows what we mean when we say that a utility function was undermined, by a compelling speech, or by acclimating to the cold water, or by falling in love or whatever. It's strange. And humans generally tend to be indifferent to these changes, they identify with process of change itself. In a sense they all must be part of one utility function (~ "when I am in bed I want to stay in bed, but if I need to pee I would prefer to get out of bed, but once I'm out of bed I will not want to go back to bed. That is my will. That is what it means to be human. If you remove this then it's a dystopia."), but somehow we have to exclude changes like... being convinced by a superintelligence's rhetoric that maximizing paperclips is more important than preserving human life. Somehow we know that's a bad change, even though we don't have any intuitive aversion to reading superintelligences' rhetoric. Even though we (Well I think I would be at least) know we'd be convinced by it, somehow we know that we want to exclude that possibility.
For me the very intuitive response to "figure out its utility function" is "this blob of meat doesn't have a utility function". A model that has a utility function is going to be a bad model of human behaviour. The module that causes humans to have desires is the very same module that causes humans to lose pennies [https://arbital.com/p/coherence_theorems/], you can't really have one without the other. I can try justifying this - I have a bunch of different sets of intuitions that all point in the same direction - but it isn't that well written up. Maybe I'll do a better job conversing with you :) https://kroma.substack.com/p/messy-post-intuitions-on-agi [https://kroma.substack.com/p/messy-post-intuitions-on-agi]

Thanks, this is a useful distinction.

Although I will note that the default case is you can't pick-and-choose what innovation will or will not happen, current institutions allow for both good and harm.

Maybe another axis would be something like, should we improve institutions ability to pick-and-choose or is this a doomed endeavour and instead should we adopt changes that slow down all tech progress.

This isn't directly answering your question, but one thing I can easily imagine happening is: some techno-pessimists spend lots of effort pulling together evidence and arguments for one premise of techno-pessimism, rush (or leave unmade) arguments for other crucial premises, and then find it frustrating that this doesn't change many people's minds. My guess is there's, say, 5+ substantive disagreements that are all upstream of typical disagreement on techno-pessimism, so I expect the bigger challenge for techno-pessimists in convincing this community will be, not necessarily having watertight arguments for any one background premise, but doing a thorough enough job at addressing the wide range of background disagreements.

I don't know about the community as a whole, so speaking just for myself, I think I'd require at least intuitive arguments (no RCTs needed) for each of the following premises to consider working on this--I'll leave it to you to judge whether that's a reasonable burden of proof:

  • That a technologically sophisticated future would be bad overall in expectation (i.e. (1)), even under ethical views that are most common in this community
  • That permanently, globally halting technological progress is tractable (in a way that isn't highly objectionable on other grounds)
  • That sufficiently improving the trajectory of AI is relatively intractable (or that AI in particular shouldn't be a dominant consideration)
  • Bonus points: directly addressing heuristics that make techno-pessimism very unintuitive (especially the heuristic that "ban all new technologies" is so un-nuanced/general that it's very suspect)

Other thoughts:

  • The bar also seems very different depending on precisely what organizations/individuals within the community you're trying to convince, and what you're trying to convince them of (e.g. the global health side of the community seems to be unresponsive to careful, intuitive arguments that are unaccompanied by RCTs, while the AI people are very interested in such arguments).
  • I think most of the difficulty comes from the generically very high burden of proof for arguing that any given cause area / intervention is the most effective use of resou
... (read more)
I can understand why the bar is very different for organisations to react to information - some need to react very fast (startups), some need to be very cautious (judiciaries). But does that also apply to individual's personal opinions? Genuinely curious. Fair, although I will note most systemic change is not constrained by capital alone. For instance if you want a policy change in the govt, money can help but you also need to genuinely change the minds of some individuals. Makes sense.
I was thinking that individuals have different background beliefs (and different attentional/incentive environments), and these things create differences in what it will take to change people's minds. E.g. some people in the community are much closer to agreement with techno-pessimists than others--they'll likely have a lower burden of proof. Agreed, I had a broad definition of "resources" in mind (including e.g. labor/careers, connections), but I could have been more clear.
Makes sense!

Thanks this is useful.

even under ethical views that are most common in this community


(in a way that isn't highly objectionable on other grounds)

This shouldn't be too hard if the default case from tech progress is extinction / totalitarianism. But fair.

That sufficiently improving the trajectory of AI is relatively intractable

Short of "solving alignment", some AI alignment researchers already believe it's intractable. And I don't know what a proof of "solving alignment" being impossible looks like, because alignment researchers deliberately state their f... (read more)

Maybe, although I suspect this assumption makes it significantly harder to argue that a technologically sophisticated future is net negative in expectation (since, at least by ethical views that seem especially common in this community, extinction leads to approximately net zero (not net negative) futures, and it seems plausible to me that a totalitarian future--with all the terrible loss of potential that would involve--would still be better than non-existence, i.e. not net negative). Just to clarify, I wouldn't demand that--I'd be looking for at least an intuitive argument that solving alignment is intractable. I agree that's still hard. As a tangent (since I want to focus on tractability rather than possibility, although impossibility would be more than enough to show intractability): the main reason I think that alignment (using roughly this definition [https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6?gi=3f07e433fb5b] of alignment) is possible is: humans can be aligned to other humans; sometimes we act in good faith to try to satisfy another's preferences. So at least some general intelligences can be aligned. And I don't see what could be so special about humans that would make this property unique to us. Returning from the tangent, I'm also optimistic about tractability because: * People haven't been trying for that long, and the field is still very small * At least some prominent, relatively new research directions (e.g. [1] [https://www.lesswrong.com/posts/k7oxdbNaGATZbtEg3/redwood-research-s-current-project] , [2] [https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge] , [3] [https://www.lesswrong.com/posts/2269iGRnWruLHsZ5r/transformer-circuits]) seem promising Yup this seems plausible, you get the bonus points :)
I'm not sure I follow. The alternative is to slow down tech progress and try to preserve the status quo. Or maybe even regress technologically, "spend more time with nature" etc. I assume all ethical views prefer status quo to extinction or totalitarianism - it's just the extent to which they prefer it and the amount of work/time/resources they're willing to put in (relative to other opportunities). Oh I agree, I feel like superintelligence cannot be trusted, atleast the kind that's capable of global power-grabs. So I wouldn't want to give this power to a human being either. I think it's largely because humans don't want consistent things, and cannot possibly want consistent things, short of neurosurgery. So I wouldn't want to delegate power to an agent that is extremely capable of consistent reasoning.
I wonder if we might be using "net negative" differently? By "net negative" I mean "worse than non-existence," not "worse than status quo." So even though we may prefer a stable status quo to imminent extinction, we might still think the latter leaves us at roughly net zero (i.e. not net negative, or at least not significantly net negative). I also suspect that, under many ethical views, some forms of totalitarianism would be better than non-existence (i.e. not net-negative). For example, a totalitarian world in which freedoms/individuality are extremely limited--but most people are mostly happy, and extreme suffering is very rare--seems at least a little better than non-existence, by the lights of many views about value. (A lot of what I'm saying here is based on the assumption that, according to very scope-sensitive views of value: on a scale where -100 is "worst possible future" and 0 is "non-existence" and 100 is "best possible future," a technologically unsophisticated future would be approximately 0, because humanity would miss out on the vast majority of time and space in which we could create (dis)value. Which is why, for a technologically unsophisticated future to be better than the average technologically sophisticated future, the latter has to be net negative.) I'm not sure if you mean that humans' preferences (a) are consistent at any given time but inconsistent over time, or (b) are inconsistent even if we hold time constant. I'd have different responses to the two. Re: (a), I think this would require some way to aggregate different individuals' preferences (applied to the same individual at different times)--admittedly seems tricky but not hopeless? Re: (b), I agree that alignment to inconsistent preferences is impossible. (I also doubt humans can be aligned to other humans' inconsistent preferences--if someone prefers apples to pears and they also prefer pears to apples (as an example of an inconsistent preference), I can't try to do what they wa
Thanks for replying. Yep I think you're using a stronger assumption in your ethical theories that situations are even comparable, if you ignore when they occur. Because the scores you're assigning to situations seem independent of the sequence of events that led to them or the absolute time they occur. I guess whether you make such an assumption or not - does not matter to the fundamental statement that: is better than or As long as this is true, trying to get the first thing to happen (evolve to stable society) instead of second or third is worth doing if it were the only thing we could do in 2021. Question is whether your ethical theory also demands you to do other things that are even more worth doing (such as building utopia), and because you have finite time and resources you want to do that instead. (Which I'd understand.) -- Regarding the AI alignment stuff, I mean (b), but I'm not sure if discussing here would be the best way to go about it. Maybe I should make a separate thread when I'm more confident.
Hm I wouldn't endorse that assumption. I avoided specifying "when"s to communicate more quickly, but I had them in mind something like your examples--agree the times matter. Agreed but only if we add another condition/caveat: that trying to get the first thing to happen also didn't trade off against the probability of very good scenarios not covered in these three scenarios (which it would mathematically have to do, under some assumptions). As an oversimplistic example with made-up numbers, suppose we were facing these probabilities of possible futures: * 20% -- your first scenario (tech stagnation) (10 goodness points) * 5% -- your second scenario (mass suffering) (-1,000,000 goodness points) * 20%--your third scenario (extinction) (-10 goodness points) * 55%--Status quo in 2021 evolving to technologically sophisticated utopia by 2100 (1,000,000 goodness points) And suppose the only action we could take in 2021 would change the above probabilities to the following: * 100% -- your first scenario (tech stagnation) (10 goodness points) * 0% -- your second scenario (mass suffering) (-1,000,000 goodness points) * 0%--your third scenario (extinction) (-10 goodness points) * 0%--Status quo in 2021 evolving to technologically sophisticated utopia by 2100 (1,000,000 goodness points) Then the expected value of not taking the action is 500,000 goodness points, while the expected value of taking the action is 10 goodness points, so taking the action would be very bad / not worthwhile (despite how technically the action falls under your description of "trying to get the first thing to happen (evolve to stable society) instead of second or third [...] if it were the only thing we could do in 2021").
Oh okay, I understood you now. You're right.
3 comments, sorted by Click to highlight new comments since: Today at 9:11 AM

I recently convinced myself to be fairly technopessimistic in the short term (at least relative to some people I talk to in EA, unclear how this compares to e.g. online EAs or the population overall), though it's not a very exciting position and I don't know if I should prioritize writing up this argument over other things I can do that's productive. 

Thanks, this is useful.

Can you give examples of technopessimists "in the wild"? I'm sure there are plenty of examples of "folk technopessimism" but if you mean something more fleshed-out than that I don't think I've seen it expressed or argued for a lot. (That said, I'm not very widely-read, so I'm sure there's lots of stuff out there I don't hear about.)