Why can't we accept the human condition as it existed in 2010?

Thanks for pointing that out. Just to elaborate a little, the table below from Newberry 2021 has some estimates of how valuable the future can be. Even if one does not endorse the total view, person-affecting views may be dominated by possibilities of large future populations of necessary people.

How could AI stop factory farms (aside from making humans extinct)? I'm honestly interested in the connection there. If you're referring to cellular agriculture, I'm not sure why any form of AI would be needed to accomplish that.

I think dying is bad too, and that's why I want to abolish AI. It's an existential risk to humanity and other sentient species on Earth, and anywhere close enough to be reached via interstellar travel at any point in the future. 

"No life extension" and "no AGI" aren't inherently linked, but they are practically linked in some important ways. These are: 

1. Human intelligence may not be enough to solve the hard problems of aging and cancer, meaning we may never develop meaningful life extension tech.

2. Humanity may not have enough time or cultural stability to commit to a megaproject like this (which will likely take centuries at purely human scales of research) before climate change, economic inequality, and other x- and s-risks greatly weaken our species. 

3. Both AGI and life extension come from the same philosophical place: the rejection of our natural limits as biological animals (put there by Nature via billions of years of natural selection). I think this is extremely dangerous, as it encourages us to seek out new x-risks to find a solution to our mortality (AGI being the most obvious, but large-scale gene editing and brain augmentation carry a high extinction chance as well). 

Basically, my argument is that any attempt to escape our mortality is likely to cause more death and suffering than it prevents. In light of this, we should accept our mortality and try to optimize society for 70-80 years of individual life and health. We should train future generations to continue human progress. In other words, we should stick to the model we've used for all of human history before 2020. 

I agree that the current state of non-human animal treatment by humans is atrocious, and animal welfare is my primary cause area because, from a moral perspective, I cannot abide by the way this society treats animals. With that said, I don't see how accelerating AGI would stop animal torture (unless you're referring to human extinction itself, but I'm not convinced AGI would be any better than humans in its treatment of non-human sentient beings). 

Thanks for your response! I did mean to limit my post by saying that I wasn't intending to discuss the practical feasibility of permanently stopping AI progress in the actual world, only the moral desirability of doing so. With that said, I don't think postmodern Western capitalism is the final word on what is possible in either the economic or moral realms. More imagination is needed, I think. 

Thanks for the further reading suggestion -- adding it to my list. 

I will admit that my comments on indefinite delay were intended to be the core of my question, with “forever” being a way to get people to think “if we never figure it out, is it so bad?”

As for the suffering costs of indefinite delay, I think most of those are pretty well-known (more deaths due to diseases, more animal suffering/death due to lack of cellular agriculture [but we don’t need AGI for this], higher x-risk from pandemics and climate effects), with the odd black swan possibility still out there. I think it’s important to consider the counterfactual conditions as well—that is, “other than extinction, what are the suffering costs of NOT indefinite delay?”

More esoteric risks aside (Basilisks, virtual hells, etc.), disinformation, loss of social connection, loss of trust in human institutions, economic crisis and mass unemployment, and a permanent curtailing of human potential by AI (making us permanently a “pet” species, totally dependent on the AGI) seem like the most pressing short-term (</= 0.01-100 years) s-risks of not-indefinite-delay. The amount of energy AI consumes can also exacerbate fossil fuel exhaustion and climate change, which carry strong s-risks (and distant x-risk) as well; this is at least a strong argument for delaying AI until we figure out fusion, high-yield solar, etc.

As for that third question, it was left out because I felt it would make the discussion too broad (the theory plus this practicality seemed like too much). “Can we actually enforce indefinite delay?” and “what if indefinite delay doesn’t reduce our x-risk?” are questions that keep me up at night, and I’ll admit that I don’t know much about the details of arguments centered on compute overhang (I need to do more reading on that specifically). I am convinced that the current path will likely lead to extinction, based on existing works on sudden capabilities increase with AGI combined with its fundamental non-connection with objective or human values.

I’ll end with this—if indefinite delay turns out to increase our x-risk (or if we just can’t do it for sociopolitical reasons), then I truly envy those who were born before 1920—they never had to see the storm that’s coming.

I agree that our civilization is unstable, and climate change, nuclear war, and resource exhaustion are certainly important risks to be considered and mitigated.

With that said, societal collapse—while certainly bad—is not extinction. Resource exhaustion and nuclear war won’t drive us to extinction, and even climate change would have a hard time killing us all (in the absence of other catastrophes, which is certainly not guaranteed).

Humans have recovered from societal collapses several times in the past, so you would have to make some argument as to why this couldn’t happen again should the same thing happen to Western techno-capitalist society.

As an example, if the P(collapse) given AGI is never achieved is 1, it would still be a preferable outcome to pursue versus creating AGI with P(extinction) of > .05 (this probability as cited in the recent AI expert survey). I’m willing to accept a very high level of s-risk(s) to avoid an x-risk with a sufficiently high probability of occurrence, because extinction would be a uniquely tragic event.

I'm not familiar enough with the arguments around this to comment on it intelligently. With that said, nuclear war is not necessarily an extinction event--it is likely that even with a full-scale nuclear exchange between the US, China, and Russia, some small breeding populations of humans would survive somewhere on Earth (source). Hostile AI takeover would likely kill every last human, however. 

Thanks for your response, Alexa! I'd recommend reading anything by Eliezer Yudkowsky (the founder of the Machine Intelligence Research Institute and one of the world's most well-known AI safety advocates), especially his open letter (linked here). This journal article by Joe Carlsmith (who is an EA and I believe a Forum participant as well) gives a more technical case for AI x-risk, and does it from a less pessimistic perspective than Yudkowsky. 

Thank you so much for this comment, Johan! It is really insightful. I agree that working with our evolutionary tendencies, instead of against them, would be the best option. The hard problem, as you mentioned, is how do we do that?

 (I'll give the chapter a read today -- if my power manages to stay on! [there's a Nor'easter hitting where I live]).

I think what has changed since 2010 has been general awareness of transcending human limits as a realistic possibility. Outside of ML researchers, MIRI and the rationality community, who back then considered AGI reshaping society in our lifetimes a serious possibility?

There is a very real psychological difference between the way the average human sees "sci-fi risks" (alien invasion, asteroids, Cthulhu rising) vs. realistic ones (war, poverty, recession, climate change). In 2010 AI was a sci-fi risk, in 2024 it is a realistic one. Most humans are still struggling with that transition, and we are getting technically closer to reaching AGI. This is extremely dangerous.