Engineer working on next-gen satellite navigation at Xona Space Systems. I write about effective-altruist and longtermist topics at nukazaria.substack.com, or you can read about videogames like Braid and The Witness at jacksonw.xyz
The Christians in this story who lived relatively normal lives ended up looking wiser than the ones who went all-in on the imminent-return-of-Christ idea. But of course, if christianity had been true and Christ had in fact returned, maybe the crazy-seeming, all-in Christians would have had huge amounts of impact.
Here is my attempt at thinking up other historical examples of transformative change that went the other way:
Muhammad's early followers must have been a bit uncertain whether this guy was really the Final Prophet. Do you quit your day job in Mecca so that you can flee to Medina with a bunch of your fellow cultists? In this case, it probably would've been a good idea: seven years later you'd be helping lead an army of 100,000 holy warriors to capture the city of Mecca. And over the next thirty years, you'll help convert/conquer all the civilizations of the middle east and North Africa.
Less dramatic versions of the above story could probably be told about joining many fast-growing charismatic social movements (like joining a political movement or revolution). Or, more relevantly to AI, about joining a fast-growing bay-area startup whose technology might change the world (like early Microsoft, Google, Facebook, etc).
You're a physics professor in 1940s America. One day, a team of G-men knock on your door and ask you to join a top-secret project to design an impossible superweapon capable of ending the Nazi regime and stopping the war. Do you quit your day job and move to New Mexico?...
You're a "cypherpunk" hanging out on online forums in the mid-2000s. Despite the demoralizing collapse of the dot-com boom and the failure of many of the most promising projects, some of your forum buddies are still excited about the possibilities of creating an "anonymous, distributed electronic cash system", such as the proposal called B-money. Do you quit your day job to work on weird libertarian math problems?...
People who bet everything on transformative change will always look silly in retrospect if the change never comes. But the thing about transformative change is that it does sometimes occur.
(Also, fortunately our world today is quite wealthy -- AI safety researchers are pretty smart folks and will probably be able to earn a living for themselves to pay for retirement, even if all their predictions come up empty.)
Kind of a repetitive stream-of-consciousness response, but I found this both interesting as a philosophical idea and also annoying/cynical/bad-faith:
This is interesting but also, IMO, kind of a strawman -- what's being attacked is some very specific form of utilitarianism, wheras I think many/most "longtermists" are just interested in making sure that we get some kind of happy long-term future for humanity and are fuzzy about the details. Torres says that "Longtermists would surely argue...", but I would like to see some real longtermists quoted as arguing this!!
Personally, I think that taking total-hedonic-utilitarianism 100% seriously is pretty dumb (if you keep doubling the number of happy people, eventually you get to such high numbers that it seems the moral value has to stop 2x-ing because you've got quadrillions of people living basically identical lives), but I still consider myself a longtermist, because I think society is underrating how bad it would be for a nuclear war or similar catastrophe to wreck civilization.
Personally I would also put some (although not overwhelming) weight on the continuity in World B on account of how it gives life more meaning (or at least it would mean that citizens of World B would be more similar to myself -- like me, they too would plan for the future and think of themselves as being part of a civilization that extends through time, rather than World A which seems like it might develop a weird "nothing matters" culture that I'd find alienating). I think a lot of EAs would agree that something feels off about World A, although the extra 10 billion people is definitely a plus, and that overall it seems like an unsolved philosophical mystery whether it matters if your civilization is stretched out in time or not, or whether there is even an objective "right answer" to that question vs being a matter of purely personal taste. At the end of the day I'm very uncertain as to whether world A vs B is better; population ethics is just a really tricky subject to think about.
So this is a good thought experiment! But it seems pretty cynical to introduce this philosophical thought experiment and then:
1. say that your political opponents would obviously/unanimously endorse World A, when actually I think if you polled EAs you might get a pretty even split or they might favor World B.
2. say that this proves they "don't actually care about the future at all", despite the myriad real-world examples of EAs who are working hard to try and reduce long-term risks from climate change, pandemics, nuclear war, rouge AI, etc.
There is also maybe a bit of a sleight-of-hand in the fact that the total population in both scenarios is only 10-20 billion, which is much smaller than the total population of the best futures we could hope for. This makes the 20-billion-people-all-at-once World A scenario feel like an imminent end of the world (nuclear war in 2100, perhaps?), which makes it feel very bad.
But the only-1-million-people-alive-at-a-time scenario is also bad; Torres just doesn't dwell on it! Maybe I should write an op-ed saying that Torres would "surely argue" in favor of stretching out modern human civilization, so that instead of all 8 billion of us hanging out together, 99.99% of us are in cryosleep at any given time and only a million humans are alive at any given time. I could write about how this proves that Torres "doesn't really care about community or cultural diversity at all", since such a small population would surely create much more of a monoculture than the present-day earth. Think about all the human connections and experiences (for example, the existence of communities built around very niche/rare hobbies, or the ability to go to a show and appreciate the talents of an artist/performer/athlete who's "one-in-a-million", or the experience of being in a big bustling city like New York, population 10m) that would be permanently ruled out in Torres's scenario! (How would we decide who's awake for each generation? Would we go by ethnicity -- first all the Chinese people, then all the Italians, then all the Norwegians, and so forth? Surely it would make more sense to make each generation be a representative slice -- but then you'd destroy very small ethnic groups, like for instance the extremely unique and low-population Hazda, by forcing only one member of the Hazda to be woken up every 8 generations! Would Torres "surely argue" in favor of this atrocity?!) But such attacks would be silly; it seems silly to attack people based on treating their opinions on really abstract unsolved philosophical questions, as if they were concrete political agendas? At the end of the day we're all on the same side just trying to make sure that we don't have a nuclear war; the abstract philosophy isn't driving many actionable political disagreements.
I think the downvotes are coming from the fact that Émile P. Torres has been making similar-ish critiques on the concept of longtermism for a while now. (Plus, in some cases, closer to bad-faith attacks against the EA movement, like I think at one point saying that various EA leaders were trying to promote white supremacism or something?) Thus, people might feel both that this kind of critique is "old news" since it's been made before, and they furthermore might feel opposed to highlighting more op-eds by Torres.
Some previous Torres content which garnered more of the critical engagement that you are seeking:
- "The Case Against Longtermism", from three years ago was one of the earlier op-eds and sparked a pretty lively discussion; this is just one of several detailed Forum posts responding to the essay.
- Some more Forum discussion of "The Dangerous Ideas of 'Longtermism' and 'Existential Risk'", two years ago.
- By contrast, the more-recent "Understanding 'longtermism': Why this suddenly influential philosophy is so toxic", from only one year ago, didn't get many responses because I think people got the impression that Torres is still making similar points and, for their part, not giving much critical engagement to various rebuttals / defenses from the people that Torres is criticizing.
Various "auto-GPT" schemes seem like a good demonstration of power-seeking behavior (and perhaps very limited forms of self-preservation or self-improvement), insofar as auto-GPT setups will often invent basic schemes like "I should try to find a way to earn some money in order to accomplish my goal of X", or "I should start a twitter account to gain some followers", or other similarly "agenty" actions/plans.
This might be a bit of a stretch, but to the extent that LLMs exhibit "sycophancy" (ie, telling people what they want to hear in response to stuff like political questions), this seems like it might be partially fueled by the LLM "specification gaming" the RLHF process? Since I'd expect that an LLM might get higher "helpful/honest/harmless" scores by trying to guess which answers the grader most wants to hear, instead of trying to give its truly most "honest" answer? (But I don't have a super-strong understanding of this stuff, and it's possible that other effects are fueling the sycophancy, such as if most of the sycophancy comes from the base model rather than emerging after RLHF.) But specification gaming seems like such a common phenomenon that there must be better examples out there.
I'm definitely not deeply familiar with any kind of "official EA thinking" on this topic (ie, I don't know any EAs that specialize in nuclear security research / grantmaking / etc). But here are some things I just thought up, which might possibly be involved:
If I had to take a wild guess, I would say that my first Tractability point (as in, "I don't know anybody who works at STRATCOM or the People's Liberation Army Rocket Force") is probably the biggest roadblock in an immediate sense. But maybe EA would have put more effort into building more influence here if we had prioritized nuclear risk more from the start -- and perhaps that lack of historical emphasis is due to some mix of the other problems I mentioned?
reposting a reply by Omnizoid from Lesswrong:
"Philosophy is pretty much the only subject that I'm very informed about. So as a consequence, I can confidently say Eliezer is eggregiously wrong about most of the controversial views I can fact check him on. That's . . . worrying."
And my reply to that:
Some other potentially controversial views that a philosopher might be able to fact-check Eliezer on, based on skimming through an index of the sequences:
Maybe you toss out half of those because they aren't seriously disputed by any legit academics. But, I am pretty sure that at least postmodern philosophers, "complexity scientists", people with bad takes on philosophy-of-science / philosophy-of-probability, and people who make "On the Impossibility of Supersized Machines"-style arguments about intelligence, are really out there! They at least consider themselves to be legit, even if you and I are skeptical! So I think EY would come across with a pretty good track record of correct philosophy at the end of the day, if you truly took the entire reference class of "controversial philosophical claims" and somehow graded how correct EY was (in practice, since we haven't yet solved philosophy -- how close he is to your own views?), and compared this to how correct the average philosopher is.
I suggest maybe re-titling this post to:
"I strongly disagree with Eliezer Yudkowsky about the philosophy of consciousness and decision theory, and so do lots of other academic philosophers"
or maybe:
"Eliezer Yudkowsky is Frequently, Confidently, Egregiously Wrong, About Metaphysics"
or consider:
"Eliezer's ideas about Zombies, Decision Theory, and Animal Consciousness, seem crazy"
Otherwise it seems pretty misleading / clickbaity (and indeed overconfident) to extrapolate from these beliefs, to other notable beliefs of Eliezer's -- such as cryonics, quantum mechanics, macroeconomics, various political issues, various beliefs about AI of course, etc. Personally, I clicked on this post really expecting to see a bunch of stuff like "in March 2022 Eliezer confidently claimed that the government of Russia would collapse within 90 days, and it did not", or "Eliezer said for years that X approach to AI couldn't possibly scale, but then it did".
Personally, I feel that beliefs within this narrow slice of philosophy topics are unlikely to correlate to being "egregiously wrong" in other fields. (Philosophy is famously hard!! So even though I agree with you that his stance on animal consciousness seems pretty crazy, I don't really hold this kind of philosophical disagreement against people when they make predictions about, eg, current events.)
I agree, and I think your point applies equally well to the original Eliezer Zombie discussion, as to this very post. In both cases, trying to extrapolate from "I totally disagree with this person on [some metaphysical philosophical questions]" to "these people are idiots who are wrong all the time, even on more practical questions", seems pretty tenuous.
But all three parts of this "takedown" are about questions of philosophy / metaphysics? How do you suggest that I "follow the actual evidence" and avoid "first principles reasoning" when we are trying to learn about the nature of consciousness or the optimal way to make decisions??
To answer with a sequence of increasingly "systemic" ideas (naturally the following will be tinged by by own political beliefs about what's tractable or desirable):
There are lots of object-level lobbying groups that have strong EA endorsement. This includes organizations advocating for better pandemic preparedness (Guarding Against Pandemics), better climate policy (like CATF and others recommended by Giving Green), or beneficial policies in third-world countries like salt iodization or lead paint elimination.
Some EAs are also sympathetic to the "progress studies" movement and to the modern neoliberal movement connected to the Progressive Policy Institute and the Niskasen Center (which are both tax-deductible nonprofit think-tanks). This often includes enthusiasm for denser ("yimby") housing construction, reforming how science funding and academia work in order to speed up scientific progress (such as advocated by New Science), increasing high-skill immigration, and having good monetary policy. All of those cause areas appear on Open Philanthropy's list of "U.S. Policy Focus Areas".
Naturally, there are many ways to advocate for the above causes -- some are more object-level (like fighting to get an individual city to improve its zoning policy), while others are more systemic (like exploring the feasibility of "Georgism", a totally different way of valuing and taxing land which might do a lot to promote efficient land use and encourage fairer, faster economic development).
One big point of hesitancy is that, while some EAs have a general affinity for these cause areas, in many areas I've never heard any particular standout charities being recommended as super-effective in the EA sense... for example, some EAs might feel that we should do monetary policy via "nominal GDP targeting" rather than inflation-rate targeting, but I've never heard anyone recommend that I donate to some specific NGDP-targeting advocacy organization.
I wish there were more places like Center for Election Science, living purely on the meta level and trying to experiment with different ways of organizing people and designing democratic institutions to produce better outcomes. Personally, I'm excited about Charter Cities Institute and the potential for new cities to experiment with new policies and institutions, ideally putting competitive pressure on existing countries to better serve their citizens. As far as I know, there aren't any big organizations devoted to advocating for adopting prediction markets in more places, or adopting quadratic public goods funding, but I think those are some of the most promising areas for really big systemic change.