There is heavy overlap among the effective altruism and rationality communities but they are not the same thing. Within the effective altruism community, especially among those who are newer to the movement and were introduced to it through a university group, I've noticed some tension between the two. I often sense the vibe that sometimes people into effective altruism who haven’t read much of the canonical LessWrong content write off the rationalist stuff as weird or unimportant.
I think this is a pretty big mistake.
Lots of people doing very valuable work within effective altruism got interested in it via first interacting with rationalist content, in particular The Sequences and Harry Potter and the Methods of Rationality. I think that is for good reason. If you haven’t come across those writings before, here’s a nudge to give The Sequences a read.
The Sequences are a (really long) collection of blog posts written by Eliezer Yudkowsky on the science and philosophy of human rationality. They are divided into sequences - a list of posts on a similar topic. Most of the posts would have been pretty useful to me on their own but I also got more value from reading posts in a particular sequence to better internalise the concepts.
There are slightly fewer posts in The Sequences than there are days in the year so reading the whole thing is a very doable thing to do in the coming year! You can also read Highlights from the Sequences which cover 50 of the best essays.
Below, I’ll list some of the parts that I have found especially helpful and that I often try to point to when talking to people into effective altruism (things I wish they had read too).
Fake Beliefs is an excellent sequence if you already know a bit about biases in human thinking. The key insight there is about making beliefs pay rent (“don’t ask what to believe—ask what to anticipate”) and that sometimes your expectations can come apart from your professed beliefs (fake beliefs). The ideas were helpful for me noticing when that happens, for example when I believe I believe something but actually do not. It happens a bunch when I start talking about abstract, wordy things but forget to ask myself what I would actually expect to see in the world if the things I am saying were true.
Noticing Confusion is a cool sequence that talks about things like:
- What is evidence? (“For an event to be evidence about a target of inquiry, it has to happen differently in a way that’s entangled with the different possible states of the target”)
- Your strength as a rationalist is your ability to be more confused by fiction than by reality - noticing confusion when something doesn’t check out and going EITHER MY MODEL IS FALSE OR THIS STORY IS WRONG
- Absence of evidence is evidence of absence, and conservation of expected evidence (“If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction”)
I am often surrounded by people who are very smart and say convincing-sounding things all the time. The ideas mentioned above have helped me better recognise when I'm confused and when a smooth-sounding argument doesn't match up with how I think the world actually works.
Against rationalisation has things that are useful to remember:
- Knowing about biases can hurt people. Exposing subjects to an apparently balanced set of pro and con arguments will exaggerate their initial polarisation. Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to some biases.
- Not to avoid your belief's real weak points. “Ask yourself what smart people who disagree would say to your first reply, and your second reply. Whenever you catch yourself flinching away from an objection you fleetingly thought of, drag it out into the forefront of your mind”
- Motivated stopping and motivated continuation. You should suspect motivated stopping when you close off search, after coming to a comfortable conclusion, and yet there’s a lot of fast cheap evidence you haven’t gathered yet. You should suspect motivated continuation when some evidence is leaning in a way you don’t like, but you decide that more evidence is needed—expensive evidence that you know you can’t gather anytime soon
- Is that your true rejection? “Is that simple straightforward-sounding reason your true rejection [for a position you disagree with], or does it come from intuition-X or professional-zeitgeist-Y?”
- Entangled truths, contagious lies
I once facilitated an effective altruism intro fellowship. Sometimes, the participants in this fellowship would have criticisms or questions that I hadn't thought of. Even so, my mind would quickly come up with a convincing-sounding response and I would feel very rational. That's rationalisation. This also happens when I'm alone, in the privacy of my own mind: the urge to find a convincing argument against something I don't want to believe and quickly move on, and the urge to put a lot of effort into gathering evidence for something I want to believe. Scary! But I notice it more now.
Cached Thoughts was useful for recognising when I am simply accepting and repeating ideas without actually evaluating or understanding them. Rohin Shah, an AI safety researcher, has previously mentioned that he estimates there are ~50 people in the world who can make a case for working on AI alignment that he wouldn't consider clearly flawed. Lots of people would disagree with Rohin about what counts as a not-clearly-flawed argument but I think the general pattern of “there are way more people who think they know the arguments and can parrot them compared to people who can actually generate them” is true in lots of areas. This is one type of thing that the ideas in this post can help with:
- What patterns are being completed, inside your mind, that you never chose to be there?
- If this idea had suddenly occurred to you personally, as an entirely new thought, how would you examine it critically?
- Try to keep your mind from completing the pattern in the standard, unsurprising, already-known way. It may be that there is no better answer than the standard one, but you can’t think about the answer until you can stop your brain from filling in the answer automatically.
- But is it true? Don’t let your mind complete the pattern! Think!
Every Cause Wants to Be a Cult points out something that happens because of human nature regardless of how worthy your cause is. It points out the need to actively push back against sliding into cultishness. It doesn’t just happen as a result of malevolence or stupidity but whenever you have a group of people with an unusual goal who aren’t always making a constant effort to resist the cult attractor. It doesn’t have suggestions on how to do this (there are other posts that cover that) but just points out that cultishness is the default unless you are actively making an effort to prevent it. For me, this is helpful to remember as I am often a part of groups with unusual goals.
Letting Go is a sequence on, well, how to let go of untrue beliefs when you change your mind instead of holding on. It has posts on The Importance of Saying “Oops”, on using the power of innocent curiosity, on leaving a line of retreat so that you can more easily evaluate the evidence for beliefs that make you uncomfortable, and how to stage a Crisis of Faith when there is a belief you have had for a while that is surrounded by a cloud of known arguments and refutations, that you have invested a lot in, and that has emotional consequences.
I first read these posts when I had doubts about my religious beliefs but they were still a huge part of my identity. The tools presented in the sequence made it easier for me to say “oops” and move on instead of just living with a cloud of doubts. I have found it useful to come back to these ideas when I start noticing uncomfortable doubts about a major belief where changing my mind on it would have emotional and social consequences.
Fake Preferences has some blog posts that I found valuable, especially Not For the Sake of Happiness Alone (helped me notice that my values aren’t reducible to just happiness), Fake Selfishness (people usually aren’t genuinely selfish - I do actually care about things outside myself), and Fake Morality (“The fear of losing a moral compass is itself a moral compass”).
The Quantified Humanism sequence has some bangers that have always been relevant for effective altruists and are especially relevant today. Ends Don’t Justify Means (Among Humans) and Ethical Injunctions caution against doing unethical things for the greater good because we run on corrupted hardware and having rules to not do certain things even if it feels like the right thing to do protects us from our own cleverness. “For example, you shouldn't rob banks even if you plan to give the money to a good cause.”
The Challenging the Difficult sequence is about solving very difficult problems, to make an extraordinary effort to do the impossible. It’s not just inspiring but also helpful for looking at how I am approaching my goals, if I am actually doing what needs to be done and aiming to win or just acting out my role and trying to try.
In summary, I think the Sequences have lots of valuable bits for people aiming to have a positive impact. I have found them valuable for my thinking. If you haven’t encountered them before, I recommend giving them a try.
To complement your recommendation, I would also add that Yudowsky's Sequences end up transmitting a somewhat packaged worldview, and I think that there are some dangers in that.
I agree that their summarization work is valuable, but some more unmediated original sources which could transmit some of the same value might be:
Because Yudkowsky's sequences are so long, and because I think that there is more value in reading the original sources, I'd probably lean towards recommending those instead.
Huh, I think this list of books covers less than half of the ideas in the sequences, so I don't really think this counts as "the original sources". Topics that get pretty extensively covered in the sequences but are absent here:
Like, I don't know, let's look at some randomly selected sequence on LessWrong:
This sequence is particularly focused on noticing confusion and modeling scientific progress. None of the books you list above really cover that at all (The Logic of Science maybe the most, but it's really not its core focus, and is also very technical and has giant holes in the middle of it due to its unfinished nature).
I have read all of the books/content you link above, and I don't think it really has that much overlap with the content of the sequences, and don't expect that someone who has read them to really have gotten close to most of the value of reading the sequences, so I don't currently think this is a good recommendation.
This comment made me more sceptical about reading the sequences. I don't think I can view anyone as an expert on all these topics. Is there a "best of" selection of the sequences somewhere?
I can't speak to Yudkowsky's knowledge of physics, economics, psychology etc, but as someone who studies philosophy I can tell you his philosophical segments are pretty weak.
It's clear that he hasn't read a lot of philosophy and he is very dismissive of the field as a whole. He also has a tendency to reinvent the wheel (e.g his 'Requiredism' is what philosophers would call compatibilism).
When I read the sequences as a teenager I was very impressed by his philosophy, but as I got older and started reading more I realized how little he actually engaged with criticisms of his favorite theories, and when he did he often only engaged with weaker criticisms.
If you want some good introductory texts on philosophy as well as criticism/alternatives to some of his/rationalists most central beliefs e.g physicalism, correspondence theory, scientific realism, the normativity of classical logic (all of which I have rejected as of the moment of this writing) then I highly recommend the Stanford Encyclopedia of Philosophy.
In fairness, my memory of the philpapers survey is that there is more consensus amongst professional philosophers on scientific realism than on almost any other philosophical theory. (Though that's going by the old survey, haven't looked at the more recent one yet.) Although of course there are prominent philosophers of science who are anti-realist.
True, here are the results you're talking about:
His views are moderately popular in general with:
I will say that PhilPapers has a rather small sample size and mostly collects data on english speaking philosophers, so I find it probable that these results are not representative of philosophers as a whole.
That's true, I would only really trust the survey for what analytic philosophers think.
I did not say that the sequences cover all content in these books! I mean, they are quite long, so they cover a lot of adjacent topics, but I would not claim that the sequences are the canonical resource on all of these.
Eliezer isn't (to my knowledge) an expert on, say, evolutionary biology. Reading the sequences will not make you an expert on evolutionary biology either.
They will, however, show you how to make a layman's understanding of evolutionary biology relevant to your life.
I agree that my book list is incomplete, and it was aimed more at topics that the OP brought up.
For each of the additional topics you mentioned, it doesn't seem like Yudkowsky's Sequences are the best introduction. E.g., for decision theory I got more out of reading a random MIRI paper trying to formalize FDT. For AI x-risk in particular it would also surprise me if you would also recommend the sequences rather than some newer introduction.
Is this literally true? In particular, have you read David's Sling?
Yeah, I think the best TDT/FDT/LDY material in-particular is probably MIRI papers. The original TDT paper is quite good, and I consider it kind of part of the sequences, since it's written around the same time, and written in a pretty similar style.
Nope, still think the sequences are by far the best (and indeed most alignment conversations I have with new people who showed up in the last 5 years tend to consist of me summarizing sequences posts, which has gotten pretty annoying after a while). There is of course useful additional stuff, but if someone wanted to start working on AI Alignment, the sequences still seem by far the best large thing to read (there are of course individual articles that do individual things best, but there isn't really anything else textbook shaped).
What are the core pieces about AI risk in the sequences? Looking through the list, I don't see any sequence about AI risk. Yudkowsky's account on the Alignment Forum doesn't have anything more than six years old, aka nothing from the sequences era.
Personally I'd point to Joe Carlsmith's report, Richard Ngo's writeups, Ajeya Cotra's writeup, some of Holden Karnofsky's writing, Concrete Problems in AI Safety and Unsolved Problems in ML Safety as the best introductions to the topic.
The primary purpose of the sequences was to communicate the generators behind AI risk and to teach the tools necessary (according to Eliezer) to make progress on it, so references to it are all over the place, and it's the second most central theme to the essays.
Later essays in the sequences tend to have more references to AI risk than earlier ones. Here is a somewhat random selection of ones that seemed crucial when looking over the list, though this is really very unlikely to be comprehensive:
There are lots more. Indeed, towards the latter half of the sequences it's hard not to see an essay quite straightforwardly about AI Alignment every 2-3 essays.
My guess is that he meant the sequences convey the kind of more foundational epistemology which helps people people derive better models on subjects like AI Alignment by themselves, though all of the sequences in The Machine in the Ghost and Mere Goodness have direct object-level relevance.
Excepting Ngo's AGI safety from first principles, I don't especially like most of those resources as introductions exactly because they offer readers very little opportunity to test or build on their beliefs. Also, I think most of them are substantially wrong. (Concrete Problems in AI Safety seems fine, but is also skipping a lot of steps. I haven't read Unsolved Problems in ML Safety.)
Out of curiosity, is this literally true? In particular, have you read David's Sling?
I have read a good chunk of David's Sling! Though it didn't really click with me a ton, and I had already been spoiled on a good chunk of it because I had a bunch of conversations about it with friends, so I didn't fully finish it.
For completeness sake, here is my reading state for all of the above:
Read
I read a bunch of general semantics stuff over the years, but I never really got into it, so a bit unclear.
Yep, read Feeling Good
Read
Read
Read
All three read more than 30%. I think I finished Stranger in a Strange Land and Moon is a Harsh Mistress, but I honestly don't remember.
Yep, read both
Read like 50% of it, but got bored because a lot of it was covering Overcoming Bias stuff that I was familiar with.
Read superforecasting. Have made 100 forecasts, though haven't been that great at keeping track.
Read 30% of it, then stopped because man, I think that book really was a huge disappointment. Would not recommend reading. See also this review by Stuart Ritchie: https://twitter.com/stuartjritchie/status/819140439827681280?lang=en
Read Nonviolent Communication
Thanks for the comment and the list of recommendations. I have read most of the things on that list and the ones that I did read, I thought were great and have recommended a bunch to others, especially Probability Theory: The Logic of Science, The Elephant in the Brain, and forecasting practice. I agree that there are some dangers in recommending something that is pretty packaged but I think there is an obvious benefit in that it feels like a distilled version of reading a bunch of valuable things. Per unit time, I found reading the sequences more insightful/useful to me than the sources which it gets its ideas from (even if I read those things before the sequences, I am fairly confident).
I don't want to oversell the sequences, I think the ideas in them have been mentioned in other places earlier. In my post, I mentioned specifically what ideas I found valuable so that people who are already familiar with them or think they are not that useful can decide not to read them. That isn't rhetorical, I have some wise friends who are pretty well-read on philosophy and economics, and a lot of the things in the sequences I found novel, they were already familiar with.
My recommendation would look very different for someone who read the sequences and then made that their whole personality. I do know of some people who overrate them but in some of my specific circles (EA uni groups for eg), I think they are underrated/not given a chance which is why I wrote this post.
Would you recommend Probability Theory: The Logic of Science to people with little math background?
Well, if you are uncertain, note that experimentation here is very cheap, because you can download a copy of each book from an online library and quickly skim it to get a sense. So I'd recommend that.