Yarrow's Quick takes

Invitation for bets

I’m willing to bet that Anthropic’s revenue growth over the next year will be slower than its revenue growth over the last 3 years. I proposed a specific bet here. Anyone who wants can offer to take the other side of that bet. Or you can make a counteroffer.

I’m also willing to make a longer-term bet that the AI industry is in a bubble. I proposed a specific bet for that, too, here. Feel free to offer to take the other side of that bet or make a counteroffer.

I’d also be open to other bets. It seems pointless to bet about whether AGI or transformative AI will be deployed within the next 5-10 years, yet, for the heck of it, I would agree to a bet against that, too. (I’ll make bets for small, nominal amounts of money to be donated to the winner’s charity of choice, since the practical and legal problems with betting are too large otherwise.)

I’d also bet against the deployment of 100,000+ SAE Level 5 fully autonomous vehicles in North America within the next 3 years, if anyone has a strong opinion on that. I’d make a similar bet against the deployment of autonomous humanoid robots in North American households, although we’d have to come up with some specific resolution criteria.

Similarly, I’d bet against any significant level of near-term labour automation by LLMs or generative AI. Or against LLMs becoming capable of performing all sorts of specific tasks well.

On any of these topics, I’m also open to invitations for a public dialogue. (More on that topic here.)

Suggestion: Leverage Research deep dive

Someone (other than me) should write a deep-dive post about the cult Leverage Research and its infiltration of effective altruism.

The story, in brief:

• Leverage Research is a cult.
• Leverage Research organized the first EA Summit in 2013 and the second EA Summit in 2014. The EA Summits were the first effective altruism conferences of any kind.
• Leverage Research also helped to organize the first EA Global conferences, which began in 2015 and continue to this day.
• In 2016, a major EA program, the Pareto Fellowship, was run largely by Leverage Research. There is some evidence the Pareto Fellowship was run in a cult-like fashion.
• Leverage Research eventually gained full control of the Centre for Effective Altruism in 2018 when one of its members, Larissa Hesketh-Rowe, became the CEO. 

The purpose of the deep-dive post would be for people in EA to understand the truth about what happened. And to learn whatever lessons they think they should learn from that.

These are the questions I would recommend asking and attempting to answer in the deep-dive post:

• Is Leverage Research a cult?
• Did it take over the Centre for Effective Altruism?
• Did it organize the EA Summits and the Pareto Fellowship? Did it play an important role in organizing the first EA Globals?
• If so, how could the EA movement, particularly the core international leadership, let this happen?
• If so, what might be the broader ramifications of this for the EA movement?
• What (if anything) is there to learn from this?

I don’t know what the chances would be of actually getting funded, but someone who wanted to spend a lot of time investigating this topic could apply for a $1,000+ grant from the EA Infrastructure Fund. 

I’m not sure if Coefficient Giving (formerly Open Philanthropy) would even consider funding something so small and so specific to EA community self-reflection, but you can look at the relevant info here.

Here are my rules of thumb for improving communication on the EA Forum and in similar spaces online:

• Say what you mean, as plainly as possible.
• Try to use words and expressions that a general audience would understand.
• Be more casual and less formal if you think that means more people are more likely to understand what you're trying to say.
• To illustrate abstract concepts, give examples.
• Where possible, try to let go of minor details that aren't important to the main point someone is trying to make. Everyone slightly misspeaks (or mis... writes?) all the time. Attempts to correct minor details often turn into time-consuming debates that ultimately have little importance. If you really want to correct a minor detail, do so politely, and acknowledge that you're engaging in nitpicking.
• When you don't understand what someone is trying to say, just say that. (And be polite.)
• Don't engage in passive-aggressiveness or code insults in jargon or formal language. If someone's behaviour is annoying you, tell them it's annoying you. (If you don't want to do that, then you probably shouldn't try to communicate the same idea in a coded or passive-aggressive way, either.)
• If you're using an uncommon word or using a word that also has a more common definition in an unusual way (such as "truthseeking"), please define that word as you're using it and — if applicable — distinguish it from the more common way the word is used.
• Err on the side of spelling out acronyms, abbreviations, and initialisms. You don't have to spell out "AI" as "artificial intelligence", but an obscure term like "full automation of labour" or "FAOL" that was made up for one paper should definitely be spelled out.
• When referencing specific people or organizations, err on the side of giving a little more context, so that someone who isn't already in the know can more easily understand who or what you're talking about. For example, instead of just saying "MacAskill" or "Will", say "Will MacAskill" — just using the full name once per post or comment is enough. You could also mention someone's profession (e.g. "philosopher", "economist") or the organization they're affiliated with (e.g. "Oxford University", "Anthropic"). For organizations, when it isn't already obvious in context, it might be helpful to give a brief description. Rather than saying, "I donated to New Harvest and still feel like this was a good choice", you could say "I donated to New Harvest (a charity focused on cell cultured meat and similar biotech) and still feel like this was a good choice". The point of all this is to make what you write easy for more people to understand without lots of prior knowledge or lots of Googling.
• When in doubt, say it shorter.^[1] In my experience, when I take something I've written that's long and try to cut it down to something short, I usually end up with something a lot clearer and easier to understand than what I originally wrote.
• Kindness is fundamental. Maya Angelou said, “At the end of the day people won't remember what you said or did, they will remember how you made them feel.” Being kind is usually more important than whatever argument you're having.

Feel free to add your own rules of thumb.

1. ^{^}

   This advice comes from the psychologist Harriet Lerner's wonderful book Why Won't You Apologize? — given in the completely different context of close personal relationships. I think it also works here.

I used to feel so strongly about effective altruism. But my heart isn't in it anymore.

I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven't been able to sustain a vegan diet for more than a short time. And so on.

But there isn't a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.

These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:

-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI's takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk has dominated the EA Forum for a while. It feels like AGI is what the movement is mostly about now, so now I just disagree with most of what effective altruism is about.

-The extent to which LessWrong culture has taken over or "colonized" effective altruism culture is such a bummer. I know there's been at least a bit of overlap for a long time, but ten years ago it felt like effective altruism had its own, unique culture and nowadays it feels like the LessWrong culture has almost completely taken over. I have never felt good about LessWrong or "rationalism" and the more knowledge and experience of it I've gained, the more I've accumulated a sense of repugnance, horror, and anger toward that culture and ideology. I hate to see that become what effective altruism is like.

-The stories about sexual harassment are so disgusting. They're really, really bad and crazy. And it's so annoying how many comments you see on EA Forum posts about sexual harassment that make exhausting, unempathetic, arrogant, and frankly ridiculous statements, if not borderline incomprehensible in some cases. You see these stories of sexual harassment in the posts and you see evidence of the culture that enables sexual harassment in the comments. Very, very, very bad. Not my idea of a community I can wholeheartedly feel I belong to.

-Kind of a similar story with sexism, racism, and transphobia. The level of underreaction I've seen to instances of racism has been crazymaking. It's similar to the comments under the posts about sexual harassment. You see people justifying or downplaying clearly immoral behaviour. It's sickening.

-A lot of the response to the Nonlinear controversy was disheartening. It was disheartening to see how many people were eager to enable, justify, excuse, downplay, etc. bad behaviour. Sometimes aggressively, arrogantly, and rudely. It was also disillusioning to see how many people were so... easily fooled.

-Nobody talks normal in this community. At least not on this forum, in blogs, and on podcasts. I hate the LessWrong lingo. To the extent the EA Forum has its own distinct lingo, I probably hate that too. The lingo is great if you want to look smart. It's not so great if you want other people to understand what the hell you are talking about. In a few cases, it seems like it might even be deliberate obscurantism. But mostly it's just people making poor choices around communication and writing style and word choice, maybe for some good reasons, maybe for some bad reasons, but bad choices either way. I think it's rare that writing with a more normal diction wouldn't enhance people's understanding of what you're trying to say, even if you're only trying to communicate with people who are steeped in the effective altruist niche. I don't think the effective altruist sublanguage is serving good thinking or good communication.

-I see a lot of interesting conjecture elevated to the level of conventional wisdom. Someone in the EA or LessWrong or rationalist subculture writes a creative, original, evocative blog post or forum post and then it becomes a meme, and those memes end up taking on a lot of influence over the discourse. Some of these ideas are probably promising. Many of them probably contain at least a grain of truth or insight. But they become conventional wisdom without enough scrutiny. Just because an idea is "homegrown" it takes on the force of a scientific idea that's been debated and tested in peer-reviewed journals for 20 years, or a widely held precept of academic philosophy. That seems just intellectually the wrong thing to do and also weirdly self-aggrandizing.

-An attitude I could call "EA exceptionalism", where people assert that people involved in effective altruism are exceptionally smart, exceptionally wise, exceptionally good, exceptionally selfless, etc. Not just above the average or median (however you would measure that), but part of a rare elite and maybe even superior to everyone else in the world. I see no evidence this is true. (In these sorts of discussions, you also sometimes see the lame argument that effective altruism is definitionally the correct approach to life because effective altruism means doing the most good and if something isn't doing the most good, then it isn't EA. The obvious implication of this argument is that what's called "EA" might not be true EA, and maybe true EA looks nothing like "EA". So, this argument is not a defense of the self-identified "EA" movement or community or self-identified "EA" thought.)

-There is a dark undercurrent to some EA thought, along the lines of negative utilitarianism, anti-natalism, misanthropy, and pessimism. I think there is a risk of this promoting suicidal ideation because it basically is suicidal ideation.

-Too much of the discourse seems to revolve around how to control people's behaviours or beliefs. It's a bit too House of Cards. I recently read about the psychologist Kurt Lewin's study on the most effective ways to convince women to use animal organs (e.g. kidneys, livers, hearts) in their cooking during meat shortages during World War II. He found that a less paternalistic approach that showed more respect for the women's was more effective in getting them to incorporate animal organs into their cooking. The way I think about this is: you didn't have to be manipulated to get to the point where you are in believing what you believe or caring this much about this issue. So, instead of thinking of how to best manipulate people, think about how you got to the point where you are and try to let people in on that in an honest, straightforward way. Not only is this probably more effective, it's also more moral and shows more epistemic humility (you might be wrong about what you believe and that's one reason not to try to manipulate people into believing it).

-A few more things but this list is already long enough.

Put all this together and the old stuff I cared about (charity effectiveness, giving what I can, expanding my moral circle) is lost in a mess of other stuff that is antithetical to what I value and what I believe. I'm not even sure the effective altruism movement should exist anymore. The world might be better off if it closed down shop. I don't know. It could free up a lot of creativity and focus and time and resources to work on other things that might end up being better things to work on.

I still think there is value in the version of effective altruism I knew around 2015, when the primary focus was on global poverty and the secondary focus was on animal welfare, and AGI was on the margins. That version of effective altruism is so different from what exists today — which is mostly about AGI and has mostly been taken over by the rationalist subculture — that I have to consider those two different things. Maybe the old thing will find new life in some new form. I hope so.

I’ve seen a few people in the LessWrong community congratulate the community on predicting or preparing for covid-19 earlier than others, but I haven’t actually seen the evidence that the LessWrong community was particularly early on covid or gave particularly wise advice on what to do about it. I looked into this, and as far as I can tell, this self-congratulatory narrative is a complete myth.

Many people were worried about and preparing for covid in early 2020 before everything finally snowballed in the second week of March 2020. I remember it personally.

In January 2020, some stores sold out of face masks in several different cities in North America. (One example of many.) The oldest post on LessWrong tagged with "covid-19" is from well after this started happening. (I also searched the forum for posts containing "covid" or "coronavirus" and sorted by oldest. I couldn’t find an older post that was relevant.) The LessWrong post is written by a self-described "prepper" who strikes a cautious tone and, oddly, advises buying vitamins to boost the immune system. (This seems dubious, possibly pseudoscientific.) To me, that first post strikes a similarly ambivalent, cautious tone as many mainstream news articles published before that post.

If you look at the covid-19 tag on LessWrong, the next post after that first one, the prepper one, is on February 5, 2020. The posts don't start to get really worried about covid until mid-to-late February.

How is the rest of the world reacting at that time? Here's a New York Times article from February 2, 2020, entitled "Wuhan Coronavirus Looks Increasingly Like a Pandemic, Experts Say", well before any of the worried posts on LessWrong:

The Wuhan coronavirus spreading from China is now likely to become a pandemic that circles the globe, according to many of the world’s leading infectious disease experts.

The prospect is daunting. A pandemic — an ongoing epidemic on two or more continents — may well have global consequences, despite the extraordinary travel restrictions and quarantines now imposed by China and other countries, including the United States.

The tone of the article is fairly alarmed, noting that in China the streets are deserted due to the outbreak, it compares the novel coronavirus to the 1918-1920 Spanish flu, and it gives expert quotes like this one:

It is “increasingly unlikely that the virus can be contained,” said Dr. Thomas R. Frieden, a former director of the Centers for Disease Control and Prevention who now runs Resolve to Save Lives, a nonprofit devoted to fighting epidemics.

The worried posts on LessWrong don't start until weeks after this article was published. On a February 25, 2020 post asking when CFAR should cancel its in-person workshop, the top answer cites the CDC's guidance at the time about covid-19. It says that CFAR's workshops "should be canceled once U.S. spread is confirmed and mitigation measures such as social distancing and school closures start to be announced." This is about 2-3 weeks out from that stuff happening. So, what exactly is being called early here?

CFAR is based in the San Francisco Bay Area, as are Lightcone Infrastructure and MIRI, two other organizations associated with the LessWrong community. On February 25, 2020, the city of San Francisco declared a state of emergency over covid. (Nearby, Santa Clara county, where most of what people consider as Silicon Valley is located, declared a local health emergency on February 10.) At this point in time, posts on LessWrong remain overall cautious and ambivalent.

By the time the posts on LessWrong get really, really worried, in the last few days of February and the first week of March, much of the rest of the world was reacting in the same way.

From February 14 to February 25, the S&P 500 dropped about 7.5%. Around this time, financial analysts and economists issued warnings about the global economy.

Between February 21 and February 27, Italy began its first lockdowns of areas where covid outbreaks had occurred.

On February 25, 2020, the CDC warned Americans of the possibility that "disruption to everyday life may be severe". The CDC made this bracing statement:

It's not so much a question of if this will happen anymore, but more really a question of when it will happen — and how many people in this country will have severe illness.

Another line from the CDC:

We are asking the American public to work with us to prepare with the expectation that this could be bad.

On February 26, Canada's Health Minister advised Canadians to stockpile food and medication.

The most prominent LessWrong post from late February warning people to prepare for covid came a few days later, on February 28. So, on this comparison, LessWrong was actually slightly behind the curve. (Oddly, that post insinuates that nobody else is telling people to prepare for covid yet, and congratulates itself on being ahead of the curve.)

In the beginning of March, the number of LessWrong posts tagged with covid-19 posts explodes, and the tone gets much more alarmed. The rest of the world was responding similarly at this time. For example, on February 29, 2020, Ohio declared a state of emergency around covid. On March 4, Governor Gavin Newsom did the same in California. The governor of Hawaii declared an emergency the same day, and over the next few days, many more states piled on.

Around the same time, the general public was becoming alarmed about covid. In the last days of February and the first days of March, many people stockpiled food and supplies. On February 29, 2020, PBS ran an article describing an example of this at a Costco in Oregon:

Worried shoppers thronged a Costco box store near Lake Oswego, emptying shelves of items including toilet paper, paper towels, bottled water, frozen berries and black beans.

“Toilet paper is golden in an apocalypse,” one Costco employee said.

Employees said the store ran out of toilet paper for the first time in its history and that it was the busiest they had ever seen, including during Christmas Eve.

A March 1, 2020 article in the Los Angeles Times reported on stores in California running out of product as shoppers stockpiled. On March 2, an article in Newsweek described the same happening in Seattle:

Speaking to Newsweek, a resident of Seattle, Jessica Seu, said: "It's like Armageddon here. It's a bit crazy here. All the stores are out of sanitizers and [disinfectant] wipes and alcohol solution. Costco is out of toilet paper and paper towels. Schools are sending emails about possible closures if things get worse.

In Canada, the public was responding the same way. Global News reported on March 3, 2020 that a Costco in Ontario ran out bottled water, toilet paper, and paper towels, and that the situation was similar at other stores around the country. The spike in worried posts on LessWrong coincides with the wider public's reaction. (If anything, the posts on LessWrong are very slightly behind the news articles about stores being picked clean by shoppers stockpiling.)

On March 5, 2020, the cruise ship the Grand Princess made the news because it was stranded off the coast of California due to a covid outbreak on board. I remember this as being one seminal moment of awareness around covid. It was a big story. At this point, LessWrong posts are definitely in no way ahead of the curve, since everyone is talking about covid now.

On March 8, 2020, Italy put a quarter of its population under lockdown, then put the whole country on lockdown on March 10. On March 11, the World Health Organization declared covid-19 a global pandemic. (The same day, the NBA suspended the season and Tom Hanks publicly disclosed he had covid.) On March 12, Ohio closed its schools statewide. The U.S. declared a national emergency on March 13. The same day, 15 more U.S. states closed their schools. Also on the same day, Canada's Parliament shut down because of the pandemic. By now, everyone knows it's a crisis.

So, did LessWrong call covid early? I see no evidence of that. The timeline of LessWrong posts about covid follow the same timeline that the world at large reacted to covid, increasing in alarm as journalists, experts, and governments increasingly rang the alarm bells. In some comparisons, LessWrong's response was a little bit behind.

The only curated post from this period (and the post with the third-highest karma, one of only four posts with over 100 karma) tells LessWrong users to prepare for covid three days after the CDC told Americans to prepare, and two days after Canada's Health Minister told Canadians to stockpile food and medication. It was also three days after San Francisco declared a state of emergency. When that post was published, many people were already stockpiling supplies, partly because government health officials had told them to. (The LessWrong post was originally published on a blog a day before, and based on a note in the text apparently written the day before that, but that still puts the writing of the post a day after the CDC warning and the San Francisco declaration of a state of emergency.)

Unless there is some evidence that I didn't turn up, it seems pretty clear the self-congratulatory narrative is a myth. The self-congratulation actually started in that post published on February 28, 2020, which, again, is odd given the CDC's warning three days before (on the same day that San Francisco declared a state of emergency), analysts' and economists' warnings about the global economy a bit before that, and the New York Times article warning about a probable pandemic at the beginning of the month. The post is slightly behind the curve, but it's gloating as if it's way ahead.

Looking at the overall LessWrong post history in early 2020, LessWrong seems to have been, if anything, slightly behind the New York Times, the S&P 500, the CDC, and enough members of the general public to clear out some stores of certain products. By the time LessWrong posting reached a frenzy in the first week of March, the world was already responding — U.S governors were declaring states of emergency, and everyone was talking about and worrying about covid.

I think people should be skeptical and even distrustful toward the claims of the LessWrong community, both on topics like pandemics and about its own track record and mythology. Obviously this myth is self-serving, and it was pretty easy for me to disprove in a short amount of time — so anyone who is curious can check and see that it's not true. The people in the LessWrong community who believe the community called covid early probably believe that because it's flattering. If they actually wondered if this is true or not and checked the timelines, it would become pretty clear that didn't actually happen.

Edited to add on Monday, December 15, 2025 at 3:20pm Eastern:

I spun this quick take out as a full post here. When I submitted the full post, there was no/almost no engagement on this quick take. In the future, I'll try to make sure to publish things only as a quick take or only as a full post, but not both. This was a fluke under unusual circumstances.

Feel free to continue commenting here, cross-post comments from here onto the full post, make new comments on the post, or do whatever you want. Thanks to everyone who engaged and left interesting comments.

The Ezra Klein Show (one of my favourite podcasts) just released an episode with GiveWell CEO Elie Hassenfeld!

Rate limiting on the EA Forum is too strict. Given that people karma downvote because of disagreement, rather than because of quality or civility — or they judge quality and/or civility largely on the basis of what they agree or disagree with — there is a huge disincentive against expressing unpopular or controversial opinions (relative to the views of active EA Forum users, not necessarily relative to the general public or relevant expert communities) on certain topics.

This is a message I saw recently:

You aren't just rate limited for 24 hours once you fall below the recent karma threshold (which can be triggered by one comment that is unpopular with a handful of people), you're rate limited for as many days as it takes you to gain 25 net karma on new comments — which might take a while, since you can only leave one comment per day, and, also, people might keep downvoting your unpopular comment. (Unless you delete it — which I think I've seen happen, but I won't do, myself, because I'd rather be rate limited than self-censor.)

The rate limiting system is a brilliant idea for new users or users who have less than 50 total karma — the ones who have little plant icons next to their names. It's an elegant, automatic way to stop spam, trolling, and other abuses. But my forum account is 2.5 years old and I have over 1,000 karma. I have 24 posts published over 2 years, all with positive karma. My average karma per post/comment is +2.3 (not counting the default karma that all post/comments start with; this is just counting karma from people's votes).

Examples of comments I've gotten downvoted into the net -1 karma or lower range include a methodological critique of a survey that was later accepted to be correct and led to the research report of an EA-adjacent organization getting revised. In another case, a comment was downvoted to negative karma when it was only an attempt to correct the misuse of a technical term in machine learning — a topic which anyone can confirm I've gotten right with a few fairly quick Google searches. People are absolutely not just downvoting comments that are poor quality or rude by any reasonable standard. They are downvoting things they disagree with or dislike for some other reason. (There are many other examples like the ones I just gave, including everything from directly answering a question to clarifying a point of disagreement to expressing a fairly anodyne and mainstream opinion that at least some prominent experts in the relevant field agree with.) Given this, karma downvoting as an automatic moderation tool with thresholds this sensitive just discourages disagreement.

One of the most important cognitive biases to look out for in a context like EA is group polarization, which is the tendency of individuals' views to become more extreme once they join a group, even if each of the individuals had less extreme views before joining the group (i.e., they aren't necessarily being converted by a few zealots who already had extreme views before joining). One way to mitigate group polarization is to have a high tolerance for internal disagreement and debate. I think the EA Forum does have that tolerance for certain topics and within certain windows of accepted opinions for most topics that are discussed, but not for other topics or only within a window that is quite narrow if you compare it to, say, the general population or expert opinion. 

For example, 76% of AI experts believe it's unlikely or very unlikely that LLMs will scale to AGI according to one survey, yet the opinion of EA Forum users seems to be the opposite of that. Not everyone on the EA Forum seems to consider the majority expert opinion an opinion worth considering too seriously. To me, that looks like group polarization in action. It's one thing to disagree with expert opinion with some degree of uncertainty and epistemic humility, it's another thing to see expert opinion as beneath serious discussion.

I don't know what specific tweaks to the rate limiting system would be best. Maybe just turn it off altogether for users with over 500 karma (and rely on reporting posts/comments and moderator intervention to handle real problems), or as Jason suggested here, have the karma threshold trigger manual review by a moderator rather than automatic rate limiting. Jason also made some other interesting suggestions for tweaks in that comment and noted, correctly:

Strong downvoting by a committed group is the most obvious way to manipulate the system into silencing those with whom you disagree.

This actually works. I am reluctant to criticize the ideas of or express disagreement with certain organizations/books because of rate limiting, and rate limiting is the #1 thing that makes me feel like just giving up on trying to engage in intellectual debate and discussion and just quit the EA Forum.

I may be slow to reply to any comments on this quick take due to the forum's rate limiting.

The NPR podcast Planet Money just released an episode on GiveWell.

If the people arguing that there is an AI bubble turn out to be correct and the bubble pops, to what extent would that change people's minds about near-term AGI? 

I strongly suspect there is an AI bubble because the financial expectations around AI seem to be based on AI significantly enhancing productivity and the evidence seems to show it doesn't do that yet. This could change — and I think that's what a lot of people in the business world are thinking and hoping. But my view is a) LLMs have fundamental weaknesses that make this unlikely and b) scaling is running out of steam.

Scaling running out of steam actually means three things:

1) Each new 10x increase in compute is less practically or qualitatively valuable than previous 10x increases in compute.

2) Each new 10x increase in compute is getting harder to pull off because the amount of money involved is getting unwieldy.

3) There is an absolute ceiling to the amount of data LLMs can train on that they are probably approaching.

So, AI investment is dependent on financial expectations that are depending on LLMs enhancing productivity, which isn't happening and probably won't happen due to fundamental problems with LLMs and due to scaling becoming less valuable and less feasible. This implies an AI bubble, which implies the bubble will eventually pop. 

So, if the bubble pops, will that lead people who currently have a much higher estimation than I do of LLMs' current capabilities and near-term prospects to lower that estimation? If AI investment turns out to be a bubble, and it pops, would you change your mind about near-term AGI? Would you think it's much less likely? Would you think AGI is probably much farther away?

The economist Tyler Cowen linked to my post on self-driving cars, so it ended up getting a lot more readers than I ever expected. I hope that more people now realize, at the very least, self-driving cars are not an uncontroversial, uncomplicated AI success story. In discussions around AGI, people often say things along the lines of: ‘deep learning solved self-driving cars, so surely it will be able to solve many other problems'. In fact, the lesson to draw is the opposite: self-driving is too hard a problem for the current cutting edge in deep learning (and deep reinforcement learning), and this should make us think twice before cavalierly proclaiming that deep learning will soon be able to master even more complex, more difficult tasks than driving.

Since my days of reading William Easterly's Aid Watch blog back in the late 2000s and early 2010s, I've always thought it was a matter of both justice and efficacy to have people from globally poor countries in leadership positions at organizations working on global poverty. All else being equal, a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from Canada with an equal level of education, an equal ability to network with the right international organizations, etc.

In practice, this is probably hard to do, since it requires crossing language barriers, cultural barriers, geographical distance, and international borders. But I think it's worth it.

So much of what effective altruism does, including around global poverty, including around the most evidence-based and quantitative work on global poverty, relies on people's intuitions, and people's intuitions formed from living in wealthy, Western countries with no connection to or experience of a globally poor country are going to be less accurate than people who have lived in poor countries and know a lot about them.

Simply put, first-hand experience of poor countries is a form of expertise and organizations run by people with that expertise are probably going to be a lot more competent at helping globally poor people than ones that aren't.

A number of podcasts are doing a fundraiser for GiveDirectly: https://www.givedirectly.org/happinesslab2025/ 

Podcast about the fundraiser: https://pca.st/bbz3num9

I just want to point out that I have a degree in philosophy and have never heard the word "epistemics" used in the context of academic philosophy. The word used has always been either epistemology or epistemic as adjective in front of a noun (never on its own, always used as an adjective, not a noun, and certainly never pluralized). 

From what I can tell, "epistemics" seems to be weird EA Forum/LessWrong jargon. Not sure how or why this came about, since this is not obscure philosophy knowledge, nor is it hard to look up.

If you Google "epistemics" philosophy, you get 1) sources like Wikipedia that talk about epistemology, not "epistemics", 2) a post from the EA Forum and a page from the Forethought Foundation, which is an effective altruist organization, 3) some unrelated, miscellaneous stuff (i.e. neither EA-related or academic philosophy-related), and 4) a few genuine but fairly obscure uses of the word "epistemics" in an academic philosophy context. This confirms that the term is rarely used in academic philosophy.

I also don't know what people in EA mean when they say "epistemics". I think they probably mean something like epistemic practices, but I actually don't know for sure.

I would discourage the use of the term "epistemics", particularly as its meaning is unclear, and would advocate for a replacement such as epistemology or epistemic practices (or whatever you like, but not "epistemics").

People in effective altruism or adjacent to it should make some public predictions or forecasts about whether AI is in a bubble. 

Since the timeline of any bubble is extremely hard to predict and isn’t the core issue, the time horizon for the bubble prediction could be quite long, say, 5 years. The point would not be to worry about the exact timeline but to get at the question of whether there is a bubble that will pop (say, before January 1, 2031). 

For those who know more about forecasting than me, and especially for those who can think of good ways to financially operationalize such a prediction, I would encourage you to make a post about this.

[Edited on Nov. 17, 2025 at 3:35 PM Eastern to add: I wrote a full-fledged post about the AI bubble that can prompt a richer discussion. It doesn’t attempt to operationalize the bubble question, but gets into the expert opinions and evidence. I also do my own analysis.]

For now, an informal poll:
 

There are two philosophies on what the key to life is.

The first philosophy is that the key to life is separate yourself from the wretched masses of humanity by finding a special group of people that is above it all and becoming part of that group.

The second philosophy is that the key to life is to see the universal in your individual experience. And this means you are always stretching yourself to include more people, find connection with more people, show compassion and empathy to more people. But this is constantly uncomfortable because, again and again, you have to face the wretched masses of humanity and say "me too, me too, me too" (and realize you are one of them).

I am a total believer in the second philosophy and a hater of the first philosophy. (Not because it's easy, but because it's right!) To the extent I care about effective altruism, it's because of the second philosophy: expand the moral circle, value all lives equally, extend beyond national borders, consider non-human creatures.

When I see people in effective altruism evince the first philosophy, to me, this is a profane betrayal of the whole point of the movement.

One of the reasons (among several other important reasons) that rationalists piss me off so much is their whole worldview and subculture is based on the first philosophy. Even the word "rationalist" is about being superior to other people. If the rationalist community has one founder or leader, it would be Eliezer Yudkowsky. The way Eliezer Yudkowsky talks to and about other people, even people who are actively trying to help him or to understand him, is so hateful and so mean. He exhales contempt. And it isn't just Eliezer — you can go on LessWrong and read horrifying accounts of how some prominent people in the community have treated their employee or their romantic partner, with the stated justification that they are separate from and superior to others. Obviously there's a huge problem with racism, sexism, and anti-LGBT prejudice too, which are other ways of feeling separate and above.

There is no happiness to be found at the top of a hierarchy. Look at the people who think in the most hierarchical terms, who have climbed to the tops of the hierarchies they value. Are they happy? No. They're miserable. This is a game you can't win. It's a con. It's a lie.

In the beautiful words of the Franciscan friar Richard Rohr, "The great and merciful surprise is that we come to God not by doing it right but by doing it wrong!"

(Richard Rohr's episode of You Made It Weird with Pete Holmes is wonderful if you want to hear more.) 

Your help requested:

I’m seeking second opinions on whether my contention in Edit #4 at the bottom of this post is correct or incorrect. See the edit at the bottom of the post for full details.

Brief info:

• My contention is about the Forecasting Research Institute’s recent LEAP survey.

• One of the headline results from the survey is about the probabilities the respondents assign to each of three scenarios.

• However, the question uses an indirect framing — an intersubjective resolution or metaprediction framing.

• The specific phrasing of the question is quite important.

• My contention is, if respondents took the question literally, as written, they did not actually report their probabilities for each scenario, and there is no way to derive their probabilities from what they did report.

• Therefore, the headline result that states the respondents’ probabilities for the three scenarios is not actually true.

If my contention is right, then it means the results of the report are being misreported in a quite significant way. If my contention is wrong, then I must make a mea culpa and apologize to the Forecasting Research Institute for my error.

So, your help requested. Am I right or wrong?

(Note: the post discusses multiple topics, but here I’m specifically asking for opinions on the intersubjective resolution/metaprediction concern raised in Edit #4.)

Self-driving cars are not close to getting solved. Don’t take my word for it. Listen to Andrej Karpathy, the lead AI researcher responsible for the development of Tesla’s Full Self-Driving software from 2017 to 2022. (Karpathy also did two stints as a researcher at OpenAI, taught a deep learning course at Stanford, and coined the term "vibe coding".)

From Karpathy’s October 17, 2025 interview with Dwarkesh Patel:

Dwarkesh Patel 01:42:55

You’ve talked about how you were at Tesla leading self-driving from 2017 to 2022. And you firsthand saw this progress from cool demos to now thousands of cars out there actually autonomously doing drives. Why did that take a decade? What was happening through that time?

Andrej Karpathy 01:43:11

One thing I will almost instantly push back on is that this is not even near done, in a bunch of ways that I’m going to get to. Self-driving is very interesting because it’s definitely where I get a lot of my intuitions because I spent five years on it. It has this entire history where the first demos of self-driving go all the way to the 1980s. You can see a demo from CMU in 1986. There’s a truck that’s driving itself on roads.

Fast forward. When I was joining Tesla, I had a very early demo of Waymo. It basically gave me a perfect drive in 2014 or something like that, so a perfect Waymo drive a decade ago. It took us around Palo Alto and so on because I had a friend who worked there. I thought it was very close and then it still took a long time.

For some kinds of tasks and jobs and so on, there’s a very large demo-to-product gap where the demo is very easy, but the product is very hard. It’s especially the case in cases like self-driving where the cost of failure is too high. Many industries, tasks, and jobs maybe don’t have that property, but when you do have that property, that definitely increases the timelines.

For example, in software engineering, I do think that property does exist. For a lot of vibe coding, it doesn’t. But if you’re writing actual production-grade code, that property should exist, because any kind of mistake leads to a security vulnerability or something like that. Millions and hundreds of millions of people’s personal Social Security numbers get leaked or something like that. So in software, people should be careful, kind of like in self-driving. In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it’s almost unbounded how terrible something could be.

I do think that they share that property. What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.

That’s why these things take so long. It’s definitely formative for me, seeing something that was a demo. I’m very unimpressed by demos. Whenever I see demos of anything, I’m extremely unimpressed by that. If it’s a demo that someone cooked up just to show you, it’s worse. If you can interact with it, it’s a bit better. But even then, you’re not done. You need the actual product. It’s going to face all these challenges when it comes in contact with reality and all these different pockets of behavior that need patching.

We’re going to see all this stuff play out. It’s a march of nines. Each nine is constant. Demos are encouraging. It’s still a huge amount of work to do. It is a critical safety domain, unless you’re doing vibe coding, which is all nice and fun and so on. That’s why this also enforced my timelines from that perspective.

Karpathy elaborated later in the interview:

The other aspect that I wanted to return to is that self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo and so on has very few cars. They’re doing that roughly speaking because they’re not economical. They’ve built something that lives in the future. They’ve had to pull back the future, but they had to make it uneconomical. There are all these costs, not just marginal costs for those cars and their operation and maintenance, but also the capex of the entire thing. Making it economical is still going to be a slog for them.

Also, when you look at these cars and there’s no one driving, I actually think it’s a little bit deceiving because there are very elaborate teleoperation centers of people kind of in a loop with these cars. I don’t have the full extent of it, but there’s more human-in-the-loop than you might expect. There are people somewhere out there beaming in from the sky. I don’t know if they’re fully in the loop with the driving. Some of the time they are, but they’re certainly involved and there are people. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.

I still think there will be some work, as you mentioned, going from environment to environment. There are still challenges to make self-driving real. But I do agree that it’s definitely crossed a threshold where it kind of feels real, unless it’s really teleoperated. For example, Waymo can’t go to all the different parts of the city. My suspicion is that it’s parts of the city where you don’t get good signal. Anyway, I don’t know anything about the stack. I’m just making stuff up.

Dwarkesh Patel 01:50:23

You led self-driving for five years at Tesla.

Andrej Karpathy 01:50:27

Sorry, I don’t know anything about the specifics of Waymo. By the way, I love Waymo and I take it all the time. I just think that people are sometimes a little bit too naive about some of the progress and there’s still a huge amount of work. Tesla took in my mind a much more scalable approach and the team is doing extremely well. I’m kind of on the record for predicting how this thing will go. Waymo had an early start because you can package up so many sensors. But I do think Tesla is taking the more scalable strategy and it’s going to look a lot more like that. So this will still have to play out and hasn’t. But I don’t want to talk about self-driving as something that took a decade because it didn’t take it yet, if that makes sense.

Dwarkesh Patel 01:51:08

Because one, the start is at 1980 and not 10 years ago, and then two, the end is not here yet.

Andrej Karpathy 01:51:14

The end is not near yet because when we’re talking about self-driving, usually in my mind it’s self-driving at scale. People don’t have to get a driver’s license, etc.

I hope the implication for discussions around AGI timelines is clear.

I’m taking a long-term, indefinite hiatus from the EA Forum.

I’ve written enough in posts, quick takes, and comments over the last two months to explain the deep frustrations I have with the effective altruist movement/community as it exists today. (For one, I think the AGI discourse is completely broken and far off-base. For another, I think people fail to be kind to others in ordinary, important ways.)

But the strongest reason for me to step away is that participating in the EA Forum is just too unpleasant. I’ve had fun writing stuff on the EA Forum. I thank the people who have been warm to me, who have had good humour, and who have said interesting, constructive things.

But negativity bias being what it is (and maybe “bias” is too biased a word for it; maybe we should call it “negativity preference”), the few people who have been really nasty to me have ruined the whole experience. I find myself trying to remember names, to remember who’s who, so I can avoid clicking on reply notifications from the people who have been nasty. And this is a sign it’s time to stop.

Psychological safety is such a vital part of online discussion, or any discussion. Open, public forums can be a wonderful thing, but psychological safety is hard to provide on an open, public forum. I still have some faith in open, public forums, but I tend to think the best safety tool is giving authors the ability to determine who is and isn’t allowed to interact with their posts. There is some risk of people censoring disagreement, sure. But nastiness online is a major threat to everything good. It causes people to self-censor (e.g. by quitting the discussion platform or by withholding opinions) and it has terrible effects on discourse and on people’s minds. 

And private discussions are important too. One of the most precious things you can find in this life is someone you can have good conversations with who will maintain psychological safety, keep your confidences, “yes, and” you, and be constructive. Those are the kind of conversations that loving relationships are built on. If you end up cooking something that the world needs to know about, you can turn it into a blog post or a paper or a podcast or a forum post. (I’ve done it before!) But you don’t have to do the whole process leading up to that end product in public.

The EA Forum is unusually good in some important respects, which is kind of sad, because it shows us a glimpse of what maybe could exist on the Internet, without itself realizing that promise.

If anyone wants to contact me for some reason, you can send me a message via the forum and I should get it as an email. Please put your email address in the message so I can respond to you by email without logging back into the forum. 

Take care, everyone.

What AI model does SummaryBot use? And does whoever runs SummaryBot use any special tricks on top of that model? It could just be bias, but SummaryBot seems better at summarizing stuff then GPT-5 Thinking, o3, or Gemini 2.5 Pro, so I'm wondering if it's a different model or maybe just good prompting or something else.

@Toby Tremlett🔹, are you SummaryBot's keeper? Or did you just manage its evil twin?

Here is the situation we're in with regard to near-term prospects for artificial general intelligence (AGI). This is why I'm extremely skeptical of predictions that we'll see AGI within 5 years. 

-Current large language models (LLMs) have extremely limited capabilities. For example, they can't score above 5% on the ARC-AGI-2 benchmark, they can't automate any significant amount of human labour,^[1] and they can only augment human productivity in minor ways in limited contexts.^[2] They make ridiculous mistakes all the time, like saying something that happened in 2025 caused something that happened in 2024, while listing the dates of the events. They struggle with things that are easy for humans, like playing hangman.

-The capabilities of LLMs have been improving slowly. There is only a modest overall difference between GPT-3.5 (the original ChatGPT model), which came out in November 2022, and newer models like GPT-4o, o4-mini, and Gemini 2.5 Pro.

-There are signs that there are diminishing returns to scaling for LLMs. Increasing the size of models and the size of the pre-training data doesn't seem to be producing the desired results anymore. LLM companies have turned to scaling test-time compute to eke out more performance gains, but how far can that go?

-There may be certain limits to scaling that are hard or impossible to overcome. For example once you've trained a model on all the text that exists in the world, you can't keep training on exponentially^[3] more text every year. Current LLMs might be fairly close to running out of exponentially^[4] more text to train on, if they haven't run out already.^[5]

-A survey of 475 AI experts found that 76% think it's "unlikely" or "very unlikely" that "scaling up current AI approaches" will lead to AGI. So, we should be skeptical of the idea that just scaling up LLMs will lead to AGI, even if LLM companies manage to keep scaling them up and improving their performance by doing so.

-Few people have any concrete plan for how to build AGI (beyond just scaling up LLMs). The few people who do have a concrete plan disagree fundamentally on what the plan should be. All of these plans are in the early-stage research phase. (I listed some examples in a comment here.)

-Some of the scenarios people are imagining where we get to AGI in the near future involve strange, exotic, hypothetical process wherein a sub-AGI AI system can automate the R&D that gets us from a sub-AGI AI system to AGI. This requires two things to be true: 1) that doing the R&D needed to create AGI is not a task that would require AGI or human-level AI and 2) that, in the near term, AI systems somehow advance to the point where they're able to do meaningful R&D autonomously. Given that I can't even coax o4-mini or Gemini 2.5 Pro into playing hangman properly, and given the slow improvement of LLMs and the signs of diminishing returns to scaling I mentioned, I don't see how (2) could be true. The arguments for (1) feel very speculative and handwavy. 

Given all this, I genuinely can't understand why some people think there's a high chance of AGI within 5 years. I guess the answer is they probably disagree on most or all of these individual points.

Maybe they think the conventional written question and answer benchmarks for LLMs are fair apples-to-apples comparisons of machine intelligence and human intelligence. Maybe they are really impressed with the last 2 to 2.5 years of progress in LLMs. Making they are confident no limits to scaling or diminishing returns to scaling will stop progress anytime soon. Maybe they are confident that scaling up LLMs is a path to AGI. Or maybe they think LLMs will soon be able to take over the jobs of researchers at OpenAI, Anthropic, and Google DeepMind. 

I have a hunch (just a hunch) that it's not a coincidence many people's predictions are converging (or herding) around 2030, give or take a few years, and that 2029 has been the prophesied year for AGI since Ray Kurzweil's book The Age of Spiritual Machines in 1999. It could be a coincidence. But I have a sense that there has been a lot of pent-up energy around AGI for a long time and ChatGPT was like a match in a powder keg. I don't get the sense that people formed their opinions about AGI timelines in 2023 and 2024 from a blank slate. 

I think many people have been primed for years by people like Ray Kurzweil and Eliezer Yudkowsky and by the transhumanist and rationalist subcultures to look for any evidence that AGI is coming soon and to treat that evidence as confirmation of their pre-existing beliefs. You don't have to be directly influenced by these people or by these subcultures to be influenced. If enough people are influenced by them or a few prominent people are influenced, then you end up getting influenced all the same. And when it comes to making predictions, people seem to have a bias toward herding, i.e., making their predictions more similar to the predictions they've heard, even if that ends up making their predictions less accurate. 

The process by which people come up with the year they think AGI will happen seems especially susceptible to herding bias. You ask yourself when you think AGI will happen. A number pops into your head that feels right. How does this happen? Who knows. 

If you try to build a model to predict when AGI will happen, you still can't get around it. Some of your key inputs to the model will require you to ask yourself a question and wait a moment for a number to pop into your head that feels right. The process by which this happens will still be mysterious. So, the model is ultimately no better than pure intuition because it is pure intuition.

I understand that, in principle, it's possible to make more rigorous predictions about the future than this. But I don't think that applies to predicting the development of a hypothetical technology where there is no expert agreement on the fundamental science underlying that technology, and not much in the way of fundamental science in that area at all. That seems beyond the realm of ordinary forecasting.

1. ^{^}

   This post discusses LLMs and labour automation in the section "Real-World Adoption".

2. ^{^}

   One study I found had mixed results. It looked at the use of LLMs to aid people working in customer support, which seems like it should be one of the easiest kinds of jobs to automate using LLMs. The study found that the LLMs increased productivity for new, inexperienced employees but decreased productivity for experienced employees who already knew the ins and outs of the job: 

   These results are consistent with the idea that generative AI tools may function by exposing lower-skill workers to the best practices of higher-skill workers. Lower-skill workers benefit because AI assistance provides new solutions, whereas the best performers may see little benefit from being exposed to their own best practices. Indeed, the negative effects along measures of chat quality—RR [resolution rate] and customer satisfaction—suggest that AI recommendations may distract top performers or lead them to choose the faster or less cognitively taxing option (following suggestions) rather than taking the time to come up with their own responses.

3. ^{^}

   I'm using "exponentially" colloquially to mean every year the LLM's training dataset grows by 2x or 5x or 10x — something along those lines. Technically, if the training dataset increased by 1% a year, that would be exponential, but let's not get bogged down in unimportant technicalities. 

^{^}

Yup, still using it colloquially.

^{^}

Epoch AI published a paper in June 2024 that predicts LLMs will exhaust the Internet's supply of publicly available human-written text between 2026 and 2032.

Show all footnotes