I’ve seen a few people in the LessWrong community congratulate the community on predicting or preparing for covid-19 earlier than others, but I haven’t actually seen the evidence that the LessWrong community was particularly early on covid or gave particularly wise advice on what to do about it. I looked into this, and as far as I can tell, this self-congratulatory narrative is a complete myth.
Many people were worried about and preparing for covid in early 2020 before everything finally snowballed in the second week of March 2020. I remember it personally.... (read more)
YARROW: Boy, one would have to be a complete moron to think that COVID-19 would not be a big deal as late as Feb 28 2020, i.e. something that would imminently upend life-as-usual. At this point had China locked down long ago, and even Italy had started locking down. Cases in the USA were going up and up, especially when you correct for the (tiny) amount of testing they were doing. The prepper community had certainly noticed, and was out in force buying out masks and such. Many public health authorities were also sounding alarms. What kind of complete moron would not see what’s happening here? Why is lesswrong patting themselves on the back for noticing something so glaringly obvious?
MY REPLY: Yes!! Yes, this is true!! Yes, you would have to be a complete moron to not make this inference!! …But man, by that definition, there sure were an awful lot of complete morons around, i.e. most everyone. LessWrong deserves credit for rising WAY above the incredibly dismal standards set by the public-at-large in the English-speaking world, even if they didn’t particularly surpass the higher standards of many virologists, preppers, etc.
The version of the claim I have heard is not that LW was early to suggest that there might be a pandemic but rather that they were unusually willing to do something about it because they take small-probability high-impact events seriously. Eg. I suspect that you would say that Wei Dai was "late" because their comment came after the nyt article etc, but nonetheless they made 700% betting that covid would be a big deal.
I think it can be hard to remember just how much controversy there was at the time. E.g. you say of March 13, "By now, everyone knows it's a crisis" but sadly "everyone" did not include the California department of public health, who didn't issue stay at home orders for another week.
[I have a distinct memory of this because I told my girlfriend I couldn't see her anymore since she worked at the department of public health (!!) and was still getting a ton of exposure since the California public health department didn't think covid was that big of a deal.]
I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.
That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.
This matters to me for two reasons.
First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.
I like this comment. This topic is always at risk to devolving into a generalized debate between rationalists and their opponents, creating a lot of heat but not light. So it's helpful to keep a fairly tight focus on potentially action-relevant questions (of which the comment identifies one).
2
Yarrow Bouchard 🔸
I've been around EA pretty deeply since 2015, and to some degree since around 2009. My impression is that overall it's what you guessed it might be: "selective memory, or post-hoc narrative construction." Particularly around AI, but also in general with such claims.
(There's a good reason to make specific, dated predictions publicly, in advance, ideally with some clear resolution criteria.)
8
niplav
Thank you, this is very good. Strong upvoted.
I don't exactly trust you to do this in an unbiased way, but this comment seems the state-of-the-art and I love retrospectives on COVID-19. Plausibly I should look into the extent that your story checks out, plus how EA itself, the relevant parts of twitter or prediction platforms like Metaculus compared at the time (which I felt was definitely ahead).
9
Linch
See eg traviswfisher's prediction on Jan 24:
https://x.com/metaculus/status/1248966351508692992
Or this post on this very forum from Jan 26:
https://forum.effectivealtruism.org/posts/g2F5BBfhTNESR5PJJ/concerning-the-recent-2019-novel-coronavirus-outbreak
I wrote this comment on Jan 27, indicating that it's not just a few people worried at the time. I think most "normal" people weren't tracking covid in January.
I think the thing to realize/people easily forget is that everything was really confusing and there was just a ton of contentious debate during the early months. So while there was apparently a fairly alarmed NYT report in early Feb, there were also many other reports in February that were less alarmed, many bad forecasts, etc.
6
Yarrow Bouchard 🔸
It would be easy to find a few examples like this from any large sample of people. As I mentioned in the quick take, in late January, people were clearing out stores of surgical masks in cities like New York.
5
Linch
Why does this not apply to your original point citing a single NYT article?
0
Yarrow Bouchard 🔸
It might, but I cited a number of data points to try to give an overall picture. What's your specific objection/argument?
6
Linch
My overall objection/argument is that you appear to selectively portray data points that show one side, and selectively dismiss data points that show the opposite view. This makes your bottom-line conclusion pretty suspicious.
I also think the rationalist community overreached and their epistemics and speed in early COVID were worse compared to, say, internet people, government officials, and perhaps even the general public in Taiwan. But I don't think the case for them being slower than Western officials or the general public in either the US or Europe is credible, and your evidence here does not update me much.
4
Yarrow Bouchard 🔸
Let's look at the data a bit more thoroughly.
It's clear that in late January 2020, many people in North America were at least moderately concerned about covid-19.
I already gave the example of some stores in a few cities selling out of face masks. That's anecdotal, but a sign of enough fear among enough people to be noteworthy.
What about the U.S. government's reaction? The CDC issued a warning about travelling to China on January 28 and on January 31, the U.S. federal government declared a public health emergency, implemented a mandatory 14-day quarantine for travelers returning to China, and implemented other travel restrictions. Both the CDC warning and the travel restrictions were covered in the press, so many people knew about it, but even before that happened, a lot of people said they were worried.
Here's a Morning Consult poll from January 24-26, 2020:
An Ipsos poll of Canadians from January 27-28 found similar results:
Were significantly more than 37% of LessWrong users very concerned about covid-19 around this time? Did significantly more than 16% think covid-19 posed a threat to themselves and their family?
It's hard to make direct, apples-to-apples comparisons between the general public and the LessWrong community. We don't have polls of the LessWrong community to compare to. But those examples you gave from January 24-January 27, 2020 don't seem different from what we'd expect if the LessWrong community was at about the same level of concern at about the same time as the general public. Even if the examples you gave represented the worries of ~15-40% of the LessWrong community, that wouldn't be evidence that LessWrong users were doing better than average.
I'm not claiming that the LessWrong community was clearly significantly behind. If it was behind at all, it was only by a few days or maybe a week tops (not much in the grand scheme of things), and the evidence isn't clear or rigorous enough to definitively draw a conclusion like that. My cla
4
Linch
Thanks, I find the polls to be much stronger evidence than the other things you've said.
2
Yarrow Bouchard 🔸
I recommend looking at the Morning Consult PDF and checking the different variations of the question to get a fuller picture. People also gave surprisingly high answers for other viruses like Ebola and Zika, but not nearly as high as for covid.
2
Yarrow Bouchard 🔸
If you want a source who is biased in the opposite direction and who generally agrees with my conclusion, take a look here and here. I like this bon mot:
This is their conclusion from the second link:
3
parconley
This is a cool write-up! I'm curious how much/if you Zvi's COVID round-ups you take into account? I wasn't around LessWrong during COVID, but, if I understand correctly, those played a large role in the information flow during that time.
2
Yarrow Bouchard 🔸
I haven't looked into it, but any and all new information that can give a fuller picture is welcome.
1
parconley
Yeah! This is the series that I am referring to: https://www.lesswrong.com/s/rencyawwfr4rfwt5C.
As I understand it, Zvi was quite ahead of the curve with COVID and moved out of New York before others. I could be wrong, though.
2
Yarrow Bouchard 🔸
The first post listed there is from March 2, 2020, so that's relatively late in the timeline we're considering, no? That's 3 days later than the February 28 post I discussed above as the first/best candidate for a truly urgent early warning about covid-19 on LessWrong. (2020 was a leap year, so there was a February 29.)
That first post from March 2 also seems fairly simple and not particularly different from the February 28 post (which it cites).
Rate limiting on the EA Forum is too strict. Given that people karma downvote because of disagreement, rather than because of quality or civility — or they judge quality and/or civility largely on the basis of what they agree or disagree with — there is a huge disincentive against expressing unpopular or controversial opinions (relative to the views of active EA Forum users, not necessarily relative to the general public or relevant expert communities) on certain topics.
This is a message I saw recently:
You aren't just rate limited for 24 hours once you fall below the recent karma threshold (which can be triggered by one comment that is unpopular with a handful of people), you're rate limited for as many days as it takes you to gain 25 net karma on new comments — which might take a while, since you can only leave one comment per day, and, also, people might keep downvoting your unpopular comment. (Unless you delete it — which I think I've seen happen, but I won't do, myself, because I'd rather be rate limited than self-censor.)
The rate limiting system is a brilliant idea for new users or users who have less than 50 total karma — the ones who have little plant icons next to their nam... (read more)
I think this highlights why some necessary design features of the karma system don't translate well to a system that imposes soft suspensions on users. (To be clear, I find a one-comment-per-day limit based on the past 20 comments/posts to cross the line into soft suspension territory; I do not suggest that rate limits are inherently soft suspensions.)
I wrote a few days ago about why karma votes need to be anonymous and shouldn't (at least generally) require the voter to explain their reasoning; the votes suggested general agreement on those points. But a soft suspension of an established user is a different animal, and requires greater safeguards to protect both the user and the openness of the Forum to alternative views.
I should emphasize that I don't know who cast the downvotes that led to Yarrow's soft suspension (which were on this post about MIRI), or why they cast their votes. I also don't follow MIRI's work carefully enough to have a clear opinion on the merits of any individual vote through the lights of the ordinary purposes of karma. So I do not intend to imply dodgy conduct by anyone. But: "Justice must not only be done, but must also be seen to be done." People who are... (read more)
I've really appreciated comments and reflections from @Yarrow Bouchard 🔸 and I think in his case at least this does feel a bit unfair. Its good to encourage new people on the forum, unless they are posting particularly egrarious thing which I don't think he has been.
2
Yarrow Bouchard 🔸
She, but thank you!
4
Thomas Kwa
Assorted thoughts
* Rate limits should not apply to comments on your own quick takes
* Rate limits could maybe not count negative karma below -10 or so, it seems much better to rate limit someone only when they have multiple downvoted comments
* 2.4:1 is not a very high karma:submission ratio. I have 10:1 even if you exclude the april fool's day posts, though that could be because I have more popular opinions, which means that I could double my comment rate and get -1 karma on the extras and still be at 3.5
* if I were Yarrow I would contextualize more or use more friendly phrasing or something, and also not be bothered too much by single downvotes
* From scanning the linked comments I think that downvoters often think the comment in question has bad reasoning and detracts from effective discussion, not just that they disagree
* Deliberately not opining on the echo chamber question
2
Yarrow Bouchard 🔸
Can you explain what you mean by "contextualizing more"? (What a curiously recursive question...)
You definitely have more popular opinions (among the EA Forum audience), and also you seem to court controversy less, i.e. a lot of your posts are about topics that aren't controversial on the EA Forum. For example, if you were to make a pseudonymous account and write posts/comments arguing that near-term AGI is highly unlikely, I think you would definitely get a much lower karma to submission ratio, even if you put just as much effort and care into them as the posts/comments you've written on the forum so far. Do you think it wouldn't turn out that way?
I've been downvoted on things that are clearly correct, e.g. the standard definitions of terms in machine learning (which anyone can Google); a methodological error that the Forecasting Research Institute later acknowledged was correct and revised their research to reflect. In other cases, the claims are controversial, but they are also claims where prominent AI experts like Andrej Karpathy, Yann LeCun, or Ilya Sutskever have said exactly the same thing as I said — and, indeed, in some cases I'm literally citing them — and it would be wild to think these sort of claims are below the quality threshold for the EA Forum. I think that should make you question whether downvotes are a reliable guide to the quality of contributions.
One-off instances of one person downvoting don't bother me that much — that literally doesn't matter, as long as it really is one-off — what bothers me is the pattern. It isn't just with my posts/comments, either, it's across the board on the forum. I see it all the time with other contributors as well. I feel uneasy dragging those people into this discussion without their permission — it's easier to talk about myself — but this is an overall pattern.
Whether reasoning is good or bad is always bound to be controversial when debating about topics that are controversial, about which there is a lo
2
Thomas Kwa
I mean it in this sense; making people think you're not part of the outgroup and don't have objectionable beliefs related to the ones you actually hold, in whatever way is sensible and honest.
Maybe LW is better at using disagreement button as I find it's pretty common for unpopular opinions to get lots of upvotes and disagree votes. One could use the API to see if the correlations are different there.
2
Yarrow Bouchard 🔸
Huh? Why would it matter whether or not I'm part of "the outgroup"...? What does that mean?
6
Thomas Kwa
I think this is a significant reason why people downvote some, but not all, things they disagree with. Especially a member of the outgroup who makes arguments EAs have refuted before and need to reexplain, not saying it's actually you
2
Yarrow Bouchard 🔸
What is "the outgroup"?
6
Thomas Kwa
Claude thinks possible outgroups include the following, which is similar to what I had in mind
2
Yarrow Bouchard 🔸
a) I’m not sure all of those count as someone who would necessarily be an outsider to EA (e.g. Will MacAskill only assigns a 50% probability to consequentialism being correct, and he and others in EA have long emphasized pluralism about normative ethical theories; there’s been an EA system change group on Facebook since 2015 and discourse around systemic change has been happening in EA since before then)
b) Even if you do consider people in all those categories to be outsiders to EA or part of "the out-group", us/them or in-group/out-group thinking seems like a bad idea, possibly leading to insularity, incuriosity, and overconfidence in wrong views
c) It’s especially a bad idea to not only think in in-group/out-group terms and seek to shut down perspectives of "the out-group" but also to cast suspicion on the in-group/out-group status of anyone in an EA context who you happen to disagree with about something, even something minor — that seems like a morally, subculturally, and epistemically bankrupt approach
8
Thomas Kwa
* You're shooting the messenger. I'm not advocating for downvoting posts that smell of "the outgroup", just saying that this happens in most communities that are centered around an ideological or even methodological framework. It's a way you can be downvoted while still being correct, especially from the LEAST thoughtful 25% of EA forum voters
* Please read the quote from Claude more carefully. MacAskill is not an "anti-utilitarian" who thinks consequentialism is "fundamentally misguided", he's the moral uncertainty guy. The moral parliament usually recommends actions similar to consequentialism with side constraints in practice.
I probably won't engage more with this conversation.
4
Mo Putera
I don't know what he meant, but my guess FWIW is this 2014 essay.
0
Yarrow Bouchard 🔸
I understand the general concept of ingroup/outgroup, but what specifically does that mean in this context?
2
Mo Putera
I don't know, sorry. I admittedly tend to steer clear of community debates as they make me sad, probably shouldn't have commented in the first place...
If the people arguing that there is an AI bubble turn out to be correct and the bubble pops, to what extent would that change people's minds about near-term AGI?
I strongly suspect there is an AI bubble because the financial expectations around AI seem to be based on AI significantly enhancing productivity and the evidence seems to show it doesn't do that yet. This could change — and I think that's what a lot of people in the business world are thinking and hoping. But my view is a) LLMs have fundamental weaknesses that make this unlikely and b) scaling is running out of steam.
Scaling running out of steam actually means three things:
2) Each new 10x increase in compute is getting harder to pull off because the amount of money involved is getting unwieldy.
3) There is an absolute ceiling to the amount of data LLMs can train on that they are probably approaching.
So, AI investment is dependent on financial expectations that are depending on LLMs enhancing productivity, which isn't happening and probably won't happen due to fundamental problems with LLMs and due t... (read more)
Here are my rules of thumb for improving communication on the EA Forum and in similar spaces online:
Say what you mean, as plainly as possible.
Try to use words and expressions that a general audience would understand.
Be more casual and less formal if you think that means more people are more likely to understand what you're trying to say.
To illustrate abstract concepts, give examples.
Where possible, try to let go of minor details that aren't important to the main point someone is trying to make. Everyone slightly misspeaks (or mis... writes?) all the time. Attempts to correct minor details often turn into time-consuming debates that ultimately have little importance. If you really want to correct a minor detail, do so politely, and acknowledge that you're engaging in nitpicking.
When you don't understand what someone is trying to say, just say that. (And be polite.)
Don't engage in passive-aggressiveness or code insults in jargon or formal language. If someone's behaviour is annoying you, tell them it's annoying you. (If you don't want to do that, then you probably shouldn't try to communicate the same idea in a coded or passive-aggressive way, either.)
I used to feel so strongly about effective altruism. But my heart isn't in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven't been able to sustain a vegan diet for more than a short time. And so on.
But there isn't a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI's takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk h... (read more)
I'd distinguish here between the community and actual EA work. The community, and especially its leaders, have undoubtedly gotten more AI-focused (and/or publicly admittted to a degree of focus on AI they've always had) and rationalist-ish. But in terms of actual altruistic activity, I am very uncertain whether there is less money being spent by EAs on animal welfare or global health and development in 2025 than there was in 2015 or 2018. (I looked on Open Phil's website and so far this year it seems well down from 2018 but also well up from 2015, but also 2 months isn't much of a sample.) Not that that means your not allowed to feel sad about the loss of community, but I am not sure we are actually doing less good in these areas than we used to.
Yes, this seems similar to how I feel: I think the major donor(s) have re-prioritized, but am not so sure how many people have switched from other causes to AI. I think EA is more left to the grassroots now, and the forum has probably increased in importance. As long as the major donors don't make the forum all about AI - then we have to create a new forum! But as donors change towards AI, the forum will inevitable see more AI content. Maybe some functions to "balance" the forum posts so one gets representative content across all cause areas? Much like they made it possible to separate out community posts?
2
Jeroen Willems🔸
Thanks for sharing this, while I personally believe the shift in focus on AI is justified (I also believe working on animal welfare is more impactful than global poverty), I can definitely sympathize with many of the other concerns you shared and agree with many of them (especially LessWrong lingo taking over, the underreaction to sexism/racism, and the Nonlinear controversy not being taken seriously enough). While I would completely understand in your situation if you don't want to interact with the community anymore, I just want to share that I believe your voice is really important and I hope you continue to engage with EA! I wouldn't want the movement to discourage anyone who shares its principles (like "let's use our time and resources to help others the most"), but disagrees with how it's being put into practice, from actively participating.
I don't think people dropped the ball here really, people were struggling honestly to take accusations of bad behaviour seriously without getting into witch hunt dynamics.
Good point, I guess my lasting impression wasn't entirely fair to how things played out. In any case, the most important part of my message is that I hope he doesn't feels discouraged from actively participating in EA.
2
Benevolent_Rain
On cause prioritization, is there a more recent breakdown of how more and less engaged EAs prioritize? Like an update of this? I looked for this from the 2024 survey but could not find it easily: https://forum.effectivealtruism.org/posts/sK5TDD8sCBsga5XYg/ea-survey-cause-prioritization
I just want to point out that I have a degree in philosophy and have never heard the word "epistemics" used in the context of academic philosophy. The word used has always been either epistemology or epistemic as adjective in front of a noun (never on its own, always used as an adjective, not a noun, and certainly never pluralized).
From what I can tell, "epistemics" seems to be weird EA Forum/LessWrong jargon. Not sure how or why this came about, since this is not obscure philosophy knowledge, nor is it hard to look up.
I agree this is just a unique rationalist use. Same with 'agentic' though that has possibly crossed over into the more mainstream, at least in tech-y discourse.
However I think this is often fine, especially because 'epistemics' sounds better than 'epistemic practices' and means something distinct from 'epistemology' (the study of knowledge).
Always good to be aware you are using jargon though!
There’s no accounting for taste, but 'epistemics' sounds worse to my ear than 'epistemic practices' because the clunky jargoniness of 'epistemics' is just so evident. It’s as if people said 'democratics' instead of 'democracy', or 'biologics' instead of 'biology'.
I also don’t know for sure what 'epistemics' means. I’m just inferring that from its use and assuming it means 'epistemic practices', or something close to that.
'Epistemology' is unfortunately a bit ambiguous and primarily connotes the subfield of philosophy rather than anything you do in practice, but I think it would also be an acceptable and standard use to talk about 'epistemology' as what one does in practice, e.g., 'scientific epistemology' or 'EA epistemology'. It’s a bit similar to 'ethics' in this regard, which is both an abstract field of study and something one does in practice, although the default interpretation of 'epistemology' is the field, not the practice, and for 'ethics' it’s the reverse.
It’s neither here nor there, but I think talking about personal 'agency' (terminology that goes back decades, long predating the rationalist community) is far more elegant than talking about a person being 'agentic'. (For AI agents, it doesn’t matter.)
I find "epistemics" neat because it is shorter than "applied epistemology" and reminds me of "athletics" and the resulting (implied) focus on being more focused on practice. I don't think anyone ever explained what "epistemics" refers to, and I thought it was pretty self-explanatory from the similarity to "athletics".
I also disagree about the general notion that jargon specific to a community is necessarily bad, especially if that jargon has fewer syllables. Most subcultures, engineering disciplines, sciences invent words or abbreviations for more efficient communication, and while some of that may be due to trying to gatekeep, it's so universal that I'd be surprised if it doesn't carry value. There can be better and worse coinages of new terms, and three/four/five-letter abbreviations such as "TAI" or "PASTA" or "FLOP" or "ASARA" are worse than words like "epistemics" or "agentic".
I guess ethics makes the distinction between normative ethics and applied ethics. My understanding is that epistemology is not about practical techniques, and that one can make a distinction here (just like the distinction between "methodology" and "methods").
I tried to figure out if there's a pair of su... (read more)
Applied ethics is still ethical theory, it’s just that applied ethics is about specific ethical topics, e.g. vegetarianism, whereas normative ethics is about systems of ethics, e.g. utilitarianism. If you wanted to distinguish theory from practice and be absolutely clear, you’d have to say something like ethical practices.
I prefer to say epistemic practices rather than epistemics (which I dislike) or epistemology (which I like, but is more ambiguous).
I don’t think the analogy between epistemics and athletics is obvious, and I would be surprised if even 1% of the people who have ever used the term epistemics have made that connection before.
I am very wary of terms that are never defined or explained. It is easy for people to assume they know what they mean, that there’s a shared meaning everyone agrees on. I really don’t know what epistemics means and I’m only assuming it means epistemic practices.
I fear that there’s a realistic chance if I started to ask different people to define epistemics, we would quickly uncover that different people have different and incompatible definitions. For example, some people might think of it as epistemic practices and some people might think of it as epistemological theory.
I am more anti-jargon and anti-acronyms than a lot of people. Really common acronyms, like AI or LGBT, or acronyms where the acronym is far better known than the spelled-out version, like NASA or DVD, are, of course, absolutely fine. PASTA and ASARA are egregious.
I’m such an anti-acronym fanatic I even spell out artificial general intelligence (AGI) and large language model (LLM) whenever I use them for the first time in a post.
My biggest problem with jargon is that nobody knows what it means. The in-group who is supposed to know what it means also doesn’t know what it means. They think they do, but they’re just fooling themselves. Ask them probing questions, and they’ll start to disagree and fight about the definition. This isn’t always true,
People in effective altruism or adjacent to it should make some public predictions or forecasts about whether AI is in a bubble.
Since the timeline of any bubble is extremely hard to predict and isn’t the core issue, the time horizon for the bubble prediction could be quite long, say, 5 years. The point would not be to worry about the exact timeline but to get at the question of whether there is a bubble that will pop (say, before January 1, 2031).
For those who know more about forecasting than me, and especially for those who can think of good w... (read more)
My leading view is that there will be some sort of bubble pop, but with people still using genAI tools to some degree afterwards (like how people kept using the internet after the dot com burst).
Still major uncertainty on my part because I don't know much about financial markets, and am still highly uncertain about the level where AI progress fully stalls.
2
Yarrow Bouchard 🔸
I just realized the way this poll is set up is really confusing. You're currently at "50% 100% probability", which when you look at it on the number line looks like 75%. Not the best tool to use for such a poll, I guess!
2
Yarrow Bouchard 🔸
Oh, sure. People will keep using LLMs.
I don’t know exactly how you’d operationalize an AI bubble. If OpenAI were a public company, you could say its stock price goes down a certain amount. But private companies can control their own valuation (or the public perception of it) to a certain extent, e.g. by not raising more money so their last known valuation is still from their most recent funding round.
Many public companies like Microsoft, Google, and Nvidia are involved in the AI investment boom, so their stocks can be taken into consideration. You can also look at the level of investment and data centre construction.
I don’t think it would be that hard to come up with reasonable resolution criteria, it’s just that this is of course always a nitpicky thing with forecasting and I haven’t spent any time on it yet.
4
Benjamin M.
I'm not exactly sure about the operationalization of this question, but it seems like there's a bubble among small AI startups at the very least. The big players might be unaffected however? My evidence for this is some mix of not seeing a revenue pathway for a lot of these companies that wouldn't require a major pivot, few barriers to entry for larger players if their product becomes successful, and having met a few people who work in AI startups who claim to be optimistic about earnings and stuff but can't really back that up.
2
Yarrow Bouchard 🔸
I don't know much about small AI startups. The bigger AI companies have a problem because their valuations have increased so much and the level of investment they're making (e.g. into building datacentres) is reaching levels that feel unsustainable.
It's to the point where the AI investment, driven primarily by the large AI companies, has significant macroeconomic effects on the United States economy. The popping of an AI bubble could be followed by a U.S. recession.
However, it's a bit complicated, in that case, as to whether to say the popping of the bubble would have "caused" the recession, since there are a lot of factors, such as tariffs. Macroeconomics and financial markets are complicated and I know very little. I'm not nearly an expert.
I don't think small AI startups creating successful products and then large AI companies copying them and outcompeting them would count as a bubble. That sounds like the total of amount of revenue in the industry would be about the same as if the startups succeeded, it just would flow to the bigger companies instead.
The bubble question is about the industry as a whole.
1
Benjamin M.
I do think there's also a significant chance of a larger bubble, to be fair, affecting the big AI companies. But my instinct is that a sudden fall in investment into small startups and many of them going bankrupt would get called a bubble in the media, and that that investment wouldn't necessarily just go into the big companies.
2
Yarrow Bouchard 🔸
I haven’t thought about my exact probability too hard yet, but for now I’ll just say 90% because that feels about right.
2
niplav
I put 30% on this possiblility, maybe 35%. I don't have much more to say than "time horizons!", "look how useful they're becoming in my dayjob & personal life!", "look at the qualitative improvement over the last six years", "we only need to automate machine learning research, which isn't the hardest thing to automate".
Worlds in which we get a bubble pop are worlds in which we don't get a software intelligence explosion, and in which either useful products come too late for the investment to sustain itself or there's not really much many useful products after what we already have. (This is tied in with "are we getting TAI through the things LLMs make us/are able to do, without fundamental insights".
5
David Mathers🔸
I haven't done the sums myself, but do we know for sure that they can't make money without being all that useful, so long as a lot of people interact with them everyday?
Is Facebook "useful"? Not THAT much. Do people pay for it? No, it's free. Instagram is even less useful than Facebook which at least used to actually be good for organizing parties and pub nights. Does META make money? Yes. Does equally useless TikTok make money? I presume so, yes. I think tech companies are pretty expert in monetizing things that have no user fee, and aren't that helpful at work. There's already a massive user base for Chat-GPT etc. Maybe they can monetize it even without it being THAT useful. Or maybe the sums just don't work out for that, I'm not sure. But clearly the market thinks they will make money in expectation. That's a boring reason for rejecting "it's a bubble" claims and bubbles do happen, but beating the market in pricing shares genuinely is quite difficult I suspect.
Of course, there could also be a bubble even if SOME AI companies make a lot of money. That's what happened with the Dot.com bubble.
4
Yarrow Bouchard 🔸
This is an important point to consider. OpenAI is indeed exploring how to put ads on ChatGPT.
My main source of skepticism about this is that the marginal revenue from an online ad is extremely low, but that’s fine because the marginal cost of serving a webpage or loading a photo in an app or whatever is also extremely low. I don’t have a good sense of the actual numbers here, but since a GPT-5 query is considerably more expensive than serving a webpage, this could be a problem. (Also, that’s just the marginal cost. OpenAI, like other companies, also has to amortize all its fixed costs over all its sales, whether they’re ad sales or sales directly to consumers.)
It’s been rumoured/reported (not sure which) that OpenAI is planning to get ChatGPT to sell things to you directly. So, if you ask, "Hey, ChatGPT, what is the healthiest type of soda?", it will respond, "Why, a nice refreshing Coca‑Cola® Zero Sugar of course!" This seems horrible. That would probably drive some people off the platform, but, who knows, it might be a net financial gain.
There are other "useless" ways companies like OpenAI could try to drive usage and try to monetize either via ads or paid subscriptions. Maybe if OpenAI leaned heavily into the whole AI "boyfriends/girlfriends" thing that would somehow pay off — I’m skeptical, but we’ve got to consider all the possibilities here.
3
Yarrow Bouchard 🔸
What do you make of the fact that METR's time horizon graph and METR's study on AI coding assistants point in opposite directions? The graph says: exponential progress! Superhuman coders! AGI soon! Singularity! The study says: overhyped product category, useless tool, tricks people into thinking it helps them when it actually hurts them.
Pretty interesting, no?
3
niplav
Yep, I wouldn't have predicted that. I guess the standard retort is: Worst case! Existing large codebase! Experienced developers!
I know that there's software tools I use >once a week that wouldn't have existed without AI models. They're not very complicated, but they'd've been annoying to code up myself, and I wouldn't have done it. I wonder if there's a slowdown in less harsh scenarios, but it's probably not worth the value of information of running such a study.
I dunno. I've done a bunch of calibration practice[1], this feels like a 30%, I'm calling 30%. My probability went up recently, mostly because some subjectively judged capabilities that I was expecting didn't start showing up.
----------------------------------------
1. My metaculus calibration around 30% isn't great, I'm overconfident there, I'm trying to keep that in mind. My fatebook is slightly overconfident in that range, and who can tell with Manifold. ↩︎
2
Yarrow Bouchard 🔸
There’s a longer discussion of that oft-discussed METR time horizons graph that warrants a post of its own.
My problem with how people interpret the graph is that people slip quickly and wordlessly from step to step in a logical chain of inferences that I don’t think can be justified. The chain of inferences is something like:
AI model performance on a set of very limited benchmark tasks → AI model performance on software engineering in general → AI model performance on everything humans do
I don’t think these inferences are justifiable.
I’m seeking second opinions on whether my contention in Edit #4 at the bottom of this post is correct or incorrect. See the edit at the bottom of the post for full details.
Brief info:
My contention is about the Forecasting Research Institute’s recent LEAP survey.
One of the headline results from the survey is about the probabilities the respondents assign to each of three scenarios.
However, the question uses an indirect framing — an intersubjective resolution or metaprediction framing.
Self-driving cars are not close to getting solved. Don’t take my word for it. Listen to Andrej Karpathy, the lead AI researcher responsible for the development of Tesla’s Full Self-Driving software from 2017 to 2022. (Karpathy also did two stints as a researcher at OpenAI, taught a deep learning course at Stanford, and coined the term "vibe coding".)
From Karpathy’s October 17, 2025 interview with Dwarkesh Patel:
Dwarkesh Patel01:42:55
You’ve talked about how you were at Tesla leading self-driving from 2017 to 2022. And you firsthand saw this progress from c
Since my days of reading William Easterly's Aid Watch blog back in the late 2000s and early 2010s, I've always thought it was a matter of both justice and efficacy to have people from globally poor countries in leadership positions at organizations working on global poverty. All else being equal, a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from Canada with an equal level of education, an equal ability to network with the right international organizations, etc.
In practice, this is probably hard to do, since it requires crossing language barriers, cultural barriers, geographical distance, and international borders. But I think it's worth it.
So much of what effective altruism does, including around global poverty, including around the most evidence-based and quantitative work on global poverty, relies on people's intuitions, and people's intuitions formed from living in wealthy, Western countries with no connection to or experience of a globally poor country are going to be less accurate than people who have lived in poor countries and know a lot about them.
Simply put, first-hand experience of poor countries is a form of expertise and organizations run by people with that expertise are probably going to be a lot more competent at helping globally poor people than ones that aren't.
I agree with most of you say here, indeed all things being equal a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from anywhere else. The problem is your caveats - things are almost never equal...
1) Education systems just aren't nearly as good in lower income countries. This means that that education is sadly barely ever equal. Even between low income countries - a Kenyan once joked with me that "a Ugandan degree holder is like a Kenyan high school leaver". If you look at the top echelon of NGO/Charity leaders from low-income who's charities have grown and scaled big, most have been at least partially educated in richer countries
2) Ability to network is sadly usually so so much higher if you're from a higher income country. Social capital is real and insanely important. If you look at the very biggest NGOs, most of them are founded not just by Westerners, but by IVY LEAGUE OR OXBRIDGE EDUCATED WESTERNERS. Paul Farmer (Partners in Health) from Harvard, Raj Panjabi (LastMile Health) from Harvard. Paul Niehaus (GiveDirectly) from Harvard. Rob Mathers (AMF) Harvard AND Cambridge. With those connections you ca... (read more)
What AI model does SummaryBot use? And does whoever runs SummaryBot use any special tricks on top of that model? It could just be bias, but SummaryBot seems better at summarizing stuff then GPT-5 Thinking, o3, or Gemini 2.5 Pro, so I'm wondering if it's a different model or maybe just good prompting or something else.
@Toby Tremlett🔹, are you SummaryBot's keeper? Or did you just manage its evil twin?
Hey! @Dane Valerie runs SummaryBot, maybe she'd like to comment.
2
Yarrow Bouchard 🔸
Thanks, Toby!
3
Dane Valerie
It used to run on Claude, but I’ve since moved it to a ChatGPT project using GPT-5. I update the system instructions quarterly based on feedback, which probably explains the difference you’re seeing. You can read more in this doc on posting SummaryBot comments.
2
Yarrow Bouchard 🔸
Thank you very much for the info! It's probably down to your prompting, then. Squeezing things into 6 bullet points might be just a helpful format for ChatGPT or for summaries (even human-written ones) in general. Maybe I will try that myself when I want to ask ChatGPT to summarize something.
I also think there's an element of "magic"/illusion to it, though, since I just noticed a couple mistakes SummaryBot made and now its powers seem less mysterious.
There are two philosophies on what the key to life is.
The first philosophy is that the key to life is separate yourself from the wretched masses of humanity by finding a special group of people that is above it all and becoming part of that group.
The second philosophy is that the key to life is to see the universal in your individual experience. And this means you are always stretching yourself to include more people, find connection with more people, show compassion and empathy to more people. But this is constantly uncomfortable because, again and again,... (read more)
[Personal blog] I’m taking a long-term, indefinite hiatus from the EA Forum.
I’ve written enough in posts, quick takes, and comments over the last two months to explain the deep frustrations I have with the effective altruist movement/community as it exists today. (For one, I think the AGI discourse is completely broken and far off-base. For another, I think people fail to be kind to others in ordinary, important ways.)
But the strongest reason for me to step away is that participating in the EA Forum is just too unpleasant. I’ve had fun writing stuff on the... (read more)
Here is the situation we're in with regard to near-term prospects for artificial general intelligence (AGI). This is why I'm extremely skeptical of predictions that we'll see AGI within 5 years.
-Current large language models (LLMs) have extremely limited capabilities. For example, they can't score above 5% on the ARC-AGI-2 benchmark, they can't automate any significant amount of human labour,[1] and they can only augment human productivity in minor ways in limited contexts.[2] They make ridiculous mistakes all the time, like saying somethin... (read more)
Have Will MacAskill, Nick Beckstead, or Holden Karnofsky responded to the reporting by Time that they were warned about Sam Bankman-Fried's behaviour years before the FTX collapse?
Slight update to the odds I’ve been giving to the creation of artificial general intelligence (AGI) before the end of 2032. I’ve been anchoring the numerical odds of this to the odds of a third-party candidate like Jill Stein or Gary Johnson winning a U.S. presidential election. That’s something I think is significantly more probable than AGI by the end of 2032. Previously, I’d been using 0.1% or 1 in 1,000 as the odds for this, but I was aware that these odds were probably rounded.
I took a bit of time to refine this. I found that in 2016, FiveThirtyEight ... (read more)
I don't think this sort of anchoring is a useful thing to do. There is no logical reason for third party presidency success and AGI success to be linked mathematically. It seems like the third party thing is based on much greater empirical grounding.
You linked them because your vague impression of the likelihood of one was roughly equal to the vague impression of the likliehood of the other: If your vague impression of the third party thing changes, it shouldn't change your opinion of the other thing. You think that AGI is 5 times less likely than you previously thought because you got more precise odds about one guy winning the presidency ten years ago?
My (perhaps controversial) view is that forecasting AGI is in the realm of speculation where quantification like this is more likely to obscure understanding than to help it.
2
Yarrow Bouchard 🔸
I don’t think AGI is five times less likely than I did a week ago, I realized the number I had been translating my qualitative, subjective intuition into was five times too high. I also didn’t change my qualitative, subjective intuition of the probability of a third-party candidate winning a U.S. presidential election. What changed was just the numerical estimate of that probability — from an arbitrarily rounded 0.1% figure to a still quasi-arbitrary but at least somewhat more rigorously derived 0.02%. The two outcomes remain logically disconnected.
I agree that forecasting AGI is an area where any sense of precision is an illusion. The level of irreducible uncertainty is incredibly high. As far as I’m aware, the research literature on forecasting long-term or major developments in technology has found that nobody (not forecasters and not experts in a field) can do it with any accuracy. With something as fundamentally novel as AGI, there is an interesting argument that it’s impossible, in principle, to predict, since the requisite knowledge to predict AGI includes the requisite knowledge to build it, which we don’t have — or at least I don't think we do.
The purpose of putting a number on it is to communicate a subjective and qualitative sense of probability in terms that are clear, that other people can understand. Otherwise, its hard to put things in perspective. You can use terms like extremely unlikely, but what does that mean? Is something that has a 5% chance of happening extremely unlikely? So, rolling a natural 20 is extremely unlikely? (There are guides to determining the meaning of such terms, but they rely on assigning numbers to the terms, so we’re back to square one.)
Something that works just as well is comparing the probability of one outcome to the probability of another outcome. So, just saying that the probability of near-term AGI is less than the probability of Jill Stein winning the next presidential election does the trick. I don’t know why I
6
MichaelDickens
I don't think this should be downvoted. It's a perfectly fine example of reasoning transparency. I happen to disagree, but the disagree-vote button is there for a reason.
1
Yarrow Bouchard 🔸
Thank you. Karma downvotes have ceased to mean anything to me.
People downvote for no discernible reason, at least not reasons that are obvious to me, nor that they explain. I'm left to surmise what the reasons might be, including (in some cases) possibly disagreement, pique, or spite.
Neutrally informative things get downvoted, factual/straightforward logical corrections get downvoted, respectful expressions of mainstream expert opinion get downvoted — everything, anything. The content is irrelevant and the tone/delivery is irrelevant. So, I've stopped interpreting downvotes as information.
4
MichaelDickens
What do you mean by this? What is it that you're 95% confident about?
2
Yarrow Bouchard 🔸
Maybe this is a misapplication of the concept of confidence intervals — math is not my strong suit, nor is forecasting, so let me know — but what I had in mind is that I'm forecasting a 0.00% to 0.02% probability range for AGI by the end of 2034, and that if I were to make 100 predictions of a similar kind, more than 95 of them would have the "correct" probability range (whatever that ends up meaning).
But now that I'm thinking about it more and doing a cursory search, I think with a range of probabilities for a given date (e.g. 0.00% to 0.02% by end of 2034) as opposed to a range of years (e.g. 5 to 20 years) or another definite quantity, the probability itself is supposed to represent all the uncertainty and the confidence interval is redundant.
As you can tell, I'm not a forecaster.
6
MichaelDickens
I kinda get what you're saying but I think this is double-counting in a weird way. A 0.01% probability means that if you make 10,000 predictions of that kind, then about one of them should come true. So your 95% confidence interval sounds like something like "20 times, I make 10,000 predictions that each have a probability between 0.00% and 0.02%; and 19 out of 20 times, about one out of the 10,000 predictions comes true."
You could reduce this to a single point probability. The math is a bit complicated but I think you'd end up with a point probability on the order of 0.001% (~10x lower than the original probability). But if I understand correctly, you aren't actually claiming to have a 0.001% credence.
I think there are other meaningful statements you could make. You could say something like, "I'm 95% confident that if I spend 10x longer studying this question, then I would end up with a probability between 0.00% and 0.02%."
2
Yarrow Bouchard 🔸
Yeah, I’m saying the probability is significantly less than 0.02% without saying exactly how much less — that’s much harder to pin down, and there are diminishing returns to exactitude here — so that means it’s a range from 0.00% to <0.02%. Or just <0.02%.
The simplest solution, and the correct/generally recommended solution, seems to be to simply express the probability, unqualified.
Yann LeCun (a Turing Award-winning pioneer of deep learning) leaving Meta AI — and probably, I would surmise, being nudged out by Mark Zuckerberg (or another senior Meta executive) — is a microcosm for everything wrong with AI research today.
LeCun is the rare researcher working on fundamental new ideas to push AI forward on a paradigm level. Zuckerberg et al. seem to be abandoning that kind of work to focus on a mad dash to AGI via LLMs, on the view that enough scaling and enough incremental engineering and R&D will push current LLMs all the way ... (read more)
LeCun is also probably one of the top people to have worsened the AI safety outlook this decade, and from that perspective perhaps his departure is a good thing for the survival of the world, and thus also Meta’s shareholders?
-4
Yarrow Bouchard 🔸
I couldn't disagree more strongly. LeCun makes strong points about AGI, AGI alignment, LLMs, and so on. He's most likely right. I think the probability of AGI by the end of 2032 is significantly less than 1 in 1,000 and the probability of LLMs scaling to AGI is even less than that. There's more explanation in a few of my posts. In order of importance: 1, 2, 3, 4, and 5.
The core ideas that Eliezer Yudkowsky, Nick Bostrom, and others came up with about AGI alignment/control/friendliness/safety were developed long before the deep learning revolution kicked off in 2012. Some of Yudkowsky's and Bostrom's key early writings about these topics are from as far back as the early 2000s. To quote Clara Collier writing in Asterisk:
So, regardless of the timeline of AGI, that's dubious.
LessWrong's intellectual approach has produced about half a dozen cults, but despite many years of effort, millions of dollars in funding, and the hard work of many people across various projects, and despite many advantages, such as connections that can open doors, it has produced nothing of objective, uncontroversial, externally confirmable intellectual, economic, scientific, technical, or social value. The perceived value of anything it has produced is solely dependent on whether you agree or disagree with its worldview — I disagree. LessWrong claims to have innovated a superior form of human thought, and yet has nothing to show for it. The only explanation that makes any sense is that they're wrong, and are just fooling themselves. Otherwise, to quote Eliezer Yudkowsky, they'd be "smiling from on top of a giant heap of utility."
Yudkowsky's and LessWrong's views on AGI are correctly seen by many experts, such as LeCun, as unserious and not credible, and, in turn, the typical LessWrong response to LeCun is unacceptably intellectually bad and doesn't understand his views on a basic level, let alone respond to them convincingly.
Why would any rational person take that seriously?
Just calling yourself rational doesn't make you more rational. In fact, hyping yourself up about how you and your in-group are more rational than other people is a recipe for being overconfidently wrong.
Getting ideas right takes humility and curiosity about what other people think. Some people pay lip service to the idea of being open to changing their mind, but then, in practice, it feels like they would rather die than admit they were wrong.
This is tied to the idea of humiliation. If disagreement is a humiliation contest, changing one's mind can fe... (read more)
Reason: Turned into a full post: https://forum.effectivealtruism.org/posts/GgesGHQmnb6G63peB/microsoft-s-ceo-satya-nadella-says-he-doesn-t-believe-in-agi
I’ve seen a few people in the LessWrong community congratulate the community on predicting or preparing for covid-19 earlier than others, but I haven’t actually seen the evidence that the LessWrong community was particularly early on covid or gave particularly wise advice on what to do about it. I looked into this, and as far as I can tell, this self-congratulatory narrative is a complete myth.
Many people were worried about and preparing for covid in early 2020 before everything finally snowballed in the second week of March 2020. I remember it personally.... (read more)
My gloss on this situation is:
YARROW: Boy, one would have to be a complete moron to think that COVID-19 would not be a big deal as late as Feb 28 2020, i.e. something that would imminently upend life-as-usual. At this point had China locked down long ago, and even Italy had started locking down. Cases in the USA were going up and up, especially when you correct for the (tiny) amount of testing they were doing. The prepper community had certainly noticed, and was out in force buying out masks and such. Many public health authorities were also sounding alarms. What kind of complete moron would not see what’s happening here? Why is lesswrong patting themselves on the back for noticing something so glaringly obvious?
MY REPLY: Yes!! Yes, this is true!! Yes, you would have to be a complete moron to not make this inference!! …But man, by that definition, there sure were an awful lot of complete morons around, i.e. most everyone. LessWrong deserves credit for rising WAY above the incredibly dismal standards set by the public-at-large in the English-speaking world, even if they didn’t particularly surpass the higher standards of many virologists, preppers, etc.
My personal experience: As som... (read more)
Thanks for collecting this timeline!
The version of the claim I have heard is not that LW was early to suggest that there might be a pandemic but rather that they were unusually willing to do something about it because they take small-probability high-impact events seriously. Eg. I suspect that you would say that Wei Dai was "late" because their comment came after the nyt article etc, but nonetheless they made 700% betting that covid would be a big deal.
I think it can be hard to remember just how much controversy there was at the time. E.g. you say of March 13, "By now, everyone knows it's a crisis" but sadly "everyone" did not include the California department of public health, who didn't issue stay at home orders for another week.
[I have a distinct memory of this because I told my girlfriend I couldn't see her anymore since she worked at the department of public health (!!) and was still getting a ton of exposure since the California public health department didn't think covid was that big of a deal.]
I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.
That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.
This matters to me for two reasons.
First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.
Second, these track-record arguments are sometimes used—explicitly or implicitly—to bolster the case for concern about AI risks. If the evidenti... (read more)
Rate limiting on the EA Forum is too strict. Given that people karma downvote because of disagreement, rather than because of quality or civility — or they judge quality and/or civility largely on the basis of what they agree or disagree with — there is a huge disincentive against expressing unpopular or controversial opinions (relative to the views of active EA Forum users, not necessarily relative to the general public or relevant expert communities) on certain topics.
This is a message I saw recently:
You aren't just rate limited for 24 hours once you fall below the recent karma threshold (which can be triggered by one comment that is unpopular with a handful of people), you're rate limited for as many days as it takes you to gain 25 net karma on new comments — which might take a while, since you can only leave one comment per day, and, also, people might keep downvoting your unpopular comment. (Unless you delete it — which I think I've seen happen, but I won't do, myself, because I'd rather be rate limited than self-censor.)
The rate limiting system is a brilliant idea for new users or users who have less than 50 total karma — the ones who have little plant icons next to their nam... (read more)
I think this highlights why some necessary design features of the karma system don't translate well to a system that imposes soft suspensions on users. (To be clear, I find a one-comment-per-day limit based on the past 20 comments/posts to cross the line into soft suspension territory; I do not suggest that rate limits are inherently soft suspensions.)
I wrote a few days ago about why karma votes need to be anonymous and shouldn't (at least generally) require the voter to explain their reasoning; the votes suggested general agreement on those points. But a soft suspension of an established user is a different animal, and requires greater safeguards to protect both the user and the openness of the Forum to alternative views.
I should emphasize that I don't know who cast the downvotes that led to Yarrow's soft suspension (which were on this post about MIRI), or why they cast their votes. I also don't follow MIRI's work carefully enough to have a clear opinion on the merits of any individual vote through the lights of the ordinary purposes of karma. So I do not intend to imply dodgy conduct by anyone. But: "Justice must not only be done, but must also be seen to be done." People who are... (read more)
The NPR podcast Planet Money just released an episode on GiveWell.
A number of podcasts are doing a fundraiser for GiveDirectly: https://www.givedirectly.org/happinesslab2025/
Podcast about the fundraiser: https://pca.st/bbz3num9
If the people arguing that there is an AI bubble turn out to be correct and the bubble pops, to what extent would that change people's minds about near-term AGI?
I strongly suspect there is an AI bubble because the financial expectations around AI seem to be based on AI significantly enhancing productivity and the evidence seems to show it doesn't do that yet. This could change — and I think that's what a lot of people in the business world are thinking and hoping. But my view is a) LLMs have fundamental weaknesses that make this unlikely and b) scaling is running out of steam.
Scaling running out of steam actually means three things:
1) Each new 10x increase in compute is less practically or qualitatively valuable than previous 10x increases in compute.
2) Each new 10x increase in compute is getting harder to pull off because the amount of money involved is getting unwieldy.
3) There is an absolute ceiling to the amount of data LLMs can train on that they are probably approaching.
So, AI investment is dependent on financial expectations that are depending on LLMs enhancing productivity, which isn't happening and probably won't happen due to fundamental problems with LLMs and due t... (read more)
Here are my rules of thumb for improving communication on the EA Forum and in similar spaces online:
- Say what you mean, as plainly as possible.
- Try to use words and expressions that a general audience would understand.
- Be more casual and less formal if you think that means more people are more likely to understand what you're trying to say.
- To illustrate abstract concepts, give examples.
- Where possible, try to let go of minor details that aren't important to the main point someone is trying to make. Everyone slightly misspeaks (or mis... writes?) all the time. Attempts to correct minor details often turn into time-consuming debates that ultimately have little importance. If you really want to correct a minor detail, do so politely, and acknowledge that you're engaging in nitpicking.
- When you don't understand what someone is trying to say, just say that. (And be polite.)
- Don't engage in passive-aggressiveness or code insults in jargon or formal language. If someone's behaviour is annoying you, tell them it's annoying you. (If you don't want to do that, then you probably shouldn't try to communicate the same idea in a coded or passive-aggressive way, either.)
- If you're using an uncommon word
... (read more)I used to feel so strongly about effective altruism. But my heart isn't in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven't been able to sustain a vegan diet for more than a short time. And so on.
But there isn't a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI's takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk h... (read more)
I'd distinguish here between the community and actual EA work. The community, and especially its leaders, have undoubtedly gotten more AI-focused (and/or publicly admittted to a degree of focus on AI they've always had) and rationalist-ish. But in terms of actual altruistic activity, I am very uncertain whether there is less money being spent by EAs on animal welfare or global health and development in 2025 than there was in 2015 or 2018. (I looked on Open Phil's website and so far this year it seems well down from 2018 but also well up from 2015, but also 2 months isn't much of a sample.) Not that that means your not allowed to feel sad about the loss of community, but I am not sure we are actually doing less good in these areas than we used to.
My memory is a large number of people to the NL controversy seriously, and the original threads on it were long and full of hostile comments to NL, and only after someone posted a long piece in defence of NL did some sympathy shift back to them. But even then there are like 90-something to 30-something agree votes and 200 karma on Yarrow's comment saying NL still seem bad: https://forum.effectivealtruism.org/posts/H4DYehKLxZ5NpQdBC/nonlinear-s-evidence-debunking-false-and-misleading-claims?commentId=7YxPKCW3nCwWn2swb
I don't think people dropped the ball here really, people were struggling honestly to take accusations of bad behaviour seriously without getting into witch hunt dynamics.
I just want to point out that I have a degree in philosophy and have never heard the word "epistemics" used in the context of academic philosophy. The word used has always been either epistemology or epistemic as adjective in front of a noun (never on its own, always used as an adjective, not a noun, and certainly never pluralized).
From what I can tell, "epistemics" seems to be weird EA Forum/LessWrong jargon. Not sure how or why this came about, since this is not obscure philosophy knowledge, nor is it hard to look up.
If you Google "epistemics" phil... (read more)
I agree this is just a unique rationalist use. Same with 'agentic' though that has possibly crossed over into the more mainstream, at least in tech-y discourse.
However I think this is often fine, especially because 'epistemics' sounds better than 'epistemic practices' and means something distinct from 'epistemology' (the study of knowledge).
Always good to be aware you are using jargon though!
I find "epistemics" neat because it is shorter than "applied epistemology" and reminds me of "athletics" and the resulting (implied) focus on being more focused on practice. I don't think anyone ever explained what "epistemics" refers to, and I thought it was pretty self-explanatory from the similarity to "athletics".
I also disagree about the general notion that jargon specific to a community is necessarily bad, especially if that jargon has fewer syllables. Most subcultures, engineering disciplines, sciences invent words or abbreviations for more efficient communication, and while some of that may be due to trying to gatekeep, it's so universal that I'd be surprised if it doesn't carry value. There can be better and worse coinages of new terms, and three/four/five-letter abbreviations such as "TAI" or "PASTA" or "FLOP" or "ASARA" are worse than words like "epistemics" or "agentic".
I guess ethics makes the distinction between normative ethics and applied ethics. My understanding is that epistemology is not about practical techniques, and that one can make a distinction here (just like the distinction between "methodology" and "methods").
I tried to figure out if there's a pair of su... (read more)
People in effective altruism or adjacent to it should make some public predictions or forecasts about whether AI is in a bubble.
Since the timeline of any bubble is extremely hard to predict and isn’t the core issue, the time horizon for the bubble prediction could be quite long, say, 5 years. The point would not be to worry about the exact timeline but to get at the question of whether there is a bubble that will pop (say, before January 1, 2031).
For those who know more about forecasting than me, and especially for those who can think of good w... (read more)
Your help requested:
I’m seeking second opinions on whether my contention in Edit #4 at the bottom of this post is correct or incorrect. See the edit at the bottom of the post for full details.
Brief info:
-
-
-
-
... (read more)My contention is about the Forecasting Research Institute’s recent LEAP survey.
One of the headline results from the survey is about the probabilities the respondents assign to each of three scenarios.
However, the question uses an indirect framing — an intersubjective resolution or metaprediction framing.
The specific phrasing of the question is q
Self-driving cars are not close to getting solved. Don’t take my word for it. Listen to Andrej Karpathy, the lead AI researcher responsible for the development of Tesla’s Full Self-Driving software from 2017 to 2022. (Karpathy also did two stints as a researcher at OpenAI, taught a deep learning course at Stanford, and coined the term "vibe coding".)
From Karpathy’s October 17, 2025 interview with Dwarkesh Patel:
... (read more)Since my days of reading William Easterly's Aid Watch blog back in the late 2000s and early 2010s, I've always thought it was a matter of both justice and efficacy to have people from globally poor countries in leadership positions at organizations working on global poverty. All else being equal, a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from Canada with an equal level of education, an equal ability to network with the right international organizations, etc.
In practice, this is probably hard to do, since it requires crossing language barriers, cultural barriers, geographical distance, and international borders. But I think it's worth it.
So much of what effective altruism does, including around global poverty, including around the most evidence-based and quantitative work on global poverty, relies on people's intuitions, and people's intuitions formed from living in wealthy, Western countries with no connection to or experience of a globally poor country are going to be less accurate than people who have lived in poor countries and know a lot about them.
Simply put, first-hand experience of poor countries is a form of expertise and organizations run by people with that expertise are probably going to be a lot more competent at helping globally poor people than ones that aren't.
I agree with most of you say here, indeed all things being equal a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from anywhere else. The problem is your caveats - things are almost never equal...
1) Education systems just aren't nearly as good in lower income countries. This means that that education is sadly barely ever equal. Even between low income countries - a Kenyan once joked with me that "a Ugandan degree holder is like a Kenyan high school leaver". If you look at the top echelon of NGO/Charity leaders from low-income who's charities have grown and scaled big, most have been at least partially educated in richer countries
2) Ability to network is sadly usually so so much higher if you're from a higher income country. Social capital is real and insanely important. If you look at the very biggest NGOs, most of them are founded not just by Westerners, but by IVY LEAGUE OR OXBRIDGE EDUCATED WESTERNERS. Paul Farmer (Partners in Health) from Harvard, Raj Panjabi (LastMile Health) from Harvard. Paul Niehaus (GiveDirectly) from Harvard. Rob Mathers (AMF) Harvard AND Cambridge. With those connections you ca... (read more)
What AI model does SummaryBot use? And does whoever runs SummaryBot use any special tricks on top of that model? It could just be bias, but SummaryBot seems better at summarizing stuff then GPT-5 Thinking, o3, or Gemini 2.5 Pro, so I'm wondering if it's a different model or maybe just good prompting or something else.
@Toby Tremlett🔹, are you SummaryBot's keeper? Or did you just manage its evil twin?
There are two philosophies on what the key to life is.
The first philosophy is that the key to life is separate yourself from the wretched masses of humanity by finding a special group of people that is above it all and becoming part of that group.
The second philosophy is that the key to life is to see the universal in your individual experience. And this means you are always stretching yourself to include more people, find connection with more people, show compassion and empathy to more people. But this is constantly uncomfortable because, again and again,... (read more)
[Personal blog] I’m taking a long-term, indefinite hiatus from the EA Forum.
I’ve written enough in posts, quick takes, and comments over the last two months to explain the deep frustrations I have with the effective altruist movement/community as it exists today. (For one, I think the AGI discourse is completely broken and far off-base. For another, I think people fail to be kind to others in ordinary, important ways.)
But the strongest reason for me to step away is that participating in the EA Forum is just too unpleasant. I’ve had fun writing stuff on the... (read more)
Here is the situation we're in with regard to near-term prospects for artificial general intelligence (AGI). This is why I'm extremely skeptical of predictions that we'll see AGI within 5 years.
-Current large language models (LLMs) have extremely limited capabilities. For example, they can't score above 5% on the ARC-AGI-2 benchmark, they can't automate any significant amount of human labour,[1] and they can only augment human productivity in minor ways in limited contexts.[2] They make ridiculous mistakes all the time, like saying somethin... (read more)
Have Will MacAskill, Nick Beckstead, or Holden Karnofsky responded to the reporting by Time that they were warned about Sam Bankman-Fried's behaviour years before the FTX collapse?
Will responded here.
Slight update to the odds I’ve been giving to the creation of artificial general intelligence (AGI) before the end of 2032. I’ve been anchoring the numerical odds of this to the odds of a third-party candidate like Jill Stein or Gary Johnson winning a U.S. presidential election. That’s something I think is significantly more probable than AGI by the end of 2032. Previously, I’d been using 0.1% or 1 in 1,000 as the odds for this, but I was aware that these odds were probably rounded.
I took a bit of time to refine this. I found that in 2016, FiveThirtyEight ... (read more)
Yann LeCun (a Turing Award-winning pioneer of deep learning) leaving Meta AI — and probably, I would surmise, being nudged out by Mark Zuckerberg (or another senior Meta executive) — is a microcosm for everything wrong with AI research today.
LeCun is the rare researcher working on fundamental new ideas to push AI forward on a paradigm level. Zuckerberg et al. seem to be abandoning that kind of work to focus on a mad dash to AGI via LLMs, on the view that enough scaling and enough incremental engineering and R&D will push current LLMs all the way ... (read more)
Just calling yourself rational doesn't make you more rational. In fact, hyping yourself up about how you and your in-group are more rational than other people is a recipe for being overconfidently wrong.
Getting ideas right takes humility and curiosity about what other people think. Some people pay lip service to the idea of being open to changing their mind, but then, in practice, it feels like they would rather die than admit they were wrong.
This is tied to the idea of humiliation. If disagreement is a humiliation contest, changing one's mind can fe... (read more)