It looks like Tesla has abandoned the idea of upgrading cars to HW5 and will stick with the current hardware, HW4: https://en.wikipedia.org/wiki/Tesla_Autopilot_hardware#AI5_(Hardware_5)

(Incidentally, AI also still isn't writing 90% of code.)

Thanks for writing this. It is... strange... and disturbing... that there is any need at all for you to write such a post, but, anyway, it's good that you did and I appreciate that.

The commonsense answer is actually stricter than what you propose: Caroline Ellison (or anyone else convicted of crimes at FTX or Alameda) should probably never be employed by any EA organization, and certainly not in a financial role shortly after her release from prison, without doing anything to try to redeem herself or prove she can be trusted. Especially not since Austin Chen explicitly said the hire was a reward and thanks for money Manifold and Manifund got from FTX. Caroline Ellison should not be rewarded and thanked for her work involving FTX or Alameda.

I also doubt that any deal was arranged in advance, but since Austin said this was one of his motivations, that makes it a quid pro quo. If people are hung up on the definition of quid pro quo and whether it applies to this case, at least we can say that this is corrupt.

I particularly agree with this part:

From the outside, to be frank, it’s quite infuriating that in the good times, whenever all of the EAs and rationalists are trying to save the world, it’s one big team with everyone largely supporting each other, but then as soon as someone does a crime or owes money, suddenly none of these orgs have ever heard of one another.

It would be so easy for EA organizations and individual EA leaders to follow suit with the Centre of Effective Altruism, which has banned Manifund from its events, or Andy Masley, whose tweet you show vowing to boycott Manifund and Manifold. Indeed, the CEA and others could go further by also banning Austin Chen and Caroline Ellison as individuals.

The main purpose of this is not to look good but to do the right thing. Do you want the EA movement to be a place where there are no (or minimal) negative consequences for corruption? There's a lot people need to be able to disagree about or do differently but still tolerate each other and try to get along. But, come on, if this isn't an ethical red line, what is?

I can empathize that people are grappling with ethical principles that are each deeply held yet seem to oppose one another. For instance:

• Every human being on Earth deserves unconditional compassion, empathy, and universal love, including patience, forgiveness, and second chances
• Crimes and severe moral transgressions must be deterred through punishment

Or:

• We should make generous assumptions about people's intentions, motivations, beliefs, and goals
• We can't automatically, uncritically believe all people all the time about their stated intentions, motivations, beliefs, or goals (e.g., otherwise, a lot of people could get away with crimes just by professing their innocence)

Or:

• We need to have a classically liberal tolerance for different views, values, principles, moral beliefs, and so on
• From the perspective of certain views, values, principles, moral beliefs, etc., the most evil acts in the world are rational and morally right, so liberal tolerance can only go so far

The push and pull of these principles can be confusing. In many cases, it is vexing how to integrate or reconcile them — how to resolve the apparent opposition. But this is not one of those vexing cases. Ultimately, it's clear and straightforward. You don't reward crime. You don't put people who've committed financial crimes involving EA in a financial management role at an EA charity. What's at stake here is not the concept of universal compassion or forgiveness. It's just common sense.

There are a million other things a person can do in the world than manage money at an EA charity. And, if for some strange reason managing money at an EA charity is their God-given destiny that they must fulfill within their lifetime, there are a million other things they can do while they work to redeem themselves and prove they can be trusted. Which could take many years and is never guaranteed to succeed.

If people are really curious and want a deeper understanding of this topic, I'm sure there are an abundance of good resources on the topic of criminal rehabilitation from people who have practical experience and a lot of knowledge. There are many people who take the perspective that our approach to criminal justice is excessively punitive and retributive, and advocate what they see as a more compassionate and forgiving alternative. Yet I'm certain they would not see Manifund's hiring of Caroline Ellison as an acceptable situation.

A comment on how variations on the simulation hypothesis like this one may actually defeat the popular formulations of the simulation argument. Here's physicist Sean Carroll's formulation of the simulation argument (which, like other popular formulations, differs from philosopher Nick Bostrom's original argument):

1. We can easily imagine creating many simulated civilizations.
2. Things that are that easy to imagine are likely to happen, at least somewhere in the universe.
3. Therefore, there are probably many civilizations being simulated within the lifetime of our universe. Enough that there are many more simulated people than people like us.
4. Likewise, it is easy to imagine that our universe is just one of a large number of universes being simulated by a higher civilization.
5. Given a meta-universe with many observers (perhaps of some specified type), we should assume we are typical within the set of all such observers.
6. A typical observer is likely to be in one of the simulations (at some level), rather than a member of the top-level civilization.
7. Therefore, we probably live in a simulation.

Carroll observes that this formulation of the argument implies a contradiction, since typical simulations or universes will be bottom-level simulations that don't have the compute to create more simulations.

If we take some further series of reasoning steps from (7) to the view that we live in a simulation of a tiny fragment of the world that simulates a short span of years before it terminates, then we arrive at the conclusion that we will not be able to create simulated universes with many conscious beings in them. Therefore, (3) becomes false. (This is similar to Sean Carroll's more general point.)

Some series of steps — whatever they are— takes us from (7) to a particular conclusion. But then that conclusion loops back around and renders one of its premises false, thereby rendering the whole argument unsound. In this way, the argument is that we live in this particular kind of simulation — whatever that argument is — ends up defeating itself, assuming it rests on the sort of popular formulation of the simulation argument that I quoted from Sean Carroll.

For any version of the simulation argument to be coherent and valid (let alone sound), care needs to be taken with the logical steps that take us from the premises to the conclusion. Careless logic will create invalid arguments and wrong conclusions.

For instance, if we end up a positing top-level, non-simulated universe (or set of universes) fundamentally and radically different from our own, which operate according to different laws of physics and perhaps other different laws of nature or different metaphysical laws, then the simulation argument seems to fall apart. This prevents us from positing that the top-level universe (or set of universes) has infinite computation, unlike — apparently — our own.

If there are no constraints on the metaphysical or natural laws that rule the top-level universe, then we could just as easily say that the top-level universe is Heaven and God created our universe supernaturally. Or any of an infinite number of similar conceivable origin stories. (This is not an idea original to me, but I can't remember who I heard it from. Maybe the physicist David Deutsch? But I'm not sure.)

So, it seems important to the simulation argument's coherence and logical validity that a) our universe will actually create simulated universes eventually and b) that the top-level universe be similar to our universe, e.g., that it shares laws of physics, other laws of nature, and any other metaphysical laws that apply. It seems that any version of the simulation argument that doesn't meet both criteria (a) and (b) can't be sound.

On top of this, it should be said that it's completely arbitrary to suppose that a simulation would focus on North American and British effective altruists as opposed to, say, Japanese librarians or Sicilian clowns. If you're going to entertain the notion of a simulation that only focuses on a narrow slice of the world for a narrow time — which seemingly can't be reached by any logically sound argument — then there are endless possible versions of that simulation. There seems to be nothing that particularly supports this imagined version of a simulation above any of the endless other conceivable varieties.

I'm surprised that this is a featured post on the Effective Altruism Forum because it's such a bizarre, crazy idea and because there is no attempt to argue that it's true. It's just asserted. Moreover, the idea rests on the simulation argument, which itself is quite flawed. The idea that effective altruists are the only people who really exist (and that everyone else is just an illusion) is not an idea that could have any rational basis. It is really strange, then, to see the employees of the Centre for Effective Altruism who run the EA Forum single this post out as one that has particularly high quality. What on Earth is going on?

I wrote in my post "The sad decline of effective altruism" that effective altruism has suffered a steep decline in its intellectual standards over the last 15 years. It's possible that the featuring of this post was some kind of fluke or mistake, but it fits the pattern of what I've observed.

If someone earnestly believes that only effective altruists exist and 99.9%+ of people on the planet are fake, from a place or compassion and genuine concern, I would recommend they talk to a qualified mental health professional. This is outside the realm of a rational difference of opinion and into mental health territory.

I would be worried if anyone I loved told me they believed they're part of an elite group comprising the less than 0.1% of humanity that is real, while everyone else is fake and doesn't really exist. I would want them to get help. I extend that same concern to anyone who truly believes the idea entertained in this post.

I will probably not read or reply to further comments here, but I welcome civil, kind messages and comments on Substack.

I don't believe in shaming, but I do believe in accountability. There's a crucial difference.

If your argument is that there's nothing EA organizations or leaders can do if a leader of an EA organization goes rogue and does something really bad, well, I can think of some things. EA orgs and leaders can decide to impose sanctions on Manifund like the Centre for Effective Altruism did. I think the CEA should go even further than it did and other EA orgs should follow suit.

Your other argument is that this is an acceptable difference of opinion rather than an ethical red line. But this is a clear-cut case of financial corruption (a quid pro quo between FTX and Manifund, which rewards a key player in FTX's crimes for those crimes because they benefited Manifund). Austin Chen conveniently admitted corrupt intent, although even without such an admission, the case for corruption would be strong. Financial corruption should be a clear ethical red line for the EA community.

The simulation argument is silly, for reasons I've previously detailed here and here. For one, the physicist Sean Carroll cleverly pointed out that the simulation argument contains a logical self-contradiction. Often, a logical self-contradiction is enough to end all discussion! Yet that's not even the end of the simulation argument's problems.

Another problem is that forecasts about future technological events within our universe don't necessarily tell us the story of the creation of our universe. Let's say we could accurately forecast that our descendants will simulate lots of universes. What does that tell us about the creation of our universe? Nothing, without further logical steps.

By analogy, let's say you plan to have a baby using IVF. Does that imply you were born thanks to IVF? No, of course not, not just by itself. Yet the simulation argument wants us to infer that if we create a universe a certain way, then our universe was also created that same way. This is an invalid inference.

But this still isn't the end of problems for the simulation argument, which just gets messier and more objectionable the more you scrutinize it. For instance, what major crimes or morally abhorrent acts might the creator of a simulation be guilty of? Torture, murder, genocide...? Wouldn't there be prohibitions against this sort of thing?

I could go on. There is just no good reason for the simulation argument to be taken seriously.

Edit (Sunday, October 18, 2026 at 8:50am UTC): I realized after writing this comment that I made a few minor mistakes, or at least there are a few minor things that should be clarified. These might be either boring and nipticky or important, depending on your perspective.

First, Nick Bostrom's original simulation argument is not an argument that we live in a simulation. It contains the possibility that simulations might be morally or legally prohibited. However, most people hear a version of the simulation argument different from what Bostrom originally argued. A good example is Sean Carroll's formulation. My point about moral or legal prohibition applies to the more popular formulations of the simulation argument, but not to Bostrom's original simulation argument, which already contains that point.

Second, my point about inferring the universe's origins from technological forecasts and whether it's a valid inference is already covered in Sean Carroll's blog post, which addresses the simulation argument's anthropic reasoning. The anthropic reasoning should be questioned, but even popular formulations of the simulation already contain the extra steps I was alluding to. In any case, it's worth thinking carefully what series of logical steps takes us from a technological forecast to a cosmological origin story. That was my point, but it was not clearly expressed.

It remains true that the popular formulation of the simulation argument discussed by Sean Carroll implies a logical contradiction. It remains true there are multiple major problems with that version of the argument. Nick Bostrom's original simulation argument is also objectionable, but less so because it is not an argument that we live in a simulation.

The bottom line remains that it's silly to think we live in a simulation. Neither Bostrom's original argument or the more popular reformulations present a serious case that we do.

I haven't read any replies to this comment and probably won't because I usually find discussions on the EA Forum to be unconstructive. However, I welcome further discussion on Substack.

Huh? Not sure I follow this comment. I have definitely stated outright what my views are about fears of near-term superhuman artificial general intelligence (AGI). I have written a lot about this here on the Effective Altruism and also on my Substack blog. See, for example, this recent post: https://strangecosmos.substack.com/p/the-sloppy-research-fuelling-fears

Maybe the idea of making a bet about a future prediction seems unusual to you. But it's not out of place on the Effective Altruism Forum, since the effective altruism community normalizes and encourages bets like this. Personally, I'm not sure prediction-based bets are all they're cracked up to be, but at least they could be a bit of fun.

You shouldn't take a bet you think you're likely to lose. That defeats the whole purpose. Personally, I also think even if you're only, say, 60% sure you'll win a bet, it sort of undermines the purpose as well. It's way more interesting if two people take opposite sides of a bet and they're both highly confident that they'll win.

It occurred to me what a drag it would be for the other person in this bet to wait until January 1, 2034 to win. I thought of a few win conditions that could call the bet in the other person's favour before the clock runs out. If any one of these conditions are satisfied, the other person would win the bet:

• Either Anthropic or OpenAI (post-IPO) reports $100 billion in annual GAAP profit (net income) or $25 billion in quarterly GAAP profit (these would have to be audited financials reported to the SEC, like public companies typically do)
• Ray Kurzweil wins his Long Bet against Mitchell Kapor because the Long Now Foundation determines that an AI system can pass the Turing test by the end of 2029
• A rigorous survey of AI experts by a reliable organization (e.g., AAAI) finds that a majority (50.1%+) think artificial general intelligence has been created (not something like an informal poll on a blog or forum, obviously)
• A household humanoid robot is sold to regular consumers that can autonomously, without human assistance, scramble eggs successfully 95%+ of the time
• Fully autonomous SAE Level 5 robotaxis are widely deployed in the U.S. or any country, with at least 100,000 Level 5 vehicles on the roads, without using any form of human assistance or remote operation, and without being geofenced to an area smaller than 95% of the country's public roads

These are rattled off just off the top of my head and we could probably come up with some better ones to add to the list.

I agree. That's an important point. Thank you. Peer review is a floor, not a ceiling. Replication is a higher standard.

There's also something complicated and idiosyncratic about the machine learning field. Machine learning papers, not just METR's, sometimes want to measure both humans' performance and AI systems' performance on the same sort of tasks, in order to compare them. There is a computer science aspect of this research and a social science aspect.

To say that an AI system achieved such-and-such a score on a benchmark is typically uncontroversial. But can ML researchers properly conduct scientific studies on humans? And what of the validity of the benchmark? Can the researchers accurately, properly score human performance? And, more importantly, does the benchmark actually constitute a meaningful, truthful, apples-to-apples comparison between human performance and AI performance on the same sort of tasks?

When ML researchers try to do scientific studies involving humans, they might not be competent to do so. They might have no experience, training, or education in that kind of research. Peer review from other ML researchers is not helpful if the reviewers are also just techy laypeople with regard to social science.

Open invitation: bet that the AI bubble won’t pop

I’m taking my second stab at formulating this AI bubble bet in a way that is attractive for people who might want to take the other side of it. I’m betting the AI bubble will pop within about 7 years. I’m looking for people who are willing to bet it won’t pop, or that there is no AI bubble.

I’m open to proposed modifications.

Stakes:

$20 to the charity of the winner’s choice.

Resolution criteria:

By January 1, 2034, all of the following news outlets will report — as fact, not opinion or editorial — that the AI bubble has popped:

• The New York Times
• The Washington Post
• The Wall Street Journal
• Bloomberg
• The Financial Times
• The Economist
• The BBC
• The CBC
• Le Monde

To repeat, all of these news outlets have to report it. And they have to report it as news. Opinion columns and editorials don’t count.

Moreover, no major news publication classified as “generally reliable” by Wikipedia’s editors — either in general factual news information or in the area of finance, technology, economics, or markets — will report that the AI bubble has not actually popped. (Let’s say a “generally reliable” major news publication needs to disagree with the above listed news outlets within a month of all the listed outlets reporting that the bubble has popped.)

Moreover, if there is any survey of experts (e.g., economists, fund managers, institutional investors, financial analysts) on the matter by an organization generally considered reliable (e.g., Bank of America, which has conducted similar surveys in the past), at least 80% of respondents will agree the AI bubble as popped. If the survey finds that 79.9% or less of respondents agree, then I lose the bet. (Let's say this survey can happen within a month of when all the listed outlets report that the AI bubble has popped.)

Finally, all the following individual experts in finance or economics will agree that the AI bubble has popped:

• Noah Smith (economics writer)
• Tyler Cowen (economist)
• Jared Bernstein (economist)
• Ryan Cummings (economist)
• Jim Covello (head of global equity research at Goldman Sachs)
• Peter Oppenheimer (chief global equity strategist at Goldman Sachs)

Let’s say after all the above listed news outlets report that the AI bubble has popped, these individual experts have a month to publicly weigh in. If they express no opinion within a month, they’ll be removed from the list.

Explanation:

Whether the AI bubble has popped is difficult to operationalize and could conceivably involve tracking many different metrics, such as valuations, AI investment, investor sentiment, and so on. This is a complex job even for an expert. Deferring to qualified “judges” is much easier, and much easier to get right than attempting to operationalize it oneself.

The resolution criteria are designed to remove all doubt that, by some random fluke, a bunch of news outlets might report that the AI bubble has popped when it really hasn’t. If the AI bubble actually pops, it will probably be so clear and uncontroversial that someone will have no reservations about conceding it happened. (Conversely, if the AI bubble doesn’t pop, it will be equally obvious.)

In fact, the resolution criteria are so slanted toward the person taking the other side of that bet that if literally everyone in the world except Tyler Cowen agreed that the AI bubble had popped, I would still lose the bet.

The stakes are low because this bet is mainly just for fun and also to make a point.

The bet’s timeframe is about 7 years because I read that many AI investments are amortized over 5 to 7 years. It’s hard to know for sure what exactly AI investors’ time horizons might be, but 7 years is probably long enough. Even if the AI bubble were to pop in 8 years rather than 6 or 7, that would still be a different enough outcome than what I’m expecting that I would say I’ve lost the bet fair and square.

Again, I'm open to proposed modifications.