Yarrow Bouchard 🔸
Bio
Pronouns: she/her or they/them.
I got interested in effective altruism back before it was called effective altruism, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now I’m trying to figure out where effective altruism can fit into my life these days and what it means to me.
Posts 19
Comments332
Topic contributions1
Can you disprove the universe is a computer simulation anymore than you can disprove that the world you think you see is just an illusion put into your mind by a demon, as famously mused by the philosopher René Descartes in his Meditations?
In theory, the beings who run the simulation could make whatever they want happen, including changing the results of scientific experiments, change the outputs of computers used for mathematics, change the brains of everyone who tries to think about certain mathematical proofs in order to trick them into believing something false, and so on. If you imagine that God or a demon might be tricking you, there is no empirical result and no mathematical or logical result that can prove you're not being tricked.
But of course the question is why we might think the universe is a computer simulation in the first place. No version of the simulation argument or simulation hypothesis has ever made sense. These are two of the objections I think are the most powerful:
- There is no logically valid inference to be made about the world outside the simulation from the world inside the simulation. That's just a logical non-sequitur. The world outside the simulation could be made of pudding for all we know, or it could be made of magic, or it could be run by Zeus. Maybe the simulation is run by a pipe-smoking rabbit and the "computer" is actually a carrot. Why should we be able to say the world outside is any particular way because of the way the world inside is? And if we can't say that, how can we infer there's a world outside, or a simulation, at all?
- There are obvious ethical concerns around simulating universes. It's not at all obvious that, if the advancement of science and technology continued indefinitely, we would eventually simulate a universe like our current universe. I think we would probably outlaw doing such a thing. If you run the simulation and can change it as you like, aren't you responsible for everything that happens in the simulation, especially the things that happen as a result of nature (e.g., infectious disease, natural disasters), and wouldn't that make you the worst murderer of all time?
While checking the Wikipedia page for the simulation hypothesis just now, I noticed the physicist Sean Carroll has a really smart reply to the simulation argument as well:
Of course one is welcome to poke holes in any of the steps of this argument. But let’s for the moment imagine that we accept them. And let’s add the observation that the hierarchy of simulations eventually bottoms out, at a set of sims that don’t themselves have the ability to perform effective simulations. Given the above logic, including the idea that civilizations that have the ability to construct simulations usually construct many of them, we inevitably conclude:
- We probably live in the lowest-level simulation, the one without an ability to perform effective simulations. That’s where the vast majority of observers are to be found.
Hopefully the conundrum is clear. The argument started with the premise that it wasn’t that hard to imagine simulating a civilization — but the conclusion is that we shouldn’t be able to do that at all. This is a contradiction, therefore one of the premises must be false.
This isn’t such an unusual outcome in these quasi-anthropic “we are typical observers” kinds of arguments. The measure on all such observers often gets concentrated on some particular subset of the distribution, which might not look like we look at all. In multiverse cosmology this shows up as the “youngness paradox.”
You could say that maybe the top-level universe is such that it can support infinite computation, so there's never a point when the nested hierarchy of simulations bottoms out. But, in this case, you'd be showing the logic of the first objection I brought up — if we can just stipulate anything we want about the top-level universe, then maybe the top-level universe is a sentient chocolate milkshake, and our universe is just a fleeting dream in its mind.
In case it's helpful, I prompted Claude Sonnet 4.5 with extended thinking to explain three of the key concepts we're discussing and I thought it gave a pretty good answer, which you can read here. (I archived that answer here, in case that link breaks.)
I gave GPT-5 Thinking almost the same prompt (I had to add some instructions because the first response it gave was way too technical) and it gave an okay answer, which you can read here. (Archive link here.)
I tried to Google for human-written explanations of the similarities and differences first, since that's obviously preferable. But I couldn't quickly find one, probably because there's no particular reason to compare these concepts directly to each other.
No, those definitions quite clearly don’t say anything about data efficiency or generalization, or the other problems I raised.
I think you have misunderstood the concept of continual learning. It doesn’t mean what you seem to think it means. You seem to be confusing the concept of continual learning with some much more expansive concept, such as generality.
If I’m wrong, you should be able to quite easily provide citations that clearly show otherwise.
Here’s a definition of continual learning from an IBM blog post:
Continual learning is an artificial intelligence (AI) learning approach that involves sequentially training a model for new tasks while preserving previously learned tasks. Models incrementally learn from a continuous stream of nonstationary data, and the total number of tasks to be learned is not known in advance.
Here’s another definition, from an ArXiv pre-print:
To cope with real-world dynamics, an intelligent system needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime. This ability, known as continual learning, provides a foundation for AI systems to develop themselves adaptively. In a general sense, continual learning is explicitly limited by catastrophic forgetting, where learning a new task usually results in a dramatic performance degradation of the old tasks.
The definition of continual learning is not related to generalization, data efficiency, the availability of training data, or the physical limits to LLM scaling.
You could have a continual learning system that is equally data inefficient as current AI systems and is equally poor at generalization. Continual learning does not solve the problem of training data being unavailable. Continual learning does not help you scale up training compute or training data if compute and data are scarce or expensive, nor does the ability to continually learn mean an AI system will automatically get all the performance improvements it would have gotten from continuing scaling trends.
I don’t know what Andrej Karpathy’s actual timeline for AGI is. In the Dwarkesh Patel interview that everyone has been citing, Karpathy says he thinks it’s a decade until we get useful AI agents, not AGI. This implies he thinks AGI is at least a decade away, but he doesn’t actually directly address when he thinks AGI will arrive.
After the interview, Karpathy made a clarification on Twitter where he said 10 years to AGI should come across to people as highly optimistic in the grand scheme of things, which maybe implies he does actually think AGI is 10 years away and will arrive at the same time as useful AI agents. However, it’s ambiguous enough I would hesitate to interpret it one way or another.
I could be wrong, but I didn’t get the impression that continual learning or online learning was Karpathy’s main reason (let alone sole reason) for thinking useful AI agents are a decade away, or for his other comments that express skepticism or pessimism — relative to people with 5-year AGI timelines — about progress in AI or AI capabilities.
Continual learning/online learning is not one of the main issues raised in my post and while I think it is an important issue, you can hand-wave away continual learning and still have problems with scaling limits, learning from video data, human examples to imitation learn from, data inefficiency, and generalization.
It’s not just Andrej Karpathy but a number of other prominent AI researchers, such as François Chollet, Yann LeCun, and Richard Sutton, who have publicly raised objections to the idea that very near-term AGI is very likely via scaling LLMs. In fact, in the preamble of my post I linked to a previous post of mine where I discuss how a survey of AI researchers found they have a median timeline for AGI of over 20 years (and possibly much, much longer than 20 years, depending how you interpret the survey), and how, in another survey, 76% of AI experts surveyed think scaling LLMs or other current techniques is unlikely or very unlikely to reach AGI. I’m not defending a fringe, minority position in the AI world, but in fact something much closer to the majority view than what you typically see on the EA Forum.
Even if you think that involvement in EA is mostly a matter of altruistic self-sacrifice (which I think is an oversimplification), it can still be true that there are women, people of colour, LGBTQ people, or people from other marginalized groups who want to make that altruistic self-sacrifice. Should they have the autonomy, the right to make that decision? I think so.
If people from these groups say that the barriers to their involvement in EA is not the amount of self-sacrifice involved, but other factors like perceived unwelcomingness toward people of their demographic, not seeing other people like them represented in EA, or a lack of trust or a sense of safety (e.g. around sexual harassment or instances of discrimination or prejudice, or the community's response to it) — or other things of that nature — then that is a moral failing on the part of EA, and not noblesse oblige.
No, none of them boil down to that, and especially not (1).
I've already read the "A Definition of AGI" paper (which the blog post you linked to is based on) and it does not even mention the objections I made in this post, let alone offer a reply.
My main objection to the paper is that it makes a false inference that tests used to assess human cognitive capabilities can be used to test whether AI systems have those same capabilities. GPT-4 scored more than 100 on an IQ test in 2023, which would imply that it is an AGI if an AI that passes a test has the cognitive capabilities a human is believed to have if it passes that same test. The paper does not anticipate this objection or try to argue against it.
(Also, this is just a minor side point, but Andrej Karpathy did not actually say AGI is a decade away on Dwarkesh Patel's podcast. He said useful AI agents are a decade away. This is pretty clear in the interview or the transcript. Karpathy did not comment directly on the timeline for AGI, although it seems to be implied that AGI can come no sooner than AI agents.
Unfortunately, Dwarkesh or his editor or whoever titles his episodes, YouTube chapters, and clips has sometimes given inaccurate titles that badly misrepresent what the podcast guest actually said.)
Thank you very much for the info! It's probably down to your prompting, then. Squeezing things into 6 bullet points might be just a helpful format for ChatGPT or for summaries (even human-written ones) in general. Maybe I will try that myself when I want to ask ChatGPT to summarize something.
I also think there's an element of "magic"/illusion to it, though, since I just noticed a couple mistakes SummaryBot made and now its powers seem less mysterious.