I haven't downvoted, but this is attracting several downvotes, and I thought I'd try to articulate some negative feelings I have here:
I guess that many of the downvotes are likely based on one or the other of these clusters (or both, although the two complaints are in some tension with one another).
(edited for clarity and to restructure the bullet point nesting)
4 is a great point, thanks.
On 1--3, I definitely agree that I may prudentially prefer some possibilities than others. I've been assuming that from a consequentialist moral perspective the distribution of future outcomes still looks like the one I give in this post, but I guess it should actually look quite different. (I think what's going on is that in some sense I don't really believe in world A, so haven't explored the ramifications properly.)
I support people poking at the foundations of these arguments. And I especially appreciated the discussion of bottlenecks, which I think is an important topic and often brushed aside in these discussions.
That said, I found that this didn't really speak to the reasons I find most compelling in favour of something like the singularity hypothesis. Thorstad says in the second blog post:
If each doubling of intelligence is harder to bring about than the last, then even if all AI research is eventually done by recursively self-improving AI systems, the pace of doubling will steadily slow.
I think this is wrong. (Though the paper itself avoids making the same mistake.) There are lots of coherent models where the effective research output of the AI systems is growing faster than the difficulty of increasing intelligence, leading to accelerating improvements despite each doubling of intelligence getting harder than the last. These are closely analogous to the models which can (depending on some parameter choices) produce a singularity in economic growth by assuming endogenous technological growth.
In general I agree with Thorstad that the notion of "intelligence" is not pinned down enough to build tight arguments on it. But I think that he goes too far in inferring that the arguments aren't there. Rather I think that the strongest versions of the arguments don't directly route through an analysis of intelligence, but something more like the economic analysis. If further investments in AI research drive the price-per-unit-of-researcher-year-equivalent down in fast enough, this could lead to hyperbolic increases in the amount of effective research progress, and this could in turn lead to rapid increases in intelligence -- however one measures that. I agree that this isn't enough to establish that things will be "orders of magnitude smarter than humans", but for practical purposes the upshot that "there will be orders of magnitude more effective intellectual labour from AI than from humans" does a great deal of work.
On the argument that extraordinary claims require extraordinary evidence, I'd have been interested to see Thorstad's takes on the analyses which suggest that long-term historical growth rates are hyperbolic, e.g. Roodman (2020). I think of that as one of the more robust long-term patterns in world history. The hypothesis which says "this pattern will approximately continue" doesn't feel to me to be extraordinary. You might say "ah, but that doesn't imply a singularity in intelligence", and I would agree -- but I think that if you condition on this kind of future hyperbolic growth in the economy, the hypothesis that there will be a very large accompanying increase in intelligence (however that's measured) also seems kind of boring rather than extraordinary.
Although I agree that all of these are challenges, I don't really believe they're enough to undermine the basic case. It's not unusual in high-paying industries for some people to make several times as much as their colleagues. So there's potentially lots of room to have higher consumption than you would working at a nonprofit while also giving away more than half your salary.
Empirically my impression is also that people who went into earning to give early tended to stay giving later in their careers 10+ years later; although that's anecdotal rather than data-driven.
I agree with "pretty rationally overall" with respect to general world modelling, but I think that some of the stuff about how it relates to its own values / future selves is a bit of a different magisterium and it wouldn't be too surprising if (1) it hadn't been selected for rationality/competence on this dimension, and (2) the general rationality didn't really transfer over.
One thought is that for something you're describing as a minimal viable takeover AI, you're ascribing it a high degree of rationality on the "whether to wait" question.
By default I'd guess that minimal viable takeover systems don't have very-strong constraints towards rationality. And so I'd expect at least a bit of a spread among possible systems -- probably some will try to break out early whether or not that's rational, and likewise some will wait even if that isn't optimal.
That's not to say that it's not also good to ask what the rational-actor model suggests. I think it gives some predictive power here, and more for more powerful systems. I just wouldn't want to overweight its applicability.
Yeah I'm arguing that with good reflective governance we should achieve a large fraction of what's accessible.
It's quite possible that that means "not quite all", e.g. maybe there are some trades so that we don't aestivate in this galaxy, but do in the rest of them; but on the aggregative view that's almost as good as aestivating everywhere.
I think in this context the natural way to interpret probabilities (or medians) of probabilities is as something like: "what would I think the probability was if I knew a lot more about the situation and got to think about it a lot more, but wasn't literally omniscient". Of course that isn't fully defined (since it's not specified exactly how much more they get to know), but I think it's approximately meaningful, and can capture something quite useful/important about credal resilience.
Relatedly, I think the journalist's presentation is misleading. I think it makes it sound like there's something like an objective 1-in-230 chance of catastrophe. I think a more accurate presentation would be "expert thinks there's a 1% chance we live in a world with a high risk". This changes the appropriate policy implications: if taken at face value they both suggest there's something important to do, but in the latter case the action is less like "work out what to do about simulation shutdown" and more like "start a serious research programme to work out how likely it is we're in the fraction of worlds that was offhand labelled as 1%".
(I also think that you may be right that the cosmologist didn't really believe their numbers. I think that issue alone would be enough reason to be suspicious of the journalist's approach. They might reasonably suspect that such off-the-cuff answers might not withstand scrutiny, so it seems bad to report them in a way that could sound like considered expert opinion.)
I think maybe yes? But I'm a bit worried that "won't react to them" is actually doing a lot of work.
We could chat about more a concrete example that you think fits this description, if you like.
I think you're missing some important ground in between "reflection process" and "PR exercise".
I can't speak for EV or other people then on the boards, but from my perspective the purpose of the legal investigation was primarily about helping to facilitate justified trust. Sam had by many been seen as a trusted EA leader, and had previously been on the board of CEA US. It seemed it wouldn't be unreasonable if people in EA (or even within EV) started worrying that leadership were covering things up. Having an external investigation was, although not a cheap signal to send, much cheaper in worlds where there was nothing to hide compared to worlds where we wanted to hide something. Between that and wanting to be able to credibly signal to external stakeholders like the Charity Commission, I think general PR was a long way down the list of considerations.