OK, thanks. My vague recollection is that A&S were making conservative guesses about the time needed to disassemble mercury, I forget the details. But mercury is 10^23 kg roughly, so e.g. a 10^9 kg starting colony would need to grow 14 OOMs bigger, which it could totally do in much less than 20 years if its doubling time is much less than a year. E.g. if it has a one-month doubling time, then the disassembly of Mercury could take like 4 years. (this ignores constraints like heat dissipation to be clear... though perhaps those can be ignored, if we are disassembling diverse cold asteroids instead)
I don't think it's infohazardous at all fwiw. This is part of a more general disagreement I have with your model of Stage 2 competition. I think you are right that dyson swarms are where the action is at, but I think that your framing is off. The exponential graph to think about is not the dyson swarm construction graph in particular, but the "Economic Might" graph more generally. If tech exists to build a dyson swarm by having self-replicating industry in space, then presumably the tech also exists to have self-replicating industry on earth, and basically everyone who has that tech will be in a race to boom their economies as fast as possible or else fall behind and face a huge disadvantage. A one-year gap in getting started will indeed be decisive, which is why there won't be such a gap between factions that have the requisite technology (they won't want to wait a whole year to start growing exponentially, that would be like the USA deciding to just... not grow their own economy for a whole century...)
I don't understand where your confidence re Dyson swarm construction times is coming from. Definitely more than 20 years? Why? Not longer than 60? Why?
It would be good to flag in the main text that the justification for this is in Appendix 2 (initially I thought it was a bare asertion). Also, it is interesting that in @kokotajlod's scenario the 'wildly superintelligent' AI maxes out at 1 million-fold AI R&D speedup; I commented to them on a draft that this seemed implausibly high to me. I have no particular take on whether 100x is too low or too high as the theoretical max, but it would be interesting to work out why there is this Forethought vs AI Futures difference.
Interesting. Some thoughts:
First of all, the argument they give in Appendix 2 seems to be more of an argument for a lower bound, not an upper bound / theoretical limit! E.g. they give examples of fruit fly doubling times and so forth as proof of concept that you could double in a day at least in principle.
Secondly, I'm not sure the thing they mean by the speed of progress is the same as the thing we mean. They are talking about doubling times for effective compute (where 100x would mean: effective compute is doubling in a day) whereas we are talking about the ratio between how fast progress goes with AIs helping/driving it vs. without. That's why we call it a "Progress Multiplier." To put it another way, they are comparing the pace of progress in the future intelligence explosion scenario to the pace of progress today, whereas we are comparing it to a hypothetical future scenario in which the powers that be say "OK we are banning AIs from assisting with AI research now, the intelligence explosion can continue but from now on humans have to be doing all the research."
UPDATE: Apparently I was wrong in the numbers here, I was extrapolating from Rose's table in the OP but I interpreted "to" as "in" and also the doubling time extrapolation was Rose's interpolation rather than what the forecasters said. The 95% most aggressive forecaster prediction for most expensive training run in 2030 was actually $250M, not $140M. My apologies! Thanks to Josh Rosenberg for pointing this out to me.
(To be clear, I think my frustration with the XPT forecasters was still basically correct, if a bit overblown; $250M is still too low and the 50th percentile forecaster thought 2030's most expensive run would only be $100M, which is not that far off from the most expensive run that had already happened at the time of the forecast being made.)
Big +1 to the gist of what you are saying here re: courage. It's something I've struggled with a lot myself, especially while I was at OpenAI. The companies have successfully made it cool to race forward and uncool to say e.g. that there's a good chance doing so will kill everyone. And peer pressure, financial incentives, status incentives, etc. really do have strong effects, including on people who think they are unaffected.
I left this comment on one of their docs about cluelessness, reposting here for visibility:
CLR embodies the best of effective altruism in my opinion. You guys are really truly actually trying to make the world better / be ethical. That means thinking hard and carefully about what it means to make the world better / be ethical, and pivoting occasionally as a result.
I am not a consequentialist myself, certainly not an orthodox EV-maximizing bayesian, though for different reasons than you describe here (but perhaps for related reasons?). I think I like your section on alternatives to 'going with your best guess,' I agree such alternatives should be thoroughly explored because 'going with your best guess' is pretty unsatisfying.
I'm not sure whether any of the alternatives will turn out to be more satisfying after more reflection, which is why I'm not ready to say I agree with you overall just yet.
I agree that as time goes on states will take an increasing and eventually dominant role in AI stuff.
My position is that timelines are short enough, and takeoff is fast enough, that e.g. decisions and character traits of the CEO of an AI lab will explain more of the variance in outcomes than decisions and character traits of the US President.
My understanding is that relatively few EAs are actual hardcore classic hedonist utilitarians. I think this is ~sufficient to explain why more haven't become accelerationists.
Have you cornered a classic hedonist utilitarian EA and asked them? Have you cornered three? What did they say?
(I forgot to mention an important part of my argument, oops -- You wouldn't have said "at least 100 years off" you would have said "at least 5000 years off." Because you are anchoring to recent-past rates of progress rather than looking at how rates of progress increase over time and extrapolating. (This is just an analogy / data point, not the key part of my argument, but look at GWP growth rates as a proxy for tech progress rates: According to this GWP doubling time was something like 600 years back then, whereas it's more like 20 years now. So 1.5 OOMs faster.) Saying "at least a hundred years off" in 1600 would be like saying "at least 3 years off" today. Which I think is quite reasonable.)
I agree with the claims "this problem is extremely fucking hard" and "humans aren't cracking this anytime soon" and I suspect Yudkowsky does too these days.
I disagree that nanotech has to predate taking over the world; that wasn't an assumption I was making or a conclusion I was arguing for at any rate. I agree it is less likely that ASIs will make nanotech before takeover than that they will make nanotech while still on earth.
I like your suggestion to model a more earthly scenario but I lack the energy and interest to do so right now.
My closing statement is that I think your kind of reasoning would have been consistently wrong had it been used in the past -- e.g. in 1600 you would have declared so many things to be impossible on the grounds that you didn't see a way for the natural philosophers and engineers of your time to build them. Things like automobiles, flying machines, moving pictures, thinking machines, etc. It was indeed super difficult to build those things, it turns out -- 'impossible' relative to the R&D capabilities of 1600 -- but R&D capabilities improved by many OOMs, and the impossible became possible.
OK, thanks. My vague recollection is that A&S were making conservative guesses about the time needed to disassemble mercury, I forget the details. But mercury is 10^23 kg roughly, so e.g. a 10^9 kg starting colony would need to grow 14 OOMs bigger, which it could totally do in much less than 20 years if its doubling time is much less than a year. E.g. if it has a one-month doubling time, then the disassembly of Mercury could take like 4 years. (this ignores constraints like heat dissipation to be clear... though perhaps those can be ignored, if we are disassembling diverse cold asteroids instead)
I don't think it's infohazardous at all fwiw. This is part of a more general disagreement I have with your model of Stage 2 competition. I think you are right that dyson swarms are where the action is at, but I think that your framing is off. The exponential graph to think about is not the dyson swarm construction graph in particular, but the "Economic Might" graph more generally. If tech exists to build a dyson swarm by having self-replicating industry in space, then presumably the tech also exists to have self-replicating industry on earth, and basically everyone who has that tech will be in a race to boom their economies as fast as possible or else fall behind and face a huge disadvantage. A one-year gap in getting started will indeed be decisive, which is why there won't be such a gap between factions that have the requisite technology (they won't want to wait a whole year to start growing exponentially, that would be like the USA deciding to just... not grow their own economy for a whole century...)
I don't understand where your confidence re Dyson swarm construction times is coming from. Definitely more than 20 years? Why? Not longer than 60? Why?
Interesting. Some thoughts:
UPDATE: Apparently I was wrong in the numbers here, I was extrapolating from Rose's table in the OP but I interpreted "to" as "in" and also the doubling time extrapolation was Rose's interpolation rather than what the forecasters said. The 95% most aggressive forecaster prediction for most expensive training run in 2030 was actually $250M, not $140M. My apologies! Thanks to Josh Rosenberg for pointing this out to me.
(To be clear, I think my frustration with the XPT forecasters was still basically correct, if a bit overblown; $250M is still too low and the 50th percentile forecaster thought 2030's most expensive run would only be $100M, which is not that far off from the most expensive run that had already happened at the time of the forecast being made.)
Big +1 to the gist of what you are saying here re: courage. It's something I've struggled with a lot myself, especially while I was at OpenAI. The companies have successfully made it cool to race forward and uncool to say e.g. that there's a good chance doing so will kill everyone. And peer pressure, financial incentives, status incentives, etc. really do have strong effects, including on people who think they are unaffected.
I left this comment on one of their docs about cluelessness, reposting here for visibility:
I agree that as time goes on states will take an increasing and eventually dominant role in AI stuff.
My position is that timelines are short enough, and takeoff is fast enough, that e.g. decisions and character traits of the CEO of an AI lab will explain more of the variance in outcomes than decisions and character traits of the US President.
Thanks for discussing with me!
(I forgot to mention an important part of my argument, oops -- You wouldn't have said "at least 100 years off" you would have said "at least 5000 years off." Because you are anchoring to recent-past rates of progress rather than looking at how rates of progress increase over time and extrapolating. (This is just an analogy / data point, not the key part of my argument, but look at GWP growth rates as a proxy for tech progress rates: According to this GWP doubling time was something like 600 years back then, whereas it's more like 20 years now. So 1.5 OOMs faster.) Saying "at least a hundred years off" in 1600 would be like saying "at least 3 years off" today. Which I think is quite reasonable.)
I agree with the claims "this problem is extremely fucking hard" and "humans aren't cracking this anytime soon" and I suspect Yudkowsky does too these days.
I disagree that nanotech has to predate taking over the world; that wasn't an assumption I was making or a conclusion I was arguing for at any rate. I agree it is less likely that ASIs will make nanotech before takeover than that they will make nanotech while still on earth.
I like your suggestion to model a more earthly scenario but I lack the energy and interest to do so right now.
My closing statement is that I think your kind of reasoning would have been consistently wrong had it been used in the past -- e.g. in 1600 you would have declared so many things to be impossible on the grounds that you didn't see a way for the natural philosophers and engineers of your time to build them. Things like automobiles, flying machines, moving pictures, thinking machines, etc. It was indeed super difficult to build those things, it turns out -- 'impossible' relative to the R&D capabilities of 1600 -- but R&D capabilities improved by many OOMs, and the impossible became possible.