I'm a stock market speculator who has been involved in transhumanist and related communities for a long time.
See my website at http://bayesianinvestor.com.
Partially aligned transformative AIs are likely to be stable under reflection
Work on corrigibility has provided a decent outline of how to do this. My response is heavily dependent on weak guesses as to how diligent AI companies will be at incorporating the best ideas.
The case for it appeals to standard premises that sound plausible when you hear them. There is no standard reason people reject it in the philosophical literature—no objection that is widely agreed to succeed.
That is probably true if you only look at the philosophical literature. But outside of a few small groups such as that, people widely reject the premise that we should have identical moral concerns for people in the distant future and people currently alive.
Premises that are truly widely shared only support a weaker version of longtermism than yours.
My donations are significantly influenced by the expectation that other people's donations will increase, due to increasing wealth and increasing awareness of the importance of AI.
Yes. Current AIs have enough general intelligence that I expect they're sufficient to replace a fair amount of manual labor.
There are still some expensive engineering problems, but they're relatively ordinary problems such as figuring out which robot models ought to be mass produced (without mass production, they'll be too expensive), substantial training to instill domain-specific knowledge, and maybe some need for more energy efficient GPUs.
As long as general-purpose AI is making rapid progress, it probably makes some sense for the robot industry to wait another year or two before choosing the specifics of what kinds of robots to mass-produce - AIs of 2028 are likely to learn domain-specific knowledge faster and more flexibly. But even if we get an AI pause by 2027, I expect at least 10% of manual labor to be replaced by robots by 2030.
Stopping AI really would be like halting technology itself, because you would be shutting off the source of nearly all growth.
If increases in basic AI intelligence were fully halted at 2027 levels, economic growth would still accelerate to something comfortably above 5%/year, due to continuing adaptations of AI to things like robotics.
None of the pause proposals look like they could be 100% effective at stopping increases in AI intelligence.
I used to give pretty high priority to economic growth, but now that growth in excess of 10%/year looks close to inevitable, I'm giving much lower priority to it.
I agree with a fair amount of what you wrote in this post, but I don't see much of an argument against slowing AI capability advances.
One good prediction that he made was in his 1986 book Engines of Creation, that a global hypertext system would be available within a decade. Hardly anyone in 1986 imagined that.
But he has almost entirely stopped trying to predict when technologies will be developed. You should read him to imagine what technologies are possible.
I'm unsure how broadly to interpret "specific values". If it's values such as democracy or equality, then both values and control are overrated.
Work on corrigibility has provided a decent outline of how to do this. My response is heavily dependent on weak guesses as to how diligent AI companies will be at incorporating the best ideas.
That is probably true if you only look at the philosophical literature. But outside of a few small groups such as that, people widely reject the premise that we should have identical moral concerns for people in the distant future and people currently alive.
Premises that are truly widely shared only support a weaker version of longtermism than yours.
My donations are significantly influenced by the expectation that other people's donations will increase, due to increasing wealth and increasing awareness of the importance of AI.
I'm concerned that the cab rank rule would cause AIs to be overly cautious.
I'm mostly satisfied with how AIs are currently adapting their responses to what they know about the person.
Yes. Current AIs have enough general intelligence that I expect they're sufficient to replace a fair amount of manual labor.
There are still some expensive engineering problems, but they're relatively ordinary problems such as figuring out which robot models ought to be mass produced (without mass production, they'll be too expensive), substantial training to instill domain-specific knowledge, and maybe some need for more energy efficient GPUs.
As long as general-purpose AI is making rapid progress, it probably makes some sense for the robot industry to wait another year or two before choosing the specifics of what kinds of robots to mass-produce - AIs of 2028 are likely to learn domain-specific knowledge faster and more flexibly. But even if we get an AI pause by 2027, I expect at least 10% of manual labor to be replaced by robots by 2030.
If increases in basic AI intelligence were fully halted at 2027 levels, economic growth would still accelerate to something comfortably above 5%/year, due to continuing adaptations of AI to things like robotics.
None of the pause proposals look like they could be 100% effective at stopping increases in AI intelligence.
I used to give pretty high priority to economic growth, but now that growth in excess of 10%/year looks close to inevitable, I'm giving much lower priority to it.
I agree with a fair amount of what you wrote in this post, but I don't see much of an argument against slowing AI capability advances.
It sure doesn't look like Anthropic has lost. See https://polymarket.com/event/anthropic-500b-valuation-in-2026 for evidence.
One good prediction that he made was in his 1986 book Engines of Creation, that a global hypertext system would be available within a decade. Hardly anyone in 1986 imagined that.
But he has almost entirely stopped trying to predict when technologies will be developed. You should read him to imagine what technologies are possible.
That's mostly bearish for bonds because it increases inflation.