Median european seed stage pre-money valuation is €5.0 million. People often raise the seed round within 1-2 years of starting. So €1M is donating 20-40% of your company.[1]
Of course a large fraction of companies never make it even so far as a seed round. It's very hard to figure out what reference class someone should be in but it is worth noting that there are a good number of people who read this forum who have achieved valuations greater than that average within a couple years of trying.
To see this, let’s be generous and say the expected number of future well-off people added by preventing an existential catastrophe is only 10^40
I think your argumentation supports the 10^40 number more than the "well-off" claim. I'm not sure for the best canonical source expressing skepticism of the "well-off" bit, but The Future Might Not Be So Great contains a long list of arguments about whether future people will be well off; I would be interested in you responding to some of those.
Anthony cites Greaves and MacAskill giving an example similar to your gunpowder one:
Consider, for example, would-be longtermists in the Middle Ages. It is plausible that the considerations most relevant to their decision – such as the benefits of science, and therefore the enormous value of efforts to help make the scientific and industrial revolutions happen sooner – would not have been on their radar. Rather, they might instead have backed attempts to spread Christianity, perhaps by violence: a putative route to value that, by our more enlightened lights today, looks wildly off the mark. The suggestion, then, is that our current predicament is relevantly similar to that of our medieval would-be longtermists.
I personally think these examples are less compelling than they first appear (e.g. the persistence literature generally finds weaker effects than what you might imagine), but I agree that a failure of EAs to find examples of sign flips doesn't mean that future ones won't exist.
Thanks! I can't tell if this is cruxy, but for what it's worth your "pessimal induction" vignettes don't resonate with me in a way which makes me less motivated by the unawareness concerns.
For example, Bostrom coined the phrase "attention hazard" in 2011. I remember someone telling me that MIRI was net-negative for this reason at EAG 2015, and I would be surprised if e.g. Habryka hadn't considered this risk before starting Lightcone. So I disagree with citing him/this as a good example of unawareness; it's more that they mis-estimated a known risk factor.
Similarly, I remember talking about SARP at one of my first EAGs. I think I came across it in Brian Tomasik's 2007 post, maybe even before I had encountered EA. Perhaps I've mis-estimated those concerns, but it doesn't seem like unawareness.
My overall experience is kind of the opposite of yours: when I got involved in EA people talked a lot about "Cause X" and "Crucial Considerations" and now they've mostly just... stopped? Like people tried to find other considerations, and there's some new stuff around s-risks and weird decision theories etc., but if you look at what people talk about at EAGs today it feels mostly like more precise versions of what was discussed in 2016, rather than a large and unpredictable jump from the older understanding. Or, more technically: it feels like we've had updates in evidence-space, but not as many updates in hypothesis-space, and I understand the latter to be motivating imprecision.
Obviously, this could be because EAs suck at cause prio research, or we just haven't been hit yet with the big update, etc., but the "pessimal induction" seems less pessimal to me.
I'm confused why you are denominating options in robotics-startup-days saved. This feels like a narrow definition of "impact". I'd encourage you to consider other ways to benefit the world; parts 4-6 of the 80k career guide might be helpful. Specifically, under the assumption that the thing you terminally value is more like "reducing suffering" than "robotics progress", I would encourage you to first consider which causes advance those values, and only then drill into job options. (The 80k career guide will walk you through this.)
Also, even to the extent you do just terminally value robotics progress, you might want to consider whether robotics will advance too quickly for your estimation to be accurate.
> do you think that, if we had a theory of sociopolitics that was about as good as 20th-century economics, then we wouldn't be clueless about how to do sociopolitical interventions (like founding AI safety movements) effectively?
No, because I think “founding AI safety movements that succeed at making the far future go better” is a pretty out-of-distribution kind of sociopolitical intervention.
Suppose instead we had a comparably good theory of the right reference class, e.g. "movements trying to shape transformative technologies." Would we still be clueless about AI safety movement-building?
More generally: you list various considerations across your posts and I have a hard time understanding which is load-bearing for your answer here. Some possibilities:
We're clueless because we haven't yet developed the relevant theory (Richard's reading IIUC, on which cluelessness is contingent and reducible)
No such theory could be validated even in principle, because we never observe the target variable (far-future value) and calibration on near-term proxies doesn't transfer
Even a validated theory wouldn't help, because impact is dominated by considerations inaccessible to any theory (e.g. unconceived hypothesis classes)
What's the clearest example of a complex cluelessness sign flip you're aware of?
(By "clear" I mean "had a very narrow confidence interval before encountering some consideration and a narrow interval after encountering that consideration but the CIs now center points with opposite signs".[1])
The clearest examples I know of (e.g. rescuing Hitler as a child) seem to me like examples of simple cluelessness. You list some examples here, but they don't seem that clear to me, e.g. I disagree that "Early awareness-raising about AGI x-risk presumably seemed robustly good" and would guess most people involved in that had CIs which comfortably straddled zero.
Thanks! "Don't arbitrarily favor some moral patients over others" was the most intuitive definition of "impartial" in my mind, so I was confused about how it was being used here.
I feel somewhat confused about what exactly the challenge here is. You say:
Grant all the premises but argue that the conclusion—that we have no impartial-altruistic reason to prefer any action over another—doesn’t follow.
My understanding is that Anthony agrees we can justifiably prefer actions, and even do so on altruistic consequence-based grounds if something like bracketing works — but he'd deny that either deontological reasons or bracketed reasons are "impartial altruistic reasons" in the sense his conclusion targets: the first aren't consequence-comparisons at all, and the second are consequence-comparisons that give up full impartiality to stay determinate.
Is my understanding correct? If so, I would find a more precise statement of the inference people are supposed to challenge helpful.
Median european seed stage pre-money valuation is €5.0 million. People often raise the seed round within 1-2 years of starting. So €1M is donating 20-40% of your company.[1]
Of course a large fraction of companies never make it even so far as a seed round. It's very hard to figure out what reference class someone should be in but it is worth noting that there are a good number of people who read this forum who have achieved valuations greater than that average within a couple years of trying.
Cool idea! I like the move of taking a selfish ambitious drive and trying to transform it into something more altruistic.
I think your argumentation supports the 10^40 number more than the "well-off" claim. I'm not sure for the best canonical source expressing skepticism of the "well-off" bit, but The Future Might Not Be So Great contains a long list of arguments about whether future people will be well off; I would be interested in you responding to some of those.
Anthony cites Greaves and MacAskill giving an example similar to your gunpowder one:
I personally think these examples are less compelling than they first appear (e.g. the persistence literature generally finds weaker effects than what you might imagine), but I agree that a failure of EAs to find examples of sign flips doesn't mean that future ones won't exist.
Thanks! I can't tell if this is cruxy, but for what it's worth your "pessimal induction" vignettes don't resonate with me in a way which makes me less motivated by the unawareness concerns.
For example, Bostrom coined the phrase "attention hazard" in 2011. I remember someone telling me that MIRI was net-negative for this reason at EAG 2015, and I would be surprised if e.g. Habryka hadn't considered this risk before starting Lightcone. So I disagree with citing him/this as a good example of unawareness; it's more that they mis-estimated a known risk factor.
Similarly, I remember talking about SARP at one of my first EAGs. I think I came across it in Brian Tomasik's 2007 post, maybe even before I had encountered EA. Perhaps I've mis-estimated those concerns, but it doesn't seem like unawareness.
My overall experience is kind of the opposite of yours: when I got involved in EA people talked a lot about "Cause X" and "Crucial Considerations" and now they've mostly just... stopped? Like people tried to find other considerations, and there's some new stuff around s-risks and weird decision theories etc., but if you look at what people talk about at EAGs today it feels mostly like more precise versions of what was discussed in 2016, rather than a large and unpredictable jump from the older understanding. Or, more technically: it feels like we've had updates in evidence-space, but not as many updates in hypothesis-space, and I understand the latter to be motivating imprecision.
Obviously, this could be because EAs suck at cause prio research, or we just haven't been hit yet with the big update, etc., but the "pessimal induction" seems less pessimal to me.
It's great that you're thinking about this!
I'm confused why you are denominating options in robotics-startup-days saved. This feels like a narrow definition of "impact". I'd encourage you to consider other ways to benefit the world; parts 4-6 of the 80k career guide might be helpful. Specifically, under the assumption that the thing you terminally value is more like "reducing suffering" than "robotics progress", I would encourage you to first consider which causes advance those values, and only then drill into job options. (The 80k career guide will walk you through this.)
Also, even to the extent you do just terminally value robotics progress, you might want to consider whether robotics will advance too quickly for your estimation to be accurate.
You respond to Richard Ngo here:
Suppose instead we had a comparably good theory of the right reference class, e.g. "movements trying to shape transformative technologies." Would we still be clueless about AI safety movement-building?
More generally: you list various considerations across your posts and I have a hard time understanding which is load-bearing for your answer here. Some possibilities:
What's the clearest example of a complex cluelessness sign flip you're aware of?
(By "clear" I mean "had a very narrow confidence interval before encountering some consideration and a narrow interval after encountering that consideration but the CIs now center points with opposite signs".[1])
The clearest examples I know of (e.g. rescuing Hitler as a child) seem to me like examples of simple cluelessness. You list some examples here, but they don't seem that clear to me, e.g. I disagree that "Early awareness-raising about AGI x-risk presumably seemed robustly good" and would guess most people involved in that had CIs which comfortably straddled zero.
Or alternatively: there are two representors with narrow but non-overlapping CIs.
Thanks! "Don't arbitrarily favor some moral patients over others" was the most intuitive definition of "impartial" in my mind, so I was confused about how it was being used here.
I feel somewhat confused about what exactly the challenge here is. You say:
My understanding is that Anthony agrees we can justifiably prefer actions, and even do so on altruistic consequence-based grounds if something like bracketing works — but he'd deny that either deontological reasons or bracketed reasons are "impartial altruistic reasons" in the sense his conclusion targets: the first aren't consequence-comparisons at all, and the second are consequence-comparisons that give up full impartiality to stay determinate.
Is my understanding correct? If so, I would find a more precise statement of the inference people are supposed to challenge helpful.