Benjamin is a research analyst at 80,000 Hours. Before joining 80,000 Hours, he worked for the UK Government and did some economics and physics research.
Thanks for this! Looks like we actually roughly agree overall :)
Thanks for this thoughtful post! I think I stand by my 1 in 10,000 estimate despite this.
A few short reasons:
If it helps at all, my subjective estimate of the risk from AI is probably around 1%, and approximately none of that comes from worrying about killer nanobots. I wrote about what an AI-caused existential catastrophe might actually look like here.
Hi! Wanted to follow up as the author of the 80k software engineering career review, as I don't think this gives an accurate impression. A few things to say:
This looks really cool, thanks Tom!
I haven't read the report in full (just the short summary) - but I have some initial scepticism, and I'd love to answers to some of the following questions, so I can figure out how much evidence this report is on takeoff speeds. I've put the questions roughly in order of subjective importance to my ability to update:
That's all the thoughts that jumped into my head when I read the summary and skimmed the report - sorry if they're all super obvious if I'd read it more thoroughly! Again, super excited to see models with this level of detail, thanks so much!
I agree with (a). I disagree that (b) is true! And as a result I disagree that existing CEAs give you an accurate signpost.
Why is (b) untrue? Well, we do have some information about the future, so it seems extremely unlikely that you won't be able to have any indication as to the sign of your actions, if you do (a) reasonably well.
Again, I don't purely mean this from an extreme longtermist perspective (although I would certainly be interested in longtermist analyses given my personal ethics). For example, simply thinking about population changes in the above report would be one way to move in this direction. Other possibilities include thinking about the effects of GHW interventions on long-term trajectories, like growth in developing countries (and that these effects may dominate short-term effects like DALYs averted for the very best interventions). I haven't thought much about what other things you'd want to measure to make these estimates, but I would love to see someone try, and it seems pretty crucial if you're going to be doing accurate CEAs.
Sure, happy to chat about this!
Roughly I think that you are currently not really calculating cost-effectiveness. That is, whether you're giving out malaria nets or preventing nuclear war, almost all of the effects of your actions will be affecting people in the future.
To clarify, by "future" I don't necessarily mean "long run future". Where you put that bar is a fascinating question. But focusing on current lives lost seems to approximately ignore most of the (positive or negative) value, so I expect your estimates to not be capturing much about what matters.
(You've probably seen this talk by Greaves, but flagging it in case you haven't! Sam isn't a huge fan, I think in part because Greaves reinvents a bunch of stuff that non-philosophers have already thought a bunch about, but I think it's a good intro to the problem overall anyway.)
I'm curious about the ethical decisions you've made in this report. What's your justification for evaluating current lives lost? I'd be far more interested in cause-X research that considers a variety of worldviews, e.g. a number of different ways of evaluating the medium or long-term consequences of interventions.
There are important reasons to think that the change by the EA community is within the measurement error of these surveys, which makes this less noteworthy.
(Like say you put +/- 10 years and +/- 10% on all these answers - note there are loads of reasons why you wouldn't actually assess the uncertainty like this, (e.g. probabilities can't go below 0 or above 1), but just to get a feel for the uncertainty this helps. Well, then you get something like:
Then many many EA timelines and shifts in EA timelines fall within those errors.)
Reasons why these surveys have huge error
1. Low response rates.
The response rates were really quite low.
2. Low response rates + selection biases + not knowing the direction of those biases
The surveys plausibly had a bunch of selection biases in various directions.
This means you need a higher sample to converge on the population means, so the surveys probably aren't representative. But we're much less certain in which direction they're biased.
Quoting me:
3. Other problems, like inconsistent answers in the survey itself
AI impacts wrote some interesting caveats here, including:
The 80k podcast on the 2016 survey goes into this too.