All of StevenKaas's Comments + Replies
We tried to write a related answer on Stampy's AI Safety Info:
How could a superintelligent AI use the internet to take over the physical world?
We're interested in any feedback on improving it, since this is a question a lot of people ask. For example, are there major gaps in the argument that could be addressed without giving useful information to bad actors?
Thanks for reporting the broken links. It looks like a problem with the way Stampy is importing the LessWrong tag. Until the Stampy page is fixed, following the links from LessWrong should work.
There's an article on Stampy's AI Safety Info that discusses the differences between FOOM and some other related concepts. FOOM seems to be used synonymously with "hard takeoff" or perhaps with "hard takeoff driven by recursive self-improvement"; I don't think it has a technical definition separate from that. At the time of the FOOM debate, it was taken more for granted that a hard takeoff would involve recursive self-improvement, whereas now there seems to be more emphasis by MIRI people on the possibility that ordinary "other-improvement" (scaling up and...
Like you say, people who are interested in AI existential risk tend to be secular/atheists, which makes them uninterested in these questions. Conversely, people who see religion as an important part of their lives tend not to be interested in AI safety or technological futurism in general. I think people have been averse to mixing AI existential ideas with religious ideas, for both epistemic reasons (worries that predictions and concepts would start being driven by meaning-making motives) and reputational reasons (worries that it would become easier for cr...
Thank you! I linked this from the post (last bullet point under "guidelines for questioners"). Let me know if you'd prefer that I change or remove that.
As I understand it, overestimation of sensitivity tails has been understood for a long time, arguably longer than EA has existed, and sources like Wagner & Weitzman were knowably inaccurate even when they were published. Also, as I understand it, although it has gotten more so over time, RCP8.5 has been considered to be much worse than the expected no-policy outcome since the beginning despite often being presented as the expected no-policy outcome. It seems to me that referring to most of the information presented by this post as "news" fails to adequ...
I think you are right that a lot of these points have been around in the scientific literature for a while. What has changed now is that they are definitely mainstream. The Sherwood et al paper has really helped to formalise the findings of the Annan and Hargreaves paper from years ago, and that has all now been recognised by the IPCC. James Annan told me that he did raise the point about priors with Weitzman a while ago but didn't get anywhere.
One thing that has changed in recent years is that whereas the IEA and others used to estimate that RCP6 was the most likely emissions scenario, it looks like RCP4.5 is the most likely scenario, on current policy. And even that may be too pessimistic
What does an eventual warming of six degrees imply for the amount of warming that will take place in (as opposed to due to emissions in), say, the next century? The amount of global catastrophic risk seems like it depends more on whether warming outpaces humanity's ability to adapt than on how long warming continues.
I was thinking e.g. of Nordhaus's result that a modest amount of mitigation is optimal. He's often criticized for his assumptions about discount rate and extreme scenarios, but neither of those is causing the difference in estimates here.
According to your link, recent famines have killed about 1M per decade, so for climate change to kill 1-5M per year through famine, it would have to increase the problem by a factor of 10-50 despite advancing technology and increasing wealth. That seems clearly wrong as a central estimate. The spreadsheet based o...
I think the upper end of Halstead's <1%-3.5% x-risk estimate is implausible for a few reasons:
1. As his paper notes and his climate x-risk writeup further discusses, extreme change would probably happen gradually instead of abruptly.
2. As his paper also notes, there's a case that issues with priors and multiple lines of evidence imply the tails of equilibrium climate sensitivity are much less fat than those used by Weitzman. As I understand it, ECS > 10 would imply paleoclimate estimates are highly misleading and estimates based on the inst...
Ah, it looks like I was myself confused by the "deaths/year" in line 20 and onward of the original, which represent an increase per year in the number of additional deaths per year. My apologies. At this point I don't understand the GWWC article's reasoning for not multiplying by years an additional time.
My prior was that, since economists argue over the relative value of mitigation (at least beyond low hanging fruit) and present consumption, and present consumption isn't remotely competitive with global health interventions, a cal...
(edit: I no longer endorse this comment)
We don’t expect to be able to recapture most emitted CO2, so a very conservative value to use would be to attribute 50 years of increased deaths to each emission. Hence, this increases the estimate of lives saved by a factor of 50x.
This seems to be the key disagreement between your estimate and GWWC's. As I understand it, if we reduce emissions for the year X by 1%, different things happen in the two calculations:
- In GWWC's calculation, every year Y for decades, we prevent 1% of the deaths during the
A piece such as this should engage with the direct cost/benefit calculations that have been done by economists and EAs (e.g. Giving What We Can), which make it seem hard to argue that climate change is competitive with global health as a cause area.
How much it would take to stay under a mostly arbitrary probability of a mostly arbitrary level of temperature change is a less relevant statistic than how much future temperatures would change in response to reduced emissions.
My nonconfident best guess at an interpretation is that, according to these estimates, for every tonne of carbon:
Future Indians suffer damages utility-equivalent to the present population of India paying a total of $76
Future Americans suffer damages utility-equivalent to the present population of the USA paying a total of $48
Future Saudis suffer damages utility-equivalent to the present population of Saudi Arabia paying a total of $47
Next are China, Brazil, and the UAE, all with $24, and then a lot of other countries, and the sum of all these numbers is $4...
I was about to say this and then saw your comment. My impression from the paper is the $417 is a sum of costs to different countries, and for each of them the cost is a present value to the people in that country, with discounting being applied based on the expected amount of economic growth in that country. So I don't think it's calibrated to present-day Americans, but I don't think it's calibrated to the world's poorest either, and I agree the argument doesn't go through.
There's another problem with the quoted claim, wh...
A nuclear exchange may have the potential to ... possibly lead to the extinction of life on Earth.
I haven't seen anyone seriously argue for this claim and I don't think it's true or true-adjacent.
3. The goal for climate change mitigation should be getting to net zero emissions as fast as possible, as anything other than that still causes warming, and this goal is absent from many EA and the 80,000 Hours write-up.
If there's already the goal of reducing emissions in general, with more reduction being better, is there any reason to add a goal about the zero level specifically? EA generally (and I think rightly) just cares about the expected amount of problem reduction, with exceptions where zero matters being things like diseases that can bounce back from a small number of cases.
What is wrong with it?
If the claims made here from p.13 on are true, it seems like the model can't be reliable. This also disagrees. In general, it seems intuitively like it would be extremely hard to do this kind of statistics and extrapolate to the future with any serious confidence or rely on it for an estimate without a lot more thought. (I haven't tried to look for critiques of the critiques and don't claim to have a rigorous argument.)
Economic activity already goes to wherever it will be the most profitable. I don't see why we wo...
My understanding is there are two somewhat separate issues, one being the improper use of uniform priors and the other being a failure to give estimates that take all evidence (GCMs, recent temperatures, paleoclimate, etc) into account, with probability distributions from mostly-independent evidence sometimes having wrongly been taken as confirmation of the same uncertainty range instead of being combined into a narrower one. Do the estimates that you're eyeballing update on every line of evidence? Annan and Hargreaves under some assumptions find numb...
If the Burke et al. article that you're largely basing the 26% number on is accurate (which I strongly doubt), it seems like trying to cause economic activity to move to more moderate climates might be an extremely effective intervention.
This source suggests we’re on for 4.1-4.8C of warming by 2100, so it seems erroneous to assume 2-4C should be our baseline assumption.
It's hard to tell where this site is getting its numbers from, but my understanding is such claims are usually based on misrepresenting the RCP 8.5 emissions scenario as representative of business as usual even though it makes a number of pessimistic assumptions about other uncertainties and is widely considered as more like a worst case scenario than a median case scenario.
As far as I can tell, claims that extremely ...
On Bayesianism - this is an important point. The very heavy tailed estimates all use a "zero information" prior with an arbitrary cut-off at eg 10 degrees or 20 degrees. (I discuss this in my write-up). This is flawed and more plausible priors are available which thin out the tails a lot.
However, I don't think you need this to get to there being substantial tail risk. Eyeballing the ECS estimates that use plausible priors, there's still something like a 1-5% chance of ECS being >5 degrees, which means that from 1.5 doublings of GHG concentrations, which seems plausible, there's a 1-5% of ~7 degrees
Since somebody was wondering if it's still possible to participate without having signed up through alignmentjam.com:
Yes, people are definitely still welcome to participate today and tomorrow, and are invited to head over to Discord to get up to speed.