All of UwU's Comments + Replies

Just want to signal boost the subreddit for s-risk discussion.

The fellowship will cover what we currently consider to be the most important sources of s-risk (TAI conflict, risks from malevolent actors).

Any reason CLR believes that to be the case specifically? For instance, it's argued on this page that botched alignment attempts/partially aligned AIs (near miss) & unforeseen instrumental drives of an unaligned AI are the 2 likeliest AGI-related s-risks, with malevolent actors (deliberately suffering-aligned AI) currently a lesser concern. I guess TAI conflict could fall under the second category, as an instru... (read more)

A bunch of scenarios are collected in the s-risk sub wiki

I agree nuclear winter risk is overblown and I'm glad to see more EAs discussing that. But I think you're also overrating the survivability of SSBNs, especially non-American ones. They are not a One Weird Trick, Just Add Water for unassailable second strike capability, with upkeep/maintenance only being one aspect of that. Geography plays a huge role in how useful they are, with the US deciding to base most of its warheads on SSBNs because they have the most favourable conditions for them (unrestricted access to and naval dominance of two oceans). In contr... (read more)

Look into suffering-focused AI safety which I think is extremely important and neglected (and s-risks).

I disagree with the claim that the overall accident risk is going down. While it's probably true early warning systems are getting more reliable (though the actual degree of this is really hard to gauge due to their complexity)^[1], a third party (China) adopting launch on warning arguably raises the risk at least 50%, if not more due to initial kinks. Also, as many have pointed out, the emerging trilateral dynamic of three nuclear peers is unprecedented in history and less stable.

Also, what would count as an accidental nuclear war? I think e.g. the US laun... (read more)

Little known detail about the Arkhipov incident. Unsure if true, but if so it sounds like he agreed to fire the torpedo and it all came down to the coincidence of the light getting wedged in the hatch making a few second-difference. Something that may not have happened if the signals officer's motor neurons had fired just slightly differently.

I think the OP is advocating a prize for solving the whole problem, not specific subproblems, which is a novel and interesting idea. Kind of like the $1M Millennium Prize Problems (presumably we should offer far more).

If you offer a prize for the final thing instead of an intermediate one people may also take more efficient paths to the goal than the one we're looking at. I see no downside to doing it, I mean you don't lose any money unless someone actually presents a real solution.

Hey Michael, sorry I didn't get around to commenting on this before you published haha. Long thought dump below:

I'm not sure if they count as "technological developments", but 2 of the largest things I see contributing to nuclear risk are development of ballistic missile defence (BMD) and proliferation of tactical nuclear weapons (TNWs).

The dangers from BMD are manyfold. One is being the cause of a conventional conflict. E.g. As the US continues to develop its maritime ICBM intercept capability, it'll pose a major threat to foreign arsenals. If a significa... (read more)

Just one thought: there are so many ways for a nuclear war to start accidentally or through miscalculations (without necessarily a conventional war) that it just seems so absurd to see estimates like 0.1%. A big part of it is even just the inscrutable failure rate of complex early warning systems composed of software, ground/space based sensors and communications infrastructure. False alarms are much likelier to be acted on during times of high tension as I pointed out. E.g., during that incident Yeltsin, despite observing a weather rocket with a similar f... (read more)

One point about the Pentagon figure, they said "700 warheads by 2027 and at least 1000 by 2030"^[1]. Meaning most likely over 1k. I actually made a bet that they will revise the estimate up again in this/next year's report, e.g. this year we may see "900-1000 by 2027 and 1200-1500 by 2030". They said the stockpile was only expected to double over the decade to the ~400s in the 2020 report to Congress, so by increasing estimates more gradually they're probably mitigating a loss of face & hard questions over the massive intelligence failure. Signs point t... (read more)

https://reducing-suffering.org/near-miss/

Just gonna boost this excellent piece by Tomasik. I think partial alignment/near-misses causing s-risk is potentially an enormous concern. This is more true the shorter timelines are and thus the more likely people are to try using "hail mary" risky alignment techniques. Also more true for less principled/Agent Foundations-type alignment directions.

See my comments here and here for a bit of analysis on targeting/risks of various locations.

Btw I want to add that it may be even more prudent to evacuate population centers preemptively than some think, as some have suggested countervalue targets are unlikely to be hit at the very start of a nuclear war/in a first strike. That's not entirely true since there are many ways cities would be hit with no warning. If Russia or China launches on warning in response to a false alarm, they would be interpreting that act as a (retaliatory) second strike and thus ma... (read more)

Also, not sure where best to post this, but here's a nice project on nuclear targets in the US (+ article). I definitely wouldn't take it at face value, but it sheds some light on which places are potential nuclear targets at least, non-exhaustively.

You mentioned the successful SM-3 intercept test in 2020. While it's true it managed to intercept an "ICBM-representative target", and can be based from ships anywhere they sail (thus posing a major potential threat to the Chinese/NK deterrent in the future), I don't know if I (or the US military) would call it a meaningful operational capability yet. For one we don't even know its success rate. The more mature (and only other) system with ICBM intercept capability, the Ground-Based Interceptor, has barely 50%. ^[1] I'm not sure what you meant by "sending i... (read more)

Don't buy the stuff about expecting a famine that kills billions at all? Especially since she didn't seem to have dug into the actual criticisms of the nuclear winter theory in her post sequence, e.g. the independent components of the theory.  I think very likely (>90%) there won't be any change in temperature at all, which will be the case if any of those components fail. And as I understand it she has since updated towards being less bullish on it since those posts, and people who succeeded her at RP don't think nuclear winter is that likely either.

Imo, evacuating to another country when a nuclear war looks literally imminent may not even be a good move because you'd have to enter a large city with an international airport with transcontinental flights, and the increased risk while you're reentering the city & waiting for your flight is probably greater than the survival benefits from arriving at your SH destination, not to mention flights would probably be booked out if things really looked that dire. A better strategy would be to evacuate whenever the risk looked heightened, but then you'd run ... (read more)

X-posting my comment from LW.

Nuclear winter is a very unlikely, highly conjunctive theory which requires many independent things to ALL happen perfectly, which are already individually suspect. E.g. that cities will all firestorm after being hit by airburst detonations (which itself relies on assumptions like adequate fuel loading per square meter, collapsed structures from the air blast not suffocating the oxygen, etc.), that this will burn in a way producing lots of black carbon, that this carbon will be nearly all lofted into the stratosphere, that this will block a high percentage... (read more)

I'm not overly concerned with the news from this morning. In fact I expected them to raise the nuclear force readiness prior to or simultaneously to commencing the invasion, not now, which is expected going into a time of conflict/high tension from normal peacetime readiness. I had about a 5% chance this will escalate to a nuclear war going into it, and it's not much different now, certainly not above 10% (For context, my odds of escalation to full countervalue exchange in a US intervention in a Taiwan reunification campaign would be about 75%). Virtually ... (read more)

and they value Chinese lives more than non-Chinese.

Right, and the alternative here (US leaders) don't do that?

For AI safety - maybe Redwood has the most room for funding? They seem to be the most interested in growth (correct me if I'm wrong). And even if the existing players don't have more room, other ways need to be thought of to scale up further through funding as the field is clearly still too small to compete in the race against the titanic field of AI capabilities.

Agree longevity needs to be funded more as well, though lots of aging billionaires like Bezos seem to be throwing tons of money at it these days too so maybe EA money would be much less useful/uniquely needed there than e.g. AI alignment.