All of Jeffrey Ladish's Comments + Replies

@Daniel_Eth asked me why I choose 1:1 offsets. The answer is that I did not have a principled reason for doing so, and do not think there's anything special about 1:1 offsets except that they're a decent schelling point. I think any offsets are better than no offsets here. I don't feel like BOTECs of harm caused as a way to calculate offsets are likely to be particularly useful here but I'd be interested in arguments to this effect if people had them. 

Really appreciate you! It's felt stressful sometimes as just someone in the community and it's hard to imagine how stressful it would feel for me in your shoes. Really appreciate your hard work, and I think the EA movement is significantly improved through your hard work maintaining and improving and moderating the forum, and all the mostly-unseen-but-important work mitigating conflicts & potential harm in the community.

I think it's worth noting that that I'd expect you would gain a significant relative advantage if you get out of cities before other people, such that acting later would be a lot less effective at furthering your survival & rebuilding goals.

I expect the bulk of the risk of an all out nuclear war to happen in the couple of weeks after the first nuclear use. If I'm right, then the way to avoid the failure mode you're identifying is returning in a few weeks if no new nuclear weapons have been used, or similar.

I think the problem is that the vagueness of the type of commitment the GWWC represents. If it's an ironclad commitment, people should lose a lot of trust in you. If it was a "best of intention" type commitment, people should only lose a modest amount of trust in you. I think the difference matters!

I super agree it's important not to conflate  "do you keep actually-thoughtful promises you think people expected you to interpret as real commitments" and "do you take all superficially-promise-like-things as serious promises"!  And while I generally want people to think harder about what they're asking for wrt commitments, I don't think going overboard on strict-promise interpretations is good. Good promises have a shared understanding between both parties. I think a big part of building trust with people is figuring out a good shared language ... (read more)

I think it will require us to reshape / redesign most ecosystems & probably pretty large parts of many / most animals. This seems difficult but well within the bounds of a superintelligence's capabilities. I think that at least within a few decades of greater-than-human-AGI we'll have superintelligence, so in the good future I think we can solve this problem.

I don't think an ordinary small/medium tech company can succeed at this. I think it's possible with significant (extraordinary) effort, but that sort of remains to be seen.

As I said in another thread:

>> I think it's an open question right now. I expect it's possible with the right resources and environment, but I might be wrong. I think it's worth treating as an untested hypothesis ( that we can secure X kind of system for Y application of resources ), and to try to get more information to test the hypothesis. If AGI development is impossible to secu... (read more)

I agree that a lot of the research today by leading labs is being published. I think the norms are slowly changing, at least for some labs. Deciding not to (initially) release the model weights of GPT-2 was a big change in norms iirc, and I think the trend towards being cautious with large language models has continued. I expect that as these systems get more powerful, and the ways they can be misused gets more obvious, norms will naturally shift towards less open publishing. That being said, I'm not super happy with where we're at now, and I think a lot o... (read more)

I think it's an open question right now. I expect it's possible with the right resources and environment, but I might be wrong. I think it's worth treating as an untested hypothesis ( that we can secure X kind of system for Y application of resources ), and to try to get more information to test the hypothesis. If AGI development is impossible to secure, that cuts off a lot of potential alignment strategies. So it seems really worth trying to find out if it's possible.

I expect most people to think either that AMF or MIRI is much more likely to do good. So from most agent's perspectives, the unilateral defection is only better if their chosen org wins. If someone has more of a portfolio approach that weights longtermist and global poverty  efforts similarly, then your point holds. I expect that's a minority position though.

Thanks!

I see you define it a few paragraphs down, but at the top would be helpful I think

Could you define ESG investing at the begining of your post?

Yeah, I would agree with that! I think radiological weapons are some of the most relevant nuclear capabilities / risks to consider from a longterm perspective, due to their risk of being developed in the future.

The part I added was:

"By a full-scale war, I mean a nuclear exchange between major world powers, such as the US, Russia, and China, using the complete arsenals of each country. The total number of warheads today (14,000) is significantly smaller than during the height of the cold war (70,000). While extinction from nuclear war is unlikely today, it may become more likely if significantly more warheads are deployed or if designs of weapons change significantly."

I also think indirect extinction from nuclear war is unlikely, but I would like to address this m... (read more)

Thanks for this perspective!

Strong agree!

I mean that the amount required to cover every part of the Earth's surface  would serve no military purpose. Or rather, it might enhance one's deterrent a little bit, but it would
1) kill all of one's own people, which is the opposite of a defense objective
2) not be a very cost effective way to improve one's deterrent. In nearly all cases it would make more sense to expand second strike capabilities by adding more submarines, mobile missile launchers, or other stealth second strike weapons.

Which isn't to say this couldn't happen! Military research team... (read more)

FWIW, my guess is that you're already planning to do this, but I think it could be valuable to carefully consider information hazards before publishing on this [both because of messaging issues similar to the one we discussed here and potentially on the substance, e.g. unclear if it'd be good to describe in detail "here is how this combination of different hazards could kill everyone"]. So I think e.g. asking a bunch of people what they think prior to publication could be good. (I'd be happy to review a post prior to publication, though I'm not sure if I'm

Thanks, fixed!

Thanks, fixed!

This may be in the Brookings estimate, which I haven't read yet, but I wonder how much cost disease + reduction in nuclear force has affected the cost per warhead / missile. My understanding is that many military weapon systems get much more expensive over time for reasons I don't well understand.

Warheads could be altered to increase the duration of radiation effects from fallout, but this would would also reduce their yield, and would represent a pretty large change in strategy. We've gone 70 years without such weapons, which the recent Russian submersibl... (read more)

I think I gave the impression that I'm making  a more expansive claim than I actually mean to make, and will edit the post to clarify this.  The main reason I wanted to write this post is that a lot of people, including a number in the EA community, start with the conception that a nuclear war is relatively likely to kill everyone, either for nebulous reason or because of nuclear winter specifically. I know most people who've examined it know this is wrong, but I wanted that information to be laid out pretty clearly, so someone could get a summar... (read more)

Some quick answers to your questions based on my current beliefs:

• Is there a high chance that human population completely collapses as a result of less than 90% of the population being wiped out in a global catastrophe?

I think the answer in the short term is no, if "completely collapses" means something like "is unable to get back to at least 1950's level technology in 500 years". I think think there are a number of things that could reduce humanity's "technological carrying capacity". I'm currently working on ex... (read more)

I want to give a brief update on this topic. I spent a couple months researching civilizational collapse scenarios and come to some tentative conclusions. At some point I may write a longer post on this, but I think some of my other upcoming posts will address some of my reasoning here.

My conclusion after investigating potential collapse scenarios:

1) There are a number of plausible (>1% probability) scenarios in the next hundred years that would result in a "civilizational collapse", where an unprecedented number of people die and key technolo... (read more)

I do know of a project here that is pretty promising, related to improving secure communication between nuclear weapons states. If you know people with significant expertise who might be interested pm me.

This seems approximately right. I have some questions around how competitive pressures relate to common-good pressures. It's sometimes the case that they are aligned (e.g. in many markets).

Also, there may be a landscape of coalitions (which are formed via competitive pressures), and some of these may be more aligned with the common good and some may be less. And their alignment with the public good may be orthogonal to their competitiveness / fitness.

It would be weird if it were completely orthogonal, but I would expect it to naturally be somewhat orthogonal.

An additional point is that "relevant roles in government" should probably mean contracting work as well. So it's possible to go work for Raytheon, get a security clearance, and do cybersecurity work for government (and that pays significantly better!)

I think working at a top security company could be a way to gain a lot of otherwise hard to get experience. Trail of bits, NCC Group, FireEye are a few that come to mind.

Our current best guess is that people who are interested should consider seeking security training in a top team in industry, such as by working on security at Google or another major tech company, or maybe in relevant roles in government (such as in the NSA or GCHQ). Some large security companies and government entities offer graduate training for people with a technical background. However, note that people we’ve discussed this with have had differing views on this topic.

This is a big area of uncertainty for me. I agree that Google & other top compa... (read more)

One potential area of biorisk + infosec work would be in improving the biotech industry's ability to secure synthesis & lab automation technology from use in creating dangerous pathogens / organisms.

This could be done via circumventing existing controls (i.e. ordering a virus which is on a banned-sequence list), or by hijacking synthesis equipment itself. So protecting this type of infrastructure may be super important. I could see this being a more policy oriented role, but one that would require infosec skills.

I expect this work to be valuable ... (read more)