All of David_Kristoffersson's Comments + Replies

BERI is doing an awesome service for university-affiliated groups, I hope more will take advantage of it!

Would you really call Jakub's response "hostile"?

Thanks for posting this. I find it quite useful to get an overview of how the EA community is being managed and developed.

Happy to see the new institute take form! Thanks for doing this, Maxime and Konrad. International long-term governance appears very high-leverage to me. Good luck, and I'm looking forward to see more of your work!

• Some "criticisms" are actually self-fulfilling prophecies
• EAs are far too inclined to abandon high-EV ideas that are <50% likely to succeed
• Over-relying on outside views over inside views.
• Picking the wrong outside view / reference class, or not even considering the different reference classes on offer.

Strong upvote for these.

What I appreciate the most about this post is simply just the understanding it shows for people in this situation.

It's not easy. Everyone has their own struggles. Hang in there. Take some breaks. You can learn, you can try something slightly different, or something very different. Make sure you have a balanced life, and somewhere to go. Make sure you have good plan B's (e.g., myself, I can always go back to the software industry). In the for-profit and wider world, there are many skills you can learn better than you would working at an EA org.

Great idea and excellent work, thanks for doing this!

This gets me wondering what other kinds of data sources could be integrated (on some other platform, perhaps). And, I guess you could fairly easily do statistics to see big picture differences between the data on the different sites.

Thanks Linch; I actually missed that the prediction had closed!

Metaculus: Will quantum computing "supremacy” be achieved by 2025? [prediction closed on Jun 1, 2018.]

While I find it plausible that it will happen, I'm not personally convinced that quantum computers will be practically very useful due the difficulties in scaling them up.

Excellent points, Carl. (And Stefan's as well.) We would love to see follow-up posts exploring nuances like these, and I put them into the Convergence list of topics worth elaborating.

Sounds like you got some pretty great engagement out of this experiment! Great work! This exact kind of project, and the space of related ideas seems well worth exploring further.

The five people that we decided to reject were given feedback about their translations as well as their motivation letters. We also provided two simple call-to-actions to them: (1) read our blog and join our newsletter, and (2) follow our FB page and attend our public events. None of these five people have so far done these actions to our awareness.

Semi-general comment regardi

Variant of Korthon's comment:

I never look at the "forum favorites" section. It seems like it's looked the same forever and it takes up a lot of screen real estate without any use for me!

Vision of Earth fellows Kyle Laskowski and Ben Harack had a poster session on this topic at EA Global San Francisco 2019: https://www.visionofearth.org/wp-content/uploads/2019/07/Vision-of-Earth-Asteroid-Manipulation-Poster.pdf

They were also working on a paper on the topic.

Thank you for this article, Michael! I like seeing the different mainline definitions of existential risk and catastrophe alongside each other, and having some common misunderstandings clarified.

Just a minor comment:

That said, at least to me, it seems that “destruction of humanity’s longterm potential” could be read as meaning the complete destruction. So I’d personally be inclined to tweak Ord’s definitions to:

• An existential catastrophe is the destruction of the vast majority of humanity’s long-term potential.
• An existential risk is a risk that threat

I think this is an excellent initiative, thank you, Michael! (Disclaimer: Michael and I work together on Convergence.)

An assortment of thoughts:

• More and more studious estimates of x-risks seem clearly very high value to me due to how much the likelihood of risks and events affect priorities and how the quality of the estimates affect our communication about these matters.
• More estimates should generally should increase our common knowledge of the risks, and individually, if people think about how to make these estimates, they will reach a deeper understa

This kind of complexity tells me that we should talk more often of risk %'s in terms of the different scenarios they are associated with. E.g., the form of current trajectory Ord is using, and also possibly better (if society would act further more wisely) and possible worse trajectories (society makes major mistakes), and what the probabilities are under these.

We can't disentangle talking about future risks and possibilities entirely from the different possible choices of society since these choices are what shapes the future. What we do affect these choices.

(Also, maybe you should edit the original post to include the quote you included here or parts of it.)

Happy to see you found it useful, Adam! Yes, general technological development corresponding to scaling of the vector is exactly the kind of intuition it's meant to carry.

But beyond the trajectories (and maybe specific distances), are you planning on representing the other elements you mention? Like the uncertainty or the speed along trajectories?

Thanks for your comment. Yes; the other elements, like uncertainty, would definitely be part of further work on the trajectories model.

I think that if I could unilaterally and definitively decide on the terms, I'd go with "differential technological development" (so keep that one the same), "differential intellectual development", and "differential development". I.e., I'd skip the word "progress", because we're really talking about something more like "lasting changes", without the positive connotations.

I agree, "development" seems like a superior word to reduce ambiguities. But as you say, this is a summary post, so it might not the best place to suggest switching up terms.

Here's two

Thanks Tobias, I think you make a really good point! You're definitely right that there are some in the cause area who don't think the technological transformation is likely.

I don't think you've established that the 'technological transformation' is essential.

What I wanted to say with this post is that it's essential to the view of a large majority in the cause area. The article is not really meant to do a good job at arguing that it should be essential to peoples' views.

It's possible I'm wrong about the size of the majority; but this was definitely my

The long term future is especially popular among EAs living in Oxford, not surprising given the focus of the Global Priorities Institute on longtermism

Even more than that, The Future of Humanity Institute has been in Oxford since 2005!

I'm not arguing "AI will definitely go well by default, so no one should work on it". I'm arguing "Longtermists currently overestimate the magnitude of AI risk".

Thanks for the clarification Rohin!

I also agree overall with reallyeli.

I'm sympathetic to many of the points, but I'm somewhat puzzled by the framing that you chose in this letter.

Why AI risk might be solved without additional intervention from longtermist

Sends me the message that longtermists should care less about AI risk.

Though, the people in the "conversations" all support AI safety research. And, from Rohin's own words:

Overall, it feels like there's around 90% chance that AI would not cause x-risk without additional intervention by longtermists.

10% chance of existential risk from AI sounds like a problem of catas

Good point, 'x-risk' is short and 'reduction' should be or should become implicit after some short steps of thinking. It will work well in many circumstances. For example, in "I work with x-risk", just as "I work with/in global poverty" works. Though some interjections that occur to me in the moment are: "the cause of x-risk" feels clumsy, "letter, dash, and then a word" feels like an odd construct, and it's a bit negatively oriented.

Thank you for your thoughtful comment!

All work is future oriented Indeed. You don't tend to employ the word 'future' or emphasize it for most work though.

One alternative could be 'full future', signifying that it encompasses both the near and long term.

I think there should be space for new and more specific terms. 'Long term' has strengths, but it's overloaded with many meanings. 'Existential risk reduction' is specific but quite a mouthful; something shorter would be great. I'm working on another article where I will offer one new alternative.

Excellent analysis, thank you! The issue definitely needs a more nuanced discussion. The increasing automation of weaponry (and other technology) won't be stopped globally and pervasively, so we should endeavor to shape how it is developed and applied in a more positive direction.

Indeed! We hope we can deliver that sooner rather than later. Though foundational research may need time to properly come to fruition.

Thanks for your detailed comment, Max!

Relative to my own intuitions, I feel like you underestimate the extent to which your "spine" ideally would be a back-and-forth between its different levels 

I agree, the "spine" glosses over a lot of the important dynamics.

I think I would find it easier to understand to what extent I agree with your recommendations if you gave specific examples of (i) what you consider to be valuable past examples of strategy research, and (ii) how you're planning to do strategy research going forward (or what

Fiscal sponsorship can be very helpful for new groups!

Though regarding attorney fees:

Official nonprofit status can take many months to get in the US, and cost $10-30k of attorney fees.

Where are you getting this from? Attorney fees are on the order of $2-5k.

https://nonprofitelite.com/how-much-will-it-cost-to-get-501c3-tax-exempt-2/

CPA’s and attorneys who specialize in nonprofit organizations routinely charge $2,500–$5,000 for preparation of IRS Form 1023 applications for small organizations, and $6,000-$15,000 for more complex ventures. 

The following two f... (read more)

Good points.

Perhaps funding organizations would like better ways of figuring out the risks of supporting new projects? I think valuable work could be done here.

One way how to think about it* is projecting the space along two axes: "project size" and "risks/establishedness".

Justin Shovelain came up with that. (Justin and I were both on the strategy team of AISC 1.)