All of David_Kristoffersson's Comments + Replies
This looks incorrect to me. Factory farming interventions winning over x-risk interventions requires both thinking (1) that animals have moral weight not too far from that of humans, and (2) that the amount of suffering in factory farming is more morally important than increasing the chances of humanity and life in general of surviving at all. These assumptions are not shared by everyone in EA.
Making the writing easy for myself: What's your response to Carl Shulman's take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
You may have responded to this before. Feel free to provide a link.
This page is giving me a 404 right now: https://pauseai.info/mitigating-pause-failures
Short answer I think trying to time this is too galaxy-brained. I think getting the meme of Pause out there ASAP is good because it pushes the Overton window and it gives people longer to chew on it. If and when warning shots occur, they will mainly advance Pause if people already had the idea that Pause would combat things like the warning shot happening before they happened.
I think takes that rely on saving up some kind of political capital and deploying it at the perfect time are generally wrong. PauseAI will gain more capital with more time and conversation, not use it up.
I’m not Holly, but my response is that getting a pause now is likely to increase, rather than decrease, the chance of getting future pauses. Quoting Evan Hubinger (2022):
...In the theory of political capital, it is a fairly well-established fact that ‘Everybody Loves a Winner.’ That is: the more you succeed at leveraging your influence to get things done, the more influence you get in return. This phenomenon is most thoroughly studied in the context of the ability of U.S. presidents to get their agendas through Congress—contrary to a naive mode
X-risk: yes. The idea of fast AI development: yes. Knowing the phrase "takeoff speed"? No. For sure, this also depends a bit on the type of role and seniority. "Moral patienthood" strikes me as one of those terms where if someone is interested in one of our jobs, they will likely get the idea, but they might not know the term "moral patienthood". So let's note here that I wrote "language", and you wrote "concepts", and these are not the same. One of the distinctions I care about is that people understand, or can easily come to understand the ideas/concepts...
Speaking as a hiring manager at a small group in AI safety/governance who made an effort to not just hire insiders (it's possible I'm in a minority -- don't take my take for gospel if you're looking for a job), it's not important to me that people know a lot about in-group language, people, or events around AI safety. It is very important to me that people agree with foundational ideas such as to actually be impact-focused and to take short-ish AI timelines and AI risk seriously and have thought about it seriously.
Thanks Holly. I agree that fixating on just trying to answer the "AI timelines" question won't be productive for most people. Though, we all need to come to terms with it somehow. I like your callout for "timeline-robust interventions". I think that's a very important point. Though I'm not sure that implies calling your representatives.
I disagree that "we know what we need to know". To me, the proper conversation about timelines isn't just "when AGI", but rather, "at what times will a number of things happen", including various stages of post-AGI technolog...
I like this principles-first approach! I think it's really valuable to have a live discussion that starts from "How do we do the most good?", even if I am kind of all-in on one cause. (Kind of: I think most causes tie together: making the future turn out well.) I think it'd be a valuable use of the time of you folks to try and clarify and refine your approach, philosophy, and incentives further, using the comments here as one input.
I have this fresh in my mind as we've had some internal discussion on the topic at Convergence. My personal take is that "consciousness" is a bit of a trap subject because it bakes in a set of distinct complex questions, people talk about it differently, it's hard to peer inside the brain, and there's slight mystification because consciousness feels a bit magical, from the inside. Sub-topics include but are not limited to: 1. Higher-order though. 2. Subjective experience. 3. Sensory integration. 3. Self-awareness. 4. Moral patienthood.
My recommendation is ...
- Some "criticisms" are actually self-fulfilling prophecies
- EAs are far too inclined to abandon high-EV ideas that are <50% likely to succeed
- Over-relying on outside views over inside views.
- Picking the wrong outside view / reference class, or not even considering the different reference classes on offer.
Strong upvote for these.
What I appreciate the most about this post is simply just the understanding it shows for people in this situation.
It's not easy. Everyone has their own struggles. Hang in there. Take some breaks. You can learn, you can try something slightly different, or something very different. Make sure you have a balanced life, and somewhere to go. Make sure you have good plan B's (e.g., myself, I can always go back to the software industry). In the for-profit and wider world, there are many skills you can learn better than you would working at an EA org.
Metaculus: Will quantum computing "supremacy” be achieved by 2025? [prediction closed on Jun 1, 2018.]
While I find it plausible that it will happen, I'm not personally convinced that quantum computers will be practically very useful due the difficulties in scaling them up.
Note that we believe that quantum supremacy has already been achieved.
As in, the quantum computer Sycamore from Google is capable of solving a (toy) problem that we currently believe unfeasible in a classical computer.
Of course, there is a more interesting question of when will we be able to solve practical problems using quantum computing. Experts believe that the median for a practical attack on modern crypto is ~2035.
I regardless believe that outside (and arguably within) quantum cryptanalysis the applications will be fairly limited.
The paper in my post...
Sounds like you got some pretty great engagement out of this experiment! Great work! This exact kind of project, and the space of related ideas seems well worth exploring further.
The five people that we decided to reject were given feedback about their translations as well as their motivation letters. We also provided two simple call-to-actions to them: (1) read our blog and join our newsletter, and (2) follow our FB page and attend our public events. None of these five people have so far done these actions to our awareness.
Semi-general comment regardi
Vision of Earth fellows Kyle Laskowski and Ben Harack had a poster session on this topic at EA Global San Francisco 2019: https://www.visionofearth.org/wp-content/uploads/2019/07/Vision-of-Earth-Asteroid-Manipulation-Poster.pdf
They were also working on a paper on the topic.
Thank you for this article, Michael! I like seeing the different mainline definitions of existential risk and catastrophe alongside each other, and having some common misunderstandings clarified.
Just a minor comment:
That said, at least to me, it seems that “destruction of humanity’s longterm potential” could be read as meaning the complete destruction. So I’d personally be inclined to tweak Ord’s definitions to:
- An existential catastrophe is the destruction of the vast majority of humanity’s long-term potential.
- An existential risk is a risk that threat
I think this is an excellent initiative, thank you, Michael! (Disclaimer: Michael and I work together on Convergence.)
An assortment of thoughts:
- More and more studious estimates of x-risks seem clearly very high value to me due to how much the likelihood of risks and events affect priorities and how the quality of the estimates affect our communication about these matters.
- More estimates should generally should increase our common knowledge of the risks, and individually, if people think about how to make these estimates, they will reach a deeper understa
This kind of complexity tells me that we should talk more often of risk %'s in terms of the different scenarios they are associated with. E.g., the form of current trajectory Ord is using, and also possibly better (if society would act further more wisely) and possible worse trajectories (society makes major mistakes), and what the probabilities are under these.
We can't disentangle talking about future risks and possibilities entirely from the different possible choices of society since these choices are what shapes the future. What we do affect these choices.
(Also, maybe you should edit the original post to include the quote you included here or parts of it.)
But beyond the trajectories (and maybe specific distances), are you planning on representing the other elements you mention? Like the uncertainty or the speed along trajectories?
Thanks for your comment. Yes; the other elements, like uncertainty, would definitely be part of further work on the trajectories model.
I think that if I could unilaterally and definitively decide on the terms, I'd go with "differential technological development" (so keep that one the same), "differential intellectual development", and "differential development". I.e., I'd skip the word "progress", because we're really talking about something more like "lasting changes", without the positive connotations.
I agree, "development" seems like a superior word to reduce ambiguities. But as you say, this is a summary post, so it might not the best place to suggest switching up terms.
Here's two
Thanks Tobias, I think you make a really good point! You're definitely right that there are some in the cause area who don't think the technological transformation is likely.
I don't think you've established that the 'technological transformation' is essential.
What I wanted to say with this post is that it's essential to the view of a large majority in the cause area. The article is not really meant to do a good job at arguing that it should be essential to peoples' views.
It's possible I'm wrong about the size of the majority; but this was definitely my
I'm sympathetic to many of the points, but I'm somewhat puzzled by the framing that you chose in this letter.
Why AI risk might be solved without additional intervention from longtermist
Sends me the message that longtermists should care less about AI risk.
Though, the people in the "conversations" all support AI safety research. And, from Rohin's own words:
Overall, it feels like there's around 90% chance that AI would not cause x-risk without additional intervention by longtermists.
10% chance of existential risk from AI sounds like a problem of catas
Good point, 'x-risk' is short and 'reduction' should be or should become implicit after some short steps of thinking. It will work well in many circumstances. For example, in "I work with x-risk", just as "I work with/in global poverty" works. Though some interjections that occur to me in the moment are: "the cause of x-risk" feels clumsy, "letter, dash, and then a word" feels like an odd construct, and it's a bit negatively oriented.
Thank you for your thoughtful comment!
All work is future oriented Indeed. You don't tend to employ the word 'future' or emphasize it for most work though.
One alternative could be 'full future', signifying that it encompasses both the near and long term.
I think there should be space for new and more specific terms. 'Long term' has strengths, but it's overloaded with many meanings. 'Existential risk reduction' is specific but quite a mouthful; something shorter would be great. I'm working on another article where I will offer one new alternative.
Thanks for your detailed comment, Max!
Relative to my own intuitions, I feel like you underestimate the extent to which your "spine" ideally would be a back-and-forth between its different levels
I agree, the "spine" glosses over a lot of the important dynamics.
I think I would find it easier to understand to what extent I agree with your recommendations if you gave specific examples of (i) what you consider to be valuable past examples of strategy research, and (ii) how you're planning to do strategy research going forward (or what...
Fiscal sponsorship can be very helpful for new groups!
Though regarding attorney fees:
Official nonprofit status can take many months to get in the US, and cost $10-30k of attorney fees.
Where are you getting this from? Attorney fees are on the order of $2-5k.
https://nonprofitelite.com/how-much-will-it-cost-to-get-501c3-tax-exempt-2/
CPA’s and attorneys who specialize in nonprofit organizations routinely charge $2,500–$5,000 for preparation of IRS Form 1023 applications for small organizations, and $6,000-$15,000 for more complex ventures.
The following two f...
Good points.
Perhaps funding organizations would like better ways of figuring out the risks of supporting new projects? I think valuable work could be done here.
One way how to think about it* is projecting the space along two axes: "project size" and "risks/establishedness".
Justin Shovelain came up with that. (Justin and I were both on the strategy team of AISC 1.)
This seems clearly incorrect to me. I'm surprised to see this claim fronted prominently inside a highly upvoted comment. It also strikes me as uncharitable by invoking the "fanatical" frame.
Prioritizing x-risk merely requires thinking the risk of existential catastrophe is close enough in time.
https:/... (read more)