I think Is Power-Seeking AI An Existential Risk is probably the best introduction, though it's probably too long as a first introduction if the person is yet that motivated. It's also written as a list of propositions, with probabilities, and that might not appeal to many people.
I also listed some shorter examples in this post for the AI Safety Public Materials Bounty we're running, that might be more suitable as a first introduction. Here are the ones most relevant to people not versed in machine learning:
Thanks for all the work you are doing here, I think some really amazing groups could come out of this. I am cautiously excited about many different kinds of groups starting.
I found it a bit surprising that the list of criteria for group organizers (including "nice to have") doesn't seem to have anything like "really cares about the objectives of their group," "really cares about improving the long term future," "is altruistic to some degree"
Being truth-seeking and open-mindedHaving a strong understanding of whatever topic their group is about, and/or being
We don't expect the work to be published anywhere when it's submitted.
For certain pieces, we may work with authors to publish them somewhere, publish them on our website, or adapt them and publish an adapted version somewhere. But this is not guaranteed.
In general, we expect that the best pieces will be generally suited for an audience of either smart people who don't know about ML, or ML researchers. Though there is a lot of room for pieces that are more optimized for particular audiences and venues, we think that more general pieces would serve as great inspiration for those later pieces.
I edited the title to say "$20k in bounties" to make it more clear.
From the original text:
Winners of the bounty will win $2,000 each, for a total of up to ten possible bounty recipients.
This doesn't mean each person who submits an entry gets $2,000. We will award this to entries that meet a high bar for quality (roughly, material that we would actually be interested in using for outreach).
I missed that part of footnote 3, it does seem to address a lot of what I said. I appreciate your response.
I do think the vast majority of people will not read footnote 3, so it's important for the main body of the text (and the visuals) to give the right impression. This means comparing averages to averages, or possible tail events to possible tail events. It sounds like this is your plan now, and if so that's great!
Yes, that's my mistake, sorry.
Posted too soon, didn't realize he had changed his mind about crossposting, please ignore.
I linkposted this when it came out, and Devin Kalish sent this comment:
A quick note, this piece was already posted to the forum briefly, and then deleted. The author said in a comment that he would rather it not be crossposted to this forum:https://astralcodexten.substack.com/p/criticism-of-criticism-of-criticism/comment/7853073I don't know if the two are related, but I might reach out to ask him if he's alright with you posting it.
Which led to me taking down my post, since I don't really like to crosspost things if people prefer that I not.
Just wanted to let you know!
Yes, I'm sorry, I talked to Claire about it and updated, sorry for the mixed messages and any stress this caused.
This point is covered quite well by Derek Parfit in his seminal book Reasons and Persons, Chapter 1, Part 17. In my view the entire chapter is excellent and worth reading, but here is an excerpt from Part 17:
Consider, for example, theft. On some versions of C [Consequentialism], it is intrinsically bad if property is stolen. On other versions of C [such as hedonistic utilitarianism], this is not so. On these versions, theft is bad only when it makes the outcome worse. Avoiding theft is not part of our ultimate moral aim. But it might be true that it would
Didn't actually know that about HIV, good to know!
This is great!
In the intro article, I don't think I really like the comparison between pandemic prevention and counterterrorism.
A couple reasons:
First, counterterrorism might be construed to include counter bio terrorism. In which case, it's not obvious to me that pandemic prevention and counterterrorism are even exclusive.
Second, both pandemics and counterterrorism are heavy-tailed and dominated by tail events. Tail events don't happen...until they do. To give an example, here is the same graph but for 2009-2019:
Essentially no deaths from COVID-19! ... (read more)
Hey, just a quick comment to say something like this line of objection is discussed in footnote 3.
I'm going to propose the following further edits:
Thank you for writing this. A lot of this feels true for me.
A quick thought: some of what you wrote can also be generalized to "working really hard, all the time, on one thing." A lot of EA community builders do this. So do a lot of student entrepreneurs, researchers, performing artists, debators, and athletes, and I think they can run into many of the same challenges. I also think some of the solutions you outlined are common for some of these communities (e.g. athletic teams often feel like friend groups). Maybe there are lessons that can be learned from people who fall into this more general category?
I would add, "people with a technical background who also have strong writing skills." Maybe this is subsumed by communicators but I wanted to flag it specifically.
A lot of the best researchers either don't like to write, are slow at writing well, or simply aren't very good at writing well. But there is much that needs to be written. For this reason I've found recently that writing appears to be one of my comparative advantages.
You do need to be somewhat technical to understand the content you're writing about, but you don't have to be a top of the line researcher.
Agreed! I wrote a post about exactly this. Julia Wise also has a good one on similar topics.
For me, a big change happened when I had been around in EA long enough, done enough things, and spoken to enough people to be able to say, "if I say something disagreeable to somebody and it turns out they are one of those people who will judge me personally for disagreeing with the dominant paradigm on x thing, it's their loss, not mine." I also feel I can say something disagreeable to people and they will tend to hear me out rather than ignore me as a newbie who doesn't know anything (in fairness, when I was just starting, I actually didn't know much at ... (read more)
Love the analogy of "f**k you money" to "I respectfully disagree with your worldview social capital" or "I respectfully disagree with your worldview concrete achievements that you cannot ignore"!
I wasn't aware of that. In that case, I'll delete this post.
This is similar to something I've thought about recently, which is that one option for a highly impactful person looks basically like having their head down and studying for many years, getting into a conventional position, and using the skills they've acquired and the leverage in that position for good. I think this is underemphasized and I wonder if that is just because it seems less exciting and different.
Anecdotally I've observed some people taking long leaves from college/talking about dropping out (edit: I took a leave from college and it was very be... (read more)
Thanks for writing this post!
I do think there are some cases where there isn't a clear line between what you call "marketing" and "skilling up."
If I do the "menial operations work" of figuring out how to easily get people to go to an EA conference, is that "marketing" or "skilling up"? It depends; if my goal is to do technical research only, then it probably isn't a useful skill, but operations is a very useful skill that you can build while doing EA community building.
If I know a group organizer has done the gruntwork of operations, I know that they can h... (read more)
I wasn't intending to single out you or any specific person when asking that question. More that the community overall seems to collectively have responded differently (in view of up/downvotes). Due to the fact that different people see different posts, it's hardly a controlled experiment, so it could have been just chance who happened to see the post first and make a first impression.
I notice a similarity to this post.
Somebody writes about an issue that happens to be a popular mainstream cause and asks, "how can I be most effective at doing good, given that I want to work specifically on this cause?"
I'm not saying the two issues are remotely equivalent. Obviously, to argue "this should be an EA cause area" would require very different arguments, and one might be much stronger than the other. With Ukraine, maybe you could justify it as being adjacent to nuclear risk, but the post wasn't talking about nuclear risk. Maybe close to being a... (read more)
I think it's essential to ask some questions first:
Only then should one even consider engaging in outreach or efforts to improve optics.
Wanted to make a very small comment on a very small part of this post.
An assistant professor in AI wants to have several PhDs funded. Hearing about the abundance of funding for AI safety research, he drafts a grant proposal arguing why the research topic his group would be working on anyway helps not only with AI capabilities, but also with AI alignment. In the process he convinces himself this is the case, and as a next step convinces some of his students.
Yes, this certainly might be an issue! This particular issue can be mitigated by having funders do lo... (read more)
Curious why people are downvoting this? If it's some substantive criticism of the work I'd be interested in hearing it.
If it's just because it's not very thought through, then what do you think the "not front page" function of the forum is for? (This might sound accusatory but I mean it genuinely).
One of the reasons I posted was because I wanted to hear thoughts/criticisms of the work overall, since I felt I didn't have a good context. Or maybe to find somebody who knew it better. But downvotes don't help with this.
This reminds me of Adorno and Horkheimer'sThe Dialectic of Enlightenment, which argues, for some of the same reasons you do, that "Enlightenment is totalitarian." A piece that feels particularly related:
For the Enlightenment, whatever does not conform to the rule of computation and utility is suspect.
They would probably say "alienation" rather than "externalization," but have some of the same criticisms.
(I don't endorse the Frankfurt School or critical theory. I just wanted to note the similarities.)
One thing to consider is moral and epistemic uncertainty.... (read more)
This document will include all of that information (some of it isn't ready yet).
This is a good point which I don't think I considered enough. This post describes this somewhat.
I do think the signal for which actions are best to take has to come from somewhere. You seem to be suggesting the signal can't come from the decisionmaker at all since people make decisions before thinking about them. I think that's possible, but I still think there's at least some component of people thinking clearly about their decision, even if what they're actually doing is trying to emulate what those around them would think.
We do want to generate actual s... (read more)
Yes, people will always have motivated reasoning, for essentially every explanation of their actions they give. That being said, I expect it to be weaker for the small set of things people actually think about deeply, rather than things they're asked to explain after the fact that they didn't think about at all. Though I could be wrong about this expectation.
If you spend a lot of time in deep thought trying to reconcile "I did X, and I want to do Y" with the implicit assumption "I am a virtuous and pure-hearted person", then you're going to end up getting way better at generating prosocial excuses via motivated reasoning.
If, instead, you're willing to consider less-virtuous hypotheses, you might get a better model of your own actions. Such a hypothesis would be "I did X in order to impress my friends, and I chose career path Y in order to make my internal model of my parents proud".
Realizing such uncomfort... (read more)
EA groups often get criticized by university students for "not doing anything." The answer usually given (which I think is mostly correct!) is that the vast majority of your impact will come from your career, and university is about gaining the skills you need to be able to do that. I usually say that EA will help you make an impact throughout your life, including after you leave college; the actions people usually think of as "doing things" in college (like volunteering), though they may be admirable, don't.
Which is why I find it strange that the po... (read more)
This is the third time I've seen a suggestion like this, and antitrust law is always brought up. I feel like maybe it's worth a post that just says "no, you can't coordinate salaries/hiring practices/etc., here's why" since that would be helpful for the general EA population to know.
Aaron didn't link it, so if people aren't aware, we are running that competition (judging in progress).
I think I disagree with this.
To me, short timelines would mean the crunch in movement building was in the past.
It's also really not obvious when exactly "crunch time" would be. 10 years before AGI? 30 years?
If AGI is in five years I expect movement building among undergrads to not matter at all. If it's in ten years maybe you could say "movement building has almost run its course" but I still think "crunch time" would probably still be in the past.
Edit: I'm referring to undergrad movement building here. Talking to tech executives, policymakers, existing ML researchers etc. would have a different timeline.
No, we're still working on it! All decisions will be sent by tomorrow, June 7th, as indicated in this post.
Total time including assignments. Don't worry, there will not be 30-40 hours of lecture videos every week!
The terminology around AI (AI, ML, DL, RL) is a bit confused sometimes. You're correct that deep reinforcement learning does indeed use deep neural nets, so it could be considered a part of deep learning. However, colloquially deep learning is often taken to mean the parts that aren't RL (so supervised, unsupervised, and self-supervised deep learning). RL is pretty qualitatively different from those in the way it is trained, so it makes sense that there would be a different term, but it can create confusion.
That shouldn't be a problem. For synchronous activities, we will have multiple sessions you can attend (we will have people from all over the world so we need to do this anyway).
Sorry, missed replying to this comment as we were working on this doc, this is indeed the resource we recommend!
It reminded me a bit of Charity Navigator shooting themselves in the foot with the phrase "defective altruism".
It's not that their claims have zero truth, but they are over the top and it harms whatever argument they did have.
The title of this post (and a link to it) was quoted here as supporting the claim that EA is mostly just longtermism.
You can certainly add it to your resume, but you wouldn't be able to get a reference letter.
The program uses public recorded online classes, and while we have TAs, none of them are professors.
Not clear right now whether we will need more TAs, but if we do, we'll make a post soon with an application. I'll reply to this if/when that happens. Thanks for your interest!
This will depend on the number of TAs we can recruit, our yield rate, and other variables so I can't give a good figure on this right now, sorry.
A confirmation email is not expected. We received your application!
Yes, but please note this on your application. In general, short periods of unavailability are fine, but we won't give any extensions for them so you will likely have to complete the material at an accelerated pace at the times when you are available.
Yes, it's possible that would be better (though I can see pros and cons to both approaches). I just saw a need and wanted to fill it, and the people I talked to about this idea beforehand seemed generally happy about it (none suggested this idea which I agree could work!).
That being said, I'm not attached to it. If you think this would be better and people on the slack seem to agree then I wouldn't be opposed to shutting down the slack.
Yes, this is what I had in mind.
The idea is for there to be a channel for each location in the slack (e.g. Oxford, Berkeley, etc.). I think that would be unwieldy as part of another slack.
Spitballing, I think it could be fine to have 1 channel with threads for each location pinned to the top where people could comment, or have 1 main channel that links to sub channels or something like this.
We may consider people in this situation, but it's not the focus of our program and we will prioritize undergraduates.
I think it's easier than it might seem to do something net negative even ignoring opportunity cost. For example, actively compete with some other better project, interfere with politics or policy incorrectly, create a negative culture shift in the overall ecosystem, etc.
Besides, I don't think the attitude that our primary problem is spending down the money is prudent. This is putting the cart before the horse, and as Habryka said might lead to people asking "how can I spend money quick?" rather than "how can I ambitiously do good?" EA certainly has a lot of money, but I think people underestimate how fast $50 billion can disappear if it's mismanaged (see, for an extreme example, Enron).
I thought this comment was valuable and it's also a concern I have.
It makes me wonder if some of the "original EA norms", like donating a substantial proportion of income or becoming vegan, might still be quite important to build trust, even as they seem less important in the grand scheme of things (mostly, the increase in the proportion of people believing in longtermism). This post makes a case for signalling.
It also seems to increase the importance of vetting people in somewhat creative ways. For instance, did they demonstrate altruistic things be... (read more)
We'll consider this if there's enough demand for it! But especially for the latter option, it might make sense for students to work through the last three weeks on their own (ML Safety lectures will be public by then).