All of ThomasWoodside's Comments + Replies

How/When Should One Introduce AI Risk Arguments to People Unfamiliar With the Idea?

I think Is Power-Seeking AI An Existential Risk is probably the best introduction, though it's probably too long as a first introduction if the person is yet that motivated. It's also written as a list of propositions, with probabilities, and that might not appeal to many people.

I also listed some shorter examples in this post for the AI Safety Public Materials Bounty we're running, that might be more suitable as a first introduction. Here are the ones most relevant to people not versed in machine learning:

... (read more)
[AMA] Announcing Open Phil’s University Group Organizer and Century Fellowships

Thanks for all the work you are doing here, I think some really amazing groups could come out of this. I am cautiously excited about many different kinds of groups starting.

I found it a bit surprising that the list of criteria for group organizers (including "nice to have") doesn't seem to have anything like "really cares about the objectives of their group," "really cares about improving the long term future," "is altruistic to some degree"

  1. Being truth-seeking and open-minded
  2. Having a strong understanding of whatever topic their group is about, and/or being
... (read more)
$20K in Bounties for AI Safety Public Materials

We don't expect the work to be published anywhere when it's submitted.

For certain pieces, we may work with authors to publish them somewhere, publish them on our website, or adapt them and publish an adapted version somewhere. But this is not guaranteed.

In general, we expect that the best pieces will be generally suited for an audience of either smart people who don't know about ML, or ML researchers. Though there is a lot of room for pieces that are more optimized for particular audiences and venues, we think that more general pieces would serve as great inspiration for those later pieces.

$20K in Bounties for AI Safety Public Materials

I edited the title to say "$20k in bounties" to make it more clear.

From the original text:

Winners of the bounty will win $2,000 each, for a total of up to ten possible bounty recipients.

This doesn't mean each person who submits an entry gets $2,000. We will award this to entries that meet a high bar for quality (roughly, material that we would actually be interested in using for outreach).

Thanks for the clarification! I might try to do something on the Orthogonality thesis if I get the chance, since I think that tends to be glossed over in a lot of popular introductions.
Some updates in EA communications

I missed that part of footnote 3, it does seem to address a lot of what I said. I appreciate your response.

I do think the vast majority of people will not read footnote 3, so it's important for the main body of the text (and the visuals) to give the right impression. This means comparing averages to averages, or possible tail events to possible tail events. It sounds like this is your plan now, and if so that's great!

No worries, appreciate ppl checking :)
[Linkpost] Criticism of Criticism of Criticism, ACX

Posted too soon, didn't realize he had changed his mind about crossposting, please ignore.

[Linkpost] Criticism of Criticism of Criticism, ACX

I linkposted this when it came out, and Devin Kalish sent this comment:

A quick note, this piece was already posted to the forum briefly, and then deleted. The author said in a comment that he would rather it not be crossposted to this forum:
I don't know if the two are related, but I might reach out to ask him if he's alright with you posting it.

Which led to me taking down my post, since I don't really like to crosspost things if people prefer that I not.

Just wanted to let you know!

[This comment is no longer endorsed by its author]Reply

Yes, I'm sorry, I talked to Claire about it and updated, sorry for the mixed messages and any stress this caused.

Posted too soon, didn't realize he had changed his mind about crossposting, please ignore.
As noted in the post, I got Scott's permission before posting this.
Consequentialists (in society) should self-modify to have side constraints

This point is covered quite well by Derek Parfit in his seminal book Reasons and Persons, Chapter 1, Part 17. In my view the entire chapter is excellent and worth reading, but here is an excerpt from Part 17:

Consider, for example, theft. On some versions of C [Consequentialism], it is intrinsically bad if property is stolen. On other versions of C [such as hedonistic utilitarianism], this is not so. On these versions, theft is bad only when it makes the outcome worse. Avoiding theft is not part of our ultimate moral aim. But it might be true that it would

... (read more)
Yep, this is one of several reasons why I think [] that Part I is perhaps the best and certainly the most underrated part of the book. :)
2Jack R6d
This is good to know - thank you for making this connection!
Some updates in EA communications

Didn't actually know that about HIV, good to know!

Some updates in EA communications

This is great! 

In the intro article, I don't think I really like the comparison between pandemic prevention and counterterrorism.

A couple reasons:

First, counterterrorism might be construed to include counter bio terrorism. In which case, it's not obvious to me that pandemic prevention and counterterrorism are even exclusive.

Second, both pandemics and counterterrorism are heavy-tailed and dominated by tail events. Tail events don't happen...until they do. To give an example, here is the same graph but for 2009-2019:

Essentially no deaths from COVID-19! ... (read more)

Hey, just a quick comment to say something like this line of objection is discussed in footnote 3.

I'm going to propose the following further edits:

  • Compare with terrorism deaths over 50 years from 1970.
  • Mention HIV/AIDS in the main text and some other tweaks.
  • Add further discussion in the footnote.
Good post, though I should point out that HIV entered the human population independently at least twice (HIV-1 and HIV-2), so your counterfactual world missing HIV might not be as likely as one might otherwise think. (There are also counterfactual worlds where SARS-CoV-1 or MERS-CoV or similar took off as well with an even higher death count.)
Agree with this completely. The fact that this same statistical manoeuvre could be used to downplay nuclear war, vaccines for diseases like polio, climate change or AI risk, should also be particularly worrying. Another angle is that the number of deaths is directly influenced by the amount of funding- the article says that "the scale of this issue differs greatly from pandemics", but it could plausibly be the case that terrorism isn't an inherently less significant/ deadly issue, but counterterrorism funding works extremely well- that's why deaths are so low.
Challenges for EA Student Group Organizers

Thank you for writing this. A lot of this feels true for me.

A quick thought: some of what you wrote can also be generalized to "working really hard, all the time, on one thing." A lot of EA community builders do this. So do a lot of student entrepreneurs, researchers, performing artists, debators, and athletes, and I think they can run into many of the same challenges. I also think some of the solutions you outlined are common for some of these communities (e.g. athletic teams often feel like friend groups). Maybe there are lessons that can be learned from people who fall into this more general category?

AGI Safety Needs People With All Skillsets!

I would add, "people with a technical background who also have strong writing skills." Maybe this is subsumed by communicators but I wanted to flag it specifically.

A lot of the best researchers either don't like to write, are slow at writing well, or simply aren't very good at writing well. But there is much that needs to be written. For this reason I've found recently that writing appears to be one of my comparative advantages.

You do need to be somewhat technical to understand the content you're writing about, but you don't have to be a top of the line researcher.

4Amber Dawn15d
Related question: I have strong writing skills, but no technical background. If I wanted to learn more technical stuff so I could help technical researchers write things, what should I learn? Where should I start? (bearing in mind that I'd be starting at the 101 level for most things).
2Severin T. Seehrich15d
Strongly agree! I actually drafted this post during a conversation of two alignment field builders when one said, "Somebody should write a forum post about this!"
Leaning into EA Disillusionment

Agreed! I wrote a post about exactly this. Julia Wise also has a good one on similar topics.

Leaning into EA Disillusionment

For me, a big change happened when I had been around in EA long enough, done enough things, and spoken to enough people to be able to say, "if I say something disagreeable to somebody and it turns out they are one of those people who will judge me personally for disagreeing with the dominant paradigm on x thing, it's their loss, not mine." I also feel I can say something disagreeable to people and they will tend to hear me out rather than ignore me as a newbie who doesn't know anything (in fairness, when I was just starting, I actually didn't know much at ... (read more)

Love the analogy of "f**k you money" to "I respectfully disagree with your worldview social capital" or "I respectfully disagree with your worldview concrete achievements that you cannot ignore"!

I wasn't aware of that. In that case, I'll delete this post.

Community Builders Spend Too Much Time Community Building

This is similar to something I've thought about recently, which is that one option for a highly impactful person looks basically like having their head down and studying for many years, getting into a conventional position, and using the skills they've acquired and the leverage in that position for good. I think this is underemphasized and I wonder if that is just because it seems less exciting and different.

Anecdotally I've observed some people taking long leaves from college/talking about dropping out (edit: I took a leave from college and it was very be... (read more)

Community Builders Spend Too Much Time Community Building

Thanks for writing this post!

I do think there are some cases where there isn't a clear line between what you call "marketing" and "skilling up."

If I do the "menial operations work" of figuring out how to easily get people to go to an EA conference, is that "marketing" or "skilling up"? It depends; if my goal is to do technical research only, then it probably isn't a useful skill, but operations is a very useful skill that you can build while doing EA community building.

If I know a group organizer has done the gruntwork of operations, I know that they can h... (read more)

Thanks, Thomas! I think that's a fair criticism. What I was trying to get at with "some actions that fall within the category of skill building would be extremely useful for some and trivial for others" is that to a certain extent, it's up to the individual to define what skill building is for them. I'm amenable to the fact that maybe these definitions weren't lenient enough to a range of career paths. I do worry, however, that too many individuals are going into ops/community building without building up other aptitudes. Referring to your last paragraph, I think it'd be better if a group organizer could get those same skills in something a bit more catered to a prospective career path, although maybe this is unrealistic. This also doesn't hold if the individual's career interest is in ops.
What actions most effective if you care about reproductive rights in America?

I wasn't intending to single out you or any specific person when asking that question. More that the community overall seems to collectively have responded differently (in view of up/downvotes). Due to the fact that different people see different posts, it's hardly a controlled experiment, so it could have been just chance who happened to see the post first and make a first impression.

What actions most effective if you care about reproductive rights in America?

I notice a similarity to this post.

Somebody writes about an issue that happens to be a popular mainstream cause and asks, "how can I be most effective at doing good, given that I want to work specifically on this cause?"

I'm not saying the two issues are remotely equivalent. Obviously, to argue "this should be an EA cause area" would require very different arguments, and one might be much stronger than the other. With Ukraine, maybe you could justify it as being adjacent to nuclear risk, but the post wasn't talking about nuclear risk. Maybe close to being a... (read more)

I agree it should apply to both; if your question is why didn't I object to the previous post I don't have any specific defense other than having no recollection of seeing the Ukraine post at the time, though maybe I saw it and forgot.
20 Critiques of AI Safety That I Found on Twitter

I think it's essential to ask some questions first:

  • Why do people hold these views? (Is it just their personality, or did somebody in this community do something wrong?)
  • Is there any truth to these views? (As can be seen here, anti-AI safety views are quite varied. For example, many are attacks on the communities that care about them rather than the object-level issues.)
  • Does it even matter what these particular people think? (If not, then leave them be.)

Only then should one even consider engaging in outreach or efforts to improve optics.

Ways money can make things worse

Wanted to make a very small comment on a very small part of this post.

An assistant professor in AI wants to have several PhDs funded. Hearing about the abundance of funding for AI safety research, he drafts a grant proposal arguing why the research topic his group would be working on anyway helps not only with AI capabilities, but also with AI alignment. In the process he convinces himself this is the case, and as a next step convinces some of his students.

Yes, this certainly might be an issue! This particular issue can be mitigated by having funders do lo... (read more)

Dialectic of Enlightenment

Curious why people are downvoting this? If it's some substantive criticism of the work I'd be interested in hearing it.

If it's just because it's not very thought through, then what do you think the "not front page" function of the forum is for? (This might sound accusatory but I mean it genuinely).

One of the reasons I posted was because I wanted to hear thoughts/criticisms of the work overall, since I felt I didn't have a good context. Or maybe to find somebody who knew it better. But downvotes don't help with this.

The totalitarian implications of Effective Altruism

This reminds me of Adorno and Horkheimer'sThe Dialectic of Enlightenment, which argues, for some of the same reasons you do, that "Enlightenment is totalitarian." A piece that feels particularly related:

For the Enlightenment, whatever does not conform to the rule of computation and utility is suspect.

They would probably say "alienation" rather than "externalization," but have some of the same criticisms.

(I don't endorse the Frankfurt School or critical theory. I just wanted to note the similarities.)

One thing to consider is moral and epistemic uncertainty.... (read more)

This is a really interesting parallel - thank you! It ties neatly into one of my major concerns with my piece -whether it can be interpreted as anti-rationality / a critique of empiricism (which is not the intention). My reflexive reaction to the claim that "enlightenment is totalitarian" is fairly heavy scepticism (whereas, obviously, I lean in the opposite direction as regards to EA), so I'm curious what distinctions there are between the arguments made in Dialectic and the arguments made in my piece. I will have a read of Dialectic and think through this further.
Introducing the ML Safety Scholars Program

This document will include all of that information (some of it isn't ready yet).

You Don't Need To Justify Everything

This is a good point which I don't think I considered enough. This post describes this somewhat.

I do think the signal for which actions are best to take has to come from somewhere. You seem to be suggesting the signal can't come from the decisionmaker at all since people make decisions before thinking about them. I think that's possible, but I still think there's at least some component of people thinking clearly about their decision, even if what they're actually doing is trying to emulate what those around them would think.

We do want to generate actual s... (read more)

(Intersubjective evaluation - the combination of multiple people's subjective evaluations - could plausibly be better than one person's subjective evaluation, especially if of themselves, assuming 'errors' are somewhat uncorrelated.)
You Don't Need To Justify Everything

Yes, people will always have motivated reasoning, for essentially every explanation of their actions they give. That being said, I expect it to be weaker for the small set of things people actually think about deeply, rather than things they're asked to explain after the fact that they didn't think about at all. Though I could be wrong about this expectation.

If you spend a lot of time in deep thought trying to reconcile "I did X, and I want to do Y" with the implicit assumption "I am a virtuous and pure-hearted person", then you're going to end up getting way better at generating prosocial excuses via motivated reasoning.

If, instead, you're willing to consider less-virtuous hypotheses, you might get a better model of your own actions. Such a hypothesis would be "I did X in order to impress my friends, and I chose career path Y in order to make my internal model of my parents proud".

Realizing such uncomfort... (read more)


EA groups often get criticized by university students for "not doing anything." The answer usually given (which I think is mostly correct!) is that the vast majority of your impact will come from your career, and university is about gaining the skills you need to be able to do that. I usually say that EA will help you make an impact throughout your life, including after you leave college; the actions people usually think of as "doing things" in college (like volunteering), though they may be admirable,  don't.

Which is why I find it strange that the po... (read more)

Jobs at EA-organizations are overpaid, here is why

This is the third time I've seen a suggestion like this, and antitrust law is always brought up. I feel like maybe it's worth a post that just says "no, you can't coordinate salaries/hiring practices/etc., here's why" since that would be helpful for the general EA population to know.

2Charles He2mo
Probably most of these brief statements about "coordinating salaries equals violating antitrust", are simplistic and may substantively be untrue or misleading. I'm LARPING wildly but the below gives a sense why: There might be several legal theories or law involved. A not entirely crazy guess is that one legal theory is that anti-trust is set to prevent for profit businesses from creating a cartel at their benefit. So these are completely separate entities, making agreements to maintain advantage in a marketplace at the expense of workers. It's totally possible this can be undermined or end-runned, and "anti-trust" doesn't apply if: * These entities are funded by very close and coordinated sources of funds * In the same way it makes no sense to have different departments of an entity claim "antitrust", defendants can argue that there is no material distinction in practice between many entities. * The "cartel" actually has to create economic harm, and this might require a natural marketplace for employees that is being undermined. If the funders and all the execs created the entire marketplace (e.g. think of many EA cause areas), that's harder to say they are unfairly benefitting. * The EA community or funders gathered together and demanded lower/higher salaries or other norms. * The entities can then act and say they are acting in their own existential interest, at the demand of donors and the community. * This doesn't apply if the coordination generally increased salaries. Law isn't this black and white thing where police drop out of helicopters. At least half of this thing, is someone actually needing to sue and show harm. That's a long and difficult process. I can see how that might not happen even if there was malign anti-trust actually going on. By the way, as suggested by preceding thought that reality sort of sucks, I probably oppose and think it's bad to set up hard salary norms from
Is the time crunch for AI Safety Movement Building now?

Aaron didn't link it, so if people aren't aware,  we are running that competition (judging in progress).

Is the time crunch for AI Safety Movement Building now?

I think I disagree with this.

To me, short timelines would mean the crunch in movement building was in the past.

It's also really not obvious when exactly "crunch time" would be. 10 years before AGI? 30 years?

If AGI is in five years I expect movement building among undergrads to not matter at all. If it's in ten years maybe you could say "movement building has almost run its course" but I still think "crunch time" would probably still be in the past.

Edit: I'm referring to undergrad movement building here. Talking to tech executives, policymakers, existing ML researchers etc. would have a different timeline.

The edit is key here. I would consider running an AI-safety arguments competition in order to do better outreach to graduate-and-above level researchers to be a form of movement building and one for which crunch time could be in the last 5 years before AGI (although probably earlier is better for norm changes). One value add from compiling good arguments is that if there is a period of panic following advanced capabilities (some form of firealarm), then it will be really helpful to have existing and high quality arguments and resources on hand to help direct this panic into positive actions. This all said, I don't think Chris's advice applies here: I think this advice likely doesn't apply because the models/strategies for this sort of AI Safety field building are very different from that of general EA community building (e.g., University groups), the background knowledge is quite different, the target population is different, the end goal is different, etc. If you are a community builder reading this and you want to transition to AI Safety community building but don't know much about it, probably learning about AI Safety for >20 hours is the best thing you can do. The AGISF curriculums [] are pretty great.
Introducing the ML Safety Scholars Program

No, we're still working on it! All decisions will be sent by tomorrow, June 7th, as indicated in this post.

1Jieru Liao2mo
Hello! I have looked through this website for thousand times, but I still regret to find that I do not seem to be accepted into this project, but I still want to learn about the courses. In addition to the links that have been posted for most courses of the project, I would like to ask whether you will establish an open community to share homework content during the project, and the final project and ML Safety course in the third week, I would like to learn these, thank you!
Introducing the ML Safety Scholars Program

Total time including assignments. Don't worry, there will not be 30-40 hours of lecture videos every week!

Complex Systems for AI Safety [Pragmatic AI Safety #3]

The terminology around AI (AI, ML, DL, RL) is a bit confused sometimes. You're correct that deep reinforcement learning does indeed use deep neural nets, so it could be considered a part of deep learning. However, colloquially deep learning is often taken to mean the parts that aren't RL (so supervised, unsupervised, and self-supervised deep learning). RL is pretty qualitatively different from those in the way it is trained, so it makes sense that there would be a different term, but it can create confusion.

Introducing the ML Safety Scholars Program

That shouldn't be a problem. For synchronous activities, we will have multiple sessions you can attend (we will have people from all over the world so we need to do this anyway).

Introducing the ML Safety Scholars Program

Sorry, missed replying to this comment as we were working on this doc, this is indeed the resource we recommend!

EA is more than longtermism

It reminded me a bit of Charity Navigator shooting themselves in the foot with the phrase "defective altruism".


It's not that their claims have zero truth, but they are over the top and it harms whatever argument they did have.

EA is more than longtermism

The title of this post (and a link to it) was quoted here as supporting the claim that EA is mostly just longtermism.

Giving money away as ineffectively as possible to own the nerds, got it 😒
Introducing the ML Safety Scholars Program

You can certainly add it to your resume, but you wouldn't be able to get a reference letter.

The program uses public recorded online classes, and while we have TAs, none of them are professors.

Introducing the ML Safety Scholars Program

Not clear right now whether we will need more TAs, but if we do, we'll make a post soon with an application. I'll reply to this if/when that happens. Thanks for your interest!

Introducing the ML Safety Scholars Program

This will depend on the number of TAs we can recruit, our yield rate, and other variables so I can't give a good figure on this right now, sorry.

Introducing the ML Safety Scholars Program

A confirmation email is not expected. We received your application!

Introducing the ML Safety Scholars Program

Yes, but please note this on your application. In general, short periods of unavailability are fine, but we won't give any extensions for them so you will likely have to complete the material at an accelerated pace at the times when you are available.

EA Housing Slack

Yes, it's possible that would be better (though I can see pros and cons to both approaches). I just saw a need and wanted to fill it, and the people I talked to about this idea beforehand seemed generally happy about it (none suggested this idea which I agree could work!).


That being said, I'm not attached to it. If you think this would be better and people on the slack seem to agree then I wouldn't be opposed to shutting down the slack.

EA Housing Slack

Yes, this is what I had in mind.

EA Housing Slack

The idea is for there to be a channel for each location in the slack (e.g. Oxford, Berkeley, etc.). I think that would be unwieldy as part of another slack.

That makes sense I think. But if it proves too much to maintain a whole slack maybe a channel in another slack with threads for each city would be an intermediate option.

Spitballing, I think it could be fine to have 1 channel with threads for each location pinned to the top where people could comment, or have 1 main channel that links to sub channels or something like this.

Introducing the ML Safety Scholars Program

We may consider people in this situation, but it's not the focus of our program and we will prioritize undergraduates.

EA and the current funding situation

I think it's easier than it might seem to do something net negative even ignoring opportunity cost. For example, actively compete with some other better project, interfere with politics or policy incorrectly, create a negative culture shift in the overall ecosystem, etc.

Besides, I don't think the attitude that our primary problem is spending down the money is prudent. This is putting the cart before the horse, and as Habryka said might lead to people asking "how can I spend money quick?" rather than "how can I ambitiously do good?" EA certainly has a lot of money, but I think people underestimate how fast $50 billion can disappear if it's mismanaged (see, for an extreme example, Enron).

That’s a fair point, thank you for bringing that up :)
EA and the current funding situation

I thought this comment was valuable and it's also a concern I have.

It makes me wonder if some of the "original EA norms", like donating a substantial proportion of income or becoming vegan, might still be quite important to build trust, even as they seem less important in the grand scheme of things (mostly, the increase in the proportion of people believing in longtermism).  This post makes a case for signalling.

It also seems to increase the importance of vetting people in somewhat creative ways. For instance, did they demonstrate altruistic things be... (read more)

3Geoffrey Miller2mo
Thomas -- excellent reply, and good points. I've written a bit about virtue signaling, and agree that there are good forms (reliable, predictive) and bad forms (cheap talk, deceptive, misguided) of virtue signaling. I also agree that EA could be more creative and broad-minded about what kinds of virtue signaling are likely to be helpful in predictive future integrity, dedication, and constructiveness in EA. Historically, a lot of EA signaling has involved living frugally, being vegan, being a good house-mate in an EA shared house, collaborating well on EA projects, getting lots of upvotes on EA Forum, etc. Assessing those signals accurately requires a lot of first-hand or second-hand knowledge, which can be hard to do at scale, as the EA movement grows. As EA grows in scale and becomes more diverse in terms of background (e.g. recruits more established professionals from other fields, not just recent college grads), we may need to get savvier about domain-specific virtue signals, e.g. how do medical researchers vs geopolitical security experts vs defense attorneys vs bioethicists vs blockchain developers show their true colors? The very tricky trade-off, IMHO, is that often the most reliable virtue signals in terms of predicting personality traits (honesty, humility, conscientiousness, kindness) are often the least efficient in terms of actually accomplishing real-world good. For example, defense attorneys who do a lot of pro bono work doing appeals for death row inmates might be showing genuine dedication and altruism -- but this might be among the least effective uses of their time in achieving criminal justice reform. So, do we want the super-trustworthy but scope-insensitive lawyers involved in EA, or the slightly less virtue-signaling but more rational and scope-sensitive lawyers? That seems like a real dilemma. Traditionally, EA has solved it mostly by expecting a fair amount of private personality-signaling (e.g. being a conscientious vegan house-mate) pl
How bad is it to fund someone untrustworthy? Obviously if they take the money and run, that would be a total loss, but I doubt that’s a particularly common occurrence (you can only do it once, and would completely shatter social reputation, so even unethical people don’t tend to do that). A more common failure mode would seem to be apathy, where once funded not much gets done, because the person doesn’t really care about the problem. However, if something gets done instead of nothing at all, then that would probably be (a fairly weak) net positive. The reason why that’s normally negative is due to that money then not being used in a more cost-effective manner, but if our primary problem is spending enough money in the first place, that may not be much of an issue at all.
Introducing the ML Safety Scholars Program

We'll consider this if there's enough demand for it! But especially for the latter option, it might make sense for students to work through the last three weeks on their own (ML Safety lectures will be public by then).

Load More