3194Joined Sep 2020



Center for AI Safety and Yale EA organizer


Pragmatic AI Safety


I've never felt comfortable in EA broadly construed, not since I encountered it about three years ago. And yet I continue to be involved to a certain extent. Why? Because I think that doing so is useful for doing good, and many of the issues that EA focuses on are sadly still far too neglected elsewhere. Many of the people who come closest to sharing my values are in EA, so even if I didn't want to be "in EA," it would be pretty difficult to remove myself entirely.

I also love my university EA group, which is (intentionally, in part by my design, in part by the design of others) different from many other groups I've encountered.

I work in AI safety, and so the benefit of staying plugged into EA for me is probably higher than it would be for somebody who wants to work in global health and development.  But I could still be making a (potentially massive) miscalculation.

If you think that EA is not serving your aims of doing good (the whole point of EA), then remember to look out the window. And even if you run an "EA" group, you don't need to feel tied to the brand. Do what you think will actually be good for the world. Best of luck.

"Living expenses while doing some of my early research" is one of the main purposes of the LTFF; to me Atlas feels like a roundabout way of getting that. LTFF asks you to have a specific high-impact project or educational opportunity for you to pursue, but as far as I know that wasn't true of Atlas.

I think The Century Fellowship would make a better comparison to the Thiel Fellowship than Atlas would. It seems aimed at similar types to the Thiel Fellowship (college age people who are prepared to start projects and need to be financially independent to do so), while Atlas targets a slightly younger demographic and gives scholarships.

Atlas is posed as a talent search and development program, so I think any evaluation of Atlas should focus on how well it is searching for and developing talent that would not otherwise exist. I personally don't know anything about how that has been turning out, or what the graduates have done/are doing with the money, so I don't feel very qualified to evaluate it myself.

In the past two years, the technical alignment organisations which have received substantial funding include

Your post does not actually say this, but when I read it I thought you were saying that these are all the organizations that have received major funding in technical alignment. I think it would have been clearer if you had said "include the following organizations based in the San Francisco Bay Area:" to make it clearer you're discussing a subset.

Anyway, here are the public numbers, for those curious, of $1 million+ grants in technical AI safety in 2021 and 2022 (ordered by total size) made by Open Philanthropy:

  • Redwood Research: $9.4 million, and then another grant for $10.7 million
  • Many professors at a lot of universities: $14.4 million
  • CHAI: $11.3 million
  • Aleksander Madry at MIT: $1.4 million
  • Hofvarpnir Studios: $1.4 million
  • Berkeley Existential Risk Initiative - CHAI collaboration: $1.1 million
  • Berkeley Existential Risk Initiative - SERI MATS Program: $1 million

The Alignment Research Center received much less: $265,000.

There isn't actually any public grant saying that Open Phil funded Anthropic. However, that isn't to say that they couldn't have made a non-public grant. It was public that FTX funded Anthropic.

having strong or intimate connections with employees of Open Philanthropy greatly enhances the chances of having funding, and it seems almost necessary

Based on spending some time in Berkeley, I think a more accurate way to describe this is as follows:

People who care about AI safety and are involved in EA tend to move to Berkeley because that is where everyone else is. It really can increase your productivity if you can easily interact with others working in your field and know what is going on, or so the established wisdom goes. The people who have been around the longest are often leading research organizations or are grantmakers at Open Phil. They go to the same parties, have the same friends, work in the same offices, and often spend nearly all of their time working with little time to socialize with anyone outside their community. Unless they make a special effort to avoid dating anyone in their social community, they may end up dating a grantmaker.

If we want these conflicts of interest to go away, we could try simply saying it should be a norm for Open Phil not to grant to organizations with possible conflicts of interest. But knowing the Berkeley social scene, this means that many Open Phil grantmakers wouldn't be able to date anyone in their social circles, since basically everyone in their social circles is receiving money from Open Phil.

The real question is as you say one of structure: whether so many of the EA-aligned AI safety organizations should be headquartered in close proximity and whether EAs should live together and be friends with basically only other EAs. That's the dynamic that created the conflicts. I don't think the answer to this is extremely obvious, but I don't really feel like trying to argue both sides of it right now.

It's possibly true that regrantors would reduce this effect in grantmaking, because you could designate regrantors in other places or who have different friends. But my suspicion would be that regrantors would by default be the same people who are already receiving grants.

GiveDirectly has a program for the US that you can donate to. I don't really know how good it is, but the organization in general seems excellent.

For what it's worth, I don't think this needed a retraction. It's true the original post was pretty overconfident about things. Instead of asserting something and defending it, it would probably have been better to assert it with the explicit aim of hearing criticism from people on here. That's what happened anyway, but your framing was more "here is a thing I think is good."

That would certainly be great if she would. I actually first heard about EA when I read Strangers Drowning in 2016! It's very well written.

I'm a bit confused about crossposting, are you saying it was always available? I don't remember seeing any crossposts a year ago, or being able to use the feature. In fact I used to crosspost a lot of things and specifically remember the first time I saw the crossposting feature. But maybe I just didn't notice this before.

Didn't know that about the dev teams, that's useful to know!

Answer by ThomasWJan 24, 20232014

I'm in favor of a clear separation between the forums. They are made for different audiences and not everything that is meant for one is meant for the other. As somebody who writes some pieces that are meant for both audiences, the cross posting feature is somewhat convenient for me (but not hugely so; I can just copy and paste). And as a reader, sometimes it's nice to see a post is cross posted so that I can go see the comments on the other forum.

I'd be interested to see how much the easy cross posting has increased the number of cross posts, and if so what kinds of posts are now more likely to be cross posted. This seems like an analysis the forum team could do and is harder to do anecdotally.

The EA Forum and LessWrong have some of the best technical infrastructure on the internet, and I think the EA Forum derives huge benefit from that. However, it does make me a little uneasy that it's made by the Lightcone team, who are in charge of LessWrong. I like the people on that team, but I expect probably some decisions that are good for LessWrong but not so good for the EA Forum might just end up propagating here by default. This is just a suspicion; I don't have any particular examples.

A lot of separation does exist. LessWrong posts are moderated pretty differently, the commenter audiences are often very different, and the types of posts are mostly different. So the connection as is isn't currently a huge concern of mine.

I still believe that there were significant problems with a section of the original statement from Max Tegmark, and they have been reinforced, not undermined, by this FAQ. To be clear, I am not referring to problems like "they left out detail x"; I am referring to the fact that a particular section was actively misleading. I understand FLI was under a lot of pressure to churn out a statement fast, so I'm not totally surprised the original statement wasn't good quality. Still, I think FLI has a responsibility not to make misleading statements that they know, or should know, are misleading.

In this FAQ, FLI states the following as a main reason they rejected the grant:

we found the term “ethnopluralism" endorsed in Nya Dagbladet.

However, in their initial statement, they wrote:

We also point out that the claim by that NDF is "pro-Nazi" [the lede in the article] is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021. This is exactly $30,000 more than the zero dollars FLI granted to them.

The invocation of Swedish government funding was never appropriate in the first place, as I wrote at the time in my only previous comment on this situation (and also there are some good replies). This is perhaps an understandable mistake for somebody might not know about the Swedish press support system or how it supports papers with essentially all political leanings. However, given the fact that FLI now states that they already knew that the Swedish government was supporting a newspaper that favors ethnopluralism (a view that wikipedia says has been linked to neo-fascist groups), surely they could not have thought that this funding was any kind of endorsement from a "center-left" government. As a result, this part of the statement appears even more misleading to me than it did when I originally pointed it out.

is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021

Disclaimer: I previously knew nothing about the Swedish press; I still know almost nothing. I just thought this seemed weird and spent about 20 minutes looking into it.

Some context which I think would be useful to evaluate this claim.

It appears that in Sweden the government subsidizes newspapers in the form of "press support." From reading the Wikipedia page on press support, which is mostly actually about Norway not Sweden, it seems like support does not really constitute a government endorsement, but rather is provided to a lot of different newspapers and is mostly to ensure a healthy press. It's possible this differs between Norway and Sweden though.

The $30,000 figure comes from the piece, which says:

Nya Dagbladet applied to the Swedish Press and Broadcasting Authority for public funding in 2020, but was turned down. The platform reacted angrily to the decision, and published a series of articles where specific officials at the agency were named and pictured. The publications caused distress among employees at the agency who felt menaced and pressured, as Dagens Nyheter reported at the time. The following year, Nya Dagbladet made another application for public funding; this time they were successful and received about $30,000 in various grants.

The article linked (archive here) Google translates the article as referring to the Norwegian press. I thought that was pretty weird, but from googling the Swedish (Myndigheten för press, radio och tv), I think the Google translate is wrong and it is indeed about the Swedish press (here is the website of the Swedish press agency). The piece might seem to imply that the government officials may have been intimidated into making the later grant, but I think that's a bit less clear if press support is supposed to be widely distributed to newspapers in any case.

Regardless, to me it does not seem like the reception of this grant really indicates that the organization is not pro-Nazi, and certainly it doesn't seem to imply endorsement of that claim from the Swedish government, at least as far as I can tell. A good understanding would require a better understanding of the Swedish press support system, which I neither I or presumably the vast majority of readers of this comment have.

Load More