All of Neel Nanda's Comments + Replies

Fun post! Note that you can see the diagrams here https://web.archive.org/web/20220523235545/https://markxu.com/dropping-out

The Wenar criticism in particular seems laughably bad, such that I find bad faith hypotheses like this fairly convincing. I do agree it's a seductive line of reasoning to follow in general though, and that this can be dangerous

I got the OpenPhil grant only after the other grant went through (and wasn't thinking much about OpenPhil when I applied for the other grant). I never thought to inform the other grant maker after I got the OpenPhil grant, which maybe I should have in hindsight out of courtesy?

This was covering some salary for a fixed period of research, partially retroactive, after an FTX grant fell through. So I guess I didn't have use for more than X, in some sense (I'm always happy to be paid a higher salary! But I wouldn't have worked for a longer period of time, so I would have felt a bit weird about the situation)

Without any context on this situation, I can totally imagine worlds where this is reasonable behaviour, though perhaps poorly communicated, especially if SFF didn't know they had OpenPhil funding. I personally had a grant from OpenPhil approved for X, but in the meantime had another grantmaker give me a smaller grant for y < X, and OpenPhil agreed to instead fund me for X - y, which I thought was extremely reasonable.

In theory, you can imagine OpenPhil wanting to fund their "fair share" of a project, evenly split across all other interested grantmakers.... (read more)

Omg what, this is amazing(though nested bullets not working does seem to make this notably less useful). Does it work for images?

I liked this, and am happy for this to have been a post. Maybe putting [short poem] in the title could help calibrate people on what to expect?

I'd be curious to hear your or Emma's case for why it's notably higher impact for a forum reader to donate via the campaign rather than to New Incentives directly (if they're inclined to make the donation at all)

To me this post ignores the elephant in the room: OpenPhil still has billions of dollars left and is trying to make funding decisions relative to where they think their last dollar is. I'd be pretty surprised if having the Wytham money liquid rather than illiquid (or even having £15mn out of nowhere!) really made a difference to that estimate.

It seems reasonable to argue that they're being too conservative, and should be funding the various things you mention in this post, but also plausible to me that they're acting correctly? More importantly, I think th... (read more)

I also work at Google, and a surprising amount of people (including EAs) aren't aware of the substantial annual donation match! I only noticed by happenstance.

I didn't know there were useful tools online for this, I agree this seems like a great thing for EA orgs/charities to have on their website if it's easy to do

It still seems like a mistake to not point out to people that they can substantially increase their donation and thus lives saved, even if it doesn't count towards the pledge

I think in hindsight the response (with the information I think the board had) was probably reasonable

Reasonable because you were all the same org, or reasonable even if EA Funds was its own org

Maybe it would have been cleaner if it wasn't about Ben, though I don't think a hypothetical person would have made the lesson as clear, and if Ben wasn't fair game for having written that article, I don't know who would be.

Thanks! This line in particular changed my mind about whether it was retributive, I genuinely can't think of anyone else it would be appropriate to do this for

They were shocked at his lack of concern for her suffering and confirmed that he would probably really hurt her career if she came forward with her information.

Re-reading that section, it was surprisingly consistent with that interpretation, but this line seems to make no sense if it's about Kat's experience - if the trauma is publishing the previous post then "probably really hurt her career if she came forward with her information" does not make sense because the trauma was a public event

I also think orgs generally should have donor diversity and more independence, so giving more funding to the orgs that OP funds is sometimes good.

I'd be curious to hear more about this - naively, if I'm funding an org, and then OpenPhil stops funding that org, that's a fairly strong signal to me that I should also stop funding it, knowing nothing more. (since it implies OpenPhil put in enough effort to evaluate the org, and decided to deviate from the path of least resistance)

Agreed re funding things without a track record, that seems clearly good for s... (read more)

Yeah, that intermediate world sounds great to me! (though a lot of effort, alas)

Ah, gotcha. If I understand correctly you're arguing for more of a "wisdom of the crowds" analogy? Many donors is better than a few donors.

If so, I agree with that, but think the major disanalogy is that the big donors are professionals, with more time experience and context, while small donors are not - big donors are more like hedge funds, small donors are more like retail investors in the efficient market analogy

Agreed!

I disagree, because you can't short a charity, so there's no way for overhyped charity "prices" to go down

Thanks for writing this up! One problem with this proposal that I didn't see flagged (but may have missed) is that if the ETG donors defer to the megadonors you don't actually get a diversified donor base. I earn enough to be a mid-sized donor, but I would be somewhat hesitant about funding an org that I know OpenPhil has passed up on/decided to stop funding, unless I understood the reasons why and felt comfortable disagreeing with them. This is both because of fear of unilateralist curse/downside risks, and because I broadly expect them to have spent more... (read more)

I upvoted this comment, since I think it's a correct critique of poor quality studies and adds important context, but I also wanted to flag that I also broadly think Athena is a worthwhile initiative and I'm glad it's happening! (In line with Lewis' argument below). I think it can create bad vibes for the highest voted comment on a post about promoting diversity to be critical

I'm confused by how this relates to Gemma's post?

Thanks for the post! This seems a useful summary, and I didn't spot anything that contradicted existing information I have (I didn't check very hard, so this isn't strong data)

For what it's worth, I interpreted the original post as Elizabeth calling it a pseudo RCT, and separately saying that commenters cited it, without implying commenters called it a pseudo RCT

+1 to Beth Barnes on dangerous capability evals

I really enjoyed this one! These feel like common EA mistakes, and I feel like I see them often

What kinds of grants tend to be most controversial among fund managers?

What are some past LTFF grants that you disagree with?

What are some types of grant that you'd love to fund, but don't tend to get as applications?

Can grantees return money if their plans change, eg they get hired during a period of upskilling? If so, how often does this happen?

Glad to hear it, I didn't think anyone still remembered this post!

Congratulations! Martin and Fernanda do great work, and I'm glad to see them being supported.

If someone is doing the shadow account thing (ie, a boiler room scam, I think), there will be exponentially fewer forecasters for each number of successful bets. I don't think this is the case for the well known ones

IMO it was tactically correct to not mention climate. The point of the letter is to get wide support, and I think many people would not be willing to put AI X-Risk on par with climate

"You’ll need to get hands-on. The best ML and alignment research engages heavily with neural networks (with only a few exceptions). Even if you’re more theoretically-minded, you should plan to be interacting with models regularly, and gain the relevant coding skills. In particular, I see a lot of junior researchers who want to do “conceptual research”. But you should assume that such research is useless until it cashes out in writing code or proving theorems, and that you’ll need to do the cashing out yourself (with threat modeling being the main exception, since it forces a different type of concreteness). ..."

This seems strongly true to me

I agree re PhD skillsets (though think that some fraction of people gain a lot of high value skills during a PhD, esp re research taste and agenda settings).

I think you're way overrating OpenAI though - in particular, Anthropic's early employees/founders include more than half of the GPT-3 first authors!! I think the company has become much more oriented around massive distributed LLM training runs in the last few years though, so maybe your inference that people would gain those skills is more reasonable now.

Strongly downvoted. I agree with the other comments. I think this post is bad as is especially in the current context of AI Safety disclose, and should be posted as part of a broader post about violent methods being ineffective (at least, assuming you're writing such a post). I personally strongly want AI Safety discourse to condemn and disavow violent methods, and think it's both immoral and ineffective. I don't think you believe that violence is a good idea here, but this post in isolation just feels like "hey, violent approaches exist, maybe worth thinking about, you wouldn't be super weird for doing them"

This seems fair, I'm significantly pushing back on this as criticism of Redwood, and as focus on the "Redwood has been overfunded" narrative. I agree that they probably consumed a bunch of grant makers time, and am sympathetic to the idea that OpenPhil is making a bunch of mistakes here.

I'm curious which academics you have in mind as slam dunks?

• I personally found MLAB extremely valuable. It was very well-designed and well-taught and was the best teaching/learning experience I've had by a fairly wide margin

Strong +1, I was really impressed with the quality of MLAB. I got a moderate amount out of doing it over the summer, and would have gotten much much more if I had done it a year or two before. I think that kind of outreach is high value, though plausibly a distraction from the core mission

Sorry for the long + rambly comment! I appreciate the pushback, and found clarifying my thoughts on this useful

I broadly agree that all of the funding ideas you point to seem decent. My biggest crux is that the counterfactual of not funding Redwood is not that one of those gets funded, and that the real constraints here around logistical effort, grantmaker time, etc. I wrote a comment downthread with further thoughts on these points.

And that it is not Redwood's job to solve this - they're pursuing a theory of change that does not depend on these, and it se... (read more)

Fwiw, my read is that a lot of "must have an ML PhD" requirements are gatekeeping nonsense. I think you learn useful skills doing a PhD in ML, and I think you learn some skills doing a non-ML PhD (but much less that's relevant, though physics PhDs are probably notably more relevant than maths). But also that eg academia can be pretty terrible for teaching you skills like ML engineering and software engineering, lots of work in academia is pretty irrelevant in the world of the bitter lesson, and lots of PhDs have terrible mentorship.

I care about people havi... (read more)

There are other TAIS labs (academic and not) that we believe could absorb and spend considerably more funding than they currently receive.

My understanding is that, had Redwood not existed, OpenPhil would not have significantly increased their funding to these other places, and broadly has more money than they know what to do with (especially in the previous EA funding environment!). I don't know whether those other places have applied for grants, or why they aren't as funded as they could be, but this doesn't seem that related to me. And more broadly th... (read more)

Neel Nanda, Tom Lieberum and others, mentored by Jacob Steinhardt

I will clarify in my personal case that I did the grokking work as an independent research project and that Jacob only became involved in the project after I had done the core research, and his mentorship was specifically about the process of distillation and writing up the results (to be clear, his mentorship here was high value! But I think that the paper benefited less from his mentorship than is implied by the reference class of having him as the final author)

Academic salaries are crazy low (which is one of my many reasons for not wanting to do a PhD lol)

Minor note that an anonymous feedback form might help to elicit negative feedback here. I appreciate the openness to criticism! (I don;t have significant negative feedback, I like constellation a lot, this is just a general note)

Of Redwood’s published research, we were impressed by Redwood's interpretability in the wild paper, but would consider it to be no more impressive than progress measures for grokking via mechanistic interpretability, executed primarily by two independent researchers, or latent knowledge in language models without supervision, performed by two PhD students.[4] These examples are cherry-picked to be amongst the best of academia and independent research, but we believe this is a valid comparison because we also picked what we consider the best of Redwood's r

Man, I have a strong negative aesthetic reaction to the new frontpage that I struggle to articulate - the old one was just so pretty and aesthetic, in a way that feels totally lost! How hard would it be to have an option to revert to the old style?

I agree with this in spirit, but think that in this case it's completely fine. a) Presumably, for some people, being zakat compatible has important cultural meaning. I generally think that the EA thing to do is to act within your constraints and belief systems and to do as much good as you can, not to need to tear down all of them. b) In my opinion, the point of impartiality is "find the most effective ways of helping people". I do not personally think that GiveDirectly is the most effective way to give, but it's not at all clear to me that the Yemeni reci... (read more)