Chief Technology Officer @ Redwood Research
5772Berkeley, CA, USAJoined Sep 2014


I'm Buck Shlegeris. I am the CTO of Redwood Research, a nonprofit focused on applied alignment research. Read more about us here: https://www.redwoodresearch.org/


Sorted by New
Buck's Shortform
· 3y ago · 1m read


Answer by BuckFeb 28, 20232014

Yes, I think this is very scary. I think this kind of risk is at least 10% as important as the AI takeover risks that I work on as an alignment researcher.


I don't think Holden agrees with this as much as you might think. For example, he spent a lot of his time in the last year or two writing a blog.


Obviously it would have been better if those organizers had planned better. It's not clear to me that it would have been better for the event to just go down in flames; OP apparently agreed with me, which is why they stepped in with more funding.

I don't think the Future Forum organizers have particularly strong relationships with OP.


The main bottleneck I'm thinking of is energetic people with good judgement to execute on and manage these projects.


I disagree, I think that making controversial posts under your real name can improve your reputation in the EA community in ways that help your ability to do good. For example, I think I've personally benefited a lot from saying things that were controversial under my real name over the years (including before I worked at EA orgs).


Stand up a meta organization for neartermism now, and start moving functions over as it is ready.

As I've said before, I agree with you that this looks like a pretty good idea from a neartermist perspective.

 Neartermism has developed meta organizations from scratch before, of course.


which is quite a bit more than neartermism had when it created most of the current meta.

I don't think it's fair to describe the current meta orgs as being created by neartermists and therefore argue that new orgs could be created by neartermists. These were created by people who were compelled by the fundamental arguments for EA (e.g. the importance of cause prioritization, cosmopolitanism, etc). New meta orgs would have to be created by people who are compelled by these arguments but also not compelled by the current arguments for longtermism, which is empirically a small fraction of the most energetic/ambitious/competent people who are compelled by arguments for the other core EA ideas.

More importantly, meta orgs that were distanced from the longtermist branch would likely attract people interested in working in GHD, animal advocacy, etc. who wouldn't currently be interested in affiliating with EA as a whole. So you'd get some experienced hands and a good number of new recruits

I think this is the strongest argument for why neartermism wouldn't be substantially weaker without longtermists subsidizing its infrastructure.

Two general points:

  • There are many neartermists who I deeply respect; for example, I feel deep gratitude to Lewis Bollard from the Open Phil farmed animal welfare team and many other farmed animal welfare people. Also, I think GiveWell seems like a competent org that I expect to keep running competently.
  • It makes me feel sad to imagine neartermists not wanting to associate with longtermists. I personally feel like I am fundamentally an EA, but I'm only contingently a longtermist. If I didn't believe I could influence the long run future, I'd probably be working on animal welfare; if I didn't believe that there were good opportunities there, I'd be working hard to improve the welfare of current humans. If I believed it was the best thing to do, I would totally be living frugally and working hard to EtG for global poverty charities. I think of neartermist EAs as being fellow travelers and kindred spirits, with much more in common with me than almost all other humans.

Fwiw my guess is that longtermism hasn’t had net negative impact by its own standards. I don’t think negative effects from AI speed up outweigh various positive impacts (e.g. promotion of alignment concerns, setting up alignment research, and non-AI stuff).


and then explains why these longtermists will not be receptive to conventional EA arguments.

I don't agree with this summary of my comment btw. I think the longtermists I'm talking about are receptive to arguments phrased in terms of the classic EA concepts (arguments in those terms are how most of us ended up working on the things we work on).


Holden Karnofsky on evaluating people based on public discourse:

I think it's good and important to form views about people's strengths, weaknesses, values and character. However, I am generally against forming negative views of people (on any of these dimensions) based on seemingly incorrect, poorly reasoned, or seemingly bad-values-driven public statements. When a public statement is not misleading or tangibly harmful, I generally am open to treating it as a positive update on the person making the statement, but not to treating it as worse news about them than if they had simply said nothing.

The basic reasons for this attitude are:

  • I think it is very easy to be wrong about the implications of someone's public statement. It could be that their statement was poorly expressed, or aimed at another audience; that the reader is failing to understand subtleties of it; or that the statement is in fact wrong, but that it merely reflects that the person who made it hasn't been sufficiently reflective or knowledgeable on the topic yet (and could become so later).
  • I think public discourse would be less costly and more productive for everyone if the attitude I take were more common. I think that one of the best ways to learn is to share one's impressions, even (especially) when they might be badly wrong. I wish that public discourse could include more low-caution exploration, without the risks that currently come with such things.
  • I generally believe in evaluating people based on what they've accomplished and what they've had the opportunity to accomplish, plus any tangible harm (including misinformation) they've caused. I think this approach works well for identifying people who are promising and people whom I should steer clear of; I think other methods add little of value and mostly add noise.

I update negatively on people who mislead (including expressing great confidence while being wrong, and especially including avoidable mischaracterizations of others' views); people who do tangible damage (usually by misleading); and people who create little of value despite large amounts of opportunity and time investment. But if someone is simply expressing a view and being open about their reasons for holding it, I try (largely successfully, I think) not to make any negative updates simply based on the substance.

FWIW I'm somewhat more judgemental than Holden, but I think the position Holden advocates is not that unusual for seniorish EAs.

Load more