All of sawyer🔸's Comments + Replies

Thanks for writing and posting this! I've had these sorts of feelings floating around in my head for a while, but this is the best term I've heard for it.

Having not read the article, this threw me and I had to go check. But unfortunately they do seem to be calling the timeline models themselves "bad". 

I focussed on one section alone: their “timelines forecast” code and accompanying methodology section. Not to mince words, I think it’s pretty bad.

I think this is the single most underrated post on the EA Forum.

Thanks for writing this! I have a more philosophical counter that I'd love for you to respond to.

The idea of haggling doesn't sit well with me or my idea of what a good society should be like. It feels competitive, uncooperative, and zero-sum, when I want to live in a society where people are honest and cooperative. Specifically, it seems to encourage deceptive pricing and reward people who are willing to be manipulative and stretch the truth.

In other words, haggling gives me bad vibes.

When you think about haggling/negotiating in altruistic context, do you... (read more)

Ah! Yes that's a good point and I misinterpreted.That's part of what I meant by "historical accident" but now I think that it was confusing to say "accident" and I should have said something like "hisotrical activities".

I agree that they're worth calling out somehow, I just think "lab" is a misleading way to doing so given their current activities. I've made some admittedly-clunky suggestions in other threads here.

I completely agree that OpenAI and Deepmind started out as labs and are no longer so.

I agree that those companies are worth distinguishing. I just think calling them "labs" is a confusing way to do so. If the purpose was only to distinguish them from other AI companies, you could call them "AI bananas" and it would be just as useful. But "AI bananas" is unhelpful and confusing. I think "AI labs" is the same (to a lesser but still important degree).

I think this is a useful distinction, thanks for raising it. I support terms like, "frontier AI company," "company making frontier AI," and "company making foundation models," all of which help distinguish OpenAI from Palantir. Also it seems pretty likely that within a few years, most companies will be AI companies!? So we'll need new terms. I just don't want that term to be "lab".

Another thing you might be alluding to is that "lab" is less problematic when talking to people within the AI safety community, and more problematic the further out you go. I thi... (read more)

Interesting point! I'd be OK with people calling them "evil mad scientist labs," but I still think the generic "lab" has more of a positive, harmless connotation than this negative one.

I'd also be more sympathetic to calling them "labs" if (1) we had actual regulations around them or (2) they were government projects. Biosafety and nuclear weapons labs have a healthy reputation for being dangerous and unfriendly, in a way "computer labs" do not. Also, private companies may have biosafety containment labs on premises, and the people working within them are ... (read more)

From everything I've seen, GWWC has totally transformed under your leadership. And I think this transformation has been one of the best things that's happened in EA during that time. I'm so thankful for everything you've done for this important organization.

Yep! Something like this is probably unavoidable, and it's what all of my examples below do (BERI, ACE, and MIRI).

There are many examples of organizations with high funding transparency, including BERI (which I run), ACE, and MIRI (transparency page and top contributors page).

(Not deeply thought through) Funders have a strong (though usually indirect) influence on the priorities and goals of the organization. Transparency about funders adds transparency about the priorities and goals of the organization. Conversely, lack of funder transparency creates the appearance that you're trying to hide something important about your goals and priorities. This sort of argument comes up a lot in US political funding, under the banners of "Citizens United", "SuperPACs", etc. I'm making a pretty similar argument to that one.

Underlying my fee... (read more)

I think this dynamic is generally overstated, at least in the existential risk space that I work in. I've personally asked all of our medium and large funders for permission, and the vast majority of them have given permission. Most of the funding comes from Open Philanthropy and SFF, both of which publicly announce all of their grants—when recipients decided not to list those funders, it's not because the funders don't want them to. There are many examples of organizations with high funding transparency, including BERI (which I run), ACE, and MIRI (transp... (read more)

Nonprofit organizations should make their sources of funding really obvious and clear: How much money you got from which grantmakers, and approximately when. Any time I go on some org's website and can't find information about their major funders, it's a big red flag. At a bare minimum you should have a list of funders, and I'm confused why more orgs don't do this.

I think people would say that the dog was stronger and faster than all previous dog breeds, not that it was "more capable". It's in fact significantly less capable at not attacking its owner, which is an important dog capability. I just think the language of "capability" is somewhat idiosyncratic to AI research and industry, and I'm arguing that it's not particularly useful or clarifying language.

More to my point (though probably orthogonal to your point), I don't think many people would buy this dog, because most people care more about not getting attacke... (read more)

What is "capabilities"? What is "safety"? People often talk about the alignment tax: the magnitude of capabilities/time/cost a developer loses by implementing an aligned/safe system. But why should we consider an unaligned/unsafe system "capable" at all? If someone developed a commercial airplane that went faster than anything else on the market, but it exploded on 1% of flights, no one would call that a capable airplane.

This idea overlaps with safety culture and safety engineering and is not new. But alongside recent criticism of the terms "safety" and "alignment", I'm starting to think that the term "capabilities" is unhelpful, capturing different things for different people.

I played the paperclips game 6-12 months before reading Superintelligence (which is what convinced me to prioritize AI x-risk), and I think the game made these ideas easier for me to understand and internalize.

This is truly crushing news. I met Marisa at a CFAR workshop in 2020. She was open, kind, and grateful to everyone, and it was joyful to be around her. I worked with her a bit revitalizing the EA Operations Slack Workspace in 2020, and had only had a few conversations with her since then, here and there at EA events. Marisa (like many young EAs) made me excited for a future that would benefit from her work, ambition, and positivity. Now she's gone. She was a good person, I'm glad she was alive, and I am so sad she's gone.

Good reasoning, well written. Reading this post convinced me to join the next NYC protest. Unfortunately I missed the one literally two days ago because I waited too long to read this. But I plan to be there in September.

One thing I think is often missing from these sorts of conversations is that "alignment with EA" and "alignment with my organization's mission" are not the same thing! It's a mistake to assume that the only people who understand and believe in your organization’s mission are members of the effective altruism community. EA ideas don’t have to come in a complete package. People can believe that one organization’s mission is really valuable and important, for different reasons, coming from totally different values, and without also believing that a bunch of o... (read more)

Within EA, work on x-risk is very siloed by type of threat: There are the AI people, the bio people, etc. Is this bad, or good?

Which of these is the correct analogy?

1. "Biology is to science as AI safety is to x-risk," or 
2. "Immunology is to biology as AI safety is to x-risk"

EAs seem to implicitly think analogy 1 is correct: some interdisciplinary work is nice (biophysics) but most biologists can just be biologists (i.e. most AI x-risk people can just do AI).

The "existential risk studies" model (popular with CSER, SERI, and lots of other non-EA academics) ... (read more)

I agree with your last sentence, and I think in some versions of this it's the vast majority of people. A lot of charity advertising seems to encourage a false sense of confidence, e.g. "Feed this child for $1," or "adopt this manatee". I think this makes use of a near-universal human bias which probably has a name but which I am not recalling at the moment. For a less deceptive version of this, note how much effort AMF and GiveDirectly seem to have put in into tracking the concrete impact of your specific donation.

Building off of Jason's comment: Another way to express this is that comparing directly to the $5,500 Givewell bar is only fair for risk-neutral donors (I think?). Most potential donors are not really risk neutral, and would rather spend $5,001 to definitely save one life than $5,000 to have a 10% chance of saving 10 lives. Risk neutrality is a totally defensible position, but so is non-neutrality. It's good to have the option of paying a "premium" for a higher confidence (but lower risk-neutral EV).

Leaving math mode...I love this post. It made me emotiona... (read more)

Very nice post. "Anarchists have no idols" strikes me as very similar to the popular anarchist slogan, "No gods, no masters." Perhaps the person who said it to you was riffing on that?

I think a simpler explanation for his bizarre actions is that he is probably the most stressed-out person on the face of the earth right now. Or he's not seeing the situation clearly, or some combination of the two. Also probably sleep-deprived, struggling to get good advice from people around him, etc.

(This is not meant to excuse any of his actions or words, I think he's 100% responsible for everything he says and does.)

This sort of falls under the second category, "Grantees who received funds, but want to set them aside to return to creditors or depositors." At least that's how I read it, though the more I think about it the more this category is kind of confusing and your wording seems more direct.

Thanks for the clarification. I agree that the FTX problems are clearly related to crypto being such a new unregulated area, and I was wrong to try to downplay that causal link.

I don't think anonymized donations would help mitigate conflicts of interest. In fact I think it would encourage COIs, since donors could directly buy influence without anyone knowing they were doing so. Currently one of our only tools for identifying otherwise-undisclosed COIs is looking at flows of money. If billionaire A donates to org B, we have a norm that org B shouldn't do st... (read more)

Downvoted because I think this is too harsh and accusatory:

I cannot believe that some of you delete your posts simply because it ends up being downvoted.

Also because I disagree in the following ways:

• Donating anonymously seems precisely opposed to transparency. At the very least, I don't think it's obvious that donor anonymity works towards the values you're expressing in your post. Personally I think being transparent about who is donating to what organizations is pretty important for transparency, and I think this is a common view.
• I don't think FTX's mist

Yep this is a great point and overlaps with Vardev's comment. If I thought that the money was gained immorally, it would be pretty bad to just return it to the people who did the immoral thing!

Yeah this seems super relevant, great point! To be honest I'm skeptical of how separate "FTX Foundation, Inc." is/was from the rest of the FTX conglomerate. Would be useful to see the Foundation's finances after this all shakes out.

What are the specific things you'd want to see on a transparency page? I think transparency is important, and I try to maintain BERI's transparency page, but I'm wondering if it meets your standards.

I'd guess the reason this was done for comments first is that posts are much longer and more complicated, such that it's often not clear what "agreeing" with the post even means. I think it's plausibly a good feature for posts, but I think it makes a lot more sense for comments.

It might be tough to implement this in a way that doesn't boost linkposts (which  I think would be counter to your purpose).

As in, you can't publish a post without at least one tag?

Love this, great work. I especially appreciate your honest opinions on what mistakes you think you made and how the survey could have been improved. If JERIS continues next year, those thoughts will enable a lot of improvement!

Consider adding the Berkeley Existential Risk Initiative (BERI) to the list, either under Professional Services or under Financial and other material support. Suggested description: "Supports university research groups working to reduce x-risk, by providing them with free services and support."

Great post. This put words to some vague concerns I've had lately with people valorizing "agent-y" characteristics. I'm agentic in some ways and very unagentic in other ways, and I'm mostly happy with my impact, reputation, and "social footprint". I like your section on not regulating consumption of finite resources: I think that modeling all aspects of a community as a free market is really bad (I think you agree with this, at least directionally).

This post, especially the section on "Assuming that it is low-cost for others to say 'no' to requests"  ... (read more)

Good catch, thanks! I can't find my original quote, so I think this was a recent change. I will edit my post accordingly.

Great points, thanks David. I especially like the compare and contrast between personal connections and academic credentials. I think probably you're more experienced with academia and non-EA philanthropy than I am, so your empirical views are different. But I also think that even if EA is better than these other communities, we should still be thinking about (1) keeping it that way, and (2) maybe getting even less reliant. This is part of what I was saying with:

None of this is unique to EA. While I think EA is particularly guilty of some of these issues,

I think the extent to which "member of the EA community" comes along with a certain way of thinking (i.e. "a lot of useful frames") is exaggerated by many people I've heard talk about this sort of thing. I think ~50% of the perceived similarity is better described as similar ways of speaking and knowledge of jargon. I think that there actually not that many people who have fully internalized new ways of thinking that are 1.) very rare outside of EA, and 2.) shared across most EA hiring managers.

Another way to put this would be: I think EA hiring managers o... (read more)

Explicitly asking for a reference the head organizer knows personally.

That feels pretty bad to me! I can imagine some reason that this would be necessary for some programs, but in general requiring this doesn't seem healthy.

I find the request for references on the EA Funds' application to be a good middle-ground. There's several sentences to it, but the most relevant one is:

References by people who are directly involved in effective altruism and adjacent communities are particularly useful, especially if we are likely to be familiar with their work and thi

Thanks Chi, this was definitely a mistake on my part and I will edit the post. I do think that your website's "Get Involved" -> "CLR Fund" might not be the clearest path for people looking for funding, but I also think I should have spent more time looking.

Thanks for the thoughtful feedback Chris!

I think that the author undervalues value alignment and how the natural state is towards one of regression to the norm unless specific action is taken to avoid this

I think there is  difference between "value alignment" and "personal connection". I agree that the former is important, and I think the latter is often used (mostly successfully) as a tool to encourage the former. I addressed one aspect of this in the Hiring Managers section.

I agree that as EA scales, we will be less able to rely personal relationshi

tension between reliance on personal connections and high rates of movement growth. You take this to be a reason for relying on personal connections less, but one may argue it is a reason for growing more slowly.

I completely agree! I think probably some combination is best, and/or it could differ between subcommunities.

Also thanks for pointing out the FTX Future Fund's experience, I'd forgotten about that. I completely agree that this is evidence against my hypothesis, specifically in the case of grantee-grantor relationships.

Great point about mitigating as opposed to solving. It's possible that my having a "solutions" section wasn't the best framing. I definitely don't think personal connections should be vilified or gotten rid of entirely (if that were even possible), and going too far in this direction would be really bad.

Thanks Stefan! I agree with those strengths of personal connections, and I think there are many others. I mainly tried to argue that there are negative consequences as well, and that the negatives might outweigh the positives at some level of use. Did any of the problems I mentioned in the post strike you as wrong? (Either you think they don't tend to arise from reliance on personal connections, or you think they're not important problems even if they do arise?)