2035Joined Sep 2014


[Linkpost] Criticism of Criticism of Criticism, ACX

As noted in the post, I got Scott's permission before posting this. 

Applications are open for CFAR workshops in Prague this fall

I strongly disagree with Greg. I think CFAR messed up very badly, but I think the way they messed up is totally consistent with also being able to add value in some situations. 

We have data I find convincing suggesting a substantial fraction of top EAs got value from CFAR. ~ 5 years have passed since I went to a CFAR workshop, and I still value what I learned and think it's been useful for my work. I would encourage other people who are curious to go (again, with the caveat that I don't know much about the new program), if they feel like they're in a place of relative strength and can take a discerning eye to what they're taught.    

If I, with (mostly) admirable candour, describe a series of grossly incompetent mistakes during my work as a doctor, the appropriate response may still be to disqualify me from future medical practice (there are sidelines re. incentives, but they don't help)

I think doctor is a really disanalogous example to use; doctors are in one of the relatively few professions where screwups regularly lead to death; we want to some somewhat risk-averse, with respect to doctors (and e.g. pilots or school bus drivers), at least if the screwups are the very dangerous kind (as opposed to like, being terrible at filing one's paperwork), and aren't based on a reasonable CBA (e.g. enrolling patients in a clinical trial with a drug that looked promising but turned out to be dangerous). For lots of other professions, this example looks way less compelling; e.g. I doubt people would think that a startup founder or movie director or author who had a bunch of failures but also some big wins should be banned from their profession or ostracized in their community. I think in-person overnight events about psychology are in a pretty in-between risk category.

Marriage, the Giving What We Can Pledge, and the damage caused by vague public commitments

You said you wouldn’t tell anyone about your friend’s secret, but this seems like a situation where they wouldn’t mind, and it would be pretty awkward to say nothing…etc.


This isn't your main point, and I agree there's a lot of motivated cognition people can fall prey to. But I think this gets a bit tricky, because people often ask for vague commitments, that are different from what they actually want and intend. For example, I think sometimes when people say "don't share this" they actually mean something more like "don't share this with people that know me personally" or "keep it in our small circle of trusted friends and advisors" or "you can tell your spouse and therapist, but no one else" (and often, this is borne out when I try to clarify). Sometimes, I think they are just trying to convey "this info is sensitive, tread with care". Or, they might mean something more intense, like "don't share this, and aim not to reveal any information that updates others substantially towards thinking it's true". 

Clarification can often be useful here (and I wish there were more verbal shorthands for different levels of intensity of commitment) but sometimes it doesn't happen and I don't think, in its absence, all agreements should be taken to be maximally strict (though I think it's extremely important to have tools for conveying when a requested agreement is very strict, and being the kind of person that can honor that). And I think some EAs get intense and overly scrupulous about obeying unimportant agreements, which can be pretty unwieldy and divorced from what anyone intended.  

I think "do you keep actually-thoughtful promises you think people expected you to interpret as real commitments" and "do you take all superficially-promise-like-things as serious promises" are fairly different qualities (though somewhat correlated), and kinda often conflated in a way that I think is unhealthy and even performative. 

Announcing the Harvard AI Safety Team

This seems really exciting, and I agree that it's an underexplored area. I hope you share resources you develop and things you learn to make it easier for others to start groups like this.

PSA for people reading this thread in the future: Open Phil is also very open to and excited about supporting AI safety student groups (as well as other groups that seem helpful for longtermist priority projects); see here for a link to the application form.

Let's not have a separate "community building" camp

I used to agree more with the thrust of this post than I do, and now I think this is somewhat overstated. 

[Below written super fast, and while a bit sleep deprived]

An overly crude summary of my current picture is "if you do community-building via spoken interactions, it's somewhere between "helpful" and "necessary" to have a substantially deeper understanding of the relevant direct work than the people you are trying to build community with, and also to be the kind of person they think is impressive, worth listening to, and admirable. Additionally, being interested in direct work is correlated with a bunch of positive qualities that help with community-building (like being intellectually-curious and having interesting and informed things to say on many topics). But not a ton of it is actually needed for many kinds of extremely valuable community building, in my experience (which seems to differ from e.g. Oliver's). And I think people who emphasize the value of keeping up with direct work sometimes conflate the value of e.g. knowing about new directions in AI safety research vs.  broader value adds from becoming a more informed person and gaining various intellectual benefits from practice engaging with object-level rather than social problems. 

Earlier on in my role at Open Phil, I found it very useful to spend a lot of time thinking through cause prioritization, getting a basic lay of the land on specific causes, thinking through what problems and potential interventions seemed most important and becoming emotionally bought-in on spending my time and effort on them. Additionally, I think the process of thinking through who you trust, and why, and doing early audits that can form the foundation for trust, is challenging but very helpful for doing EA CB work well. And I'm wholly in favor of that, and would guess that most people that don't do this kind of upfront investment are making an important mistake. 

But on the current margin, the time I spend keeping up with e.g. new directions in AI safety research feels substantially less important than spending time on implementation on my core projects, and almost never directly decision-relevant (though there are some exceptions, e.g. I could imagine information that would (and, historically, has) update(d) me a lot about AI timelines, and this would flow through to making different decisions in concrete ways). And examining what's going on with that, it seems like most decisions I make as a community-building grantmaker are too crude to be affected much by additional info at the relevant level of granularity intra-cause, and when I think about lots of other community-building-related decisions, the same seems true. 

For example, if I ask a bunch of AI safety researchers what kinds of people they would like to join their teams, they often say pretty similar versions of "very smart, hardworking people who grok our goals, who are extremely gifted in a field like math or CS".  And I'm like "wow , that's very intuitive, and has been true for years, without changing". Subtle differences between alignment agendas do not, in my experience, bear out enough in people's ideas about what kinds of recruitment are good that I've found it to be a good use of time to dig in on. This is especially true given that places where informed, intelligent people who have various important-to-me markers of trustworthiness differ are places where I find that it's particularly difficult for an outsider to gain much justified confidence.

Another testbed is that I spend a few years spending a lot of time on Open Phil's biosecurity strategy, and I formed a lot of my own, pretty nuanced and intricate views about it. I've never dived as deep on AI. But I notice that I didn't find my own set of views about biosecurity that helpful for many broader community-building tradeoffs and questions, compared to the counterfactual of trusting the people who seemed best to me to trust in the space (which I think I could have guessed using a bunch of proxies that didn't involve forming my own models of biosecurity) and catching up with them or interviewing them every 6mo about what it seems helpful to know (which is more similar to what I do with AI).  Idk, this feels more like 5-10% of my time, though maybe I absorb additional context via osmosis from social proximity to people doing direct work, and maybe this helpful in ways that aren't apparent to me. 

Let's not have a separate "community building" camp

>It's fine to have professional facilitators who are helping the community-building work without detailed takes on object-level priorities, but they shouldn't be the ones making the calls about what kind of community-building work needs to happen

I think this could be worth calling out more directly and emphatically. I think a large fraction (idk, between 25 and 70%) of people who do community-building work aren't trying to make calls about what kinds of community-building work needs to happen.

What We Owe the Past

I put a bunch of weight on  decision theories which support 2. 

A mundane example: I get value now from knowing that, even if I died, my partner would pursue certain Claire-specific projects I value being pursued because it makes me happy to know they will get pursued even if I die. I couldn't have that happiness now if I didn't believe he would actually do it, and it'd be hard for him (a person who lives with me and who I've dated for many years) to make me believe that he actually would pursue them even if it weren't true (as well as seeming sketchy from a deontological perspective). 

And, +1 to Austin's example of funders; funders occasionally have people ask for retroactive funding, and say that they only did the thing because their model of the funders suggested the funder would pay. 

Why CEA Online doesn’t outsource more work to non-EA freelancers

Thanks for this! Most of what you wrote here matches my experience and what I've seen grantees experience. It often feels weird and frustrating (and counter to econ 101 intuitions) to be like "idk, you just can't exchange money for good and services the obvious way, sorry, no, you can't just pay more money to get out of having to manage that person and have them still do their work well" and I appreciate this explanation of why.

Good practices for changing minds

Riffing off of the alliance mindset point, one shift I've personally found really helpful (though I could imagine it backfiring for other people) in decision-making settings is switching from thinking "my job is to come up with the right proposal or decision" to "my job is to integrate the evidence I've observed (firsthand, secondhand, etc.) and reason about it as clearly and well as I'm able". 

The first framing made me feel like I was failing if other people contributed; I was "supposed" to get to the best decision, but instead I came to the wrong one that needed to be, humiliatingly, "fixed". The frame is more individualistic, and has more of a sense of some final responsibility that increases emotional heat and isn't explained just by bayesian reasoning.  

The latter frame evokes thoughts like "of course, what I'm able to observe and think of is only a small piece of the puzzle, of course others have lots of value of add" and shifts my experience of changing decisions from embarrassing or a sign of failure to natural and inevitable, and my orientation towards others from defensiveness to curiosity and eagerness to elicit their knowledge. And it shifts my orientation towards myself from a stakesy attempt to squeeze out an excellent product via the sheer force of emotional energy, to something more reflective, internally quiet, and focused on the outer world, not what my proposals will say about me. 

I could imagine this causing people to go easy on themselves or try less hard, but for me it's been really helpful. 

Load More