All of Holden Karnofsky's Comments + Replies

My current impressions on career choice for longtermists

Thanks for the thoughtful comments, Linch.

Response on point 1: I didn't mean to send a message that one should amass the most impressive conventional credentials possible in general - only that for many of these aptitudes, conventional success is an important early sign of fit and potential.

I'm generally pretty skeptical by default of advanced degrees unless one has high confidence that one wants to be on a track where the degree is necessary (I briefly give reasons for this skepticism in the "political and bureaucratic aptitudes" section). This piece only... (read more)

My current impressions on career choice for longtermists

I like this; I agree with most of what you say about this kind of work.

I've tried to mostly list aptitudes that one can try out early on, stick with if they're going well, and pretty reliably build careers (though not necessarily direct-work longtermist careers) around. I think the aptitude you're describing here might be more of later-career/"secondary" aptitude that often develops as someone moves up along an "organization building/running/boosting" or "political/bureaucratic" track. But I agree it seems like a cluster of skills that can be intentionally developed to some degree and used in a lot of different contexts.

My current impressions on career choice for longtermists

Thanks for the thoughtful comments!

On your first point: the reason I chose to emphasize longtermism is because:

  • It's what I've been thinking about the most (note that I am now professionally focused on longtermism, which doesn't mean I don't value other areas, but does mean that that's where my mental energy goes).
  • I think longtermism is probably the thorniest, most frustrating area for career choice, so I wanted to focus my efforts on helping people in that category think through their options.
  • I thought a lot of what I was saying might generalize further, b
... (read more)
My current impressions on career choice for longtermists

I think a year of full-time work is likely enough to see the sort of "signs of life" I alluded to, but it could take much longer to fulfill one's potential. I'd generally expect a lot of people in this category to see steady progress over time on things like (a) how open-ended and poorly-scoped of a question they can tackle, which in turn affects how important a question they can tackle; (b) how efficiently and thoroughly they can reach a good answer; (c) how well they can communicate their insights; (d) whether they can hire and train other people to do c... (read more)

My current impressions on career choice for longtermists

I didn't mean to express a view one way or the other on particular current giving opportunities; I was instead looking for something a bit more general and timeless to say on this point, since especially in longtermism, giving opportunities can sometimes look very appealing at one moment and much less so at another (partly due to room-for-more-funding considerations). I think it's useful for you to have noted these points, though.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

This is still a common practice. The point of it isn't to evaluate employees by # of hours worked; the point is for their manager to have a good understanding of how time is being used, so they can make suggestions about what to go deeper on, what to skip, how to reprioritize tasks, etc.

Several employees simply opt out from this because they prefer not to do it. It's an optional practice for the benefit of employees rather than a required practice used for performance assessment.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

I'm referring to the possibility of supporting academics (e.g. philosophers) to propose and explore different approaches to moral uncertainty and their merits and drawbacks. (E.g., different approaches to operationalizing the considerations listed at https://www.openphilanthropy.org/blog/update-cause-prioritization-open-philanthropy#Allocating_capital_to_buckets_and_causes , which may have different consequences for how much ought to be allocated to each bucket)

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

Keep in mind that Milan worked for GiveWell, not OP, and that he was giving his own impressions rather than speaking for either organization in that post.

That said:

*His "Flexible working schedule" point sounds pretty consistent with how things are here.

*We continue to encourage time tracking (but we don't require it and not everybody does it).

*We do try to explicitly encourage self-care.

Does that respond to what you had in mind?

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

GiveWell's CEA was produced by multiple people over multiple years - we wouldn't expect a single person to generate the whole thing :)

I do think you should probably be able to imagine yourself engaging in a discussion over some particular parameter or aspect of GiveWell's CEA, and trying to improve that parameter or aspect to better capture what we care about (good accomplished per dollar). Quantitative aptitude is not a hard requirement for this position (there are some ways the role could evolve that would not require it), but it's a major plus.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

The role does include all three of those things, and I think all three things are well served by the job qualifications listed in the posting. A common thread is that all involve trying to deliver an informative, well-calibrated answer to an action-relevant question, largely via discussion with knowledgeable parties and critical assessment of evidence and arguments.

In general, we have a list of the projects that we consider most important to complete, and we look for good matches between high-ranked projects and employees who seem well suited to them. I ex... (read more)

1Ben Pace3yThanks Holden!
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We do formal performance reviews twice per year, and we ask managers to use their regular (~weekly) checkins with reports to sync up on performance such that nothing in these reviews should be surprising. There's no unified metric for an employee's output here; we set priorities for the organization, set assignments that serve these priorities, set case-by-case timelines and goals for the assignments (in collaboration with the people who will be working on them), and compare output to the goals we had set.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

All bios here: https://www.openphilanthropy.org/about/team

Grants Associates and Operations Associates are likely to report to Derek or Morgan. Research Analysts are likely to report to people who have been in similar roles for a while, such as Ajeya, Claire, Luke and Nick. None of this is set in stone though.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

A few things that come to mind:

  1. The work is challenging, and not everyone is able to perform at a high enough level to see the career progression they want.

  2. The culture tends toward direct communication. People are expected to be open with criticism, both of people they manage and of people who manage them. This can be uncomfortable for some people (though we try hard to create a supportive and constructive context).

  3. The work is often solitary, consisting of reading/writing/analysis and one-on-one checkins rather than large-group collaboration. It's pos

... (read more)
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We don't control the visa process and can't ensure that people will get sponsorship. We don't expect sponsorship requirements to be a major factor for us in deciding which applicants to move forward with.

2rjmk3yThanks for the response. I understand OPP doesn't control the visa process, but do you have a rough sense of how likely a successful applicant would be to get a visa after being sponsored, or is it a complete unknown?
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

There will probably be similar roles in the future, though I can't guarantee that. To become a better candidate, one can accomplish objectively impressive things (especially if they're relevant to effective altruism); create public content that gives a sense for how they think (e.g., a blog); or get to know people in the effective altruism community to increase the odds that one gets a positive & meaningful referral.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

Most of the roles here involve a lot of independent work, consisting of reading/writing/analysis and one-on-one checkins rather than large-group collaboration. It’s possible that this will change for some roles in the future (e.g. it’s possible that we’ll want more large-group collaboration as our cause prioritization team grows), but we’re not sure of that. I think you should probably be prepared for a fair amount of work along the lines of what I've described here.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

They're different organizations and I don't know nearly as much about the GiveWell role. One big difference is the causes we work on.

If you're interested in both, I'd recommend applying to both, and if you are offered both roles, there will be lots of opportunities to learn more about each at that point in order to inform the decision.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

I answered a similar question here: http://effective-altruism.com/ea/1mf/hi_im_holden_karnofsky_ama_about_jobs_at_open/dpl

In general, people who have been in the Research Analyst role for a while will be the managers and primary mentors of new Research Analysts. There will be regular (~weekly) scheduled checkins as well as informal interaction as needed (e.g., over Slack).

There's no hard line between training and "just doing the work" - every assignment should have some direct value and some training value. We expect to lean pretty hard toward t... (read more)

1Denise_Melchin3yIt is still unclear to me whether the statutory holidays are supposed to be included in the 25 days paid days off or in addition to.
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We're flexible. People don't clock in or out; we evaluate performance based on how much people get done on a timescale of months. We encourage people to work hard but also prioritize work-life balance. The right balance varies by the individual.

Most people here work more than one would in a traditional 9-5 job. (A common figure is 35-40 "focused" hours per week.) I think that reflects that they're passionate about their work rather than that they feel pressure from management to work a lot. We regularly check in with people about work-life balance and encourage them to work less if it seems this would be good for their happiness.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We're in the process of reviewing our policies, but we're likely to settle on something like 25 paid days off (including sick days), 10 holiday days (with the option to work on holidays and use the paid time off elsewhere), several months of paid parental leave, and a flexible unpaid leave policy for people who want to take more time off. We are also flexible with respect to working from home.

0Khorton3yTo clarify, are the 10 holiday days part of the 25 paid days off, or in addition to?
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

Perhaps other staff will chime in here, but my take: our pay is competitive and takes cost of living into account, and we are near public transportation, so I don't think the rents or commutes are a major issue. As a former NYC resident, I think the Bay Area is a great place to live (weather, food, etc.) and has a very strong effective altruist community. I don't see a lot of drawbacks to living here if you can make it work.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

Hm, I'm not sure why our form asks for more detail on undergrad relative to grad - we copied the form from GiveWell and may not have thought about it. It's possible this is because the form was being used in an earlier GiveWell search where few applicants had been to grad schools. I'll ask around about this.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

Broadly speaking, we're going to try to give people assignments that are relevant to our work and that we think include a lot of the core needed skills - things like evaluating a potential grant (or renewal) and writing up the case for or against. We'll evaluate these assignments, give substantial feedback, and iterate so that people improve. We'll also be providing resources for gaining background knowledge, such as "flex time," recommended reading lists and optional Q&A sessions. We've seen people improve a lot in the past and become core contributors, and think this basic approach is likely to lead to more of that.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

I would rate those about equally, though I'd add that GiveWell would prefer not to hire people whose main goal is to go to OP.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We currently have a happy hour every 3 weeks and host group activities as well, including occasional parties and a multiple-day staff retreat this year. We want to make it easy for staff to socialize and be friends, without making it a requirement or an overly hard nudge (if people would rather stick to their work, that's fine by us).

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We could certainly imagine ramping up grantmaking without a much better answer. As an institution we're often happy to go with a "hacky" approach that is suboptimal, but captures most of the value available under multiple different assumptions.

If someone at Open Phil has an idea for how to make useful progress on this kind of question in a reasonable amount of time, we'll very likely find that worthwhile and go forward. But there are lots of other things for Research Analysts to work on even if we don't put much more time into researching or reflecting on moral uncertainty.

Also note that we may pursue an improved understanding via grantmaking rather than via researching the question ourselves.

1SiebeRozendal3yI'm very curious about how that improved understanding would come about via grantmaking. Any write-up you have about this? I can see how you'd learn about tractability, and maybe about neglectedness, but I wonder how you incorporate this in your decision-making. Anyway, this might go a little too off-topic so I'd understand if you replied to other questions first :)
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

All else equal, we consider applicants stronger when they have degrees in challenging fields from strong institutions. It’s not the only thing we’re looking at, even at that early stage. And the early stage is for filtering; ultimately, things like work trial assignments will be far more important to hiring decisions.

2Khorton3yWhat does a "challenging field" mean here? Do you have a ranking of fields by degree of challenge or rely mainly on intuition?
1SiebeRozendal3yNot sure if interpreting Khorton correctly, but interested anyway: Why focus on undergrad and not on postgrad (or highest level achieved/pursuing)?
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

This varies by the individual. We have some Research Analysts who are always working on a variety of things, and some who have become quite specialized. It varies largely by the interests/preferences of the employee.

Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

We're certainly not using the same standards as academia! In general, we aim to base assignments on a combination of 1. How we judge what's most important to do (in terms of accomplishing as much good as possible) 2. What the employees themselves are motivated and interested to work on (including their own judgments of how to do as much good as possible).

1Michael_Cannon3yDefinitely not academia! Sounds like a healthy working environment, and I admire your focus on researcher autonomy.
Hi, I'm Holden Karnofsky. AMA about jobs at Open Philanthropy

I'd recommend that recent grads looking to help with AI governance and policy apply for the Research Analyst position. With Research Analysts, we'll first focus on mentorship & training, then try to figure out where everyone can do the most good based on their interests and skills. Someone who has a high aptitude & interest for AI strategy would likely end up putting substantial time into that within a year or so (maybe less).

You can also check out roles at the Future of Humanity Institute.

2branperr3yThank you for your reply Holden! I submitted my application yesterday after hearing you all were talent constrained, so I hope to hear from you all sometime soon. What does the training process look like for people who are hired on as research analysts?
Upcoming AMA with Luke Muehlhauser on consciousness and moral patienthood (June 28, starting 9am Pacific)

I super highly recommend reading this report. In full, including many of the appendices (and footnotes :) )

I thought it was really interesting, and helpful for thinking this question through and understanding the state of what evidence and arguments are out there (unfortunately there is much less to go on than I’d even expected, though).

I was the most proximate audience for the report, so discount my recommendation as much as feels appropriate with that in mind.

Some Thoughts on Public Discourse

The principles were meant as descriptions, not prescriptions.

I'm quite sympathetic to the idea expressed by your Herbert Simon quote. This is part of what I was getting at when I stated: "I think that one of the best ways to learn is to share one's impressions, even (especially) when they might be badly wrong. I wish that public discourse could include more low-caution exploration, without the risks that currently come with such things." But because the risks are what they are, I've concluded that public discourse is currently the wrong venue fo... (read more)

2RomeoStevens4yMost people wear their hearts on their sleeve to a greater degree than they might realize. Public conservatism of discourse seems a pretty reasonable proxy measure of private conservatism of discourse in most cases. As I mentioned, I am very happy to hear evidence this is not the case for openPhil. I do not think the model of creativity as a deliberate, trainable set of practices is widely known, so I go out of my way to bring it up WRT projects that are important.
Some Thoughts on Public Discourse

Michael, this post wasn't arguing that there are no benefits to public discourse; it's describing how my model has changed. I think the causal chain you describe is possible and has played out that way in some cases, but it seems to call for "sharing enough thinking to get potentially helpful people interested" rather than for "sharing thinking and addressing criticisms comprehensively (or anything close to it)."

The EA Forum counts for me as public discourse, and I see it as being useful in some ways, along the lines described in the post.

Some Thoughts on Public Discourse

Hi John, thanks for the thoughts.

I agree with what you say about public discourse as an "advertisement" and "critical first step," and allude to this somewhat in the post. And we plan to continue a level of participation of public discourse that seems appropriate for that goal - which is distinct from the level of public discourse that would make it feasible for readers to understand the full thinking behind the many decisions we make.

I don't so much agree that there is a lot of low-hanging fruit to be had in terms of getting more poten... (read more)

Some Thoughts on Public Discourse

Thanks for the thoughts!

I'm not sure I fully understand what you're advocating. You talk about "only selectively engag[ing] with criticism" but I'm not sure whether you are in favor of it or against it. FWIW, this post is largely meant to help understand why I only selectively engage with criticism.

I agree that "we should be skeptical of our stories about why we do things, even after we try to correct for this." I'm not sure that the reasons I've given are the true ones, but they are my best guess. I note that the reasons I give here ar... (read more)

7RomeoStevens4yWhoops, I somehow didn't see this until now. Scattered EA discourse, shrug. I am in support of only engaging selectively. great! agreed the whole thing. Principles are better as descriptions and not prescriptions :) WRT preventing views from calcifying, I think it is very very important to actively cultivate something similar to I've been researching top and breakout performance and this sort of thing keeps coming up again and again. Fortunately, creative reasoning is not magic. It has been studied and has some parameters that can be intentionally inculcated. This talk gives a brief overview: https://vimeo.com/89936101 [https://vimeo.com/89936101] And I recommend skimming one of Edward deBono's books, such as six thinking hats. He outlined much of the sort of reasoning of 0 to 1, the Lean Startup, and others way back in the early nineties. It may be that openPhil is already having such conversations internally. In which case, great! That would make me much more bullish on the idea that openPhil has a chance at outsize impact. My main proxy metric is an Umeshism: if you never output any batshit crazy ideas your process is way too conservative.
Some Thoughts on Public Discourse

Thanks for the thoughts, Vipul! Responses follow.

(1) I'm sorry to hear that you've found my writing too vague. There is always a tradeoff between time spent, breadth of issues covered, and detail/precision. The posts you hold up as more precise are on narrower topics; the posts you say are too vague are attempts to summarize/distill views I have (or changes of opinions I've had) that stem from a lot of different premises, many hard to articulate, but that are important enough that I've tried to give people an idea of what I'm thinking. In many cases their ... (read more)

Some Thoughts on Public Discourse

Thanks for the comments, everyone!

I appreciate the kind words about the quality and usefulness of our content. To be clear, we still have a strong preference to share content publicly when it seems it would be useful and when we don't see significant downsides. And generally, the content that seems most likely to be helpful has fairly limited overlap with the content that poses the biggest risks.

I have responded to questions and criticisms on the appropriate threads.

GiveWell and the problem of partial funding

(Continued from previous comment)

Thoughts on your recommendations. I appreciate your making suggestions, and providing helpful context on the spirit in which you intend them. Here I only address suggestions for Open Phil.

  • Maintaining a list of open investigations: I see some case for this, but at the moment we don't plan on it. I don't think we can succinctly and efficiently maintain such a list without incurring a number of risks (e.g., causing people to excessively plan on our support; causing controversy due to hasty communication or miscommunication).
... (read more)
GiveWell and the problem of partial funding

Hi Ben,

Thanks for putting so much thought into this topic and sharing your feedback.

I'm going to discuss the reasoning behind the "splitting" recommendation that was made in 2015, as well as our current stance, and how they relate to your points. I'll start with the latter because I think that will make this comment easier to follow. I'll then address some more specific points and suggestions.

I'm not addressing recommendations addressed to GiveWell - I think it will make more sense for someone more involved in GiveWell to do that - though I will... (read more)

5BenHoffman4yThanks for the detailed response! I wanted to quickly point out something you did here that I think is good practice, and wish more people did: Marking which parts of someone's argument you think are relevant and which you think aren't - and, relatedly, which branches of a disjunction you accept and which you reject - are an important part of how arguments can lead to shared models. A lot of people neglect this sort of thing, because it's not a clear way to score points for their side. You took care to address it here. Thanks. (More to follow when I've had time to take this in.)

(Continued from previous comment)

Thoughts on your recommendations. I appreciate your making suggestions, and providing helpful context on the spirit in which you intend them. Here I only address suggestions for Open Phil.

  • Maintaining a list of open investigations: I see some case for this, but at the moment we don't plan on it. I don't think we can succinctly and efficiently maintain such a list without incurring a number of risks (e.g., causing people to excessively plan on our support; causing controversy due to hasty communication or miscommunication).
... (read more)
How Should a Large Donor Prioritize Cause Areas?

I think the more uncertain you are, the more learning and option value matter, as well as some other factors I will probably discuss more in the future. I agree that committing to a cause, and helping support a field in its early stages, can increase room for more funding, but I think it's a pretty slow and unpredictable process. In the future we may see enough room for more funding in our top causes to transition to a more concentrated funding, but I think the tradeoffs implied in OP have very limited relevance to the choices we're making today.

How Should a Large Donor Prioritize Cause Areas?

Thanks, all, for the very thoughtful post and comments!

At some point this year, I hope to make a post about our general reasons for wanting to put some resources into the causes that look best according to different plausible background worldviews and epistemology. Dan Keys and Telofy touched on a lot of these reasons (especially Dan's #3 and #4).

I think our biggest disagreement with Michael is that he seems to see a couple of particular categories of giving (those relating to farm animal suffering and direct existential risk) as massively and clearly bett... (read more)

1MichaelDickens5yThanks for the response, Holden. I appreciate it when you engage with public comments on GiveWell/Open Phil. I'm probably more confident than you are about cause prioritization, but I don't believe that's necessary for my arguments. You just have to be weakly confident that one area is better, and that it has more room for funding than you can fill in the long term. But if you're only weakly confident in one cause area being better than another then that makes Dan's #3 look more compelling, so diversifying may be the right call in that case. I'll add that I agree with you that there's almost certainly not $50 million worth of "shovel-ready" grants in AI safety, and definitely not in wild-animal suffering, but the problems are big enough that they could easily absorb this much funding if more people were working on the problems. Committing money to the problems is probably one of the best ways to incentivize people to work on them—Open Phil already seems to be doing this a bit with AI safety. I don't know as much about grantmaking as you do but my understanding is that you can create giving opportunities by committing to cause areas, which was part of Open Phil's motivation for making such commitments.
Why the Open Philanthropy Project isn't currently funding organizations focused on promoting effective altruism

Thanks for the comments, all.

Telofy and kgallas: I'm not planning to write up an exhaustive list of the messages associated with EA that we're not comfortable with. We don't have full internal agreement on which messages are good vs. problematic, and writing up a list would be a bit of a project in itself. But I will give a couple of examples, speaking only for myself:

  1. I'm generally uncomfortable with (and disagree with) the "obligation" frame of EA. I'm particularly uncomfortable with messages along the lines of "The arts are a waste when

... (read more)
4Denis Drescher6yI’m looking forward to news from Lewis then! Agreed on point 2. About point 1: “I think messages along these lines make EA sound overly demanding/costly to affiliate with”: This strategic issue is one that I have no informed opinion on. Intuitively I would also think that people work that way, but the practice of hazing, e.g., initiation rites of fraternities, suggests that such costliness might counter recidivism, and that’s an important factor. Moral frameworks that have this obligation aspect also seem relatively more simple and consistent to me, which might make it easier to defend them convincingly in outreach. “As well as intellectually misguided”: From a moral antirealist’s perspective, this depends on the person’s moral framework. Taking Brian’s critique of the demandingness critique [http://crucialconsiderations.org/ethics/demandingness/] into account, this does apply to mine, so whether to demand the same from others, again only boils down to the strategic question above. Do you have an ethical or epistemic reason why it would be misguided even from a broadly utilitarian viewpoint?