AdamGleave

Wiki Contributions

Comments

EA Infrastructure Fund: May–August 2021 grant recommendations

That being said, we might increase our funding threshold if we learn that few grants have been large successes, or if more funders are entering the space.

My intuition is that more funders entering the space should lower your bar for funding, as it'd imply there's generally more money in this space going after the same set of opportunities. I'm curious what the reasoning behind this is, e.g. unilateralist curse considerations?

Democratising Risk - or how EA deals with critics

First of all, I'm sorry to hear you found the paper so emotionally draining. Having rigorous debate on foundational issues in EA is clearly of the utmost importance. For what it's worth when I'm making grant recommendations I'd view criticizing orthodoxy (in EA or other fields) as a strong positive so long as it's well argued. While I do not wholly agree with your paper, it's clearly an important contribution, and has made me question a few implicit assumptions I was carrying around.

The most important updates I got from the paper:

  1. Put less weight on technological determinism. In particular, defining existential risk in terms of a society reaching "technological maturity" without falling prey to some catastrophe frames technological development as being largely inevitable. But I'd argue even under the "techno-utopian" view, many technological developments are not needed for "technological maturity", or at least not for a very long time. While I still tend to view development of things like advanced AI systems as hard to stop (lots of economic pressures, geographically dispersed R&D, no expert consensus on whether it's good to slow down/accelerate), I'd certainly like to see more research into how we can affect the development of new technologies, beyond just differential technological advancement.
  2. "Existential risk" is ambiguous, so hard to study formally, we might want to replace it with more precise terms like "extinction risk" that are down-stream of some visions of existential risk. I'm not sure how decision relevant this ends up being, I think disagreement about how the world will unfold explains more of the disagreement on x-risk probabilities than definitions of x-risk, but it does seem worth trying to pin down more precisely.
  3. "Direct" vs "indirect" x-risk is a crude categorization, as most hazards lead to risks via a variety of pathways. Taking AI: there are some very "direct" risks such as a singleton AI developing some superweapon, but also some more "indirect" risks such as an economy of automated systems gradually losing alignment with collective humanity.

My main critiques:

  1. I expect a fairly broad range of worldviews end up with similar conclusions to the "techno-utopian approach" (TUA). The key beliefs seem to be that: (a) substantially more value is present in the future than exists today; (b) we have a moral obligation to safeguard that. The TUA is a very strong version of this, where there is many orders of magnitude more value in the future (transhumanism, total utilitarianism) and moral obligation is equal in the future and present (strong longtermism). But a non-transhumanist who wants 8 billion non-modified, biological humans to continue happily living on Earth for the next 100,000 years and values future generations at 1% of current generations would for many practical purposes make the same decisions.
  2. I frequently found myself unsure if there was actually a concrete disagreement between your views and those in the x-risk community, including those you criticize, beyond a choice of framing and emphasis. I understand it can be hard to nail down a disagreement, but this did leave me a little unsatisfied. For example, I'm still unsure what it really means to "democratise research and decision-making in existential risk" (page 26). I think almost all x-risk researchers would welcome more researchers from complementary academic disciplines or philosophical bents, and conversely I expect you would not suggest that random citizen juries should start actively participating in research. One concrete question I had is what axes you'd be most excited for the x-risk research field to become more diverse on at the margin: academic discipline, age, country, ethnicity, gender, religion, philosophical views, ...?
  3. Related to the above, it frequently felt like the paper was arguing against uncharitable versions of someone else's views -- VWH is an example others have brought up. On reflection, I think there is value to this, as many people may be holding those versions of the person's views even if the individual themselves had a more nuanced perspective. But it did often make me react "but I subscribe to <view X> and don't believe <supposed consequence Y>"! One angle you could consider taking in future work is to start by explaining your most core disagreements with a particular view, and then go on to elaborate on problems with commonly held adjacent positions.

I'd also suggest that strong longtermism is a meaningfully different assumption to e.g. transhumanism and total utilitarianism. In particular, the case for existential or extinction risk research seems many orders of magnitude weaker under a near-termist than strong longtermist worldview. Provided you think strong longtermism is at least credible, it seems reasonable to assume it when doing x-risk research, even though you should discount the impact of such interventions based on your credence in longtermism when making a final decision on where to allocate resources. If there is a risk that seems very likely to occur (e.g. AI, bio) such that it is plausible under both near-termist and longtermist grounds then perhaps it makes sense to drop this assumption, but even then I suspect it is often easier to just run two different analyses, given the different outcome metrics of concerns (e.g. % x-risk averted vs QALYs saved).

2017 Donor Lottery Report

Since writing this post, I have benefited both from 4 years of hindsight, and also significantly more grantmaking experience with just over a year at the long-term future fund. My main updates:

  • Exploit networks: I think small individual donors are often best off donating to people in their network that larger donors don't have access to. In particular I 70% believe it would have been better for me to wait 1-3 years and donate the money to opportunities as and when they came up. For example, there have been a few cases where something would help CHAI but couldn't be funded institutionally (for various bureaucratic or political reasons) -- I think we always managed to find a way to make it work, but me just having effectively discretionary funding would have made things simpler.
  • Efficient Markets in Grantmaking: When I wrote the post I tended to think the small orgs were getting overlooked by major donors, because it wasn't worth the time cost of evaluating. There's some truth to this, but I think more often the major donors actually had good reasons against wanting to fund the orgs more. -
  • Impact from Post: The post had less direct impact than I hoped, e.g. I haven't seen much analysis following on from it or heard of any major donations influenced by it. Although I've not tried very hard to track this, so I may have missed it. However, it did have a pretty big indirect impact, of making me more interested in grantmaking and likely helping me get a position on the long-term future fund. Notably you can write posts about what orgs are good to donate to even if you don't have $100k to donate... so I'd encourage people to do this if they have an interest in grantmaking, or scrutinize how good the grants made by existing grantmakers are. In general I'd like to see more discussion and diversity of opinions around where to make grants.
Long-Term Future Fund: May 2021 grant recommendations

Thanks for raising this. I think we communicated the grant decision to Po-Shen in late March/early April, when the pandemic was still significantly more active in the US. I was viewing this as "last chance to trial this idea", and I think I still stand by that given what we knew at the time, although I'd be less excited by this with the benefit of hindsight (the pandemic has developed roughly along my median expectations, but that still means I put significant credence on case rates being much higher than they currently are.)

In general our grant write-ups will appear at least a few weeks after the grant decision has actually been communicated to the grantee, as CEA needs to conduct due dilligence, we need to draft the write-up, have the write-up reviewed by the grantee, etc. For time-sensitive grants, the lag between making the grant and writing-up can be longer.

I'll also plug that the LTFF is one of the few funders that are able to make grants on short notice, so people with similarly ephemeral opportunities in the future should feel free to apply to us, there's an option in the application to flag it as time-sensitive.

How to PhD

Sorry for the (very) delayed reply here. I'll start with the most important point first.

But compared to working with a funder who, like you, wants to solve the problem and make the world be good, any of the other institutions mentioned including academia look extremely misaligned.

I think overall the incentives set up by EA funders are somewhat better than run-of-the-mill academic incentives, but I think the difference is smaller than you seem to believe, and I think we're a long way from cracking it. I think this is something we can get better at, but it's something that I expect will take significant infrastructure and iteration: e.g. new methods for peer review, experimenting with different granter-grantee relationships, etc.

Concretely, I think EA funders are really good (way better than most of academia or mainstream funders) at picking important problems like AI safety or biosecurity. I also think they're better at reasoning about possible theories of change (if this project succeeds, would it actually help?) and considering a variety of paths to impact (e.g. maybe a blog post can have more impact than a paper in this case, or maybe we'd even prefer to distribute some results privately).

However, I think most EA funders are actually worse at evaluating whether the research agenda is being executed well than the traditional academic structure. I help the LTFF evaluate grants, many of which are for independent research, and while I try to understand people's research agenda and how successful they've been, I think it's fair to say I spend at least an order of magnitude less time on this per applicant than someone's academic advisor.

Even worse, I have basically zero visibility into the process -- I only see the final write-up, and maybe have an interview with the person. If I see a negative result, it's really hard for me to tell if the person executed on the agenda well but the idea just didn't pan out, or if they bungled the process. Whereas I find it quite easy to form an opinion on projects I advise, as I can see the project evolve over time, and how the person responds to setbacks. Of course, we can (and do) ask for references, but if they're executing independently they may not have any, and there's always some CoI on advisors providing a reference.

Of course, when it comes to evaluating larger research orgs, funders can do a deeper dive and the stochasticity of research matters less (as it's averaged over a longer period of time). But this is just punting the problem to those who are running the org. In general I still think evaluating research output is a really hard problem.

I do think one huge benefit EA has is that people are mostly trying to "play fair", whereas in academia there is sadly more adversarial behavior (on the light side, people structuring their papers to dodge reviewer criticism; on the dark side, actual collusion in peer review or academic fraud). However, this isn't scalable, and I wouldn't want to build systems that rely on it.

In that more general comparison article, I think I may have still cautioned about academic incentives in particular. Because they seem, for lack of a better word, sneakier?

This is a fair point. I do think people kid themselves a bit about how much "academic freedom" they really have, and this can lead to people in effect internalizing the incentives more.

I've observed folks [...] behave as if they believe a research project to be directly good when I (and others) can't see the impact proposition, and the behavior feels best explained by publishing incentives.

Believing something is "directly good" when others disagree seems like a classic case of wishful thinking. There are lots of reasons why someone might be motivated to work on a project (despite it not, in fact, being "directly good"). Publication incentives are certainly a big one, and might well be the best explanation for the cases you saw. But in general I think it could also be that they just find that topic intellectually interesting, have been working on it for a while and are suffering from sunk cost fallacy, etc.

How to PhD

If you mean that once you are on the Junior Faculty track in CS, you don't really need to worry about well-received publications, this is interesting and doesn't line up with my models. Can you think of any examples which might help illustrate this?

To clarify, I don't think tenure is guaranteed, more that there's significant margin of error. I can't find much good data on this, but this post surveys statistics gathered from a variety of different universities, and finds anywhere between 65% of candidates get tenure (Harvard) to 90% (Cal State, UBC). Informally, my impression is that top schools in CS are the higher end of this: I'd have guessed 80%. Given this, the median person in the role could divert some of their research agenda to less well-received topics and still get tenure. But I don't think they could work on something that no one in the department or elsewhere cared about.

I've not noticed much tenure switch in CS but have never actually studied this, would love to see hard data here. I do think there's a significant difference in research agendas between junior and senior professors, but it's more a question of what was in vogue when they were in grad school and shaped their research agenda, than tenured vs non-tenured per se. I do think pre-tenure professors tend to put their students under more publication pressure though.

How to PhD

Publishing good papers is not the problem, deluding yourself is.

Big +1 to this. Doing things you don't see as a priority but which other people are excited about is fine. You can view it as kind of a trade: you work on something the research community cares about, and the research community is more likely to listen on (and work on) things you care about in the future.

But to make a difference you do eventually need to work on things that you find impactful, so you don't want to pollute your own research taste by implicitly absorbing incentives or others opinions unquestioningly.

How to PhD

You approximately can't get directly useful/ things done until you have tenure.

At least in CS, the vast majority of professors at top universities in tenure-track positions do get tenure. The hardest part is getting in. Of course all the junior professors I know work extremely hard, but I wouldn't characterize it as a publication rat race. This may not be true in other fields and outside the top universities.

The primary impediment to getting things done that I see is professors are also working as administrator and teaching, and that remains a problem post-tenure.

How to PhD

One important factor of a PhD that I don't see explicitly called out in this post is what I'd describe as "research taste": how to pick what problems to work on. I think this is one of if not the most important part of a PhD. You can only get so much faster at executing routine tasks or editing papers. But the difference between the most and mediam importance research problems can be huge.

Andrej Karpathy has a nice discussion of this:

When it comes to choosing problems you’ll hear academics talk about a mystical sense of “taste”. It’s a real thing. When you pitch a potential problem to your adviser you’ll either see their face contort, their eyes rolling, and their attention drift, or you’ll sense the excitement in their eyes as they contemplate the uncharted territory ripe for exploration. In that split second a lot happens: an evaluation of the problem’s importance, difficulty, its sexiness, its historical context (and possibly also its fit to their active grants). In other words, your adviser is likely to be a master of the outer loop and will have a highly developed sense of taste for problems. During your PhD you’ll get to acquire this sense yourself.

Clearly we might care about some of these criteria (like grants) less than others, but I think the same idea holds. I'd also recommend Chris Olah's exercises on developing research taste.

Load More