All of zdgroff's Comments + Replies

I'm a grantmaker at Longview and manage the Digital Sentience Fund—thought I'd share my thinking here: “backchaining from… making the long-term future go well conditional on no Al takeover” is my goal with the fund (with the restriction of being related to the wellbeing of AIs in a somewhat direct way), though we might disagree on how that’s best achieved through funding. Specifically, the things you’re excited about would probably be toward the top of the list of things I’m excited about, but I also think broader empirical and philosophical work and field... (read more)

Thanks for writing this Will. I feel a bit torn on this so will lay out some places where I agree and some where I disagree:

• I agree that some of these AI-related cause areas beyond takeover risk deserve to be seen as their own cause areas as such and that lumping them all under "AI" risks being a bit inaccurate.
  • That said, I think the same could be said of some areas of animal work—wild animal welfare, invertebrate welfare, and farmed vertebrate welfare should perhaps get their own billing. And then this can keep expanding—see, e.g., OP's focus areas, which

I'd add to this that you do also have the possibility that 1-3 happen, but they happen much later than many people currently think. My personal take is that the probability that 'either AGI's impact comes in more than ten years or it's not that radical' is >50%, certainly far more than 0%.

I've been interested to see this book since I came across the idea. I think the argument for this being a problem from a variety of perspectives is pretty compelling.

• For me, probably the key chapter is "Dodging the asteroid. And other benefits of other people." I'm also interested in how population issues could interact with AI-driven changes.
• The Peter Singer quote is interesting—I'm a bit surprised given his past views on population ethics. I'm wondering if he's updated his views.

Agree that it's not certain or obvious that AI risk is the most pressing issue (though it is 80k's best guess & my personal best guess

Yeah, FWIW, it's mine too. Time will tell how I feel about the change in the end. That EA Forum post on the 80K-EA community relationship feels very appropriate to me, so I think my disagreement is about the application.

I think that existential risks from various issues with AGI (especially if one includes trajectory changes) are high enough that one needn't accept fanatical views to prioritise them

I think the argument you linked to is reasonable. I disagree, but not strongly. But I think it's plausible enough that AGI concerns (from an impartial cause prioritization perspective) require fanaticism that there should still be significant worry about it. My take would be that this worry means an initially general EA org should not overwhelmingly prioritize AGI.

By my read, that post and the excerpt from it are about the rhetorical motivation for existential risk rather than the impartial ethical motivation. I basically agree that longtermism is not the right framing in most conversations, and it's also not necessary for thinking existential risk work would be more valuable than the marginal public dollar.

I included the qualifier "From an altruistic cause prioritization perspective" because I think that from an impartial cause prioritization perspective, the case is different. If you're comparing existential risk to animal welfare and global health, the links in my comment I think make the case pretty persuasively that you need longtermism.

I'm not sure exactly what this change will look like, but my current impression from this post leaves me disappointed. I say this as someone who now works on AI full-time and is mostly persuaded of strong longtermism. I think there's enough reason for uncertainty about the top cause and value in a broad community that central EA organizations should not go all-in on a single cause. This seems especially the case for 80,000 Hours, which brings people in by appealing to a general interest in doing good.

Some reasons for thinking cause diversification by the c... (read more)

I think I agree with the Moral Power Laws hypothesis, but it might be irrelevant to the question of whether to try to improve the value of the future or work on extinction risk.

My thought is this: the best future is probably a convergence of many things going well, such as people being happy on average, there being many people, the future lasting a long time, and maybe some empirical/moral uncertainty stuff. Each of these things plausibly has a variety of components, creating a long tail. Yet you'd need expansive, simultaneous efforts on many fronts to get... (read more)

The value of the future conditional on civilization surviving seems positive to me, but not robustly so. I think the main argument for its being positive is theoretical (e.g., Spreading happiness to the stars seems little harder than just spreading), but the historical/contemporary record is ambiguous.

The value of improving the future seems more robustly positive if it is tractable. I suspect it is not that much less tractable than extinction risk work. I think a lot of AI risk satisfies this goal as well as the x-risk goal for reasons Will MacAskill gives... (read more)

This is a stimulating and impressively broad post.

I want to press a bit on whether these trends are necessarily bad—I think they are, but there are a few reasons why I wonder about it.

1) Secrecy: While secrecy makes it difficult or impossible to know if a system is a moral patient, it also prevents rogue actors from quickly making copies of a sentient system or obtaining a blueprint for suffering. (It also prevents rogue actors from obtaining a blueprint for flourishing, which supports your point.) How do you think about this?

2 and 3) If I understand corre... (read more)

I think I'd be more worried about pulling out entirely than a delayed release, but either one seems possible (but IMO unlikely).

What seems less likely to work?

• Work with the EU and the UK
  • Trump is far less likely to take regulatory inspiration from European countries and generally less likely to regulate. On the other-hand perhaps under a 2028 Dem administration we would see significant attention on EU/UK regulations.
  • The EU/UK are already scaling back the ambitions of their AI regulations out of fear that Trump would retaliate if they put limits on US companies.

Interesting—I've had the opposite take for the EU. The low likelihood of regulation in the US seems like it would... (read more)

I think I might add this to my DIY, atheist, animal-rights Haggadah.

[Edited to add the second sentence of the paragraph beginning, "Putting these together."]

The primary result doesn't speak to this, but secondary results can shed some light on it. Overall, I'd guess persistence is a touch less for policies with much more support, but note that the effect of proposing a policy on later policy is likely much larger than the effect of passing a policy conditional on its having been proposed.

The first thing to note is that there are really two questions here we might want to ask:

1. What is the effect of passing a policy change, c

Yep, at the risk of omitting others, Lukas Freund as well.

Yes, it's a good point that benefits and length of the period are not independent, and I agree with the footnote too.

I would note that the factors I mentioned there don't seem like they should change things that much for most issues. I could see using 50-100 years rather than, e.g., 150 years as my results would seem to suggest, but I do think 5-10 years is an order of magnitude off.

Easy Q to answer so doesn't take much time! In economics, the norm is not to publish your job market paper until after the market for various reasons. (That way, you don't take time away from improving the paper, and the department that hires you gets credit.)

We will see before long how it publishes!

1. I look at some things you might find relevant here. I try to measure the scale of the impact of a referendum. I do this two ways. I have just a subjective judgment on a five-point scale, and then I also look at predictions of the referendum's fiscal impact from the secretary of state. Neither one is predictive. I also look at how many people would be directly affected by a referendum and how much news coverage there was before the election cycle. These predict less persistence.
2. This is something I plan to do more, but they can't vary that much because when

I do look at predictors a bit—though note that it's not about what makes it harder to repeal but rather about what makes a policy change/choice influential decades later.

The main takeaway is there aren't many predictors—the effect is remarkably uniform. I can't look at things around the structure of the law (e.g., integration in a larger bill), but I'd be surprised if something like complexity of language or cross-party support made a difference in what I'm looking at.

Yeah, Jack, I think you're capturing my thinking here (which is an informal point for this audience rather than something formal in the paper). I look at measures of how much people were interested in a policy well before the referendum or how much we should expect them to be interested after the referendum. It looks like both of these predict less persistence. So the thought is that things that generally are less salient when not on the ballot are more persistent.

See my reply to Neil Dullaghan—I think that gives somewhat of a sense here. Some other things:

• I don't have a ton of observations on any one specific policy, so I can't say much about whether some special policy area (e.g., pollution regulation) exhibits a different pattern.
• I look at whether this policy, or a version of it, is in place. This should capture anything that would be a direct and obvious substitute, but there might be looser substitutes that end up passing if you fail to pass an initial policy. The evidence I do have on this suggests it's small,

I didn't write down a prior. I think if I had, it would have been less persistence. I think I would have guessed five years was an underestimate. (I think probably many people making that assumption would also have guessed it was an underestimate but were airing on the side of conservatism.)

Yes, basically (if I understand correctly). If you think a policy has impact X for each year it's in place, and you don't discount, then the impact of causing it to pass rather than fail is something on the order of 100 * X. The impact of funding a campaign to pass it is bigger, though, because you presumably don't want to count the possibility that you fund it later as part of the counterfactual (see my note above about Appendix Figure D20).

 Some things to keep in mind:

1. Impacts might change over time (e.g., a policy stops mattering in 50 years even if

A few things:

• I do find these patterns when I look at a few different types of policies (referendums, legislation, state vs. Congress, U.S. vs. international), so there's some reason to think it's not just state referendums. 
• There's a paper on the repeals of executive orders that finds an even lower rate of repeals there, but that doesn't tell us the counterfactual (i.e., would someone else have done this if the president in question did not).
• There's suggestive evidence that when policies are more negotiable, there's less persistence. In my narrative/c

I think there are probably ways to tackle that but don't have anything shovel-ready. I'd want to look at the general evidence on campaign spending and what methods have been used there, then see if any of those would apply (with some adaptations) to this case.

Thanks a lot! And good luck on the job market to you—let's connect when we're through with this (or if we have time before then).

I'm very glad to see you working and thinking about this—it seems pretty neglected within the EA community. (I'm aware of and agree with the thought that speeding up space settlement is not a priority, but making sure it goes well if it happens does seem important.)

Oh, that's a good idea. I had thought of something quite different and broader, but this also seems like a promising approach.

Yeah, I think that would reduce the longevity in expectation, maybe by something like 2x. My research includes things that could hypothetically fall under congressional authority and occasionally do. (Anything could fall under congressional authority, though some might require a constitutional amendment.) So I don't think this is dramatically out of sample, but I do think it's worth keeping in mind.

The former, though I don't have estimates of the counterfactual timeline of corporate campaigns. (I'd like to find a way to do that and have toyed with it a bit but currently don't have one.)

I believe 4 years is very conservative. I'm working on a paper due November that should basically answer the question in part 1, but suffice it to say I think the ballot measures should look many times more cost-effective than corporate campaigns.

From what I can tell, the climate change one seems like the one with the most support in the literature. I'm not sure how much the consensus in favor of the human cause of megafauna extinctions (which I buy) generalizes to the extinction of other species in the Homo genus. Most of the Homo extinctions happened much earlier than the megafauna ones. But it could be—I have not given much thought to whether this consensus generalizes.

The other thing is that "extinction" sometimes happened in the sense that the species interbred with the larger population of Homo sapiens, and I would not count that as the relevant sort of extinction here.

Yeah, this is an interesting one. I'd basically agree with what you say here. I looked into it and came away thinking (a) it's very unclear what the actual base rate is, but (b) it seems like it probably roughly resembles the general species one I have here. Given (b), I bumped up how much weight I put on the species reference class, but I did not include the human subspecies as a reference class here given (a).

From my exploration, it looked like there had been loose claims about many of them going extinct because of Homo sapiens, but it seemed like this w... (read more)

Very strong +1 to all this. I honestly think it's the most neglected area relative to its importance right now. It seems plausible that the vast majority of future beings will be digital, so it would be surprising if longtermism does not imply much more attention to the issue.

I take 5%-60% as an estimate of how much of human civilization's future value will depend on what AI systems do, but it does not necessarily exclude human autonomy. If humans determine what AI systems do with the resources they acquire and the actions they take, then AI could be extremely important, and humans would still retain autonomy.

I don't think this really left me more or less concerned about losing autonomy over resources. It does feel like this exercise made it starker that there's a large chance of AI reshaping the world beyond human extinction. ... (read more)

I guess I would think that if one wants to argue for democracy as an intrinsic good, that would get you global democracy (and global control of EA funds), and it's practical and instrumental considerations (which, anyway, are all the considerations in my view) that bite against it.

Yes, that seems right.

It seems like the critics would claim that EA is, if not coercing or subjugating, at least substantially influencing something like the world population in a way that meets the criteria for democratisation. This seems to be the claim in arguments about billionaire philanthropy, for example. I'm not defending or vouching for that claim, but I think whether we are in a sufficiently different situation may be contentious.

Well I think MIC relies on some sort of discontinuity this century, and when we start getting into the range of precedented growth rates, the discontinuity looks less likely.

But we might not be disagreeing much here. It seems like a plausibly important update, but I'm not sure how large.

This is a valuable point, but I do think that giving real weight to a world where we have neither extinction nor 30% growth would still be an update to important views about superhuman AI. It seems like evidence against the Most Important Century thesis, for example.

It might be challenging to borrow (though I'm not sure), but there seem to be plenty of sophisticated entities that should be selling off their bonds and aren't. The top-level comment does cut into the gains from shorting (as the OP concedes), but I think it's right that there are borrowing-esque things to do.

I'm trying to make sure I understand: Is this (a more colorful version) of the same point as the OP makes at the end of "Bet on real rates rising"?

The other risk that could motivate not making this bet is the risk that the market – for some unspecified reason – never has a chance to correct, because (1) transformative AI ends up unaligned and (2) humanity’s conversion into paperclips occurs overnight. This would prevent the market from ever “waking up”.

However, to be clear, expecting this specific scenario requires both: 

1. Buying into spe

It doesn't seem all that relevant to me whether traders have a probability like that in their heads. Whether they have a low probability or are not thinking about it, they're approximately leaving money on the table in a short-timelines world, which should be surprising. People have a large incentive to hunt for important probabilities they're ignoring.

Of course, there are examples (cf. behavioral economics) of systemic biases in markets. But even within behavioral economics, it's fairly commonly known that it's hard to find ongoing, large-scale biases in financial markets.

Do you have a sense of whether the case is any stronger for specifically using cortical and pallial neurons? That's the approach Romain Espinosa takes in this paper, which is among the best work in economics on animal welfare.

Yes, and thank you for the detailed private proposal you sent the research team. I didn't see it but heard about it, and it seems like it was a huge help and just a massive amount of volunteer labor. I know they really appreciated it.

I'm an ACE board member, so full disclosure on that, though what I say here is in my personal capacity.

I'm very glad about a number of improvements to the eval process that are not obvious from this post. In particular, there are now numeric cost-effectiveness ratings that I found clarifying, overall explanations for each recommendation, and clearer delineation of the roles the "programs" and "cost-effectiveness" sections play in the reviews. I expect these changes to make recommendations more scope sensitive. This leaves me grateful for and confident in the new review framework.

As I noted on the nuclear post, I believe this is based on a (loosely speaking) person-affecting view (mentioned in Joel and Ben's back-and-forth below). That seems likely to me to bias the cost-effectiveness downward.

Like Fin, I'm very surprised by how well this performs given takes in other places (e.g. The Precipice) on how asteroid prevention compares to other x-risk work.