All of Holden Karnofsky's Comments + Replies

Hm. I contacted Nick and replaced it with another link - does that work?

I didn't make a claim that constant replacement occurs "empirically." As far as I can tell, it's not possible to empirically test whether it does or not. I think we are left deciding whether we choose to think of ourselves as being constantly replaced, or not - either choice won't contradict any empirical observations. My post was pointing out that if one does choose to think of things that way, a lot of other paradoxes seem to go away.

I personally like Radiohead a lot, but I don't feel like my subjective opinions are generally important here; with Pet Sounds I tried to focus on what seemed like an unusually clear-cut case (not that the album has nothing interesting going on, but that it's an odd choice for #1 of all time, especially in light of coming out a year after A Love Supreme).

I think this is interesting and plausible, but I'm somewhat skeptical in light of the fact that there doesn't seem to have been much (or at least, very effective) outcry over the rollback of net neutrality.

I think this is often a good approach!

I think "people can test their fit without much experience, but would get lots of value out of that experience for actually doing this work" is pretty valid, though I'll also comment that I think there are diminishing returns to direct experience - I think getting some experience (or at least exposure, e.g. via conversation with insiders) is important, but I don't think one necessarily needs several years inside key institutions in order to be helpful on problems like these.

I don't have anything available for this offhand - I'd have to put serious thought into what questions are at the most productive intersection of "resolvable", "a good fit for Metaculus" and "capturing something important." Something about warning signs ("will an AI system steal at least $10 million?") could be good.

Thanks! I'd estimate another 10-15 hours on top of the above, so 20-30 hours total. A good amount of this felt like leisure time and could be done while not in front of a computer, which was nice. I didn't end up with "solutions" I'd be actually excited about for substantive progress on alignment, but I think I accomplished my goal of understanding the ELK writeup well enough to nitpick it.

The link works for me in incognito mode (it is a Google Drive file).

Thanks, this is helpful! I wasn't aware of that usage of "moral quasi-realism."

Personally, I find the question of whether principles can be described as "true" unimportant, and don't have much of a take on it. My default take is that it's convenient to sometimes use "true" in this way, so I sometimes do, while being happy to taboo it anytime someone wants me to or I otherwise think it would be helpful to.

I share a number of your intuitions as a starting point, but this dialogue (and previous ones) is intended to pose challenges to those intuitions. To follow up on those:

On Challenge 1A (and as a more general point) - if we take action against climate change, that presumably means making some sort of sacrifice today for the sake of future generations. Does your position imply that this is "simply better for some and worse for others, and not better or worse on the whole?" Does that imply that it is not particularly good or bad to take action on climate chan... (read more)

I think that's a fair point. These positions just pretty much end up in the same place when it comes to valuing existential risk.

It's interesting that you have that intuition! I don't share it, and I think the intuition somewhat implies some of the "You shouldn't leave your house" type things alluded to in the dialogue.

I agree with this argument for discount rates, but I think it is a practical rather than philosophical argument. That is, I don't think it undermines the idea that if we were to avert extinction, all of the future lives thereby enabled should be given "full weight."

You're right that I haven't comprehensively addressed risk aversion in this piece. I've just tried to give an intuition for why the pro-risk-aversion intuition might be misleading.

Good point, thanks! Edited.

Thanks, this is appreciated!

Sorry for the long delay, I let a lot of comments to respond to pile up!

APS seems like a category of systems that includes some of the others you listed (“Advanced capability: they outperform the best humans on some set of tasks which when performed at advanced levels grant significant power in today’s world (tasks like scientific research, business/military/political strategy, engineering, and persuasion/manipulation) … “). I still don’t feel clear on what you have in mind here in terms of specific transformative capabilities. If we condition on not ha... (read more)

Comments on Debating myself on whether “extra lives lived” are as good as “deaths prevented” will go here.

Fixed, thanks!

Comments on Defending One-Dimensional Ethics will go here.

(Placeholder for comments on "To Match the Greats, Don’t Follow In Their Footsteps")

Placeholder for comments on Beach Boys post

Comments for Cost disease and civilizational decline will go here.

Comments on Reader reactions and update on "Where's Today's Beethoven" will go here.

Comments for Book non-review: The Dawn of Everything will go here.

The latter.

This version of the mice analogy was better than mine, thanks!

I largely agree with this comment, and I didn't mean to say that different intellectual property norms would create more "Beethoven-like" figures critical-acclaim-wise. I more meant to say it would just be very beneficial to consumers. (And I do think music is in a noticeably better state (w/r/t the ease of finding a lot that one really likes) than film or books, though this could be for a number of reasons.)

Sorry, just saw this! This did not in fact work out on the hoped-for timeline, and I didn't have a grantee in mind - I think the right way to try to do something here would've been through direct dialogue with policymakers.

In response to the paragraph starting "I see how ..." (which I can't copy-paste easily due to the subscripts):

I think there are good pragmatic arguments for taking actions that effectively hold Ht responsible for the actions of Ht-1. For example, if Ht-1 committed premeditated murder, this gives some argument that Ht is more likely to harm others than the average person, and should be accordingly restricted for their benefit. And it's possible that the general practice of punishing Ht for Ht-1's actions would generally deter crime, while not creating other... (read more)

I think this is a good point, but it doesn't totally knock me out of feeling sympathy for the "rowing" case.

It looks quite likely to me that factory farming is going to end up looking something like air pollution - something that got worse, then better, as capabilities/wealth improved. I expect the combination of improving "animal product alternatives" (Impossible, Beyond, eventually clean meat) with increasing wealth to lead this way.

Granted, this is no longer a "pure trend extrapolation," but I think the consistent and somewhat mysterious improvement in ... (read more)

Interesting, thanks! Yep, those probabilities definitely seem too high to me :) How much would you shade them down for 5 years instead of 15? It seems like if your 5-year probabilities are anywhere near your 15-year probabilities, then the next 5 years have a lot of potential to update you one way or the other (e.g., if none of the "paths to PONR" you're describing work out in that time, that seems like it should be a significant update).

I'm not going to comment comprehensively on the paths you laid out, but a few things:

• I think EfficientZero is sample-

I generally put this comment up in advance of the post, so that I can link to it from the post. The post is up now!

Comments for Empowerment and Stakeholder Management will go here.

Comments for Jan. 14 Cold Links will go here.

[Placeholder for Why it matters if "ideas are getting harder to find" comments]

Hey Josh, I think this is a good point - it would be great to have some common knowledge of what sort of commitment this is.

Here's where I am so far:

1. I read through the full report reasonably carefully (but only some of the appendices).

2. I spent some time thinking about potential counterexamples. It's hard to say how much; this mostly wasn't time I carved out, more something I was thinking about while taking a walk or something.

3. At times I would reread specific parts of the writeup that seemed important for thinking about whether a particular idea was

[Placeholder for How artistic ideas could get harder to find comments]

Comments for AI alignment research links will go here.

My understanding is that "mixture of experts" essentially comes down to training multiple distinct models, and having some "meta" procedure for assigning problems (or pieces of problems) to them.

Since training expense grows with something like the square of model size, it's much more expensive to train one big model than N smaller models that are each 1/N as big (plus a procedure for choosing between the N smaller models).

A human brain is about 100x the "size" of a mouse brain. So for a metaphor, you can think of "mixture of experts" as though it's trying ... (read more)

Comments for Where's Today's Beethoven? will go here.

I broadly agree that my summary has this issue. If there were causal stories I were confident in, I would try to include them in the summary; but in fact I feel very hazy on a lot of multiple-step causal stories about history, and have defaulted to leaving them out when I think the case is quite unclear. I'm sure this leaves my summary less informative than it would ideally be (and than it would be if I knew more about history and were more confident about some of these multiple-step causal stories).

I agree with what you say here, as a general matter. I'm not sure I want to make an edit, as I really do think there are some "bad" parts of myself that I'd prefer to "expose and downweight," but I agree that it's easy to get carried away with that sort of thing.

It might be true that the right expected utility calculation would endorse being overconfident, but "Bayesian mindset" isn't about behaving like a theoretically ideal utility maximizer - it's about actually writing down probabilities and values and taking action based on those. I think trying to actually make decisions this way is a very awkward fit with an overconfident attitude: even if the equation you write down says you'll do best by feeling overconfident, that might be tough in practice.